Diagnosing skin conditions using machine-learned models

Abstract

To provide diagnosing skin conditions using machine-learned models.SOLUTION: A diagnosis system trains a set of machine-learned diagnosis models that are configured to receive an image of a patient and generate predictions on whether the patient has one or more health conditions. In one embodiment, the set of machine-learned models are trained to generate predictions for images that contain two or more underlying health conditions of the patient. In one instance, the symptoms for the two or more health conditions are shown as two or more overlapping skin abnormalities on the patient. By using architectures of the set of diagnosis models described in the present specification, the diagnosis system can generate more accurate predictions for images that contain overlapping symptoms for the two or more health conditions in comparison to existing systems.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 869,553, filed July 1, 2019, which is incorporated by reference in its entirety. [Background technology]

[0002] The present disclosure relates generally to diagnosing health conditions, and more particularly, to diagnosing overlapping health conditions in images.

[0003] The computerized diagnostic system is configured to receive an image of a patient and generate a prediction as to whether the patient has one or more health conditions based on the presence of a symptom in the image. The computerized diagnostic system can be applied to predict health conditions in various anatomical parts of the patient. For example, a skin diagnostic system can be applied to predict whether an image of a patient's skin has one or more skin abnormalities, such as a rash, mole, eczema, acne, and herpes. As another example, a tumor diagnostic system can be applied to predict whether an image of a patient's anatomy contains a tumor.

[0004] In many cases, a patient may have multiple health conditions that appear in the same image. For example, an image may contain a psoriatic plaque and a birthmark, both skin abnormalities, located on the same arm, that are spatially separated from each other. As another example, an image may contain skin abnormalities that overlap each other, such as a psoriatic plaque with a mole within its borders. However, existing computerized diagnostic systems have difficulty generating accurate diagnoses in these cases, for example, because the computerized diagnostic systems are trained to generate predictions regarding a single health condition. Furthermore, when health conditions overlap, the presentation and other symptoms in the image may be the result of the conditions, either individually or both in combination, and it may be difficult for the computerized diagnostic system to accurately differentiate the symptoms in the image into separate health conditions. Summary of the Invention [Means for solving the problem]

[0005] The diagnostic system receives images of a patient and trains a set of machine-learned diagnostic models configured to generate predictions about whether the patient has one or more health conditions. In one embodiment, the set of machine-learned models is trained to generate predictions for images that contain two or more underlying health conditions of the patient. In one instance, symptoms for two or more health conditions are manifested as two or more overlapping skin abnormalities on a patient. By using the architecture of the set of diagnostic models described herein, a diagnostic system can generate more accurate predictions for images containing overlapping symptoms for two or more health conditions compared to existing systems.

[0006] In an embodiment, the diagnostic system receives a request from a client device to diagnose a skin anomaly in an input image. The input image includes overlapping symptoms for two or more skin anomalies on the skin. The diagnostic system accesses a set of machine-learned diagnostic models from a database. Each machine-learned model includes an individual set of trained weights determined through a training process. The diagnostic system generates an individual prediction for each of the two or more skin anomalies in the input image by applying the set of diagnostic models to the input image. The prediction indicates a likelihood that the individual skin anomaly among the two or more skin anomalies is exhibited in the input image. The diagnostic system generates a diagnosis of the skin anomaly from the prediction for the input image and provides the diagnosis to the client device.

[0007] In one embodiment, the diagnostic system receives images of a patient or individual and trains a set of machine-learned indicator models configured to generate an indication as to whether the input image presents two or more health conditions. In one embodiment, the diagnostic system receives a request for diagnosis of a health condition in the input image. The diagnostic system accesses an indicator model or a diagnostic model with indicator functionality from a database. The diagnostic system generates an indication representing a likelihood that the input image includes two or more health conditions by applying the indicator model to the input image. The diagnostic system makes a determination as to whether the input image presents two or more health conditions from the indication generated for the input image. Based on the determination, the diagnostic system generates a result for the request and provides the result to a user of the client device.

[0008] Because an input image presenting multiple health conditions may be difficult to diagnose, a user of a client device may receive such an indication in addition to, or as an alternative to, active diagnosis of the image. The user may then decide to provide the input image to a medical professional to obtain a more accurate diagnosis. Thus, it may be advantageous for a user of a computerized diagnostic system to receive information about whether an input image includes symptoms from more than one health condition, so that a more accurate diagnosis may be made by a different specialized system or medical professional. The present invention provides, for example, the following items. (Item 1) 1. A method for diagnosing overlapping skin anomalies in an input image, the method comprising: receiving a request from a client device to diagnose skin anomalies in an input image, the input image including overlapping skin anomalies on a patient's skin; accessing a set of machine-learned models from a database, each machine-learned model including a distinct set of trained weights; generating a respective prediction for each of two or more skin anomalies in the input image by applying the set of machine learning models to the input image, the prediction indicating a likelihood that the respective skin anomaly among the two or more skin anomalies is exhibited in the input image; generating a diagnosis of the overlapping skin anomaly from the prediction for the input image; and providing the diagnosis to the client device; A method comprising: (Item 2) The set of machine-learned models is an ensemble set of neural network models, and generating the predictions for the two or more skin anomalies further comprises: for each neural network model in the ensemble set, generating one or more predictions from the neural network model by applying the neural network model to the input image; combining the predictions for the ensemble set of neural network models to generate the predictions for the two or more skin anomalies. The method according to item 1, comprising: (Item 3) The set of machine learned models includes a second machine learned model and a third machine learned model, and generating the predictions for the two or more skin anomalies further comprises: generating an image tensor by applying the second machine-learned model to the input image, the image tensor characterizing a plurality of spatial features within the input image; and Extracting a plurality of components from the image tensor; generating a separate tensor for each of the two or more skin anomalies; and generating the predictions for the two or more skin anomalies by applying the third machine-learned model to the respective tensors for each of the two or more skin anomalies; and The method according to item 1, comprising: (Item 4) 4. The method of claim 3, wherein the components are extracted from the image tensor by performing an independent component analysis (ICA) on the image tensor. (Item 5) 4. The method of claim 3, wherein the sets of trained weights for the second machine-learned model and the third machine-learned model are trained jointly. (Item 6) The set of machine-learned models includes a recurrent neural network model, and generating the predictions for the two or more skin anomalies further comprises: repeatedly applying the recurrent neural network model to the input image to generate the individual prediction for a first skin anomaly among the two or more skin anomalies at a first time and to generate the individual prediction for a second skin anomaly among the two or more skin anomalies at a second time subsequent to the first time. The method according to item 1, comprising: (Item 7) the set of machine learned models includes a second machine learned model, a third machine learned model, and a fourth machine learned model, and generating the predictions for the two or more skin anomalies further includes: generating a prediction as to whether the input image contains an amorphous skin anomaly or a focal anomaly; in response to determining that the input image includes an amorphous skin anomaly, generating a prediction regarding the amorphous skin anomaly by applying the amorphous anomaly model to the input image; in response to determining that the input image includes a focal skin anomaly, generating a prediction regarding the focal skin anomaly by applying the focal anomaly model to the input image; The method according to item 1, comprising: (Item 8) 2. The method of claim 1, wherein at least one of the set of machine-learned models is configured as a neural network architecture including a set of layers of nodes, each layer connected to the previous layer via a subset of weights. (Item 9) 2. The method of claim 1, wherein the input image is at least one of a radiological image, a computed tomography (CT) scan, a medical magnetic resonance imaging (MRI) scan, an X-ray image, an ultrasound or sonography image, a tactile image, or a thermographic image. (Item 10) 2. The method of claim 1, wherein the input image is an image captured by a user of the client device, the client device being a smartphone. (Item 11) 1. A computer program product for diagnosing overlapping skin anomalies in an input image, the computer program product comprising: a computer readable storage medium; receiving a request from a client device to diagnose skin anomalies in an input image, the input image including overlapping skin anomalies on a patient's skin; accessing a set of machine-learned models from a database, each machine-learned model including a distinct set of trained weights; generating a respective prediction for each of two or more skin anomalies in the input image by applying the set of machine learning models to the input image, the prediction indicating a likelihood that the respective skin anomaly among the two or more skin anomalies is exhibited in the input image; generating a diagnosis of the overlapping skin anomaly from the prediction for the input image; and providing the diagnosis to the client device; A computer program product comprising computer program code for: (Item 12) The set of machine-learned models is an ensemble set of neural network models, and generating the predictions for the two or more skin anomalies further comprises: for each neural network model in the ensemble set, generating one or more predictions from the neural network model by applying the neural network model to the input image; combining the predictions for the ensemble set of neural network models to generate the predictions for the two or more skin anomalies. Item 12. The computer program product of item 11, comprising: (Item 13) The set of machine learned models includes a second machine learned model and a third machine learned model, and generating the predictions for the two or more skin anomalies further comprises: generating an image tensor by applying the second machine-learned model to the input image, the image tensor characterizing a plurality of spatial features within the input image; and Extracting a plurality of components from the image tensor; generating a separate tensor for each of the two or more skin anomalies; and generating the predictions for the two or more skin anomalies by applying the third machine-learned model to the respective tensors for each of the two or more skin anomalies; and Item 12. The computer program product of item 11, comprising: (Item 14) Item 14. The computer program product of item 13, wherein the plurality of components are extracted from the image tensor by performing an independent component analysis (ICA) on the image tensor. (Item 15) Item 14. The computer program product of item 13, wherein the sets of trained weights for the second machine-learned model and the third machine-learned model are trained jointly. (Item 16) The set of machine-learned models includes a recurrent neural network model, and generating the predictions for the two or more skin anomalies further comprises: repeatedly applying the recurrent neural network model to the input image to generate the individual prediction for a first skin anomaly among the two or more skin anomalies at a first time and to generate the individual prediction for a second skin anomaly among the two or more skin anomalies at a second time subsequent to the first time. Item 12. The computer program product of item 11, comprising: (Item 17) the set of machine learned models includes a second machine learned model, a third machine learned model, and a fourth machine learned model, and generating the predictions for the two or more skin anomalies further includes: generating a prediction as to whether the input image contains an amorphous skin anomaly or a focal anomaly; in response to determining that the input image includes an amorphous skin anomaly, generating a prediction regarding the amorphous skin anomaly by applying the amorphous anomaly model to the input image; in response to determining that the input image includes a focal skin anomaly, generating a prediction regarding the focal skin anomaly by applying the focal anomaly model to the input image; Item 12. The computer program product of item 11, comprising: (Item 18) Item 12. The computer program product of item 11, wherein at least one of the set of machine-learned models is configured as a neural network architecture including a set of layers of nodes, each layer connected to a previous layer via a subset of weights. (Item 19) Item 12. The computer program product of item 11, wherein the input image is at least one of a radiological image, a computed tomography (CT) scan, a medical magnetic resonance imaging (MRI) scan, an X-ray image, an ultrasound or sonography image, a tactile image, or a thermographic image. (Item 20) Item 12. The computer program product of item 11, wherein the input image is an image captured by a user of the client device, the client device being a smartphone. (Item 21) 1. A method for generating an indication of two or more skin anomalies in an input image, the method comprising: receiving, from a client device, a request for diagnosis of a skin anomaly presented in an input image; accessing a machine-learned model from a database, the machine-learned model including a distinct set of trained weights; applying the machine learning model to the input image to generate an indication, the indication representing a likelihood that the input image exhibits two or more skin anomalies; and generating a determination from the indication generated with respect to the input image as to whether the input image exhibits two or more skin anomalies; and generating a result with respect to the request based on the determination and providing the result to the client device; A method comprising: (Item 22) 22. The method of claim 21, further comprising determining a treatment option for the input image based on the determination and providing the treatment option as the result to the client device. (Item 23) in response to determining that the input image exhibits one skin anomaly rather than two or more skin anomalies, accessing a diagnostic model from the database configured to generate a prediction regarding a single skin anomaly; generating a prediction regarding the skin anomaly in the input image by applying the diagnostic model to the input image; generating a diagnosis regarding the input image from the prediction and providing the diagnosis as the result to the client device; 22. The method of claim 21, further comprising: (Item 24) in response to determining that the input image exhibits two or more skin anomalies, accessing from the database a set of diagnostic models configured to generate a respective prediction for each of the two or more skin anomalies; generating predictions regarding the two or more skin anomalies in the input image by applying the set of diagnostic models to the input image; generating a diagnosis regarding the input image from the prediction and providing the diagnosis as the result to the client device; 22. The method of claim 21, further comprising: (Item 25) 22. The method of claim 21, further comprising, in response to determining that the input image presents two or more skin anomalies, providing information to the client device that the input image cannot be diagnosed. (Item 26) 22. The method of claim 21, further comprising, in response to determining that the input image presents two or more skin anomalies, providing information about the indication to the client device without providing a prediction regarding the two or more skin anomalies in the input image. (Item 27) 22. The method of claim 21, wherein the machine-learned model is also a diagnostic model further configured to receive an image and generate a prediction regarding one or more skin anomalies presented in the image. (Item 28) generating the indication further includes applying the diagnostic model to the input image to generate an output vector including a set of elements, the prediction regarding the one or more skin anomalies being represented by values ​​of the elements in the output vector; the input image is determined to exhibit two or more skin anomalies when a difference between values ​​of the set of elements in the output vector is below a predetermined threshold. Item 28. The method according to item 27. (Item 29) 28. The method of claim 27, wherein generating the indication further comprises applying the diagnostic model to the input image and generating an output vector comprising a set of elements, wherein the prediction regarding the one or more skin abnormalities is represented by values ​​for a subset of elements in the output vector and the indication is represented by values ​​of the remaining elements in the output vector. (Item 30) the prediction regarding the one or more skin abnormalities is represented by one or more categories in a set of categories; generating the indication further includes applying the diagnostic model to the input image to generate an output value; the input image is determined to exhibit two or more skin anomalies when the output values ​​are assigned to the remaining categories in the set of categories. Item 28. The method according to item 27. (Item 31) 1. A computer program product for generating an indication of two or more skin anomalies in an input image, the computer program product comprising: a computer readable storage medium; receiving a request from a client device for diagnosis of a skin anomaly in an input image; accessing a machine-learned model from a database, the machine-learned model including a distinct set of trained weights; applying the machine-learned model to the input image to generate an indication, the indication representing a likelihood that the input image includes two or more skin anomalies; and generating a determination from the indication generated with respect to the input image as to whether the input image exhibits two or more skin anomalies; and generating a result with respect to the request based on the determination and providing the result to the client device; A computer program product comprising computer program code for: (Item 32) The computer readable storage medium further comprises: 32. The computer program product of claim 31, comprising computer program code for determining treatment options for the individual in the input image based on the determination and providing the treatment options to the client device as a result. (Item 33) The computer readable storage medium further comprises: in response to determining that the input image exhibits one skin anomaly rather than two or more skin anomalies, accessing a diagnostic model from the database configured to generate a prediction regarding a single skin anomaly; generating a prediction regarding the skin anomaly in the input image by applying the diagnostic model to the input image; generating a diagnosis for the input image from the prediction and providing the diagnosis to the client device as the result; 32. A computer program product according to item 31, containing computer program code for: (Item 34) The computer readable storage medium further comprises: in response to determining that the input image exhibits two or more skin anomalies, accessing from the database a set of diagnostic models configured to generate a respective prediction for each of the two or more skin anomalies; generating predictions regarding the two or more skin anomalies in the input image by applying the set of diagnostic models to the input image; generating a diagnosis regarding the input image from the prediction and providing the diagnosis as the result to the client device; 32. A computer program product according to item 31, containing computer program code for: (Item 35) Item 32. The computer program product of item 31, wherein the computer-readable storage medium further contains computer program code for, in response to determining that the input image presents two or more skin abnormalities, providing to the client device information that the input image cannot be diagnosed. (Item 36) Item 32. The computer program product of item 31, wherein the computer-readable storage medium further contains computer program code for, in response to determining that the input image presents two or more skin anomalies, providing to the client device information about the indication without providing a prediction regarding the two or more skin anomalies in the input image. (Item 37) Item 32. The computer program product of item 31, wherein the machine-learned model is also a diagnostic model further configured to receive an image and generate a prediction regarding one or more skin anomalies presented in the image. (Item 38) generating the indication further includes applying the diagnostic model to the input image to generate an output vector comprising a set of elements, the prediction regarding the one or more skin anomalies being represented by values ​​of the elements in the output vector; the input image is determined to exhibit two or more skin anomalies when a difference between values ​​of the set of elements in the output vector is below a predetermined threshold. Item 38. A computer program product according to item 37. (Item 39) 38. The computer program product of claim 37, wherein generating the indication further comprises applying the diagnostic model to the input image and generating an output vector comprising a set of elements, wherein the prediction regarding the one or more skin anomalies is represented by values ​​for a subset of elements in the output vector and the indication is represented by values ​​of the remaining elements in the output vector. (Item 40) the prediction regarding the one or more skin abnormalities is represented by one or more categories in a set of categories; generating the indication further includes applying the diagnostic model to the input image to generate an output value; the input image is determined to exhibit two or more skin anomalies when the output values ​​are assigned to the remaining categories in the set of categories. Item 38. A computer program product according to item 37. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a high-level block diagram of a system environment for a diagnostic system, according to one embodiment.

[0010] [Diagram 2] FIG. 2 illustrates a general inference process for a diagnostic model, according to one embodiment.

[0011] [Diagram 3] FIG. 3 illustrates an example inference process for a set of machine-learned diagnostic models with an ensemble architecture, according to an embodiment.

[0012] [Figure 4] FIG. 4 illustrates an exemplary inference process using independent component analysis (ICA) in conjunction with a set of diagnostic models, according to one embodiment.

[0013] [Diagram 5] FIG. 5 illustrates an example inference process for a diagnostic model involving a recurrent neural network (RNN) architecture, according to one embodiment.

[0014] [Figure 6] FIG. 6 illustrates an example inference process for a set of diagnostic models, including differentiator models for amorphous and focal skin abnormalities, according to an embodiment.

[0015] [Figure 7] FIG. 7 illustrates an example inference process for a diagnostic model with non-maximum suppression, according to an embodiment.

[0016] [Figure 8] FIG. 8 illustrates an example inference process for an uncertainty model, according to one embodiment.

[0017] [Figure 9] FIG. 9 is a block diagram of an architecture of a diagnostic system, according to one embodiment.

[0018] [Figure 10] FIG. 10 illustrates an example training process for the set of diagnostic models shown in FIG. 4, according to one embodiment.

[0019] [Figure 11] FIG. 11 illustrates an example data flow for diagnosing overlapping skin anomalies in an image, according to an embodiment.

[0020] [Figure 12] FIG. 12 illustrates an example data flow for generating an indication of two or more skin anomalies in an input image, according to an embodiment.

[0021] The drawings depict various embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the present invention as described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Overview Figure 1 is a high-level block diagram of a system environment for a diagnostic system, according to one embodiment. The system environment 100 illustrated by Figure 1 includes one or more client devices 110, a network 120, and a diagnostic system 130. In alternative configurations, different and / or additional components may be included in the system environment 100.

[0023] The diagnostic system 130 is a system for providing various types of computerized diagnoses to a user of the client device 110. The diagnostic system 130 may receive images of an anatomical part of a patient and generate a prediction as to whether the patient has one or more health conditions based on the images. For example, the diagnostic system 130 may receive images of the patient's skin and generate a prediction as to whether the patient has one or more skin abnormalities, such as a rash, a mole, eczema, acne, and herpes. As another example, the diagnostic system 130 may receive radiological images of the patient's brain and generate a prediction as to whether the patient has a brain tumor. The predictions generated by the diagnostic system 130 can be used as stand-alone diagnoses or can be used to assist, for example, a hospital physician in interpreting the patient's images.

[0024] The diagnostic system 130 receives images of a patient and trains a set of machine-learned diagnostic models configured to generate predictions for one or more health conditions. The images received by the diagnostic system 130 may be medical images acquired in a hospital, such as radiological images, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, x-ray images, ultrasound or sonography images, tactile images, or thermography images. In another case, the images may be photographs taken by individual users of the client devices 110 (e.g., in their homes or offices). The images may show symptoms or other presentations on the patient's anatomy that the diagnostic system 130 may use to infer health conditions responsible for generating those symptoms.

[0025] In one embodiment, the diagnostic system 130 trains a set of machine-learned models to generate predictions for images that include symptoms for two or more health conditions of a patient. Specifically, a patient may have symptoms from multiple health conditions appearing in the same image. For example, an image may contain a psoriatic plaque and a birthmark, both skin abnormalities, located on the same arm, spatially separated from one another. As another example, an image may contain skin abnormalities that overlap one another, such as a psoriatic plaque with a mole within its borders. However, existing computerized diagnostic systems may have difficulty generating accurate diagnoses in these cases, especially when symptoms overlap, because the symptoms may be an unknown combination of an individual's results or the patient's underlying health conditions. Furthermore, existing diagnostic systems may be configured to generate predictions for a single condition. Thus, even if an existing diagnostic system is capable of detecting one of the health conditions in an image, the detected condition will be to the exclusion of the remaining health conditions in the image, leading to a missed diagnosis that may lead to significant health issues for the subject of the image.

[0026] During the inference process, the diagnostic system 130 receives a request to diagnose a health condition in an input image. The input image includes overlapping symptoms from two or more health conditions of a patient. The diagnostic system 130 accesses a set of machine-learned diagnostic models from a database. Each machine-learned model includes an individual set of trained weights determined through a training process. The diagnostic system 130 generates an individual prediction for each of the two or more health conditions in the input image by applying the set of diagnostic models to the input image. The prediction indicates the likelihood that the input image exhibits symptoms from the individual health conditions. The diagnostic system 130 generates a diagnosis for the input image from the predictions and provides the diagnosis back to the client device 110.

[0027] In one particular embodiment, which will be referred to throughout the remainder of this specification, the two or more health conditions are two or more skin abnormalities that have overlapping symptoms that present on the skin of a patient. For example, a mole may appear in an area of ​​skin with a rash. As another example, an area of ​​skin with eczema may overlap with a portion of skin with contact dermatitis. However, it should be understood that in other embodiments, the two or more health conditions may be any type of health-related condition, such as a disease or allergy, that may produce overlapping or spatially separated symptoms in the same anatomical part of a patient. For example, the two or more health conditions may be different types of tumors, blood disorders, and the like, that may present symptoms on a patient individually or in combination.

[0028] 2 illustrates a general inference process for diagnostic models, according to one embodiment. A diagnostic system 130 receives an input image 210 of a patient suspected of having a skin anomaly. Specifically, the input image 210 shows a first skin anomaly 212 and a second skin anomaly 214 overlapping each other on the patient's arm. The diagnostic system 130 accesses a set of machine-learned diagnostic models from a database. The diagnostic system 130 applies the diagnostic models to the input image 210 and generates two or more predictions v'1, v'2, ..., v' for the input image 210. n Generates prediction v' imay indicate the likelihood that the input image 210 contains a symptom from the individual skin anomaly i. In the example shown in FIG. 2, prediction v'1 may indicate a high likelihood that the input image 210 contains a mole, and prediction v'2 may indicate a high likelihood that the input image 210 also contains a rash. The remaining predictions for other skin anomalies may be associated with significantly lower probabilities. The diagnostic system 130 determines a diagnosis that the input image 210 contains a mole and a skin rash by selecting a subset of the predictions with probabilities above a pre-determined threshold, and provides the diagnosis to the client device 110.

[0029] Returning to the system environment of FIG. 1, the diagnostic system 130 generally uses a training corpus of images and signs to train a separate set of weights for a diagnostic model to reduce a loss function. The loss function for the diagnostic model indicates the difference between an estimated output generated by applying the diagnostic model with the estimated set of weights to input data in the training corpus that the diagnostic model is configured to receive, and the actual signs in the training corpus that represent the type of data that the diagnostic model is configured to predict. The estimated set of weights for the diagnostic model is iteratively updated to reduce the loss function until a pre-determined criterion is reached. The training process for a set of diagnostic models is described in more detail in conjunction with FIG. 9.

[0030] As described herein, Figures 3-7 illustrate various architectures of a set of diagnostic models that may be used by the diagnostic system 130 to generate predictions regarding two or more health conditions present in an image during an inference process. Each architecture described below may include a set of trained weights determined through a training process, which will be described in conjunction with Figure 9. By using the architectures of the set of diagnostic models described herein, the diagnostic system can generate more accurate predictions regarding images containing two or more overlapping health conditions compared to existing systems.

[0031] FIG. 3 illustrates an exemplary inference process for a set of machine-learned diagnostic models with an ensemble architecture, according to an embodiment. In one embodiment, the set of diagnostic models includes an ensemble set of machine-learned diagnostic models. Specifically, each diagnostic model in the ensemble is configured to receive an input image and generate one or more predictions about whether a patient has one or more health conditions by applying a set of trained weights for the diagnostic model. In the example shown in FIG. 3, the ensemble set of diagnostic models includes "Diagnostic Model 1", "Diagnostic Model 2", and "Diagnostic Model 3".

[0032] A diagnostic model in the ensemble set is configured to generate predictions for a different or identical set of health conditions from other diagnostic models in the ensemble set. In the example shown in Figure 3, "Diagnostic Model 1" generates a prediction v' that indicates the likelihood that an input image contains a mole. M1 1 and a prediction v' indicating the likelihood that the input image contains a rash. M1 As another example, “Diagnostic Model 2” may be configured to generate a single prediction v′ that indicates the likelihood that the input image contains a rash. M2 1, where the “diagnostic model 3” generates a prediction v′ indicating the likelihood that the input image contains a mole. M3 1 and a prediction v' indicating the likelihood that the input image contains acne. M3 It may be configured to generate 2.

[0033] During the inference process, the diagnostic system 130 generates an individual prediction for each of two or more health conditions in the input image by applying an ensemble set of diagnostic models to the input image and combining predictions from the ensemble set for each individual health condition. In one instance, a prediction for an individual health condition is generated by averaging the predictions for the health conditions generated by the ensemble set. In the example shown in FIG. 3, a prediction v'1 indicating the likelihood that a patient has a rash is generated by averaging the predictions v'1 from "Diagnostic Model 1" and "Diagnostic Model 2". M1 2. v'M2 1. However, it should be understood that in other embodiments, predictions from diagnostic models in the ensemble may be combined in any other way other than averaging. The diagnostic system 130 determines a diagnosis by selecting a subset of predictions with probabilities above a pre-determined threshold.

[0034] In one embodiment, the diagnostic models in the ensemble set can be configured to generate multiple outputs that do not necessarily correspond to the same set of health conditions each time the diagnostic models are applied to an input image. In such an embodiment, the diagnostic system 130 may group predictions from the ensemble set by similarity of the predictions and intermediate features, and a prediction for two or more health conditions present in the input image may be generated by combining the predictions from each identified group. In another case, the diagnostic system 130 may group predictions from the ensemble set by similar locations, for example using an attention model, and a prediction for two or more health conditions in the input image may be generated by combining the predictions for each identified location.

[0035] FIG. 4 illustrates an exemplary inference process using independent component analysis (ICA) in conjunction with a set of diagnostic models, according to an embodiment. In one embodiment, the set of diagnostic models includes a feature extractor model, an independent component analysis (ICA) model, and a feature classifier model. The feature extractor model is configured to receive an input image and generate an image tensor characterizing a plurality of spatial features in the input image by applying a first set of trained weights. The ICA model is configured to receive the image tensor and extract a plurality of components from the image tensor. The feature classifier model is configured to receive the components and generate a prediction indicative of a likelihood of an individual health state for the components by applying a second set of trained weights.

[0036] During the inference process, the diagnostic system 130 generates an image tensor 416 for the input image by applying a feature extractor model to the input image. The diagnostic system 130 performs ICA to extract multiple components CF1, CF2, CF3, ..., CF from the image tensor 416, considering each spatial location in the image tensor 416 as an observation. n For each extracted component, the diagnostic system 130 calculates the component's contribution to the image tensor 416 and generates an individual tensor for the component. In one example, the individual tensor for the component is generated by calculating the contribution of the remaining components among the components and subtracting the contribution of the remaining components from the image tensor 416. In the example shown in FIG. 4, the individual tensor 418 for component CF1 is generated by subtracting the contribution of the remaining components CF2, CF, ..., CF n CF1, and subtracting the contribution from the image tensor 416. While FIG. 4 only illustrates an individual tensor 418 for component CF1, the process is similar for the remaining components CF2, CF3, ..., CF n may be iterated to generate separate tensors for each of

[0037] For each tensor, diagnostic system 130 generates a prediction by applying the feature classifier model to the component-related tensor. In the example shown in FIG. 4, the prediction v'1 for component CF1 is generated by applying the feature classifier model to the component-related tensor 418. Although FIG. 4 illustrates only the prediction v'1 for component CF1, the process also generates predictions for the remaining components CF2, CF3, ..., CF n The diagnostic system 130 may iterate to generate an individual prediction from each of the predictions v'1, v'2, ..., v' by selecting a subset with a probability above a threshold. n The diagnosis is based on:

[0038] Based on the number of components extracted from the image tensor, the diagnoses may each correspond to a different health condition underlying the input image, or a subset of diagnoses may correspond to the same health condition if the number of extracted components exceeds the number of health conditions included in the input image. In one example, the diagnostic system 130 extracts a predetermined number of components based on an estimated number of health conditions in the input image. In another example, the diagnostic system 130 performs ICA with an increasing number of components and iteratively generates predictions and diagnoses with each new component. The diagnostic system 130 completes the inference process when further increasing the number of components does not increase the number of unique diagnoses and there is convergence of the criterion function for the ICA. In one example, there may be a threshold number of added components to determine convergence. For example, three components may be added while the set of diagnoses remains unchanged to be considered for convergence.

[0039] In another embodiment, the components are pre-extracted during a training process. During the inference process, the diagnostic system 130 decomposes the image tensor into a mixture of components, calculates the contribution of each pre-determined component to the image tensor, and generates a separate tensor for each component. The diagnostic system 130 identifies a subset of components that have a contribution above a threshold. For each tensor, the diagnostic system 130 generates a prediction by applying a feature classifier model to the tensor for the components in the subset. In some embodiments, each pre-determined component may be pre-classified and stored using the feature classifier model. The classification result is then determined by using the stored components associated with the contributing components.

[0040] FIG. 5 illustrates an exemplary inference process for a diagnostic model with a recurrent neural network (RNN) architecture, according to an embodiment. In one embodiment, the set of diagnostic models includes a diagnostic model with an RNN architecture, including one or more neural network layers with a set of weights. The RNN architecture is configured to receive a series of images and generate predictions successively for two or more health conditions presented in the series of images. The series of input images may be a series of identical input images repeated, or may be a series of images capturing an anatomical part of a patient upon issuance of different views, viewpoints, filters, and the like. In the example shown in FIG. 5, the series of images 510(1), 510(2), ..., 510(n) are the same images of the skin of a patient's arm. However, in another example, the series of images may include images of the same arm captured from different viewpoints or with different colors or scales.

[0041] Specifically, for a current image in the sequence, the RNN architecture computes the hidden state h i-1 and applies the first set of trained weights to obtain a hidden state h i The RNN architecture is further configured to generate a current image h i , and predicts the individual health states v by applying a second set of trained weights. i After each iteration, the RNN architecture may be further configured to determine whether the diagnostic model should generate predictions for other health conditions in the input image or complete the inference process. If the RNN architecture determines to continue generating predictions, it receives the next image 510(i+1) in the sequence and generates a prediction v(i+1) for another health condition. i+1This process is repeated until the RNN architecture decides to complete the inference process. In the example shown in Figure 5, a prediction v'1 may indicate that the input image contains a mole, while a next prediction v'2 may indicate that the input image contains a rash.

[0042] In one embodiment, the RNN architecture further comprises a hidden state h i-1 In addition, at each iteration, the memory vector m i and receiving a prediction v regarding an individual health state from the current image 510(i). i It is configured to generate a memory vector m i may represent a vector that contains information about the health states that have already been classified up to the current iteration i. For example, a memory vector m i may be an aggregation of previous predictions or previous diagnoses up to the current iteration. In the example shown in FIG. 5, the memory vector m2 in the second iteration may indicate that a prediction or diagnosis regarding a mole has been made up to the second iteration based on v1. By further configuring the RNN architecture to receive the memory vectors, the diagnostic system 130 may structure a diagnostic model to generate predictions regarding previously unpredicted health conditions in each iteration of the inference process.

[0043] During the inference process, the diagnostic system 130 generates an individual prediction for each of the two or more health conditions by repeatedly applying the RNN architecture to the series of input images. In one embodiment, when the RNN architecture is further configured to receive a memory vector, the diagnostic system 130 may obtain an individual memory vector for an iteration based on the predictions generated up to the current iteration. The diagnostic system 130 generates an individual prediction for each of the two or more health conditions by repeatedly applying the RNN architecture to the series of images and memory vectors.

[0044] FIG. 6 illustrates an exemplary inference process for a set of diagnostic models including differentiator models for amorphous and focal skin anomalies, according to an embodiment. In one embodiment, the set of machine-learned diagnostic models includes a differentiator model, an amorphous anomaly model, and a focal anomaly model. The differentiator model is configured to receive an input image and generate an indication of whether the input image includes an amorphous skin anomaly or a focal skin anomaly by applying a first set of trained weights. The amorphous anomaly model is configured to receive an input image and generate a prediction of whether the input image includes one or more amorphous skin anomalies, such as a rash, eczema, etc., by applying a second set of trained weights. The focal anomaly model is configured to receive an input image and generate a prediction of whether the input image includes one or more focal skin anomalies, such as a mole, acne, etc., by applying a third set of trained weights.

[0045] Specifically, amorphous skin abnormalities, such as rashes, eczema, dermatitis, etc., may appear on the skin without a distinct shape or morphology, while focal skin abnormalities, such as moles, may be clearly defined by localized points on the skin. Thus, due to the differences in shape and morphology, the diagnostic system 130 may obtain more accurate predictions by training two separate models, one model generating predictions for amorphous skin abnormalities and another model generating predictions for focal skin abnormalities.

[0046] During the inference process, the diagnostic system 130 generates an indication as to whether the input image contains a focal or amorphous skin anomaly by applying the differentiator model to the input image. Rather than generating a prediction that can be used to diagnose a specific type of skin anomaly (e.g., mole, acne, rash), the output of the differentiator model may simply indicate whether the input image contains any skin anomaly that can be classified as amorphous and / or any skin anomaly that can be classified as focal. In response to determining that the input image contains an amorphous skin anomaly, the diagnostic system 130 generates a specific prediction regarding one or more amorphous skin anomalies by applying the amorphous anomaly model to the input image. Furthermore, in response to determining that the input image contains a focal skin anomaly, the diagnostic system 130 generates a specific prediction regarding one or more focal skin anomalies by applying the focal anomaly model to the input image.

[0047] 7 illustrates an example inference process for a diagnostic model with non-maximal suppression, according to an embodiment. In one embodiment, a machine-learned diagnostic model is configured to receive an input image and generate a vector including predictions for a list of health conditions. In the example shown in FIG. 7, the prediction v′ generated by the diagnostic model may include one or more elements, each corresponding to a prediction for a distinct health condition. For example, a first element may correspond to a prediction that the input image includes eczema, a second element may correspond to a prediction that the input image includes dermatitis, etc.

[0048] During the inference process, the diagnostic system 130 generates individual predictions for two or more health conditions by applying diagnostic models to the input image. In one embodiment, the diagnostic system 130 determines the diagnosis by using non-maximal suppression. Specifically, the diagnostic system 130 divides the different health conditions represented by the vector v' into groups and selects, for each group, only the health condition with the highest probability as part of the diagnosis. In one case, the health conditions are grouped according to the degree of similarity in which their symptoms are visually presented on the patient, and thus each group contains health conditions for which the diagnostic model is likely to be confused in the image. In the example shown in FIG. 7, the health conditions for the first through fourth elements in the vector are assigned to "group 1", and the health conditions for the n-3 to n elements are assigned to "group m". The diagnostic system 130 determines the diagnosis v' by selecting the health condition for the second element in "group 1" and the n-1 element in "group m" as each element has the highest probability in each group, and setting the remaining elements to zero. nm Determine.

[0049] Returning to the system diagram of FIG. 1, the diagnostic system 130 may also train a set of machine-learned indicator models configured to receive images of a patient and generate an indication as to whether the input image presents two or more health conditions. Rather than actively diagnosing a specific health condition itself, the diagnostic system 130 may generate an indication as to whether the input image presents two or more health conditions and provide the result to a user of the client device 110. Specifically, rather than generating a prediction that can be used to diagnose a specific type of health condition, the output of the indicator models may simply determine whether the input image exhibits symptoms (e.g., overlapping symptoms) from two or more health conditions. Because input images presenting multiple health conditions may be difficult to diagnose, the user of the client device 110 may receive such an indication in addition to or as an alternative to an active diagnosis. The user may then decide to provide the input image to a medical professional to obtain a more accurate diagnosis.

[0050] FIG. 8 illustrates an exemplary inference process for an indicator model, according to an embodiment. In one embodiment, the diagnostic model trains a machine-learned indicator model configured to receive an input image and generate an indication e′ of whether the input image presents two or more health conditions by applying a set of trained weights. In the example shown in FIG. 8, the diagnostic system 130 generates the indication e′ by applying the indicator model to the input image. The diagnostic system 130 may simply provide information to a user of the client device 110 that the image contains too many health conditions for an accurate diagnosis.

[0051] In another embodiment, rather than training a separate indicator model as described in conjunction with FIG. 8, the diagnostic system 130 further incorporates the indicator functionality into the diagnostic model. For example, the diagnostic model may be configured to receive an input image and generate a prediction for one or more health conditions contained in the image, and an indication of whether the input image presents more than one health condition. For example, the set of diagnostic models described in conjunction with FIGS. 2-7 may be further configured to generate such an indication in addition to the prediction regarding the health condition. As another example, the diagnostic model may be configured to generate a single output at a time, and may generate such an indication as a single output in response to receiving an input image indicating more than one health condition.

[0052] In one instance, the diagnostic model is configured to generate such an indication by categorizing an input image exhibiting two or more health conditions as separate categories and generating probabilities for the separate categories in response to receiving the input image. In another instance, the diagnostic model is configured to generate such an indication when predictions for the health conditions are output with relatively equal probabilities or confidence levels indicating ambiguity of the predictions. For example, the diagnostic system 130 may determine that the input image includes symptoms from two or more health conditions when predictions for having a mole, a rash, and a skin disease are output as being 0.32, 0.29, and 0.35, respectively.

[0053] In one embodiment, the diagnostic system 130 may determine that the input image presents more than one condition if the indication likelihood exceeds a threshold likelihood, otherwise the input image presents a single health condition. In some embodiments, the indication may represent two possibilities: a first possibility for whether the input image presents more than one health condition, and a second possibility for whether the input image presents a single health condition. The diagnostic system 130 may determine that the input image presents more than one condition if the first likelihood exceeds a threshold, and may determine that the input image presents a single health condition if the second likelihood exceeds a threshold.

[0054] Thus, in one embodiment, the diagnostic system 130 may receive a request for diagnosis of a health condition in an input image. The diagnostic system 130 may access an indicator model or a diagnostic model with indicator functionality from a database. The diagnostic system 130 may generate an indication representing the likelihood that the input image includes two or more health conditions by applying the indicator model to the input image. The diagnostic system 130 makes a determination as to whether the input image presents two or more health conditions from the indication generated for the input image. The diagnostic system 130 generates a result for the request based on the determination and provides the result to the client device 110. The result may include medical treatment options for the individuals in the input image based on the determination, and the suggested treatment options may be provided to the client device 110.

[0055] In one embodiment, the diagnostic system 130 may determine that the indication is uncertain in that the diagnostic model 130 cannot make a definitive decision as to whether the input image presents two or more health conditions. For example, this may occur when a first possibility (e.g., the likelihood that the image presents two or more health conditions) and a second possibility (e.g., the likelihood that the image presents a single health condition) are both below a threshold (e.g., 80%). In such an embodiment, the diagnostic system 130 may output a result to the client device that a diagnosis cannot be made and may suggest, for example, that the user of the client device 110 obtain medical expertise.

[0056] In some embodiments, in response to determining that the input image presents two or more skin anomalies, the diagnostic system 130 may provide information about the indication to the client device 110 without providing a prediction regarding the two or more health conditions in the input image. For example, the diagnostic system 130 may review and review government regulations and policies (e.g., regulations issued by the Food and Drug Administration (FDA), or guidelines issued by the Centers for Disease Control (CDC), etc.) and generate the required results to comply with these regulations. Thus, even if the diagnostic system 130 is capable of generating predictions regarding two or more health conditions, the diagnostic system 130 may omit these results in output to the client device 110 if the regulations, for example, constrain the computerized diagnosis of the two or more health conditions presented in the image. In such an embodiment, the diagnostic system 130 may simply provide information to the user that the input image contains two or more health conditions, but due to government regulations, a diagnosis cannot be made.

[0057] In some embodiments, in response to determining that the input image exhibits a single health condition, the diagnostic system 130 may access and select, from a database, a diagnostic model configured to generate a prediction for the single health condition. The diagnostic system 130 generates a prediction for the input image by applying the selected diagnostic model to the input image. The diagnostic system 130 generates a diagnosis for the single skin anomaly based on the prediction, and provides the resulting diagnosis to the client device.

[0058] In some embodiments, in response to determining that the input image exhibits two or more health conditions, the diagnostic system 130 may access and select from a database a set of diagnostic models configured to generate a respective prediction for each of the two or more health conditions. For example, the diagnostic system 130 may access the models described in conjunction with FIGS. 2-7 that are configured to generate predictions for the two or more health conditions. The diagnostic system 130 generates predictions for the two or more health conditions in the input image by applying the set of diagnostic models to the input image. The diagnostic system 130 generates a diagnosis for the input image from the predictions and provides the resulting diagnosis to the client device 110.

[0059] Although the machine-learned models described in conjunction with Figures 2-8 are configured to receive input data in the form of images, the machine-learned models may also be configured to receive additional types of input data. For example, the additional types of input data may include patient historical data, such as the patient's height or weight, the patient's medical history, or patient responses to questions requesting information about existing medical conditions known to the patient, and the like. For example, the patient's medical history may indicate that the patient is predisposed to certain health conditions, which may affect the model's predictive outcomes.

[0060] Returning to the diagram of Figure 1, client device 110 provides images of a patient's anatomical parts to diagnostic system 130 so that diagnostic system 130 may generate and display a prediction or other indication on client device 110. A user of client device 110 may be a medical professional in a hospital who desires to be assisted in diagnosing a patient's health condition using a computerized diagnostic system. As another example, a user of client device 110 may be an individual in a home or office who desires to obtain a diagnosis of an underlying health condition based on symptoms presenting on the individual's body.

[0061] In one embodiment, the client device 110 includes a browser that allows a user of the client device 110 to interact with the diagnostic system 130 using standard internet protocols. In another embodiment, the client device 110 includes a dedicated application that is specifically designed (e.g., by an entity involved with the diagnostic system 130) to enable interaction between the client device 110 and a server. In one embodiment, the client device 110 includes a user interface that allows a user of the client device 110 to interact with the diagnostic system 130 and view predictions of health conditions within an input image. For example, the user interface may be configured to overlay predictions generated by the diagnostic system 130 over locations on the image that are predicted to have individual health conditions.

[0062] The client device 110 may be a computing device such as a smartphone with an operating system such as ANDROID® or APPLE® IOS®, a tablet computer, a laptop computer, a desktop computer, or any other type of network-enabled device that includes or can be configured to connect with a camera. In another embodiment, the client device 110 is a headset that includes a computing device or a smartphone camera for generating an augmented reality (AR) environment for a user, or a headset that includes a computing device for generating a virtual reality (VR) environment for a user. A typical client device 110 includes the hardware and software required to connect to a network 122 (e.g., via WiFi and / or 4G or 5G or other wireless telecommunications standards).

[0063] Network 122 provides a communications infrastructure between client device 110 and diagnostic system 130. Network 122 is typically the Internet, but may be any network including, but not limited to, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network. (Diagnostic System Architecture)

[0064] 9 is a block diagram of an architecture of a diagnostic system 130, according to an embodiment. The diagnostic system 130 illustrated by FIG. 9 includes a training module 920, a data management module 925, a training module 330, a prediction module 930, and a treatment output module 935. The diagnostic system 130 also includes a training data store 960 and a model data store 965. In alternative configurations, different and / or additional components may be included in the diagnostic system 130.

[0065] The data management module 920 acquires and manages training data stored in a training data store 960 that may be used to train a set of models described in conjunction with the diagnostic system 130 of FIGS. 2-8. The training data store 960 includes information extracted from images or videos of an individual's anatomical parts and known diagnoses of health conditions presented in these images. For example, the training data store 960 may include information extracted from medical images, such as radiological images, computed tomography (CT) scans, medical magnetic resonance imaging (MRI) scans, x-ray images, ultrasound or sonography images, tactile images, or thermographic images, acquired by the data management module 920 from hospitals, medical image repositories, research image repositories, and the like. As another example, the training data store 960 may include information extracted from images taken by an individual, such as images taken on a smartphone camera, acquired by the data management module 920 from a user of the client device 110.

[0066] The training module 925 uses the information in the training data store 960 to train a set of models. In particular, for a given diagnostic model M i With respect to the training data S i is the input data x j∈S (e.g., training images or videos) and the input data x j∈S A label y that is a known characterization (e.g., diagnostic) of j∈Si Let us consider multiple training instances j=1, 2, …, |S i Specifically, the input data x j may include information extracted from training images capturing anatomical parts of an individual. In one embodiment, when the machine-learned model is configured to receive additional types of data in addition to, or as an alternative to, the input images, the input data x j∈S may also include additional types of data about the patients in the training images (e.g., demographic information about the patients). j∈Si may be obtained by a human operator, such as a physician, medical professional, or other individual, who may validate the characterization for the input data. In another case, the signature itself may be a prediction generated by a machine learning model that is subsequently validated by a human operator.

[0067] The training module 925 trains the set of weights for the model by repeatedly iterating between forward pass and backpropagation steps. During the forward pass step, the training module 920 trains a corresponding subset S of the training data. i , and apply the model with the estimated set of weights to the input data x j∈S By applying j∈S The training module 920 generates the estimated output y' for multiple training instances. j∈S and label y j∈SDuring the backpropagation step, the training module 920 iteratively updates a set of weights for the model with a backpropagation error term obtained from the loss function. This process is repeated until the change in the loss function meets a pre-determined criterion. For example, the criterion may be triggered if the change in the loss function at each iteration falls below a threshold.

[0068] In one embodiment, the loss function is, for example, the label y j When is continuous, it can be determined by: [ka] In the formula, θ p is a set of weights for the diagnostic model. In another embodiment, the loss function is, for example, j When is a binary value, it can be found by: [ka] However, it should be understood that the loss function can be any other function, such as the L1-norm, the L-infinity norm, etc., that indicates the difference between the actual labels and the estimated outputs generated during each training interaction.

[0069] As described below, the training module 925 trains a set of diagnostic models, described in conjunction with FIGS. 2-7, that may be used to determine active diagnoses for the input images. In one embodiment, one or more diagnostic models are configured to generate predictions y′=v′ as an output vector, with each element in the output vector v′ corresponding to a prediction for a distinct health state. In such an embodiment, the training indicators y j =v jmay be encoded as a one-hot encoded vector, with each element in the vector being a non-zero binary value (e.g., a value of 1) if the training image is diagnosed with a distinct health condition for the element, and zero if no diagnosis is present. Thus, for training images exhibiting more than one health condition, each distinct element may be set to an appropriate value such that more than one element in the vector has a non-zero value. For example, training data for a diagnostic model configured to predict rashes and moles may include indicators "0001", with a first element indicating the presence of a rash in the corresponding training image and a second element indicating the absence of a mole in the training image. In another embodiment, the diagnostic model may be configured to generate a prediction y'=v' as an output value that may be assigned to one or more categories, each representing a distinct health condition. In such an embodiment, an indicator y for the diagnostic model may be set to an appropriate value such that more than one element in the vector has a non-zero value. j =v j may be values ​​that represent categories for distinct health conditions diagnosed in the training images. For example, training data for an example diagnostic model may include a label "1" indicating the presence of a rash in the training image, or a "2" indicating the presence of a mole in the training image.

[0070] The training module 920 trains the set of weights for an ensemble set of diagnostic models as described in conjunction with FIG. 3. Specifically, each diagnostic model in the ensemble set may be configured to generate predictions for a distinct set of one or more health conditions. The diagnostic models in the ensemble set may differ from one another with respect to the architecture of the model, the set of health conditions that the model is configured to predict, and the like. In one embodiment, the diagnostic models are each configured as a neural network architecture including a set of layers of nodes, each layer connected to the previous layer via a set of weights. Thus, the diagnostic models in such an ensemble set may differ from one another with respect to the number of layers, nodes, and connections in the architecture.

[0071] The training module 925 receives input data x j∈Siand corresponding labels y indicating the presence of the specific health condition that the diagnostic model is configured to predict. j∈S =v j∈S During the forward pass step, the training module 925 trains the set of weights for the diagnostic models in the ensemble set using a training data set including: j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the diagnostic model using terms obtained from the loss function.

[0072] The training module 920 trains the sets of weights for the feature extractor model and the feature classifier model described in conjunction with FIG. 4. In one embodiment, the feature extractor model and the feature classifier model are trained by training a neural network architecture that includes a first portion and a second portion. The set of weights for the first portion of the neural network architecture may be stored as a set of weights for the feature extractor model, and the set of weights for the second portion may be stored as a set of weights for the feature classifier model. The set of weights for the feature classifier model is subsequently retrained using the components extracted from the training images.

[0073] FIG. 10 illustrates an exemplary training process for the set of diagnostic models shown in FIG. 4, according to an embodiment. The training module 925 trains a neural network architecture 1036, which includes a set of layers of nodes, each layer connected to the previous layer via a set of weights. The neural network architecture 1036 may be configured to generate predictions for a discrete set of one or more health conditions. In one embodiment, the neural network architecture 1036 is configured as a deep neural network (DNN), a convolutional neural network (CNN), or any other type of neural network architecture that can receive image data and generate an output.

[0074] The training module 925 receives input data x j∈Si and the corresponding labels y indicating the presence of the specific health condition that the feature classifier model is configured to predict. j∈S =v j∈S During the forward pass step, the training module 925 trains a set of weights for the neural network architecture using a training data set including: j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the neural network architecture using terms obtained from the loss function.

[0075] After the training process for the neural network architecture is completed, the training module 925 stores the first portion of the neural network architecture as a set of weights for the feature extractor model and the second portion of the neural network architecture as a set of weights for the feature classifier model. In one embodiment, the first portion and the second portion are selected such that the second portion includes a set of weights for layers that are located after the layers for the first portion of the neural network architecture. For example, the first portion may include a set of weights for the first three layers of the neural network architecture, while the second portion may include a set of weights for the last five layers of the neural network architecture.

[0076] The training module 925 then j∈Si Multiple components are extracted from the training image as j∈S and a corresponding indicator y indicating the presence of a specific health condition in the training images that the feature classifier model is configured to predict. j∈S =v j∈Sand retrain the set of weights for the feature classifier model using the training dataset, which includes: In one example, the components for the training images are extracted by applying the trained feature extractor model to the training images, generating an image tensor for the training images, and performing ICA on the image tensor to generate the components. During the forward pass step, the training module 925 trains the feature classifier model to the components CF j∈S By applying j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the feature classifier model using terms obtained from the loss function.

[0077] The training module 925 trains the set of weights for the RNN architecture described in conjunction with FIG. j∈S and the corresponding labels y indicating the presence of the specific health states that the RNN architecture is configured to predict. j∈S =v j∈S and train the set of weights for the RNN architecture using a training dataset including: j may be a one-hot encoded vector that has a non-zero value for each distinct one of the health states present in the training images. During the forward pass step, the training module 925 generates a set of estimated outputs y' by repeatedly applying the RNN architecture to the set of training images. j∈S =v' j∈S In one embodiment, the label v j is the difference, e.g., |v j∈S -v' j∈SThe training modules 925 may be reordered during the forward pass step of the training process such that | is reduced or minimized across all or a subset of the possible orderings. The training module 925 determines a loss function by combining the differences between the estimated outputs and the labels at each successive iteration. During the backpropagation step, the training module 925 iteratively updates the set of weights for the RNN architecture with terms obtained from the loss function.

[0078] The training module 925 trains the differentiator model, the amorphous anomaly model, and the focal anomaly model, which are described in conjunction with FIG. 6. The training module 925 trains the differentiator model, the amorphous anomaly model, and the focal anomaly model, which are described in conjunction with FIG. j∈S and corresponding labels y indicating the presence of amorphous skin anomalies or the presence of focal skin anomalies. j∈S =v j∈S and train a set of weights for the differentiator model using a training data set including: In one example, the differentiator model is configured to generate an indication as a vector, the first element of which corresponds to the likelihood that the image contains an amorphous skin anomaly, and the second element of which corresponds to the likelihood that the image contains a focal skin anomaly. In such an example, the indication y j∈S may be encoded such that the first element of the vector is a non-zero value (e.g., a value of 1) if the training images contain amorphous skin anomalies, or zero if the images do not. Similarly, the second element of the vector is a non-zero value (e.g., a value of 1) if the training images contain focal skin anomalies, or zero if the images do not. During the forward pass step, the training module 925 generates the estimated output y' by applying the differentiator model to the training images. j∈S =v' j∈S and iteratively updates the set of weights for the differentiator model during the backpropagation step using terms obtained from the loss function.

[0079] The training module 925 receives input data x j∈Siand corresponding labels y indicating the presence of the specific amorphous skin anomalies that the model is configured to predict. j∈S =v j∈S and train the set of weights for the amorphous anomaly model using a training data set including: The training data for the amorphous anomaly model may include training images with a single amorphous skin anomaly or multiple amorphous skin anomalies. During the forward pass step, the training module 925 trains the set of weights for the amorphous anomaly model by applying the amorphous anomaly model to the training images. j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the amorphous anomaly model using terms obtained from the loss function.

[0080] The training module 925 receives input data x j∈S and corresponding labels y indicating the presence of the specific focal skin anomaly that the model is configured to predict. j∈S =v j∈S The training data set containing: and trains the set of weights for the focal anomaly model. The training data for the focal anomaly model may include training images with a single focal skin anomaly or multiple focal skin anomalies. During the forward pass step, the training module 925 trains the set of weights for the focal anomaly model by applying the focal anomaly model to the training images. j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the localized anomaly model using terms obtained from the loss function.

[0081] As described further below, the training module 925 further trains an indicator model, described in conjunction with FIG. 8, that provides an indication of whether an image exhibits two or more health conditions. The training module 925 receives input data x j∈S and a corresponding label y indicating whether the training image presents two or more health conditions. j∈S =v j∈SA training data set containing: y, y′ ... j∈Si may be encoded as a non-zero value if the training images exhibit two or more health conditions, or as zero if the training images exhibit a single health condition. During the forward pass step, the training module 925 derives the estimated output y' by applying the indicator model to the training images. j∈S =v' j∈S During the backpropagation step, the training module 925 iteratively updates the set of weights for the indicator model using terms obtained from the loss function.

[0082] The training module 925 can also train diagnostic models that incorporate the indicator functionality. The diagnostic models may be any models configured to generate computerized predictions for one or more health conditions in an image, such as the set of diagnostic models of FIGS. 2-7, or diagnostic models configured to predict a single health condition. Although the diagnostic models can generally be trained using diagnostic training data (e.g., training images and corresponding health condition indicators), the training module 925 further trains the diagnostic models to incorporate the indicator functionality by including instructional training data for the diagnostic models in addition to the diagnostic training data.

[0083] In one embodiment, the instruction training data S a is a set of training images presenting two or more health conditions and a corresponding label y indicating the presence of two or more health conditions in the training images. j∈Sa =v j∈Sa In one example, the indicators y j∈S =v j∈S is a one-hot encoded vector, the indicator y j∈Sa=v j∈Sa may be a vector having non-zero values ​​for all or most of the elements in the vector. For example, training data for a diagnostic model configured to predict a single health condition may include the indicator “

[0011] ”, which includes non-zero values ​​for all elements, when training images exhibit more than one health condition. In this manner, the diagnostic model is configured to generate an indication by predicting relatively similar or equal likelihoods for all or most of the health conditions.

[0084] In another example, the label y j∈Sa =v j∈Sa may include additional elements or categories for assigning images exhibiting more than one health condition. For example, the label y j∈S =v j∈S When y is a one-hot encoded vector, the indicator may be further configured to include a separate element for an indication in addition to the elements for the different health conditions. The separate element is a non-zero value if the training images present two or more health conditions and is zero if the training images do not present them. For example, training data for a diagnostic model configured to predict rashes and / or moles may include an indicator “

[0111] ” where a first element indicates the presence of a rash in a corresponding training image, a second element indicates the presence of a mole in the training image, and a third separate element indicates the presence of two or more health conditions in the training image. As another example, an indicator for diagnosis y j∈S =v j∈S and when the indicator is categorical, the indicator may be further configured to include separate categories of indications in addition to the categories for different health conditions. For example, training data for an example diagnostic model may include an indicator "1" indicating the presence of a rash in the training image, an indicator "2" indicating the presence of a mole in the training image, and an indicator "3" indicating the presence of two or more health conditions in the training image.

[0085] The training module 925 may store the trained machine-learned models in the model data store 965 so that they can be deployed during the inference process in the manner described in conjunction with Figures 2-8.

[0086] The prediction module 930 receives a request from the client device 110 to provide a computerized diagnosis of a health condition captured within the input image. In response to a request for a diagnosis, the prediction module 930 may select a set of diagnostic models that may be used to generate a diagnosis for the input image using diagnostic models trained by the diagnostic system 130. The prediction module 930 may apply the selected diagnostic models to generate a prediction for the health condition presented in the input image, similar to the example inference process described in conjunction with Figures 2-8. In one instance, the prediction module 930 determines a diagnosis for the health condition if the likelihood of the prediction is above a threshold. For example, the prediction module 930 may conclude that the individual health condition is indeed present in the input image if the predictability is above 0.80 or 80%. The prediction module 930 may conclude that the individual health condition is otherwise absent from the input image.

[0087] In some embodiments, the prediction module 930 may determine that a prediction for an individual health condition is inconclusive and generate information to the client device 110 that a diagnosis request cannot be made, or alternatively, output a diagnosis for the remaining health conditions that have a definitive prediction to the client device 110. For example, the prediction module 930 may determine that a diagnosis is inconclusive if the individual prediction has a probability between a first threshold and a second threshold (e.g., 20%-80%). In another case, the prediction module 930 determines a diagnosis for a health condition if the probability of the prediction is within a threshold percentage among other probabilities generated by the diagnostic model.

[0088] In one embodiment, the prediction module 930 is programmed to comply with relevant government regulations or policies governing computerized diagnosis and generates a diagnosis result to comply with the current regulations or policies. For example, the regulations or policies may dictate various prediction thresholds used to determine whether an individual health condition is present in the input image, and these thresholds may vary between different health conditions. In one case, an administrator of the diagnostic system 130 may review these regulations or policies and program the prediction module 930 to comply with them. In another case, the prediction module 930 may collect information from a website or database of a relevant body (e.g., the FDA) and generate a prediction according to rules parsed from the collected information. Thus, the prediction module 930 may review the predictions generated by the model and generate a diagnosis result on request to comply with these regulations or policies. For example, the prediction module 930 may determine a diagnosis for acne if the predictability exceeds a threshold of 80%, while the prediction module 930 may determine a diagnosis for a rash if the predictability only exceeds a threshold of 95%, in accordance with the policy set by the FDA.

[0089] Further, in response to a request for diagnosis, the prediction module 930 may select an indicator model or a diagnostic model with indicator functionality and generate an indication for the input image. The prediction module 930 may apply the selected model and generate an indication. In one case, the prediction module 930 determines that the input image presents more than one health condition if the likelihood of the indication is above a threshold. In another case, the prediction module 930 determines that the input image presents more than one health condition if the likelihood of the indication is within a threshold percentage among other outputs generated by the selected model. As described in conjunction with FIG. 8, the prediction module 930 may generate a result for the request based on the determination such that an appropriate diagnostic model may be selected to generate a diagnosis or information that a diagnosis cannot be made due to compliance with government regulations or policies for computerized diagnosis is output to a user of the client device 110.

[0090] The treatment output module 935 provides potential treatment options in conjunction with the diagnosis provided by the prediction module 930. Given the diagnosis result, it may be advantageous for the user of the client device 110 to receive potential options for treatment with the diagnosis, for example, so that the health condition may be treated within a short period of time, if required. In one embodiment, the treatment output module 935 may generate treatment options by consulting a human medical professional, deriving options from expert system reasoning, or previous systems. In one embodiment, the treatment output module 935 may suggest treatment options that are known to reduce or avoid side effects for the health condition presented by the patient. (Methods for diagnosing overlapping skin disorders)

[0091] FIG. 11 illustrates a method of diagnosing overlapping skin anomalies in an input image. The diagnostic system 130 receives 1102 a request from a client device (e.g., via the network 120) to diagnose skin anomalies in an input image. The input image includes overlapping skin anomalies on the skin of a patient. The diagnostic system 130 accesses 1104 a set of machine-learned models (e.g., via the network 120 using the prediction module 930) from a database (e.g., the model data store 965, where the models 965 are trained using the training data 960 and may include models described in conjunction with FIG. 2-8). Each machine-learned model includes an individual set of trained weights. The diagnostic model 130 generates 1106 an individual prediction for each of two or more skin anomalies in the input image (e.g., using the prediction module 930) by applying the set of machine-learned models to the input image (e.g., the inference process described in conjunction with FIG. 2-8). The prediction indicates the likelihood that the individual skin anomaly among the two or more skin anomalies is shown in the input image. The diagnostic system 130 generates 1108 a diagnosis of the overlapping skin anomaly from the prediction (e.g., using the prediction module 930). The diagnostic system 130 provides 1100 the diagnosis to the client device (e.g., via the network 120). (Method of generating two or more indications of skin abnormalities)

[0092] FIG. 12 illustrates a method of generating an indication of two or more skin anomalies in an input image. The diagnostic system 130 receives a request for diagnosis of skin anomalies in an input image from a client device (e.g., via the network 120) 1202. The diagnostic system 130 accesses a machine-learned model (e.g., via the network 120 using the prediction module 930) from a database (e.g., a model data store 965, where the model 965 is trained using the training data 960 and may include a model described in conjunction with FIG. 8) 1204. The machine-learned model includes a separate set of trained weights. The diagnostic model 130 generates an indication (e.g., using the prediction module 930) 1206 by applying the machine-learned model to the input image (e.g., the inference process described in conjunction with FIG. 8). The diagnostic system 130 generates a result 1208 with respect to the request based on a determination of overlapping skin anomalies from the prediction (e.g., using the prediction module 930). The diagnostic system 130 provides 1210 the results to the client device (eg, via the network 120). (overview)

[0093] The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will appreciate that numerous modifications and variations are possible in light of the above disclosure.

[0094] Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or logically, will be understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. In addition, it has also proven convenient at times, without loss of generality, to refer to arrangements of these operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0095] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented with a computer program product comprising a computer-readable medium containing computer program code that may be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0096] Embodiments of the present invention may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions that may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design for increased computing power.

[0097] Embodiments of the invention may also relate to products produced by the computing processes described herein. Such products may comprise information resulting from the computing processes, which information may be stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of the computer program product or other data combination described herein.

[0098] Finally, the language used herein has been selected primarily for readability and instructional purposes, and may not be described to precisely delineate or limit the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but rather by any claims issuing on an application based thereon. Thus, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

1. A method performed by a diagnostic system having one or more processors, the method comprising: receiving, from a client device, a request for diagnosis of a skin anomaly presented in an input image; the diagnostic system generating an indication by providing the input image as an input to a machine-learned model, the machine-learned model outputting the indication, the indication representing a likelihood that the input image exhibits two or more skin anomalies; and in response to the indication being of a single skin anomaly within the input image, said diagnostic system generating a computerized diagnosis of said single skin abnormality; and the diagnostic system outputting the computerized diagnosis for display on the client device; in response to the indication showing two or more skin anomalies in the input image, the diagnostic system outputting a message for display; A method comprising:

2. The method comprises: in response to the indication being of the single skin abnormality, the diagnostic system accessing a diagnostic model from a database; the diagnostic system generating a prediction for the skin anomaly in the input image by providing the input image as an input to the diagnostic model, the diagnostic model outputting the prediction; generating the computerized diagnosis for the input image from the prediction; and The method of claim 1 further comprising:

3. The method of claim 1, wherein the machine-learned model is configured as a neural network architecture including at least a set of multiple layers of multiple nodes, each layer connected to the previous layer through a separate set of trained weights.

4. The method of claim 1, wherein the output message indicates that a computerized diagnosis cannot be output with respect to the input image.

5. The method of claim 1, further comprising, in response to the indication not indicating a single skin abnormality or two or more skin abnormalities, the diagnostic system outputting a second message for display.

6. The method of claim 1, wherein generating the indication further includes outputting a first possibility and a second possibility from the machine-learned indicator model, the method further including determining, when the first possibility exceeds a first threshold, that the indication indicates that the input image presents a single skin anomaly, and determining, when the second possibility exceeds a second threshold, that the indication indicates that the input image presents two or more skin anomalies.

7. The method of claim 6, further comprising, in response to the first likelihood falling below the first threshold and the second likelihood falling below the second threshold, the diagnostic system outputting a second message for display.

8. The method of claim 1, wherein the machine-learned model is also a diagnostic model further configured to receive an image as input and output a prediction regarding one or more skin anomalies presented in the image.

9. The method of claim 8, wherein generating the indication further comprises providing the input image as input to the diagnostic model, outputting an output vector comprising a set of elements, wherein the prediction for the one or more skin abnormalities is represented by a value for a subset of elements in the output vector, and the indication is represented by a value for one or more remaining elements in the output vector, wherein the diagnostic model has been trained using training data comprising a plurality of training images and a plurality of corresponding vectors, each vector among the plurality of corresponding vectors comprising a set of elements, the set of elements comprising a subset of the plurality of elements and the one or more remaining elements, wherein a value for an element of the subset of elements of each vector indicates whether an individual skin abnormality is indicated in a training image among the plurality of training images, and a value for the one or more remaining elements of each vector indicates whether two or more skin abnormalities are indicated in a training image among the plurality of training images.

10. Generating the indication further comprises providing the input image as an input to the diagnostic model, thereby outputting an output vector including a set of a plurality of elements, wherein the prediction regarding the one or more skin abnormalities is represented by values ​​for the plurality of elements in the output vector; The method of claim 8 , wherein the input image is determined to exhibit two or more skin anomalies when a difference between values ​​of the set of elements in the output vector is below a threshold.

11. A computer-readable storage medium having stored thereon computer program code that, when executed by one or more processors, causes the one or more processors to perform a plurality of operations: The plurality of operations include: receiving, from a client device, a request for diagnosis of a skin anomaly presented in an input image; generating an indication by providing the input image as an input to a machine-learned model, the machine-learned model outputting the indication, the indication representing a likelihood that the input image exhibits two or more skin anomalies; in response to the indication being of a single skin anomaly within the input image, generating a computerized diagnosis of the single skin abnormality; and outputting the computerized diagnosis for display on the client device; outputting a message for display in response to the indication showing two or more skin anomalies in the input image; and A computer-readable storage medium comprising:

12. The plurality of operations include: accessing a diagnostic model from a database in response to the indication being of the single skin abnormality; generating a prediction regarding the skin anomaly in the input image by providing the input image as an input to the diagnostic model, the diagnostic model outputting the prediction; generating said computerized diagnosis for said input image from said prediction; The computer-readable storage medium of claim 11 , further comprising:

13. The computer-readable storage medium of claim 11, wherein the machine-learned model is configured as a neural network architecture including at least a set of multiple layers of multiple nodes, each layer connected to the previous layer through a separate set of trained weights.

14. The computer-readable storage medium of claim 11, wherein the output message indicates that a computerized diagnosis cannot be output with respect to the input image.

15. The computer-readable storage medium of claim 11, wherein the plurality of actions further include outputting a second message for display in response to the indication not indicating a single skin anomaly or two or more skin anomalies.

16. The computer readable storage medium of claim 11, wherein generating the indication further includes outputting a first possibility and a second possibility from the machine learning indicator model, and the computer readable storage medium further comprises computer program code for determining, when the first possibility exceeds a first threshold, that the indication indicates that the input image presents a single skin anomaly, and determining, when the second possibility exceeds a second threshold, that the indication indicates that the input image presents two or more skin anomalies.

17. The computer-readable storage medium of claim 16, wherein the plurality of actions further include outputting a second message for display in response to the first likelihood falling below the first threshold and the second likelihood falling below the second threshold.

18. The computer-readable storage medium of claim 11, wherein the machine-learned model is also a diagnostic model further configured to receive an image as input and output a prediction regarding one or more skin anomalies presented in the image.

19. The computer-readable storage medium of claim 18, wherein generating the indication further comprises providing the input image as an input to the diagnostic model, outputting an output vector comprising a set of elements, wherein the prediction for the one or more skin abnormalities is represented by a value for a subset of elements in the output vector, and the indication is represented by a value for one or more remaining elements in the output vector, wherein the diagnostic model has been trained using training data comprising a plurality of training images and a plurality of corresponding vectors, each vector among the plurality of corresponding vectors comprising a set of elements, the set of elements comprising a subset of the plurality of elements and the one or more remaining elements, wherein a value for an element of the subset of elements of each vector indicates whether a particular skin abnormality is indicated in a training image among the plurality of training images, and a value for the one or more remaining elements of each vector indicates whether two or more skin abnormalities are indicated in a training image among the plurality of training images.

20. Generating the indication further comprises providing the input image as an input to the diagnostic model, thereby outputting an output vector including a set of a plurality of elements, wherein the prediction regarding the one or more skin abnormalities is represented by values ​​for the plurality of elements in the output vector; 20. The computer-readable storage medium of claim 18, wherein the input image is determined to exhibit two or more skin anomalies when a difference between values ​​of the set of elements in the output vector is below a threshold.

Images