Diagnosing skin conditions using machine learning models

Abstract

Providing a solution for diagnosing skin conditions using machine learning models. [Solution] The diagnostic system receives images of a patient and trains a set of machine learning-based diagnostic models configured to generate predictions about whether the patient has one or more health conditions. In one embodiment, the set of machine learning models is trained to generate predictions for images containing two or more underlying health conditions of the patient. In one example, symptoms related to two or more health conditions are represented as two or more overlapping skin abnormalities on the patient. By using the architecture of the set of diagnostic models described herein, the diagnostic system can generate more accurate predictions for images containing overlapping symptoms related to two or more health conditions compared to existing systems.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the benefit and priority of U.S. Provisional Application No. 62 / 869,553, filed on 1 July 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0002] This disclosure relates, in general, to the diagnosis of health abnormalities, and more specifically, to the diagnosis of overlapping health conditions in images.

[0003] A computerized diagnostic system is configured to receive images of a patient and, based on the presence of symptoms in the images, generate a prediction about whether the patient has one or more health conditions. The computerized diagnostic system can be applied to predict health conditions in various anatomical parts of a patient. For example, a skin diagnostic system can be applied to predict whether images of a patient's skin have one or more skin abnormalities such as rashes, moles, eczema, acne, and herpes. In another embodiment, a tumor diagnostic system can be applied to predict whether images of a patient's biological structures contain tumors.

[0004] In many cases, a patient may have multiple health conditions appearing in the same image. For example, an image may contain a psoriatic plaque and a nevus, both of which are skin abnormalities, located on the same arm, and spatially separated from each other. In another example, an image may contain overlapping skin abnormalities, such as a psoriatic plaque with a mole within its boundaries. However, existing computerized diagnostic systems have difficulty generating accurate diagnoses in these cases, for example, because they are trained to generate predictions for a single health condition. Furthermore, when health conditions overlap, the presentation and other symptoms in the image may be consequences of the condition individually or in combination, and it may be difficult for the computerized diagnostic system to accurately differentiate the symptoms in the image into individual health conditions. [Overview of the Initiative] [Means for solving the problem]

[0005] The diagnostic system receives images of a patient and trains a set of machine learning-based diagnostic models configured to generate predictions about whether the patient has one or more health conditions. In one embodiment, the set of machine learning models is trained to generate predictions for images containing two or more underlying health conditions of the patient. In one case, symptoms relating to two or more health conditions are presented as two or more overlapping skin abnormalities on a patient. By using the architecture of the set of diagnostic models described herein, a diagnostic system can generate more accurate predictions for images containing overlapping symptoms relating to two or more health conditions compared to existing systems.

[0006] In one embodiment, the diagnostic system receives a request from a client device to diagnose skin abnormalities in an input image. The input image contains overlapping symptoms relating to two or more skin abnormalities on the skin. The diagnostic system accesses a set of machine-learned diagnostic models from a database. Each machine-learned model contains a distinct set of trained weights determined through the training process. The diagnostic system generates a distinct prediction for each of the two or more skin abnormalities in the input image by applying the set of diagnostic models to the input image. The prediction indicates the likelihood that a particular skin abnormality among the two or more skin abnormalities is present in the input image. The diagnostic system generates a diagnosis of the skin abnormalities from the predictions with respect to the input image and provides the diagnosis to the client device.

[0007] In one embodiment, the diagnostic system receives an image of a patient or individual and trains a set of machine-learned indicator models configured to generate indications about whether the input image presents two or more health conditions. In one embodiment, the diagnostic system receives a request for a diagnosis of health conditions in an input image. The diagnostic system accesses an indicator model or a diagnostic model with indicator functionality from a database. The diagnostic system applies the indicator model to the input image to generate indications that represent the possibility that the input image contains two or more health conditions. The diagnostic system makes a decision from the indications generated regarding the input image whether the input image presents two or more health conditions. Based on the decision, the diagnostic system generates a result regarding the request and provides the result to the user of the client device.

[0008] Since input images presenting multiple health conditions can be difficult to diagnose, users of client devices may receive such indications in addition to, or as an alternative to, active diagnosis of the images. The user can then decide to provide the input images to a medical professional to obtain a more accurate diagnosis. Therefore, it may be advantageous for users of a computerized diagnostic system to receive information about whether the input images contain symptoms from two or more health conditions, so that a more accurate diagnosis can be made by different specialized systems or medical professionals. The present invention provides, for example, the following items: (Item 1) A method for diagnosing overlapping skin abnormalities in an input image, wherein the method is: Receiving a request from a client device to diagnose skin abnormalities in an input image, wherein the input image includes overlapping skin abnormalities on the patient's skin. This involves accessing a set of machine-learned models from a database, where each machine-learned model contains a separate set of trained weights. Applying the aforementioned set of machine-learned models to the input image generates individual predictions for each of two or more skin abnormalities in the input image, wherein the predictions indicate the likelihood that one of the two or more skin abnormalities is present in the input image. With respect to the input image, the diagnosis of the overlapping skin abnormalities is generated from the prediction, To provide the aforementioned diagnosis to the aforementioned client device Methods that include... (Item 2) The aforementioned set of machine-learned models is an ensemble set of neural network models, and the ability to generate the aforementioned predictions regarding two or more skin abnormalities is further... With respect to each neural network model in the ensemble set, the neural network model is applied to the input image to generate one or more predictions from the neural network model. The predictions for the ensemble set of the neural network models are combined to generate the predictions for two or more skin abnormalities. The method described in item 1, including the method described in item 1. (Item 3) The set of machine-learned models includes a second machine-learned model and a third machine-learned model, and further generates the predictions regarding the two or more skin abnormalities. The method involves applying the second machine learning model described above to the input image to generate an image tensor, wherein the image tensor characterizes a plurality of spatial features in the input image. Extracting multiple components from the aforementioned image tensor, To generate a separate tensor for each of the two or more skin abnormalities, By applying the third machine learning model to the individual tensors relating to each of the two or more skin abnormalities, the predictions are generated for the two or more skin abnormalities. The method described in item 1, including the method described in item 1. (Item 4) The method according to item 3, wherein the plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor. (Item 5) The set of trained weights for the second machine-learned model and the third machine-learned model are trained jointly, as described in item 3. (Item 6) The set of machine-learned models includes a recurrent neural network model, and further generates the predictions regarding the two or more skin abnormalities. The recurrent neural network model is repeatedly applied to the input image to generate individual predictions for the first skin abnormality among the two or more skin abnormalities in a first time period, and to generate individual predictions for the second skin abnormality among the two or more skin abnormalities in a second time period following the first time period. The method described in item 1, including the method described in item 1. (Item 7) The set of machine-learned models includes a second machine-learned model, a third machine-learned model, and a fourth machine-learned model, and further generates the predictions regarding the two or more skin abnormalities. To generate a prediction as to whether the input image contains amorphous or localized skin abnormalities, In response to determining that the input image contains an amorphous skin abnormality, the amorphous abnormality model is applied to the input image to generate a prediction regarding the amorphous skin abnormality. In response to determining that the input image includes a localized skin abnormality, the localized abnormality model is applied to the input image to generate a prediction regarding the localized skin abnormality. The method described in item 1, including the method described in item 1. (Item 8) The method according to item 1, wherein at least one of the set of machine-learned models is configured as a neural network architecture comprising a set of layers of nodes, each layer being connected to the previous layer via a subset of weights. (Item 9) The method according to item 1, wherein the input image is at least one of the following: radiographic image, computed tomography (CT) scan, medical resonance imaging (MRI) scan, X-ray image, ultrasound or ultrasound image, tactile image, or thermographic image. (Item 10) The method according to item 1, wherein the input image is an image captured by the user of the client device, and the client device is a smartphone. (Item 11) A computer program product for diagnosing overlapping skin abnormalities in an input image, wherein the computer program product comprises a computer-readable storage medium, and the computer-readable storage medium is Receiving a request from a client device to diagnose skin abnormalities in an input image, wherein the input image includes overlapping skin abnormalities on the patient's skin. This involves accessing a set of machine-learned models from a database, where each machine-learned model contains a separate set of trained weights. Applying the aforementioned set of machine-learned models to the input image generates individual predictions for each of two or more skin abnormalities in the input image, wherein the predictions indicate the likelihood that one of the two or more skin abnormalities is present in the input image. With respect to the input image, the diagnosis of the overlapping skin abnormalities is generated from the prediction, To provide the aforementioned diagnosis to the aforementioned client device A computer program product containing computer program code for [a specific purpose]. (Item 12) The aforementioned set of machine-learned models is an ensemble set of neural network models, and the ability to generate the aforementioned predictions regarding two or more skin abnormalities is further... With respect to each neural network model in the ensemble set, the neural network model is applied to the input image to generate one or more predictions from the neural network model. The predictions for the ensemble set of the neural network models are combined to generate the predictions for two or more skin abnormalities. Computer program products, including those listed in item 11. (Item 13) The set of machine-learned models includes a second machine-learned model and a third machine-learned model, and further generates the predictions regarding the two or more skin abnormalities. The method involves applying the second machine-learned model described above to the input image to generate an image tensor, wherein the image tensor characterizes a plurality of spatial features in the input image. Extracting multiple components from the aforementioned image tensor, To generate a separate tensor for each of the two or more skin abnormalities, By applying the third machine learning model to the individual tensors relating to each of the two or more skin abnormalities, the predictions are generated for the two or more skin abnormalities. Computer program products, including those listed in item 11. (Item 14) The aforementioned plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor, as described in item 13 of the computer program product. (Item 15) The set of trained weights relating to the second machine-learned model and the third machine-learned model are jointly trained in the computer program product described in item 13. (Item 16) The set of machine-learned models includes a recurrent neural network model, and further generates the predictions regarding the two or more skin abnormalities. The recurrent neural network model is repeatedly applied to the input image to generate individual predictions for the first skin abnormality among the two or more skin abnormalities in a first time period, and to generate individual predictions for the second skin abnormality among the two or more skin abnormalities in a second time period following the first time period. Computer program products, including those listed in item 11. (Item 17) The set of machine-learned models includes a second machine-learned model, a third machine-learned model, and a fourth machine-learned model, and further generates the predictions regarding the two or more skin abnormalities. To generate a prediction as to whether the input image contains amorphous or localized skin abnormalities, In response to determining that the input image contains an amorphous skin abnormality, the amorphous abnormality model is applied to the input image to generate a prediction regarding the amorphous skin abnormality. In response to determining that the input image includes a localized skin abnormality, the localized abnormality model is applied to the input image to generate a prediction regarding the localized skin abnormality. Computer program products, including those listed in item 11. (Item 18) The computer program product described in item 11, wherein at least one of the set of machine-learned models is configured as a neural network architecture comprising a set of layers of nodes, each layer being connected to the previous layer via a subset of weights. (Item 19) The computer program product described in item 11, wherein the input image is at least one of the following: radiographic images, computed tomography (CT) scans, medical resonance imaging (MRI) scans, X-ray images, ultrasound or ultrasound images, tactile images, or thermographic images. (Item 20) The input image is an image captured by the user of the client device, and the client device is a smartphone, as described in item 11 of the computer program product. (Item 21) A method for generating indications of two or more skin abnormalities in an input image, wherein the method is: The client device receives a request for diagnosis of skin abnormalities presented in the input image, Accessing a machine-learned model from a database, wherein the machine-learned model includes a separate set of trained weights. The process involves applying the machine-learned model to the input image to generate an indication, wherein the indication represents the possibility that the input image exhibits two or more skin abnormalities. From the indication generated with respect to the input image, a decision is made as to whether the input image presents two or more skin abnormalities. Based on the above decision, generate a result with respect to the above request and provide the above result to the client device. Methods that include... (Item 22) The method according to item 21, further comprising determining a treatment option with respect to the input image based on the aforementioned determination, and providing the treatment option to the client device as a result. (Item 23) In response to determining that the input image presents one skin abnormality rather than two or more, access a diagnostic model configured to generate a prediction for a single skin abnormality from the database; By applying the diagnostic model to the input image, predictions are generated regarding the skin abnormalities in the input image. Based on the prediction, a diagnosis is generated with respect to the input image, and as a result, the diagnosis is provided to the client device. The method described in item 21, further including the method described in item 21. (Item 24) In response to determining that the input image presents two or more skin abnormalities, access a set of diagnostic models configured to generate separate predictions for each of the two or more skin abnormalities from the database, By applying the set of diagnostic models to the input image, predictions are generated regarding two or more skin abnormalities in the input image. Based on the prediction, a diagnosis is generated with respect to the input image, and as a result, the diagnosis is provided to the client device. The method described in item 21, further including the method described in item 21. (Item 25) The method according to item 21, further comprising providing the client device with information that the input image cannot be diagnosed in response to determining that the input image presents two or more skin abnormalities. (Item 26) The method according to item 21, further comprising providing the client device with information about the indication without providing predictions about the two or more skin abnormalities in the input image, in response to determining that the input image presents two or more skin abnormalities. (Item 27) The method according to item 21, wherein the machine-learned model is also a diagnostic model configured to receive an image and generate predictions regarding one or more skin abnormalities presented in the image. (Item 28) Generating the indication further includes applying the diagnostic model to the input image and generating an output vector comprising a set of elements, wherein the prediction for one or more skin abnormalities is represented by the values ​​of the elements in the output vector. The input image is determined to present two or more skin abnormalities when the difference between the values ​​of the set of elements in the output vector falls below a predetermined threshold. The method described in item 27. (Item 29) The method according to item 27, wherein generating the indication further comprises applying the diagnostic model to the input image to generate an output vector comprising a set of elements, the prediction for one or more skin abnormalities being represented by values ​​relating to a subset of elements in the output vector, and the indication being represented by values ​​relating to the remaining elements in the output vector. (Item 30) The prediction regarding the one or more skin abnormalities is represented by one or more categories from a set of categories, Generating the indication further includes applying the diagnostic model to the input image and generating an output value. The input image is determined to present two or more skin abnormalities when the output value is assigned to the remaining category in the set of categories. The method described in item 27. (Item 31) A computer program product for generating indications of two or more skin abnormalities in an input image, wherein the computer program product comprises a computer-readable storage medium, and the computer-readable storage medium is Receiving a request from the client device for diagnosis of skin abnormalities in the input image, Accessing a machine-learned model from a database, wherein the machine-learned model includes a separate set of trained weights. The process involves applying the machine-learned model to the input image to generate an indication, wherein the indication represents the possibility that the input image contains two or more skin abnormalities. From the indication generated with respect to the input image, a decision is made as to whether the input image presents two or more skin abnormalities. Based on the above decision, generate a result with respect to the above request and provide the above result to the client device. A computer program product containing computer program code for [a specific purpose]. (Item 32) The computer-readable storage medium further, A computer program product according to item 31, comprising computer program code for determining treatment options for an individual in the input image based on the aforementioned determination, and for providing the aforementioned treatment options to the client device as a result. (Item 33) The computer-readable storage medium further, In response to determining that the input image presents one skin abnormality rather than two or more, access a diagnostic model configured to generate a prediction for a single skin abnormality from the database; By applying the diagnostic model to the input image, predictions are generated regarding the skin abnormalities in the input image. The process involves generating a diagnosis regarding the input image based on the prediction, and providing the diagnosis to the client device as a result. A computer program product as described in item 31, which contains computer program code for the purpose of... (Item 34) The computer-readable storage medium further, In response to determining that the input image presents two or more skin abnormalities, access a set of diagnostic models configured to generate separate predictions for each of the two or more skin abnormalities from the database, By applying the set of diagnostic models to the input image, predictions are generated regarding two or more skin abnormalities in the input image. Based on the prediction, a diagnosis is generated with respect to the input image, and as a result, the diagnosis is provided to the client device. A computer program product as described in item 31, which contains computer program code for the purpose of... (Item 35) The computer program product according to item 31, wherein the computer-readable storage medium further includes computer program code for providing the client device with information that the input image cannot be diagnosed in response to determining that the input image presents two or more skin abnormalities. (Item 36) The computer program product according to item 31, wherein the computer-readable storage medium further includes computer program code for providing information about the indication to the client device without providing predictions about the two or more skin abnormalities in the input image, in response to the computer-readable storage medium determining that the input image presents two or more skin abnormalities. (Item 37) The machine-learned model is also a diagnostic model configured to receive an image and generate predictions regarding one or more skin abnormalities presented in the image, as described in item 31. (Item 38) Generating the indication further includes applying the diagnostic model to the input image and generating an output vector comprising a set of elements, wherein the prediction for one or more skin abnormalities is represented by the values ​​of the elements in the output vector. The input image is determined to present two or more skin abnormalities when the difference between the values ​​of the set of elements in the output vector falls below a predetermined threshold. Computer program products as described in item 37. (Item 39) The computer program product according to item 37, wherein generating the indication further comprises applying the diagnostic model to the input image and generating an output vector comprising a set of elements, the prediction for one or more skin abnormalities being represented by values ​​relating to a subset of elements in the output vector, and the indication being represented by values ​​relating to the remaining elements in the output vector. (Item 40) The prediction regarding the one or more skin abnormalities is represented by one or more categories from a set of categories, Generating the indication further includes applying the diagnostic model to the input image and generating an output value. The input image is determined to present two or more skin abnormalities when the output value is assigned to the remaining category in the set of categories. Computer program products as described in item 37. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a high-level block diagram of a system environment for a diagnostic system according to one embodiment.

[0010] [Figure 2] Figure 2 illustrates a general reasoning process for a diagnostic model according to one embodiment.

[0011] [Figure 3] Figure 3 illustrates an exemplary inference process for a set of machine learning-generated diagnostic models with an ensemble architecture, according to one embodiment.

[0012] [Figure 4] Figure 4 illustrates an exemplary reasoning process using independent component analysis (ICA) in conjunction with a set of diagnostic models, according to one embodiment.

[0013] [Figure 5] Figure 5 illustrates an exemplary inference process for a diagnostic model with a recurrent neural network (RNN) architecture according to one embodiment.

[0014] [Figure 6] Figure 6 illustrates an exemplary reasoning process for a set of diagnostic models, including differentiator models for amorphous and localized skin abnormalities, according to one embodiment.

[0015] [Figure 7] Figure 7 illustrates an exemplary reasoning process for a diagnostic model with non-maximal suppression according to one embodiment.

[0016] [Figure 8] Figure 8 illustrates an exemplary inference process for an uncertainty model according to one embodiment.

[0017] [Figure 9] Figure 9 is a block diagram of the architecture of a diagnostic system according to one embodiment.

[0018] [Figure 10] Figure 10 illustrates an exemplary training process for a set of diagnostic models shown in Figure 4, according to one embodiment.

[0019] [Figure 11] Figure 11 illustrates an exemplary data flow for diagnosing overlapping skin abnormalities in an image according to one embodiment.

[0020] [Figure 12] Figure 12 illustrates an exemplary data flow for generating indications of two or more skin abnormalities in an input image according to one embodiment.

[0021] The drawings illustrate various embodiments of the invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein can be adopted without departing from the principles of the invention described herein. [Modes for carrying out the invention]

[0022] Overview Figure 1 is a high-level block diagram of a system environment for a diagnostic system according to one embodiment. The system environment 100 shown in Figure 1 includes one or more client devices 110, a network 120, and a diagnostic system 130. In alternative configurations, different and / or additional components may be included in the system environment 100.

[0023] The diagnostic system 130 is a system for providing various types of computerized diagnoses to the user of the client device 110. The diagnostic system 130 may receive images of anatomical parts of a patient and, based on the images, generate predictions about whether the patient has one or more health conditions. For example, the diagnostic system 130 may receive images of the patient's skin and generate predictions about whether the patient has one or more skin abnormalities such as rashes, moles, eczema, acne, and herpes. In another embodiment, the diagnostic system 130 may receive radiographic images of the patient's brain and generate predictions about whether the patient has a brain tumor. The predictions generated by the diagnostic system 130 can be used as a standalone diagnosis or, for example, to assist a hospital physician in interpreting the patient's images.

[0024] The diagnostic system 130 receives images of a patient and trains a set of machine learning-based diagnostic models configured to generate predictions about one or more health conditions. The images received by the diagnostic system 130 may be medical images acquired in a hospital, such as radiographs, computed tomography (CT) scans, medical resonance imaging (MRI) scans, X-ray images, ultrasound or oscilloscope images, tactile images, or thermographic images. In another case, the images may be photographs taken by individual users of client devices 110 (e.g., at home or in the office). The images may show symptoms or other presentations on anatomical parts of the patient that the diagnostic system 130 can use to infer health conditions involved in causing these symptoms.

[0025] In one embodiment, the diagnostic system 130 trains a set of machine learning models to generate predictions regarding images containing symptoms relating to two or more health conditions of a patient. Specifically, a patient may have symptoms from multiple health conditions appearing in the same image. For example, an image may contain a psoriatic plaque and a nevus, both of which are skin abnormalities, located on the same arm and spatially separated from each other. In another embodiment, an image may contain overlapping skin abnormalities, such as a psoriatic plaque with a mole within its boundaries. However, existing computerized diagnostic systems may have difficulty generating accurate diagnoses in these cases, particularly when symptoms overlap, because the symptoms may be an unknown combination of individual outcomes or the patient's underlying health conditions. Furthermore, existing diagnostic systems may be configured to generate predictions for a single condition. Therefore, even if an existing diagnostic system is capable of detecting one of the health conditions in an image, the detected condition may be in the exclusion of the remaining health conditions in the image, leading to missed diagnoses that could result in significant health problems for the subject of the image.

[0026] During the inference process, the diagnostic system 130 receives a request to diagnose a health condition in an input image. The input image contains overlapping symptoms from two or more health conditions of a patient. The diagnostic system 130 accesses a set of machine learning-trained diagnostic models from a database. Each machine learning-trained model contains a distinct set of trained weights determined through the training process. The diagnostic system 130 generates a distinct prediction for each of the two or more health conditions in the input image by applying the set of diagnostic models to the input image. The prediction indicates the likelihood that the input image exhibits symptoms from the distinct health conditions. The diagnostic system 130 generates a diagnosis of the input image from the prediction and provides the diagnosis back to the client device 110.

[0027] In one particular embodiment, as referenced throughout the remainder of this specification, two or more health conditions are two or more skin abnormalities having overlapping symptoms presented on the patient's skin. For example, moles may appear within an area of ​​skin with a rash. In another embodiment, an area of ​​skin with eczema may overlap with a portion of skin with contact dermatitis. However, it should be understood that in other embodiments, two or more health conditions may be any type of health-related condition, such as disease or allergy, that may produce symptoms overlapping or spatially separated on the same anatomical part of the patient. For example, two or more health conditions may be different types of tumors, blood disorders, and equivalents that may present symptoms on the patient individually or in combination.

[0028] Figure 2 illustrates a general inference process for a diagnostic model according to one embodiment. The diagnostic system 130 receives an input image 210 of a patient suspected of having a skin abnormality. Specifically, the input image 210 shows a first skin abnormality 212 and a second skin abnormality 214 overlapping each other on the patient's arm. The diagnostic system 130 accesses a set of machine learning-trained diagnostic models from a database. The diagnostic system 130 applies the diagnostic model to the input image 210 and makes two or more predictions v'1, v'2, ..., v' for the input image 210. n This generates the prediction v'. i This may indicate that the input image 210 contains symptoms from individual skin abnormalities i. In the embodiment shown in Figure 2, prediction v'1 may indicate a high probability that the input image 210 contains moles, and prediction v'2 may indicate a high probability that the input image 210 also contains a rash. The remaining predictions for other skin abnormalities may be associated with significantly lower probabilities. The diagnostic system 130 determines that the input image 210 contains moles and a skin rash by selecting a subset of predictions with probabilities exceeding a predetermined threshold, and provides the diagnosis to the client device 110.

[0029] Returning to the system environment in Figure 1, the diagnostic system 130 generally uses a training corpus of images and labels to train a separate set of weights for the diagnostic model and reduce the loss function. The loss function for the diagnostic model represents the difference between the estimated output, which is generated by applying the diagnostic model with the estimated set of weights to the input data in the training corpus that the diagnostic model is configured to receive, and the actual labels in the training corpus that represent the type of data that the diagnostic model is configured to predict. The estimated set of weights for the diagnostic model is repeatedly updated to reduce the loss function until a predetermined criterion is reached. The training process for the set of diagnostic models is described in more detail in conjunction with Figure 9.

[0030] As described herein, Figure 3-7 illustrates various architectures of a set of diagnostic models that may be used by the diagnostic system 130 to generate predictions regarding two or more health conditions presented in an image during the inference process. Each architecture described below may include a set of trained weights determined through a training process, which will be described in conjunction with Figure 9. By using the architectures of the sets of diagnostic models described herein, the diagnostic system can generate more accurate predictions regarding images containing two or more overlapping health conditions compared to existing systems.

[0031] Figure 3 illustrates an exemplary inference process for a set of machine learning-trained diagnostic models with an ensemble architecture according to one embodiment. In one embodiment, the set of diagnostic models includes an ensemble set of machine learning-trained diagnostic models. Specifically, each diagnostic model in the ensemble is configured to receive an input image and apply a set of trained weights for the diagnostic model to generate one or more predictions about whether a patient has one or more health conditions. In the embodiment shown in Figure 3, the ensemble set of diagnostic models includes “Diagnostic Model 1”, “Diagnostic Model 2”, and “Diagnostic Model 3”.

[0032] Diagnostic models within an ensemble set are configured to generate predictions regarding different or identical sets of health states from other diagnostic models within the ensemble set. In the embodiment shown in FIG. 3, “Diagnostic Model 1” generates a prediction v’ M1 1 indicating the possibility that the input image contains moles and a prediction v’ M1 2 indicating the possibility that the input image contains rashes. As another example, “Diagnostic Model 2” may be configured to generate a single prediction v’ M2 1 indicating the possibility that the input image contains rashes, and “Diagnostic Model 3” may be configured to generate a prediction v’ M3 1 indicating the possibility that the input image contains moles and a prediction v’ M3 2 indicating the possibility that the input image contains acne.

[0033] During the inference process, diagnostic system 130 applies an ensemble set of diagnostic models to the input image and generates individual predictions for each of two or more health states within the input image by combining the predictions from the ensemble set regarding each individual health state. In one example, the prediction regarding an individual health state is generated by calculating the average of the predictions regarding the health state generated by the ensemble set. In the embodiment shown in FIG. 3, the prediction v’1 indicating the possibility that the patient has rashes is generated by averaging the predictions v’ M1 2 and v’ M2 1 from “Diagnostic Model 1” and “Diagnostic Model 2”. However, it should be understood that in other embodiments, the predictions from the diagnostic models within the ensemble may be combined in any other way other than averaging. Diagnostic system 130 determines the diagnosis by selecting a subset of predictions that have a likelihood of exceeding a pre-determined threshold.

[0034] In one embodiment, the diagnostic model in the ensemble set may be configured to generate multiple outputs that do not necessarily correspond to the same set of health conditions each time the diagnostic model is applied to an input image. In such an embodiment, the diagnostic system 130 may group predictions from the ensemble set by similarity of predictions and intermediate features, and predictions for two or more health conditions presented in the input image may be generated by combining predictions from each identified group. In another example, the diagnostic system 130 may group predictions from the ensemble set by similar locations, for example, using a focus model, and predictions for two or more health conditions in the input image may be generated by combining predictions for each identified location.

[0035] Figure 4 illustrates an exemplary inference process using Independent Component Analysis (ICA) in conjunction with a set of diagnostic models according to one embodiment. In one embodiment, the set of diagnostic models includes a feature extractor model, an Independent Component Analysis (ICA) model, and a feature classifier model. The feature extractor model is configured to receive an input image and generate an image tensor characterizing multiple spatial features in the input image by applying a first set of trained weights. The ICA model is configured to receive the image tensor and extract multiple components from the image tensor. The feature classifier model is configured to receive the components and generate predictions indicating the likelihood of individual health conditions with respect to the components by applying a second set of trained weights.

[0036] During the inference process, the diagnostic system 130 generates an image tensor 416 with respect to the input image by applying a feature extractor model to the input image. The diagnostic system 130 performs ICA and considers each spatial location in the image tensor 416 as an observation, and extracts multiple components CF1, CF2, CF3, ..., CF from the image tensor 416. nThe components are extracted. For each extracted component, the diagnostic system 130 calculates the component's contribution to the image tensor 416 and generates a separate tensor for that component. In one example, the separate tensor for a component is generated by calculating the contribution of the remaining components among the multiple components and subtracting the contribution of the remaining components from the image tensor 416. In the embodiment shown in Figure 4, the separate tensor 418 for component CF1 is generated by the remaining components CF2, CF, ..., CF n This is generated by calculating the contribution and subtracting it from the image tensor 416. Figure 4 shows only the individual tensor 418 for component CF1, but this process is carried out for the remaining components CF2, CF3, ..., CF n This process may be repeated to generate a separate tensor for each of the terms.

[0037] For each tensor, the diagnostic system 130 generates a prediction by applying the feature classifier model to the tensor relating to the component. In the embodiment shown in Figure 4, the prediction v'1 for component CF1 is generated by applying the feature classifier model to the tensor 418 relating to the component. Although Figure 4 only illustrates the prediction v'1 for component CF1, this process applies the feature classifier model to the tensor relating to each component, thereby generating predictions for the remaining components CF2, CF3, ..., CF n The process may be repeated to generate individual predictions from each of them. The diagnostic system 130 selects a subset that has a high probability of exceeding a threshold, thereby generating predictions v'1, v'2, ..., v' n The diagnosis will be determined based on these factors.

[0038] Based on the number of components extracted from the image tensor, each diagnosis may correspond to a different health condition underlying the input image, or a subset of diagnoses may correspond to the same health condition if the number of extracted components exceeds the number of health conditions present in the input image. In one case, the diagnostic system 130 extracts a predetermined number of components based on an estimated number of health conditions in the input image. In another case, the diagnostic system 130 performs an ICA with an increasing number of components, iteratively generating predictions and diagnoses with each new component. The diagnostic system 130 completes the inference process when further increasing the number of components does not increase the number of unique diagnoses and there is convergence of a criterion function with respect to the ICA. In one case, there may be a threshold number of additional components to determine convergence. For example, three components may be added, while the set of diagnoses remains invariant to be considered for convergence.

[0039] In another embodiment, multiple components are pre-extracted during the training process. During the inference process, the diagnostic system 130 decomposes the image tensor into a mixture of multiple components, calculates the contribution of each pre-determined component to the image tensor, and generates a separate tensor for each component. The diagnostic system 130 identifies a subset of components whose contribution exceeds a threshold. For each tensor, the diagnostic system 130 generates a prediction by applying a feature classifier model to the tensor relating to the components in the subset. In some embodiments, each pre-determined component may be pre-classified and stored using the feature classifier model. The classification result is then determined by using the stored components associated with the contributing components.

[0040] Figure 5 illustrates an exemplary inference process for a diagnostic model with a recurrent neural network (RNN) architecture according to one embodiment. In one embodiment, the set of diagnostic models includes a diagnostic model with an RNN architecture that includes one or more neural network layers with a set of weights. The RNN architecture is configured to receive a series of images and sequentially generate predictions regarding two or more health conditions presented in the series of images. The series of input images may be a series of identical input images repeated, or a series of images capturing an anatomical part of a patient when different views, viewpoints, filters, and equivalents are issued. In the embodiment shown in Figure 5, the series of images 510(1), 510(2), ..., 510(n) are the same images of the skin on the patient's arm. However, in another embodiment, the series of images may include images of the same arm captured from different viewpoints or in different colors or scales.

[0041] Specifically, with respect to the current image in a series, the RNN architecture considers the current image 510(i) and the hidden state h for the previous iteration. i-1 By receiving and applying the first set of trained weights, the hidden state h with respect to the current image 510(i) is determined. i The RNN architecture is configured to generate the current image h i By receiving the hidden state regarding and applying a second set of trained weights, predictions can be made regarding the individual health status. i The RNN architecture is configured to generate predictions. After each iteration, the RNN architecture may further be configured to decide whether the diagnostic model should generate predictions regarding other health conditions in the input image or complete the inference process. If the RNN architecture decides to continue generating predictions, the RNN architecture receives the next image in the sequence 510(i+1) and makes a prediction regarding another health condition v i+1This process is repeated until the RNN architecture decides to complete the inference process. In the embodiment shown in Figure 5, prediction v'1 may indicate that the input image contains a mole, while the next prediction v'2 may indicate that the input image contains a rash.

[0042] In one embodiment, the RNN architecture further includes hidden states h for the current image 510(i) and the previous iteration. i-1 In addition, in each iteration, the memory vector m i Upon receiving the current image 510(i), predictions regarding the individual health status are made. i It is configured to generate the memory vector m. i This can represent a vector containing information that has already been classified up to the current iteration i, where the health status is. For example, a memory vector m i This may be an aggregation of previous predictions or diagnoses up to the current iteration. In the embodiment shown in Figure 5, the memory vector m2 in the second iteration may indicate that predictions or diagnoses regarding moles have been made up to the second iteration based on v1. By further configuring the RNN architecture to receive memory vectors, the diagnostic system 130 may structure a diagnostic model and generate predictions for unpredicted health conditions in each iteration of the inference process.

[0043] During the inference process, the diagnostic system 130 generates individual predictions for each of two or more health conditions by iteratively applying the RNN architecture to a series of input images. In one embodiment, when the RNN architecture is further configured to receive memory vectors, the diagnostic system 130 may obtain individual memory vectors for each iteration based on the predictions generated up to the current iteration. The diagnostic system 130 generates individual predictions for each of two or more health conditions by iteratively applying the RNN architecture to a series of images and memory vectors.

[0044] Figure 6 illustrates an exemplary reasoning process for a set of diagnostic models, including differentiator models for amorphous and localized skin abnormalities, according to one embodiment. In one embodiment, the machine-learned set of diagnostic models includes a differentiator model, an amorphous abnormality model, and a localized abnormality model. The differentiator model is configured to receive an input image and, by applying a first set of trained weights, generate an indication of whether the input image contains an amorphous or localized skin abnormality. The amorphous abnormality model is configured to receive an input image and, by applying a second set of trained weights, generate a prediction of whether the input image contains one or more amorphous skin abnormalities, such as a rash or eczema. The localized abnormality model is configured to receive an input image and, by applying a third set of trained weights, generate a prediction of whether the input image contains one or more localized skin abnormalities, such as a mole or acne.

[0045] Specifically, amorphous skin abnormalities such as rashes, eczema, and dermatitis may appear on the skin without a clear shape or form, while localized skin abnormalities such as moles can be clearly defined by a localized point on the skin. Therefore, due to differences in shape and form, the diagnostic system 130 may obtain more accurate predictions by training two separate models, one model generating predictions for amorphous skin abnormalities and the other model generating predictions for localized skin abnormalities.

[0046] During the inference process, the diagnostic system 130 generates an indication of whether the input image contains localized or amorphous skin abnormalities by applying a differentiator model to the input image. Rather than generating predictions that can be used to diagnose specific types of skin abnormalities (e.g., moles, acne, rashes), the output of the differentiator model may simply indicate whether the input image contains any skin abnormalities that can be classified as amorphous and / or localized. In response to determining that the input image contains amorphous skin abnormalities, the diagnostic system 130 generates specific predictions for one or more amorphous skin abnormalities by applying an amorphous abnormality model to the input image. Furthermore, in response to determining that the input image contains localized skin abnormalities, the diagnostic system 130 generates specific predictions for one or more localized skin abnormalities by applying a localized abnormality model to the input image.

[0047] Figure 7 illustrates an exemplary reasoning process for a diagnostic model with non-maximal suppression according to one embodiment. In one embodiment, a machine learning-trained diagnostic model is configured to receive an input image and generate a vector containing predictions about a list of health conditions. In the embodiment shown in Figure 7, the prediction v' generated by the diagnostic model may include one or more elements, each corresponding to a prediction about an individual health condition. For example, a first element may correspond to a prediction that the input image contains eczema, a second element may correspond to a prediction that the input image contains dermatitis, and so on.

[0048] During the inference process, the diagnostic system 130 generates individual predictions for two or more health conditions by applying a diagnostic model to the input image. In one embodiment, the diagnostic system 130 determines the diagnosis by using non-maximal suppression. Specifically, the diagnostic system 130 divides the different health conditions represented by the vector v' into groups and, for each group, selects only the health conditions that have the highest probability as part of the diagnosis. In one example, health conditions are grouped according to the degree of similarity in which their symptoms are visually presented on the patient, and thus each group contains health conditions that the diagnostic model is likely to confuse in the image. In the embodiment shown in Figure 7, the health conditions relating to the first to fourth elements in the vector are assigned to "group 1", and the health conditions relating to the (n-3) to nth elements are assigned to "group m". The diagnostic system 130 determines the diagnosis v' by selecting the second element in "group 1" and the (n-1)th element in "group m" as health conditions, since each element has the highest probability in each group, and setting the remaining elements to zero. nm To decide.

[0049] Returning to the diagram in Figure 1, the diagnostic system 130 may also train a set of machine learning indicator models configured to receive patient images and generate indications about whether the input images present two or more health conditions. Rather than actively diagnosing specific health conditions themselves, the diagnostic system 130 may generate indications about whether the input images present two or more health conditions and provide the results to the user of the client device 110. Specifically, rather than generating predictions that can be used to diagnose specific types of health conditions, the output of the indicator model may simply determine whether the input image shows symptoms from two or more health conditions (e.g., overlapping symptoms). Since input images presenting multiple health conditions can be difficult to diagnose, the user of the client device 110 may receive such indications in addition to, or as an alternative to, an active diagnosis. The user can then decide to provide the input images to a medical professional to obtain a more accurate diagnosis.

[0050] Figure 8 illustrates an exemplary inference process for an indicator model according to one embodiment. In one embodiment, the diagnostic model is trained to generate an indication e' about whether the input image presents two or more health conditions by receiving an input image and applying a trained set of weights. In the embodiment shown in Figure 8, the diagnostic system 130 generates the indication e' by applying the indicator model to the input image. The diagnostic system 130 may simply provide the user of the client device 110 with information that the image contains an excessive number of health conditions for accurate diagnosis.

[0051] In another embodiment, instead of training a separate indicator model as described in conjunction with Figure 8, the diagnostic system 130 further incorporates indicator functionality into the diagnostic model. For example, the diagnostic model may be configured to receive an input image and generate predictions about one or more health conditions contained in the image, and an indication of whether the input image presents two or more health conditions. For example, the set of diagnostic models described in conjunction with Figure 2-7 may be further configured to generate such indications in addition to predictions about health conditions. In another embodiment, the diagnostic model may be configured to generate a single output at a time, and may generate such indications as a single output in response to receiving an input image showing two or more health conditions.

[0052] In one example, the diagnostic model is configured to generate such an indication by categorizing input images that present two or more health conditions as separate categories and, in response to receiving the input images, generating probabilities for each separate category. In another example, the diagnostic model is configured to generate such an indication when predictions for health conditions are output with relatively equal probability or confidence levels, indicating ambiguity in the predictions. For example, the diagnostic system 130 may determine that an input image contains symptoms from two or more health conditions if the predictions for having moles, rashes, and skin diseases are output as 0.32, 0.29, and 0.35, respectively.

[0053] In one embodiment, the diagnostic system 130 may determine that the input image presents two or more states if the indication probability exceeds a threshold probability, and that the input image presents a single health state otherwise. In some embodiments, the indication may represent two possibilities: a first possibility that the input image presents two or more health states, and a second possibility that the input image presents a single health state. The diagnostic system 130 may determine that the input image presents two or more states if the first possibility exceeds a threshold, and may determine that the input image presents a single health state if the second possibility exceeds a threshold.

[0054] Therefore, in one embodiment, the diagnostic system 130 may receive a request for a diagnosis of a health condition in an input image. The diagnostic system 130 may access an indicator model or a diagnostic model with indicator functionality from a database. By applying the indicator model to the input image, the diagnostic system 130 can generate an indication that represents the possibility that the input image contains two or more health conditions. From the indication generated with respect to the input image, the diagnostic system 130 makes a decision as to whether the input image presents two or more health conditions. Based on the decision, the diagnostic system 130 generates a result with respect to the request and provides the result to the client device 110. The result may include medical treatment options for the individual in the input image based on the decision, and the proposed treatment options may be provided to the client device 110.

[0055] In one embodiment, the diagnostic system 130 may determine that the indication is unclear, in that the diagnostic model 130 cannot make a definitive decision as to whether the input image represents two or more health conditions. For example, this may occur when both the first possibility (e.g., the image representing two or more health conditions) and the second possibility (e.g., the image representing a single health condition) are below a threshold (e.g., 80%). In such an embodiment, the diagnostic system 130 may output to the client device that a diagnosis cannot be made, and for example, the user of the client device 110 may be suggested to acquire medical expertise.

[0056] In some embodiments, in response to determining that an input image presents two or more skin abnormalities, the diagnostic system 130 may provide the client device 110 with information about the indication without providing predictions regarding the two or more health conditions in the input image. For example, the diagnostic system 130 may scrutinize and verify government regulations and policies (e.g., regulations published by the Food and Drug Administration (FDA) or guidelines issued by the Centers for Disease Control and Prevention (CDC)) and generate results requiring compliance with these regulations. Therefore, even if the diagnostic system 130 can generate predictions regarding two or more health conditions, the diagnostic system 130 may omit these results in its output to the client device 110 if the regulations, for example, impose constraints on the computerized diagnosis of the two or more health conditions presented in the image. In such embodiments, the diagnostic system 130 may simply provide the user with information that the input image contains two or more health conditions, but a diagnosis cannot be made due to government regulations.

[0057] In some embodiments, in response to determining that the input image represents a single health condition, the diagnostic system 130 may access and select a diagnostic model from a database configured to generate a prediction for the single health condition. The diagnostic system 130 generates a prediction for the input image by applying the selected diagnostic model to the input image. Based on the prediction, the diagnostic system 130 generates a diagnosis for the single skin abnormality and consequently provides the diagnosis to the client device.

[0058] In some embodiments, in response to determining that the input image presents two or more health conditions, the diagnostic system 130 may access and select a set of diagnostic models from a database configured to generate separate predictions for each of the two or more health conditions. For example, the diagnostic system 130 may access a model, as illustrated in conjunction with Figure 2-7, configured to generate predictions for two or more health conditions. The diagnostic system 130 generates predictions for two or more health conditions in the input image by applying the set of diagnostic models to the input image. The diagnostic system 130 generates a diagnosis for the input image from the predictions and consequently provides the diagnosis to the client device 110.

[0059] While the machine learning model described in conjunction with Figure 2-8 is configured to receive input data in the form of images, the machine learning model may also be configured to receive additional types of input data. For example, additional types of input data may include patient history data such as the patient's height or weight, the patient's medical history, or patient responses to questions requesting information about pre-existing medical conditions known to the patient, and equivalents. For example, the patient's medical history may indicate that the patient is prone to certain health conditions that may affect the model's predictive outcomes.

[0060] Returning to the diagram in Figure 1, the client device 110 provides the diagnostic system 130 with images of the patient's anatomical parts so that the diagnostic system 130 can generate and display predictions or other indications on the client device 110. The user of the client device 110 could be a hospital medical professional who desires to be assisted in diagnosing a patient's health condition using a computerized diagnostic system. In another embodiment, the user of the client device 110 could be an individual at home or in an office who desires to obtain a diagnosis of their basic health condition based on symptoms presented on their body.

[0061] In one embodiment, the client device 110 includes a browser that allows the user of the client device 110 to interact with the diagnostic system 130 using standard internet protocols. In another embodiment, the client device 110 includes a dedicated application that is specifically designed (e.g., by the organization involved with the diagnostic system 130) to enable interaction between the client device 110 and the server. In one embodiment, the client device 110 includes a user interface that allows the user of the client device 110 to interact with the diagnostic system 130 and visualize predictions of health conditions in input images. For example, the user interface may be configured to overlay predictions generated by the diagnostic system 130 on top of locations in the image where individual health conditions are estimated to be present.

[0062] The client device 110 may be a computing device such as a smartphone, tablet computer, laptop computer, desktop computer, or any other type of network-enabled device that includes or can be configured to connect to an operating system such as ANDROID® or APPLE® IOS®. In another embodiment, the client device 110 is a computing device for creating an augmented reality (AR) environment for the user or a headset including a smartphone camera, or a headset including a computing device for creating a virtual reality (VR) environment for the user. A typical client device 110 includes the hardware and software required to connect to the network 122 (e.g., via WiFi and / or 4G or 5G or other radio telecommunications standards).

[0063] Network 122 provides the communication infrastructure between the client device 110 and the diagnostic system 130. Network 122 is typically the internet, but may be any network, including, but is not limited to, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network. (Diagnostic system architecture)

[0064] Figure 9 is a block diagram of the architecture of a diagnostic system 130 according to one embodiment. The diagnostic system 130 shown in Figure 9 includes a training module 920, a data management module 925, a training module 330, a prediction module 930, and a treatment output module 935. The diagnostic system 130 also includes a training data store 960 and a model data store 965. In alternative configurations, different and / or additional components may be included in the diagnostic system 130.

[0065] The data management module 920 retrieves and manages training data stored in the training data store 960, which can be used to train a set of models described in conjunction with the diagnostic system 130 in Figure 2-8. The training data store 960 includes images or videos of an individual's anatomical parts and information extracted from known diagnoses of health conditions presented in these images. For example, the training data store 960 may include information extracted from medical images such as radiographs, computed tomography (CT) scans, medical resonance imaging (MRI) scans, X-ray images, ultrasound or ultrasound images, tactile images, or thermographic images, which are retrieved by the data management module 920 from hospitals, medical image repositories, research image repositories, and equivalents. In another embodiment, the training data store 960 may include information extracted from images taken by an individual, such as images taken on a smartphone camera, which are retrieved by the data management module 920 from a user of a client device 110.

[0066] The training module 925 uses the information in the training data store 960 to train a set of models. Specifically, given a diagnostic model M i Regarding the training data S, i is the input data x j∈S (For example, training images or videos) and input data x j∈S A known characteristic (e.g., diagnosis) of marker y j∈Si Multiple training instances j=1, 2, ..., |S | include and respectively. i Includes |. Specifically, input data x j This may include information extracted from training images that capture an individual's anatomical parts. In one embodiment, when a machine-learned model is configured to receive additional types of data in addition to or alternative to input images, the input data x j∈S It may also include additional types of data about the patient in the training images (e.g., demographic information about the patient). In one example, label y j∈Si The characteristics of the input data may be obtained by a human operator, such as a physician, medical professional, or other individual, who can verify the characterization of the input data. In another case, the labels themselves may be predictions generated by a machine learning model, which are later validated by a human operator.

[0067] The training module 925 trains a set of weights with respect to the model by repeatedly iterating between a forward path step and a backpropagation step. During the forward path step, the training module 920 trains a corresponding subset S of the training data. i A model with a set of weighted values ​​is input to x across the input data x. j∈S By applying this, the estimated output y' j∈S This generates the estimated output y' for multiple training instances. The training module 920 generates the estimated output y' for multiple training instances. j∈S and sign y j∈SA loss function is determined that shows the difference between the given value and the target value. During the backpropagation step, the training module 920 iteratively updates the set of weights for the model by a backpropagation error term obtained from the loss function. This process is repeated until the change in the loss function meets a predetermined criterion. For example, the criterion may be triggered if the change in the loss function in each iteration falls below a threshold.

[0068] In one embodiment, the loss function is, for example, the indicator y j When is a continuous value, it can be calculated as follows: [ka] In the formula, θ p is a set of weights relating to the diagnostic model. In another embodiment, the loss function is, for example, the label y j When it is a binary value, it can be calculated as follows: [ka] However, it should be understood that the loss function can be any other function, such as the L1-norm or L-infinite norm, that represents the difference between the actual marker and the estimated output that occurs between each training interaction.

[0069] As described below, the training module 925 trains a set of diagnostic models, as illustrated in conjunction with Figure 2-7, which can be used to determine an active diagnosis of an input image. In one embodiment, one or more diagnostic models are configured to produce a prediction y'=v' as an output vector, where each element in the output vector v' corresponds to a prediction about an individual health condition. In such an embodiment, the training indicator y for the diagnostic model is j =v jThe data may be encoded as a one-hot encoded vector, where each element in the vector is a non-zero binary value (e.g., a value of 1) if the training image is diagnosed with an individual health condition with respect to the element, and zero if no diagnosis exists. Thus, with respect to training images presenting two or more health conditions, each individual element may be set to an appropriate value such that two or more elements in the vector have non-zero values. For example, training data for a diagnostic model configured to predict rashes and moles may include a label "

[0001] " where the first element indicates the presence of a rash in the corresponding training image, and the second element indicates the absence of a mole in the training image. In another embodiment, the diagnostic model may be configured to generate a prediction y'=v' as an output value that can be assigned to one or more categories, each representing an individual health condition. In such an embodiment, the label y for the diagnostic model may be j =v j This may be a value representing a category related to an individual health condition diagnosed within the training image. For example, training data for an exemplary diagnostic model may include a marker "1" indicating the presence of a rash in the training image, or a marker "2" indicating the presence of a mole in the training image.

[0070] The training module 920 trains a set of weights on an ensemble of diagnostic models, as illustrated in conjunction with Figure 3. Specifically, each diagnostic model in the ensemble may be configured to generate predictions on a separate set of one or more health conditions. The diagnostic models in the ensemble may differ from one another in terms of the model architecture, the set of health conditions on which the model is configured to predict, and their equivalents. In one embodiment, each diagnostic model is configured as a neural network architecture, comprising a set of layers of nodes, with each layer connected to the previous layer via a set of weights. Thus, the diagnostic models in such an ensemble may differ from one another in terms of the number of layers, nodes, and connections in the architecture.

[0071] Training module 925 uses input data x j∈SiA training image as a training image, and a corresponding label y indicating the presence of a specific health condition that the diagnostic model is configured to predict. j∈S =v j∈S Using the training dataset, which includes the above, a set of weights is trained for the diagnostic model in the ensemble set. During the forward path step, the training module 925 estimates the output y' by applying the diagnostic model to the training images. j∈S =v' j∈S This generates the following. During the backpropagation step, the training module 925 repeatedly updates the set of weights for the diagnostic model using terms obtained from the loss function.

[0072] The training module 920 trains a set of weights for a feature extractor model and a feature classifier model, as described in conjunction with Figure 4. In one embodiment, the feature extractor model and the feature classifier model are trained by training a neural network architecture, which includes a first part and a second part. The set of weights for the first part of the neural network architecture may be stored as a set of weights for the feature extractor model, and the set of weights for the second part may be stored as a set of weights for the feature classifier model. The set of weights for the feature classifier model is then retrained using multiple components extracted from training images.

[0073] Figure 10 illustrates an exemplary training process for a set of diagnostic models shown in Figure 4, according to one embodiment. The training module 925 trains a neural network architecture 1036, which includes a set of layers of nodes, each layer connected to the previous layer via a set of weights. The neural network architecture 1036 may be configured to generate predictions for one or more distinct sets of health states. In one embodiment, the neural network architecture 1036 is configured as a deep neural network (DNN), a convolutional neural network (CNN), or any other type of neural network architecture capable of receiving image data and generating outputs.

[0074] Training module 925 uses input data x j∈Si The training image is used as a training image, and the corresponding label y indicates the presence of a specific health condition that the feature classifier model is configured to predict. j∈S =v j∈S Using the training dataset, including the above, a set of weights is trained for the neural network architecture. During the forward pass step, the training module 925 estimates the output y' by applying the neural network architecture to the training images. j∈S =v' j∈S This generates the following. During the backpropagation step, the training module 925 repeatedly updates the set of weights for the neural network architecture using terms obtained from the loss function.

[0075] After the training process for the neural network architecture is complete, the training module 925 stores the first part of the neural network architecture as a set of weights for the feature extractor model and the second part of the neural network architecture as a set of weights for the feature classifier model. In one embodiment, the first and second parts are selected such that the second part includes a set of weights for layers that are placed after the layers for the first part of the neural network architecture. For example, the first part may include a set of weights for the first three layers of the neural network architecture, while the second part may include a set of weights for the last five layers of the neural network architecture.

[0076] Training module 925 then processes the input data x j∈Si Multiple components CF extracted from training images j∈S And, corresponding labels y indicate the presence of specific health conditions in the training images, which are configured for the feature classifier model to predict. j∈S =v j∈SThe set of weights for the feature classifier model is retrained using the training dataset, which includes the following. In one example, multiple components of a training image are extracted by applying the trained feature extractor model to the training image, generating an image tensor about the training image, performing ICA on the image tensor, and generating multiple components. During the forward path step, the training module 925 retrains the feature classifier model for multiple components CF j∈S By applying this, the estimated output y' j∈S =v' j∈S This generates the following. During the backpropagation step, the training module 925 repeatedly updates the set of weights for the feature classifier model using terms obtained from the loss function.

[0077] Training module 925 trains a set of weights with respect to the RNN architecture described in conjunction with Figure 5. Training module 925 uses input data x j∈S A series of training images as and corresponding labels y indicating the presence of specific health conditions that the RNN architecture is configured to predict. j∈S =v j∈S A set of weights is trained for the RNN architecture using a training dataset that includes the following. In one embodiment, when the training data for the RNN architecture includes a series of training images presenting two or more health states, the label v in each iteration of the series is trained. j This may be a one-hot encoded vector having non-zero values ​​for individual health states present in the training images. During the forward pass step, the training module 925 iteratively applies the RNN architecture to a series of training images to obtain a series of estimated outputs y' j∈S =v' j∈S This generates a label v j This is a difference, for example, |v j∈S -v' j∈S| may be reordered during the forward pass steps of the training process so that all or a subset of possible orderings are reduced or minimized. The training module 925 determines the loss function by combining the differences between the estimated output and the marker at each iteration in a series of iterations. During the backpropagation step, the training module 925 iteratively updates the set of weights for the RNN architecture using terms obtained from the loss function.

[0078] Training module 925 trains the differentiator model, amorphous anomaly model, and localized anomaly model, as described in conjunction with Figure 6. Training module 925 uses input data x j∈S A training image as such, and a corresponding label y indicating the presence of amorphous or localized skin abnormalities. j∈S =v j∈S A set of weights is trained on the differentiator model using a training dataset that includes . In one case, the differentiator model is configured to generate indications as vectors, where the first element corresponds to the likelihood that the image contains amorphous skin abnormalities, and the second element corresponds to the likelihood that the image contains localized skin abnormalities. In such a case, the indicator y j∈S The first element of the vector may be encoded such that if the training image contains an amorphous skin abnormality, the first element of the vector is a non-zero value (e.g., a value of 1), or zero if the image does not contain it. Similarly, the second element of the vector is a non-zero value (e.g., a value of 1) if the training image contains a localized skin abnormality, or zero if the image does not contain it. During the forward pass step, the training module 925 estimates the output y' by applying the differentiator model to the training image. j∈S =v' j∈S This generates a signal, and the set of weights for the differentiator model is repeatedly updated during the backpropagation step using the terms obtained from the loss function.

[0079] Training module 925 uses input data x j∈SiA training image as, and a corresponding label y indicating the presence of a specific amorphous skin abnormality configured to be predicted by the model j∈S =v j∈S Using a training dataset including the above, train a set of weights for the amorphous abnormality model. The training data for the amorphous abnormality model may include training images with a single amorphous skin abnormality or multiple amorphous skin abnormalities. During the forward pass step, the training module 925 applies the amorphous abnormality model to the training image to generate an estimated output y’ j∈S =v’ j∈S During the backpropagation step, the training module 925 repeatedly updates the set of weights for the amorphous abnormality model using terms obtained from the loss function.

[0080] The training module 925 uses a training dataset containing the training image as input data x j∈S and a corresponding label y indicating the presence of a specific focal skin abnormality configured to be predicted by the model j∈S =v j∈S to train a set of weights for the focal abnormality model. The training data for the focal abnormality model may include training images with a single focal skin abnormality or multiple focal skin abnormalities. During the forward pass step, the training module 925 applies the focal abnormality model to the training image to generate an estimated output y’ j∈S =v’ j∈S During the backpropagation step, the training module 925 repeatedly updates the set of weights for the focal abnormality model using terms obtained from the loss function.

[0081] As further described below, the training module 925 further trains an indicator model, described in conjunction with FIG. 8, that provides an indication of whether the image presents two or more health states. The training module 925 uses a training dataset containing the training image as input data x j∈S and a corresponding label y indicating whether the training image presents two or more health states j∈S =v j∈SUsing a training dataset containing the same, train a set of weights for the indicator model. The training data may include training images with symptoms from a single health state or multiple health states. In one example, the indicator model is configured to generate an indication indicating that the image may present two or more health states. In such an example, the label y j∈Si may be encoded as a non-zero value if the training image presents two or more health states, or as zero if the training image presents a single health state. During the forward pass step, the training module 925 applies the indicator model to the training image to generate an estimated output y' j∈S = v' j∈S . During the backpropagation step, the training module 925 repeatedly updates the set of weights for the indicator model using terms obtained from the loss function.

[0082] The training module 925 can also train a diagnostic model incorporated with indicator functionality. The diagnostic model can be any model configured to generate a computerized prediction for one or more health states in an image, such as the set of diagnostic models of FIGS. 2-7, or a diagnostic model configured to predict a single health state. The diagnostic model can generally be trained using diagnostic training data (e.g., training images and corresponding health state labels), but the training module 925 further trains the diagnostic model to incorporate indicator functionality by including, in addition to the diagnostic training data, indicator training data for the diagnostic model.

[0083] In one embodiment, the indicator training data S a includes training images presenting two or more health states and corresponding labels y j∈Sa = v j∈Sa indicating the presence of two or more health states in the training image. In one example, when the label y j∈S = v j∈S for the diagnostic training data is a one-hot encoded vector, the label y j∈Sa=v j∈Sa The vector may have non-zero values ​​for all or most of its elements. For example, training data for a diagnostic model configured to predict a single health condition may include a label "

[0011] " which contains non-zero values ​​for all elements if the training images represent two or more health conditions. Thus, the diagnostic model is configured to generate indications by predicting relatively similar or equal probabilities for all or most health conditions.

[0084] In another example, marker y related to instruction training data j∈Sa =v j∈Sa It may include additional elements or categories for assigning images that represent two or more health conditions. For example, label y for diagnostic training data. j∈S =v j∈S However, when the vector is one-hot encoded, the marker may further be configured to include a separate element regarding indication, in addition to elements regarding different health conditions. The separate element is a non-zero value if the training image presents two or more health conditions, and zero if the training image does not present them. For example, training data for a diagnostic model configured to predict rashes and / or moles may include the marker "

[0111] " where the first element indicates the presence of a rash in the corresponding training image, the second element indicates the presence of a mole in the training image, and the third separate element indicates the presence of two or more health conditions in the training image. In another embodiment, a marker y regarding diagnosis j∈S =v j∈S And when the indicator is a category, the indicator may further be configured to include separate categories related to indications in addition to categories related to different health conditions. For example, training data for an exemplary diagnostic model may include indicator "1" indicating the presence of a rash in the training image, indicator "2" indicating the presence of a mole in the training image, and indicator "3" indicating the presence of two or more health conditions in the training image.

[0085] The training module 925 may store the trained machine learning model in the model data store 965 so that it can be deployed during the inference process, in the manner described in conjunction with Figure 2-8.

[0086] The prediction module 930 receives a request from the client device 110 and provides a computerized diagnosis of a health condition captured in the input image. In response to a request for diagnosis, the prediction module 930 may select a set of diagnostic models that can be used to generate a diagnosis regarding the input image, using diagnostic models trained by the diagnostic system 130. The prediction module 930 may apply the selected diagnostic models and generate a prediction about the health condition presented in the input image, similar to the exemplary reasoning process described in conjunction with Figure 2-8. In one example, the prediction module 930 makes a diagnosis regarding a health condition if the probability of prediction exceeds a threshold. For example, the prediction module 930 may conclude that a particular health condition is indeed present in the input image if the predictability exceeds 0.80 or 80%. The prediction module 930 may conclude that a particular health condition is otherwise missing from the input image.

[0087] In some embodiments, the prediction module 930 may determine that the prediction for an individual health condition is not definitive and generate information to the client device 110 that a diagnostic request cannot be made, or alternatively, output a diagnosis to the client device 110 regarding the remaining health conditions for which there are definitive predictions. For example, the prediction module 930 may determine that a diagnosis is not definitive if the individual prediction has a probability between a first threshold and a second threshold (e.g., 20% to 80%). In another example, the prediction module 930 may determine a diagnosis for a health condition if the probability of the prediction is within a threshold percentage of the other predictabilities generated by the diagnostic model.

[0088] In one embodiment, the prediction module 930 is programmed to comply with relevant government regulations or policies governing computerized diagnostics and generates diagnostic results in accordance with current regulations or policies. For example, the regulations or policies may determine various prediction thresholds used to determine whether an individual health condition is presented in the input image, and these thresholds may vary between different health conditions. In one example, the administrator of the diagnostic system 130 may scrutinize these regulations or policies and program the prediction module 930 to comply with them. In another example, the prediction module 930 may collect information from the website or database of a relevant organization (e.g., the FDA) and generate predictions according to rules analyzed from the collected information. Thus, the prediction module 930 may scrutinize the predictions generated by the model and generate diagnostic results regarding the requirements in accordance with these regulations or policies. For example, the prediction module 930 may determine a diagnosis for acne if the predictability exceeds an 80% threshold, in accordance with policies set by the FDA, while the prediction module 930 may determine a diagnosis for rash if the predictability exceeds only a 95% threshold.

[0089] Furthermore, in response to a diagnostic request, the prediction module 930 may select an indicator model or a diagnostic model with indicator functionality and generate an indication regarding the input image. The prediction module 930 may apply the selected model and generate an indication. In one example, the prediction module 930 determines that the input image presents two or more health conditions if the likelihood of indication exceeds a threshold. In another example, the prediction module 930 determines that the input image presents two or more health conditions if the likelihood of indication is within a threshold percentage of other outputs generated by the selected model. As illustrated in conjunction with Figure 8, the prediction module 930 may generate results regarding the request based on its decision, such that information is output to the user of the client device 110 indicating that an appropriate diagnostic model may be selected to generate a diagnosis, or that a diagnosis cannot be made, for example, due to compliance with government regulations or policies regarding computerized diagnostics.

[0090] The treatment output module 935 provides potential treatment options in conjunction with the diagnosis provided by the prediction module 930. Based on the diagnostic results, it may be advantageous for the user of the client device 110 to receive potential treatment options using the diagnosis, for example, so that the health condition can be treated within a short time cycle if necessary. In one embodiment, the treatment output module 935 may generate treatment options by consulting with a human medical professional, using the logic of a professional system, or by deriving options from a previous system. In one embodiment, the treatment output module 935 may propose treatment options that are known to reduce and avoid side effects to the health condition presented by the patient. (Methods for diagnosing overlapping skin abnormalities)

[0091] Figure 11 illustrates a method for diagnosing overlapping skin abnormalities in an input image. The diagnostic system 130 receives a request from a client device (e.g., via network 120) to diagnose skin abnormalities in an input image 1102. The input image contains overlapping skin abnormalities on the patient's skin. The diagnostic system 130 accesses a set of machine-learned models (e.g., via network 120, using a prediction module 930) from a database (e.g., a model data store 965, which may contain models such as model 965, trained using training data 960 and described in conjunction with Figure 2-8) 1104. Each machine-learned model contains a distinct set of trained weights. The diagnostic model 130 generates distinct predictions for each of two or more skin abnormalities in the input image (e.g., using a prediction module 930) by applying the set of machine-learned models to the input image (e.g., an inference process described in conjunction with Figure 2-8) 1106. The predictions indicate the likelihood that a distinct skin abnormality among the two or more skin abnormalities is present in the input image. The diagnostic system 130 generates a diagnosis of overlapping skin abnormalities from predictions (for example, using the prediction module 930) 1108. The diagnostic system 130 provides the diagnosis to client devices (for example, via the network 120) 1100. (Methods for inducing indications of two or more skin abnormalities)

[0092] Figure 12 illustrates how to generate indications of two or more skin abnormalities in an input image. The diagnostic system 130 receives a request from a client device for a diagnosis of skin abnormalities in an input image (e.g., via network 120) 1202. The diagnostic system 130 accesses a machine learning model from a database (e.g., a model data store 965, which may include a model 965 trained using training data 960, as described in conjunction with Figure 8) (e.g., via network 120, using a prediction module 930) 1204. The machine learning model includes a separate set of trained weights. The diagnostic model 130 generates indications (e.g., using a prediction module 930) by applying the machine learning model to the input image (e.g., an inference process described in conjunction with Figure 8) 1206. The diagnostic system 130 generates results for the request (e.g., using a prediction module 930) based on the determination of overlapping skin abnormalities from the predictions 1208. The diagnostic system 130 provides the results to a client device 1210 (for example, via the network 120). (overview)

[0093] The foregoing description of embodiments of the present invention is presented for illustrative purposes only and is not intended to be comprehensive or to limit the invention to the precise forms disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.

[0094] Several parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the field of data processing to effectively communicate the nature of the work to others skilled in the field. These operations are described functionally, computationally, or logically, but are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or equivalents. Furthermore, it has also proven that, without loss of generality, it is convenient from time to refer to sequences of these operations as modules. The operations described and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0095] Any of the steps, operations, or processes described herein may be performed or implemented, either alone or in combination with other devices, using one or more hardware or software modules. In one embodiment, the software module is implemented with a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described herein.

[0096] Embodiments of the present invention may also relate to apparatus for carrying out the operations described herein. The apparatus may, in particular, comprise a general-purpose computing device that may be constructed for a required purpose and / or selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transient, tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referenced herein may comprise a single processor or may be an architecture employing multiple processor designs for increased computing power.

[0097] Embodiments of the present invention may also relate to products produced by computing processes described herein. Such products may comprise information arising from the computing processes, which is stored on a non-transient, tangible, computer-readable storage medium and may include any embodiment of computer program products or other data combinations described herein.

[0098] Finally, the language used herein has been selected primarily for readability and explanatory purposes and may not be used to accurately describe or limit the subject matter of the invention. Therefore, the scope of the invention is intended to be limited not by the embodiments for carrying out the invention, but rather by any claims arising in an application thereunder. Thus, the disclosure of embodiments of the invention is intended to be illustrative, not a limitation of the scope of the invention as described in the following claims.

Claims

[Claim 1] The invention described herein.

Images