Using machine learning models to diagnose skin conditions

Abstract

A diagnostic system trains a set of machine learning diagnostic models that are configured to receive images of a patient and generate a prediction as to whether the patient has one or more health conditions. In one embodiment, the set of machine learning models is trained to generate predictions for images containing two or more potential health conditions for the patient. In one example, symptoms of the two or more health conditions are shown as two or more overlapping skin abnormalities on the patient. By using the architecture of the set of diagnostic models described herein, the diagnostic system can generate more accurate predictions for images containing overlapping symptoms of two or more health conditions compared to existing systems.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 869,553, filed on July 1, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] The present disclosure relates generally to diagnosing health anomalies, and more particularly to diagnosing overlapping health conditions in images.

[0004] The computerized diagnostic system is configured to receive an image of a patient and generate a prediction about whether the patient has one or more health conditions based on the presence of symptoms in the image. The computerized diagnostic system can be applied to predict health conditions in different anatomical parts of the patient. For example, a skin diagnostic system can be applied to predict whether an image of a patient's skin has one or more skin abnormalities, such as a rash, a nevus, eczema, acne, and a cold sore. As another example, a tumor diagnostic system can be applied to predict whether an image of a patient's anatomical structure contains a tumor.

[0005] Often, a patient may have multiple health conditions that appear in the same image. For example, an image may contain a psoriasis plaque and a mole, both skin abnormalities located on the same arm, spatially separated from each other. As another example, an image may contain skin abnormalities that overlap each other, such as a psoriasis plaque with a pigmented mole within its borders. However, in these instances, existing computerized diagnostic systems have difficulty generating accurate diagnoses because, for example, the computerized diagnostic systems have been trained to generate predictions for a single health condition. In addition, when health conditions overlap, the presentation and other symptoms in the image may be the result of either condition alone or a combination of both conditions, and the computerized diagnostic system may have difficulty accurately distinguishing the symptoms in the image as the corresponding health conditions. Summary of the invention

[0006] The diagnostic system trains a set of machine learning diagnostic models that are configured to receive an image of a patient and generate a prediction as to whether the patient has one or more health conditions. In one embodiment, the set of machine learning models is trained to generate predictions for an image that includes two or more potential health conditions for the patient. In one instance, symptoms of the two or more health conditions are shown as two or more overlapping skin abnormalities on the patient. By using the architecture of the set of diagnostic models described herein, the diagnostic system can generate more accurate predictions for images that include overlapping symptoms of two or more health conditions compared to existing systems.

[0007] In an embodiment, a diagnostic system receives a request from a client device to diagnose a skin abnormality in an input image. The input image includes overlapping symptoms of two or more skin abnormalities on the skin. The diagnostic system accesses a set of machine learning diagnostic models from a database. Each machine learning model includes a corresponding set of trained weights, which are determined by a training process. The diagnostic system generates a corresponding prediction for each of the two or more skin abnormalities in the input image by applying the set of diagnostic models to the input image. The prediction indicates the likelihood that a corresponding skin abnormality of the two or more skin abnormalities is shown in the input image. The diagnostic system generates a diagnosis of the skin abnormality from the prediction of the input image and provides the diagnosis to the client device.

[0008] In one embodiment, a diagnostic system trains a set of machine-learned indicator models that are configured to receive an image of a patient or individual and generate an indication of whether the input image presents two or more health conditions. In one embodiment, the diagnostic system receives a request to diagnose a health condition in an input image. The diagnostic system accesses an indicator model or a diagnostic model with indicator functionality from a database. By applying the indicator model to the input image, the diagnostic system generates an indication representing the likelihood that the input image includes two or more health conditions. The diagnostic system determines from the indication generated for the input image whether the input image presents two or more health conditions. The diagnostic system generates a result of the request based on the determination and provides the result to a user of a client device.

[0009] Because input images presenting multiple health conditions may be difficult to diagnose, a user of a client device may receive such an indication as a supplement or alternative to an active diagnosis of the image. The user may then determine to provide the input image to a medical expert for a more accurate diagnosis. Thus, it would be advantageous for a user of a computerized diagnostic system to receive information regarding whether an input image includes symptoms from two or more health conditions so that a different expert system or medical expert can make a more accurate diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a high-level block diagram of a system environment for a diagnostic system according to an embodiment.

[0011] Figure 2 The general reasoning process of the diagnosis model according to the embodiment is illustrated.

[0012] Figure 3 An example reasoning process of a set of machine learning diagnostic models with an integrated architecture according to an embodiment is illustrated.

[0013] Figure 4 An example inference process using independent component analysis (ICA) in conjunction with a set of diagnostic models according to an embodiment is illustrated.

[0014] Figure 5 An example reasoning process of a diagnostic model with a recurrent neural network (RNN) architecture according to an embodiment is illustrated.

[0015] Figure 6 An example reasoning process of a diagnostic model set including discriminator models for amorphous and localized skin abnormalities according to an embodiment is illustrated.

[0016] Figure 7 An example inference process of a diagnostic model with non-maximum suppression according to an embodiment is illustrated.

[0017] Figure 8 An example reasoning process of an uncertainty model according to an embodiment is illustrated.

[0018] Fig. 9 is a block diagram of the architecture of a diagnostic system according to an embodiment.

[0019] Fig.10 The diagram shows a method for Figure 4 An example training process for the set of diagnostic models shown in .

[0020] Fig.11 An exemplary data flow for diagnosing overlapping skin abnormalities in an image is illustrated in accordance with an embodiment.

[0021] Fig.12 An exemplary data flow for generating an indication of two or more skin abnormalities in an input image is illustrated, according to an embodiment.

[0022] The accompanying drawings depict various embodiments of the present invention for illustration purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods described herein may be employed without departing from the principles of the present invention described herein. DETAILED DESCRIPTION

[0023] Overview

[0024] Figure 1 is a high-level block diagram of a system environment for a diagnostic system according to an embodiment. Figure 1 The illustrated system environment 100 includes one or more client devices 110, a network 120, and a diagnostic system 130. In alternative configurations, different and / or additional components may be included in the system environment 100.

[0025] The diagnostic system 130 is a system for providing various types of computerized diagnoses to users of the client device 110. The diagnostic system 130 can receive an image of an anatomical part of a patient and generate a prediction about whether the patient has one or more health conditions based on the image. For example, the diagnostic system 130 can receive an image of the patient's skin and generate a prediction about whether the patient has one or more skin abnormalities, such as a rash, a nevus, eczema, acne, and a cold sore. As another example, the diagnostic system 130 can receive a radiological image of the patient's brain and generate a prediction about whether the patient has a brain tumor. The prediction generated by the diagnostic system 130 can be used as a stand-alone diagnosis, or can be used to assist, for example, a doctor in a hospital in interpreting the patient's images.

[0026] The diagnostic system 130 trains a set of machine learning diagnostic models that are configured to receive images of a patient and generate predictions of one or more health conditions. The images received by the diagnostic system 130 may be medical images obtained in a hospital, such as radiological images, computed tomography (CT) scans, medical resonance imaging (MRI) scans, X-ray images, ultrasound or sonographic images, tactile images, or thermal imaging images. In another example, the images may be pictures taken by a personal user of the client device 110 (e.g., at home or in the office). The images may show symptoms or other presentations on the patient's anatomical parts, which the diagnostic system 130 may use to infer the health conditions that are the cause of the symptoms.

[0027] In one embodiment, the diagnostic system 130 trains a set of machine learning models to generate predictions for an image that includes symptoms of two or more health conditions of a patient. Specifically, a patient may have symptoms from multiple health conditions that appear in the same image. For example, an image may contain psoriasis plaques and moles, both skin abnormalities located on the same arm, spatially separated from each other. As another example, an image may include skin abnormalities that overlap each other, such as psoriasis plaques with pigmented moles within their boundaries. However, in these instances, existing computerized diagnostic systems may have difficulty generating accurate diagnoses, especially when symptoms overlap, because symptoms may be the result of an unknown combination of underlying health conditions of a patient or the result of an individual. In addition, existing diagnostic systems may be configured to generate predictions for a single condition. Therefore, even if an existing diagnostic system is able to detect one of the health conditions in an image, the detected condition will exclude the remaining health conditions in the image, resulting in missed diagnoses, which can lead to significant health problems for the subject of the image.

[0028] During the reasoning process, the diagnostic system 130 receives a request to diagnose a health condition in an input image. The input image includes overlapping symptoms of two or more health conditions from a patient. The diagnostic system 130 accesses a set of machine learning diagnostic models from a database. Each machine learning model includes a corresponding set of trained weights, which are determined by a training process. The diagnostic system 130 generates a corresponding prediction for each of the two or more health conditions in the input image by applying the diagnostic model set to the input image. The prediction indicates the likelihood that the input image shows symptoms from the corresponding health condition. The diagnostic system 130 generates a diagnosis for the input image from the prediction and provides the diagnosis back to the client device 110.

[0029] In a specific embodiment mentioned in the rest of this specification, the two or more health conditions are two or more skin abnormalities that have overlapping symptoms presented on the patient's skin. For example, a mole may appear in an area of ​​skin with a rash. As another example, an area of ​​skin with eczema may overlap with a portion of skin with contact dermatitis. However, it should be understood that in other embodiments, the two or more health conditions can be any type of health-related condition, such as a disease or allergy that can produce overlapping or spatially separated symptoms in the same anatomical part of a patient. For example, the two or more health conditions can be different types of tumors, blood diseases, etc., which can present symptoms in a patient alone or in combination.

[0030] Figure 2 The general reasoning process of the diagnostic model according to the embodiment is illustrated. The diagnostic system 130 receives an input image 210 of a patient suspected of having a skin abnormality. Specifically, the input image 210 shows a first skin abnormality 212 and a second skin abnormality 214 overlapping each other on the patient's arm. The diagnostic system 130 accesses a set of machine learning diagnostic models from a database. The diagnostic system 130 applies the diagnostic model to the input image 210 to generate two or more predictions v'1, v'2, ..., v' for the input image 210. n The prediction v'1 may indicate the likelihood that the input image 210 includes symptoms from the corresponding skin abnormality i. Figure 2 In the example shown in , prediction v'1 may indicate a high likelihood that the input image 210 includes a nevus, while prediction v'2 may indicate a high likelihood that the input image 210 also includes a rash. The remaining predictions for other skin abnormalities may be associated with significantly low probabilities. By selecting a subset of predictions with probabilities above a predetermined threshold and providing a diagnosis to the client device 110, the diagnostic system 130 determines that the input image 210 includes a nevus and a rash.

[0031] Back to Figure 1In a system environment, the diagnostic system 130 typically uses a training corpus of images and labels to train a corresponding set of weights for a diagnostic model to reduce a loss function. The loss function for the diagnostic model indicates the difference between an estimated output and actual labels in the training corpus, the estimated output being generated by applying a diagnostic model with an estimated set of weights to input data that the diagnostic model is configured to receive in the training corpus, the actual labels in the training corpus representing the type of data that the diagnostic model is configured to predict. The estimated set of weights for the diagnostic model is repeatedly updated to reduce the loss function until a predetermined criterion is reached. In combination Fig. 9 The training process for the diagnostic model set is described in more detail.

[0032] As stated in this article, Figures 3 to 7 Various architectures of a set of diagnostic models are illustrated that the diagnostic system 130 can use during the inference process to generate predictions of two or more health conditions present in the image. Each of the architectures described below can include a set of trained weights determined through a training process that will be combined with Fig. 9 By using the architecture of the diagnostic model set described herein, the diagnostic system can generate more accurate predictions for images that include two or more overlapping health conditions compared to existing systems.

[0033] Figure 3 An example reasoning process for a machine learning diagnostic model set with an integrated architecture according to an embodiment is illustrated. In one embodiment, the diagnostic model set includes an integrated set of machine learning diagnostic models. Specifically, each diagnostic model in the ensemble is configured to receive an input image and generate one or more predictions about whether a patient has one or more health conditions by applying a trained set of weights to the diagnostic model. Figure 3 In the example shown in , the diagnostic model ensemble set includes “diagnostic model 1”, “diagnostic model 2”, and “diagnostic model 3”.

[0034] The diagnostic models in the ensemble are configured to generate predictions for different or the same set of health conditions as other diagnostic models in the ensemble. Figure 3 In the example shown in , "Diagnostic Model 1" can be configured to generate a prediction v' M1 1 and predicted v' M1 2. Predict v' M1 1 indicates the possibility that the input image includes a mole, and predicts v' M1 2 indicates the likelihood that the input image includes a rash. As another example, “diagnosis model 2” may be configured to generate a single prediction v′ indicating the likelihood that the input image includes a rash M2 1, and "Diagnostic Model 3" can be configured to generate predictions v' M3 1 and predicted v' M32. Predict v' M3 1 indicates the possibility that the input image includes a mole, and predicts v' M3 2 indicates the likelihood that the input image includes acne.

[0035] During the inference process, the diagnostic system 130 generates a corresponding prediction for each of two or more health conditions in the input image by applying the ensemble of diagnostic models to the input image and combining the predictions for each corresponding health condition from the ensemble. In one example, the prediction for the corresponding health condition is generated by calculating the average of the predictions for the health condition generated by the ensemble. Figure 3 In the example shown in , by comparing the predictions v' from "Diagnostic Model 1" and "Diagnostic Model 2" M1 2, v' M2 1 are averaged to generate a prediction v'1 indicating the likelihood that the patient has a rash. However, it should be appreciated that in other embodiments, the predictions from the diagnostic models in the ensemble may be combined in any other manner other than an average. The diagnostic system 130 determines a diagnosis by selecting a subset of predictions that have a likelihood above a predetermined threshold.

[0036] In one embodiment, the diagnostic models in the ensemble set can be configured to generate multiple outputs that do not necessarily correspond to the same sequence of health conditions each time the diagnostic models are applied to the input image. In such an embodiment, the diagnostic system 130 can group the predictions from the ensemble set by similarity of the predictions and intermediate features, and can generate predictions for two or more health conditions presented in the input image by combining the predictions from each identified group. In another instance, the diagnostic system 130 can group the predictions from the ensemble set at similar locations using, for example, an attention model, and can generate predictions for two or more health conditions in the input image by combining the predictions for each identified location.

[0037] Figure 4 An example reasoning process using independent component analysis (ICA) in combination with a diagnostic model set according to an embodiment is illustrated. In one embodiment, the diagnostic model set includes a feature extractor model, an independent component analysis (ICA) model, and a feature classifier model. The feature extractor model is configured to receive an input image and generate an image tensor characterizing multiple spatial features in the input image by applying a first trained weight set. The ICA model is configured to receive an image tensor and extract multiple components from the image tensor. The feature classifier model is configured to receive a component and generate a prediction by applying a second trained weight set, which indicates the likelihood of a corresponding health condition for the component.

[0038] During the inference process, the diagnostic system 130 generates an image tensor 416 for the input image by applying the feature extractor model to the input image. Considering each spatial position in the image tensor 416 as an observation position, the diagnostic system 130 performs ICA to extract multiple components CF1, CF2, CF3, ..., CF from the image tensor 416. n For each extracted component, the diagnostic system 130 calculates the contribution of the component to the image tensor 416 to generate a corresponding tensor for the component. In one example, the corresponding tensor for the component is generated by calculating the contribution of the remaining components in the plurality of components and subtracting the contribution of the remaining components from the image tensor 416. Figure 4 In the example shown in FIG. 4 , the corresponding tensor 418 for component CF1 is obtained by calculating the remaining components CF2, CF3, ..., CF n and subtracting these contributions from the image tensor 416. Figure 4 Only the corresponding tensor 418 for component CF1 is illustrated, but the process can be repeated to generate tensors 418 for the remaining components CF2, CF3, ..., CF n The corresponding tensor for each of .

[0039] For each tensor, the diagnostic system 130 generates a prediction by applying a feature classifier model to the tensor for the component. Figure 4 In the example shown in , a prediction v'1 for component CF1 is generated by applying the feature classifier model to the tensor 418 for the component. Figure 4 Only the prediction v'1 for component CF1 is illustrated, but the process can be repeated to obtain the predictions v'1 from the remaining components CF2, CF3, ..., CF n By selecting a subset whose likelihood is above a threshold, the diagnostic system 130 generates a corresponding prediction based on the predictions v'1, v'2, ..., v' n to confirm the diagnosis.

[0040] Based on the number of components extracted from the image tensor, the diagnoses can each correspond to a different health condition under the input image, or if the number of extracted components is greater than the number of health conditions included in the input image, a subset of diagnoses can correspond to the same health condition. In one instance, the diagnostic system 130 extracts a predetermined number of components based on the estimated number of health conditions in the input image. In another instance, the diagnostic system 130 performs ICA with an increasing number of components and iteratively generates predictions and diagnoses with each new component. The diagnostic system 130 completes the reasoning process when further increasing the number of components does not increase the number of unique diagnoses, and there is convergence for the criterion function of ICA. In one instance, there may be a threshold number of components added for determining convergence. For example, adding three components when the diagnostic set remains unchanged can be considered convergence.

[0041] In another embodiment, multiple components are pre-extracted during the training process. During the inference process, the diagnostic system 130 decomposes the image tensor into a mixture of multiple components, and calculates the contribution of each predetermined component to the image tensor to generate a corresponding tensor for each component. The diagnostic system 130 identifies a subset of components with contributions above a threshold. For each tensor, the diagnostic system 130 generates a prediction by applying a feature classifier model to the tensors for the components in the subset. In some embodiments, each predetermined component can be pre-classified and stored using the feature classifier model. Then, the classification result is determined by using the stored components associated with the contributing components.

[0042] Figure 5 An example reasoning process of a diagnostic model having a recurrent neural network (RNN) architecture according to an embodiment is illustrated. In one embodiment, the diagnostic model set includes a diagnostic model having an RNN architecture, which includes one or more neural network layers with a set of weights. The RNN architecture is configured to receive an image sequence and sequentially generate predictions for two or more health conditions presented in the image sequence. The sequence of input images can be a repeated sequence of the same input images, or can be a sequence of images that capture the anatomical part of the patient in question with different views, perspectives, filters, etc. Figure 5 In the example shown in , the image sequence 510(1), 510(2), ..., 510(n) is the same image of the patient's arm skin. However, in another example, the image sequence may include images of the same arm captured from different viewpoints or with different color scales.

[0043] Specifically, for the current image in the sequence, the RNN architecture is configured to receive the current image 510(i) and the hidden state h of the previous iteration. i-1 , and generates a hidden state h for the current image 510(i) by applying the first trained weight seti The RNN architecture is also configured to receive the hidden state h of the current image i And generate a prediction v for the corresponding health condition by applying the second trained weight set i After each iteration, the RNN architecture may also be configured to determine whether the diagnostic model should generate a prediction for another health condition in the input image or complete the inference process. If the RNN architecture determines to continue generating predictions, the RNN architecture receives the next image 510 (i+l) in the sequence and generates a prediction v for another health condition. i+1 This process is repeated until the RNN architecture determines that the reasoning process is complete. Figure 5 In the example shown in , the prediction v'1 may indicate that the input image includes a mole, while the next prediction v'2 may indicate that the input image includes a rash.

[0044] In one embodiment, the RNN architecture is further configured to, at each iteration, in addition to the current image 510 (i) and the hidden state h of the previous iteration i-1 In addition, it also receives the memory vector m i , and generates a prediction vi of the corresponding health condition from the current image 510 (i). The memory vector m i can represent a vector that includes information about which health conditions have been classified until the current iteration i. For example, the memory vector m i It can be the total number of previous predictions or previous diagnoses up to the current iteration. Figure 5 In the example shown in , the memory vector m2 at the second iteration can indicate that a prediction or diagnosis has been made for a nevus based on v1 by the second iteration. By further configuring the RNN architecture to receive the memory vector, the diagnosis system 130 can construct a diagnosis model to generate a prediction for a health condition that has not been predicted in each iteration of the reasoning process.

[0045] During the inference process, by repeatedly applying the RNN architecture to the input image sequence, the diagnostic system 130 generates a corresponding prediction for each of the two or more health conditions. In one embodiment, when the RNN architecture is also configured to receive a memory vector, the diagnostic system 130 can obtain a corresponding memory vector for an iteration based on the predictions that have been generated up to the current iteration. By repeatedly applying the RNN architecture to the image sequence and the memory vector, the diagnostic system 130 generates a corresponding prediction for each of the two or more health conditions.

[0046] Figure 6An example reasoning process of a diagnostic model set including a discriminator model for amorphous and localized skin abnormalities according to an embodiment is illustrated. In one embodiment, the machine learning diagnostic model set includes a discriminator model, an amorphous abnormality model, and a localized abnormality model. The discriminator model is configured to receive an input image and generate an indication of whether the input image includes an amorphous skin abnormality or a localized skin abnormality by applying a first trained weight set. The amorphous abnormality model is configured to receive an input image and generate a prediction of whether the input image includes one or more amorphous skin abnormalities such as rashes and eczema by applying a second trained weight set. The localized abnormality model is configured to receive an input image and generate a prediction of whether the input image includes one or more localized skin abnormalities such as moles and acne by applying a third trained weight set.

[0047] Specifically, while amorphous skin abnormalities such as rashes, eczema, and dermatitis may appear on the skin without a definite shape or form, localized skin abnormalities such as nevi may be more clearly defined as localized spots on the skin. Therefore, due to the differences in shape and form, the diagnostic system 130 may obtain more accurate predictions by training two separate models, one of which generates predictions for amorphous skin abnormalities and the other for localized skin abnormalities.

[0048] During the inference process, by applying the discriminator model to the input image, the diagnostic system 130 generates an indication of whether the input image includes a localized or amorphous skin abnormality. Instead of generating a prediction that can be used to diagnose a specific type of skin abnormality (e.g., nevus, acne, rash), the output of the discriminator model can simply indicate whether the input image includes any skin abnormality that can be classified as amorphous and / or any skin abnormality that can be classified as localized. In response to determining that the input image includes an amorphous skin abnormality, by applying the amorphous abnormality model to the input image, the diagnostic system 130 generates a specific prediction of one or more amorphous skin abnormalities. In addition, in response to determining that the input image includes a localized skin abnormality, by applying the localized abnormality model to the input image, the diagnostic system 130 generates a specific prediction of one or more localized skin abnormalities.

[0049] Figure 7 An example inference process for a diagnostic model with non-maximum suppression according to an embodiment is illustrated. In one embodiment, a machine learning diagnostic model is configured to receive an input image and generate a vector including predictions for a list of health conditions. Figure 7 In the example shown in , the prediction v' generated by the diagnostic model may include one or more elements, each element corresponding to a prediction of a corresponding health condition. For example, the first element may correspond to a prediction that the input image includes eczema, the second element may correspond to a prediction that the input image includes dermatitis, and so on.

[0050] During the inference process, the diagnostic system 130 generates corresponding predictions for two or more health conditions by applying the diagnostic model to the input image. In one embodiment, the diagnostic system 130 determines the diagnosis by using non-maximum suppression. Specifically, the diagnostic system 130 groups the different health conditions represented by the vector v' and, for each group, selects only the health condition with the highest likelihood as part of the diagnosis. In one example, the health conditions are grouped according to how similar their symptoms appear visually on the patient, so that each group contains health conditions that the diagnostic model may have been confused by the image. Figure 7 In the example shown in , the health conditions for the first to fourth elements in the vector are assigned to "Group 1", while the health conditions for the n-3rd to nth elements are assigned to "Group m". By selecting the health conditions for the second element in "Group 1" and the n-1th element in "Group m" (because each element has the highest probability in each group) and setting the remaining elements to zero, the diagnosis system 130 determines the diagnosis v' nm .

[0051] Back to Figure 1 As shown in the system diagram of FIG. 1 , the diagnostic system 130 may also train a set of machine-learned indicator models that are configured to receive images of a patient and generate an indication of whether the input image presents two or more health conditions. Rather than proactively diagnosing a particular health condition itself, the diagnostic system 130 may generate an indication of whether the input image presents two or more health conditions and provide the result to a user of the client device 110. Specifically, instead of generating a prediction that can be used to diagnose a specific type of health condition, the output of the indicator model may simply determine whether the input image shows symptoms (e.g., overlapping symptoms) from two or more health conditions. Because input images presenting multiple health conditions may be difficult to diagnose, a user of the client device 110 may receive such an indication as a supplement or alternative to proactive diagnosis. The user may then determine to provide the input image to a medical specialist for a more accurate diagnosis.

[0052] Figure 8 An example reasoning process for an indicator model according to an embodiment is illustrated. In one embodiment, the diagnostic model trains a machine-learned indicator model that is configured to receive an input image and generate an indication e' of whether the input image presents two or more health conditions by applying a trained set of weights. Figure 8 In the example shown in , the diagnostic system 130 generates the indication e' by applying the indicator model to the input image. The diagnostic system 130 may simply provide information to the user of the client device 110 that the image includes too many health conditions for accurate diagnosis.

[0053] In another embodiment, instead of combining Figure 8 In addition to training a separate indicator model, the diagnostic system 130 also incorporates indicator functionality into the diagnostic model. For example, the diagnostic model can be configured to receive an input image and generate a prediction of one or more health conditions included in the image, and generate an indication of whether the input image presents two or more health conditions. Figures 2 to 7 The described diagnostic model set may also be configured to generate such an indication.As another example, the diagnostic model may be configured to generate a single output at a time, and may generate such an indication as a single output in response to receiving an input image showing two or more health conditions.

[0054] In one instance, the diagnostic model is configured to generate such an indication by classifying an input image presenting two or more health conditions into separate categories in response to receiving the input image and generating a likelihood for the separate category. In another instance, the diagnostic model is configured to generate such an indication when predictions for a health condition are output with relatively equal likelihoods or confidence levels to indicate ambiguity in the prediction. For example, if the predictions for having a pigmented nevus, a rash, and a skin disease are output as 0.32, 0.29, and 0.35, respectively, the diagnostic system 130 can determine that the input image includes symptoms from two or more health conditions.

[0055] In one embodiment, if the indicated likelihood is above a threshold likelihood, the diagnostic system 130 may determine that the input image presents two or more conditions, otherwise the input image presents a single health condition. In some embodiments, the indication may mark two possibilities: a first possibility as to whether the input image presents two or more health conditions, and a second possibility as to whether the input image presents a single health condition. If the first likelihood is above the threshold, the diagnostic system 130 may determine that the input image presents two or more conditions, and if the second likelihood is above the threshold, it may be determined that the input image presents a single health condition.

[0056] Thus, in one embodiment, the diagnostic system 130 may receive a request to diagnose a health condition in an input image. The diagnostic system 130 may access an indicator model or a diagnostic model with indicator functionality from a database. By applying the indicator model to the input image, the diagnostic system 130 may generate an indication representing the likelihood that the input image includes two or more health conditions. The diagnostic system 130 determines from the indication generated for the input image whether the input image presents two or more health conditions. The diagnostic system 130 generates a result of the request based on the determination and provides the result to the client device 110. The result may include a health care option for the individual in the input image based on the determination, and this suggested care option may be provided to the client device 110.

[0057] In one embodiment, the diagnostic system 130 may determine that the indication is ambiguous because the diagnostic model 130 cannot make a conclusive determination as to whether the input image presents two or more health conditions. For example, this may occur when both the first likelihood (e.g., the likelihood that the image presents two or more health conditions) and the second likelihood (e.g., the likelihood that the image presents a single health condition) are below a threshold (e.g., 80%). In such an embodiment, the diagnostic system 130 may output a result that no diagnosis can be made to the client device, and may, for example, suggest that the user of the client device 110 obtain medical expertise.

[0058] In some embodiments, in response to determining that the input image presents two or more skin abnormalities, the diagnostic system 130 may provide information about the indication to the client device 110 without providing a prediction of the two or more health conditions in the input image. For example, the diagnostic system 130 may review and confirm government regulations and policies (e.g., regulations issued by the Food and Drug Administration (FDA) or guidelines issued by the Centers for Disease Control (CDC), etc.), and generate results requiring compliance with such regulations. Therefore, even if the diagnostic system 130 can generate a prediction of two or more health conditions, the diagnostic system 130 may omit these results from the output to the client device 110 if the regulations impose restrictions on computerized diagnosis of the two or more health conditions presented in the image, for example. In such an embodiment, the diagnostic system 130 may simply provide the user with information that the input image contains two or more health conditions, but a diagnosis cannot be made due to government regulations.

[0059] In some embodiments, in response to determining that the input image presents a single health condition, the diagnostic system 130 can access a database and select a diagnostic model configured to generate a prediction of the single health condition. The diagnostic system 130 generates a prediction of the input image by applying the selected diagnostic model to the input image. The diagnostic system 130 generates a diagnosis of the single skin abnormality based on the prediction and provides the diagnosis as a result to the client device.

[0060] In some embodiments, in response to determining that the input image presents two or more health conditions, the diagnostic system 130 can access and select a set of diagnostic models configured to generate a corresponding prediction for each of the two or more health conditions from a database. For example, the diagnostic system 130 can access a combination of Figures 2 to 7 The described model is configured to generate predictions of two or more health conditions. The diagnostic system 130 generates predictions of two or more health conditions in the input image by applying the diagnostic model set to the input image. The diagnostic system 130 generates a diagnosis for the input image from the prediction and provides the diagnosis as a result to the client device 110.

[0061] Although combined Figures 2 to 8 The described machine learning models are configured to receive input data in the form of images, but these machine learning models can also be configured to receive additional types of input data. For example, the additional types of input data can include biographical data of the patient, such as the patient's height or weight, the patient's medical history, or the patient's answers to questions requesting information about existing medical conditions known to the patient. For example, the patient's medical history may indicate that the patient is prone to certain health conditions that may affect the model's prediction results.

[0062] Return to Figure 1 In the system diagram of FIG. 1 , the client device 110 provides an image of the patient's anatomical part to the diagnostic system 130 so that the diagnostic system 130 can generate and show a prediction or other indication on the client device 110. The user of the client device 110 may be a medical professional in a hospital who wishes to use the computerized diagnostic system to assist in diagnosing a patient's health condition. As another example, the user of the client device 110 may be an individual at home or in the office who wishes to obtain a diagnosis of a potential health condition based on symptoms present on the individual's body.

[0063] In one embodiment, the client device 110 includes a browser that allows a user of the client device 110 to interact with the diagnostic system 130 using standard Internet protocols. In another embodiment, the client device 110 includes a dedicated application designed specifically (e.g., by the organization responsible for the diagnostic system 130) to enable interaction between the client device 110 and the server. In one embodiment, the client device 110 includes a user interface that allows a user of the client device 110 to interact with the diagnostic system 130 to view predictions of health conditions in an input image. For example, the user interface can be configured to overlay predictions generated by the diagnostic system 130 on locations where the image is estimated to have a corresponding health condition.

[0064] Client device 110 may be a computing device, such as a or , a tablet computer, a laptop computer, a desktop computer, or any other type of network-enabled device that includes or can be configured to connect to a camera. In another embodiment, the client device 110 is a headset that includes a computing device or a smartphone camera for generating an augmented reality (AR) environment for the user, or a headset that includes a computing device for generating a virtual reality (VR) environment for the user. A typical client device 110 includes the hardware and software required to connect to the network 122 (e.g., via WiFi and / or 4G or 5G or other wireless telecommunications standards).

[0065] The network 122 provides the communication infrastructure between the client device 110 and the diagnostic system 130. The network 122 is typically the Internet, but can be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network.

[0066] Diagnostic system architecture

[0067] Fig. 9 is a block diagram of the architecture of the diagnostic system 130 according to an embodiment. Fig. 9 The illustrated diagnostic system 130 includes a training module 920, a data management module 925, a training module 330, a prediction module 930, and a processed output module 935. The diagnostic system 130 also includes a training data repository 960 and a model data repository 965. In alternative configurations, different and / or additional components may be included in the diagnostic system 130.

[0068] The data management module 920 obtains and manages the training data stored in the training data repository 960. The training data can be used to train the combined Figures 2 to 8The training data repository 960 includes information extracted from images or videos of individual anatomical parts and known diagnoses of health conditions presented in these images. For example, the training data repository 960 may include information extracted from medical images, such as radiological images, computed tomography (CT) scans, medical resonance imaging (MRI) scans, X-ray images, ultrasound or sonographic images, tactile images, or thermal imaging images, which are obtained by the data management module 920 from hospitals, medical image repositories, research image repositories, etc. As another example, the training data repository 960 may include information extracted from images taken by individuals, such as images obtained on a smartphone camera, which are obtained by the data management module 920 from users of the client device 110.

[0069] The training module 925 uses the information in the training data repository 960 to train the model set. Specifically, for a given diagnostic model M i , training data S i Including multiple training instances j1, 2, ..., |S i |, each of which includes the input data x j∈S (e.g., training images or videos) and labels y j∈Si , which are the input data x j∈S Specifically, the input data X j This may include information extracted from training images capturing individual anatomical parts. In one embodiment, when the machine learning model is configured to receive additional types of data in addition to or in lieu of input images, the input data x j∈S Additional types of data for the patients in the training images may also be included (e.g., demographic information for the patients). j∈Si It can be obtained by a human operator, such as a doctor, medical expert, or other individual who can verify the characteristics of the input data. In another example, the label itself can be a prediction generated by a machine learning model that is subsequently verified by a human operator.

[0070] The training module 925 trains the weight set for the model by repeatedly iterating between the forward propagation step and the back propagation step. During the forward propagation step, the training module 920 applies the model with the estimated weight set to the input data x across the corresponding subset Si of the training data. j∈S To generate the estimated output y' j∈S The training module 920 determines a loss function, which indicates the estimated output y' j∈S With the labels y of multiple training instances j∈SDuring the back propagation step, the training module 920 repeatedly updates the weight set of the model by back propagating the error term obtained from the loss function. This process is repeated until the change in the loss function meets a predetermined criterion. For example, if the change in the loss function is below a threshold at each iteration, the criterion can be triggered.

[0071] In one embodiment, the loss function can be given by:

[0072]

[0073] For example, when the label y j is a continuous value, where θ p is a set of weights used in the diagnostic model. In another embodiment, for example, when the label y j When is a binary value, the loss function can be given by:

[0074]

[0075] However, it should be appreciated that the loss function may be any other function, such as L1-norm, L-infinity norm, which indicates the difference between the actual labels and the estimated outputs generated during each training iteration.

[0076] As described below, the training module 925 trains the combination Figures 2 to 7 The diagnostic models described herein can be used to determine an active diagnosis for an input image. In one embodiment, one or more diagnostic models are configured to generate a prediction y'=v' as an output vector, where each element in the output vector v' corresponds to a prediction for a corresponding health condition. In such an embodiment, the training labels y for the diagnostic models are j =v j Can be encoded as a one-hot encoded vector, where each element in the vector is a binary non-zero value (e.g., a value of 1) if the training image is diagnosed with the corresponding health condition for the element, and zero if there is no diagnosis. Therefore, for training images presenting two or more health conditions, each corresponding element can be set to an appropriate value so that two or more elements in the vector have non-zero values. For example, training data for a diagnostic model configured to predict rashes and nevi may include a label "[0 1]", where the first element indicates the presence of a rash in the corresponding training image and the second element indicates the absence of a nevus in the training image. In another embodiment, the diagnostic model can be configured to generate a prediction y'=v' as an output value, which can be assigned to one or more categories, each category representing a corresponding health condition. In this embodiment, the label y for the diagnostic model j =v jThe value of may label the category of the corresponding health condition diagnosed in the training image. For example, the training data of the example diagnostic model may include a label "1" indicating the presence of a rash in the training image, or a label "2" indicating the presence of a mole in the training image.

[0077] Training module 920 training combination Figure 3 The invention relates to an integrated set of diagnostic models described herein. Specifically, each diagnostic model in the integrated set can be configured to generate a prediction for a corresponding set of one or more health conditions. The diagnostic models in the integrated set can differ from each other in terms of the model architecture, the set of health conditions that the model is configured to predict, etc. In one embodiment, each diagnostic model is configured as a neural network architecture, which includes a set of node layers, each layer connected to the previous layer via a set of weights. Therefore, the diagnostic models in such an integrated set may differ from each other in terms of the number of layers, nodes, and connections within the architecture.

[0078] The training module 925 uses the training data set to train the weight set of the diagnostic model in the ensemble, the training set including the input data x j∈Si The training images and corresponding labels y j∈S =v j∈S , label y j∈S =v j∈S Indicates the presence of a specific health condition that the diagnostic model is configured to predict. During the forward propagation step, the training module 925 generates an estimated output y' by applying the diagnostic model to the training images. j∈S =v' j∈S During the back-propagation step, the training module 925 repeatedly updates the set of weights for the diagnostic model using the terms obtained from the loss function.

[0079] Training module 920 training combination Figure 4 The weight sets for the feature extractor model and the feature classifier model described above. In one embodiment, the feature extractor model and the feature classifier model are trained by training a neural network architecture including a first part and a second part. The weight set for the first part of the neural network architecture can be stored as the weight set of the feature extractor model, and the weight set for the second part can be stored as the weight set of the feature classifier model. Subsequently, the weight set for the feature classifier model is retrained using multiple components extracted from the training image.

[0080] Fig.10 The diagram shows a method according to an embodiment of the present invention. Figure 4. The training module 925 trains a neural network architecture 1036 comprising a set of node layers, each layer connected to the previous layer via a set of weights. The neural network architecture 1036 can be configured to generate a prediction for a corresponding set of one or more health conditions. In one embodiment, the neural network architecture 1036 is configured as a deep neural network (DNN), a convolutional neural network (CNN), or any other type of neural network architecture that can receive image data and generate an output.

[0081] The training module 925 trains a set of weights for a neural network architecture using a training data set, the training data set including as input data x j∈S The training images and corresponding labels y j∈S =v j∈S , the label indicates the presence of a specific health condition that the feature classifier model is configured to predict. During the forward propagation step, the training module 925 generates an estimated output y' by applying the neural network architecture to the training images j∈S =v' j∈S During the back-propagation step, the training module 925 uses the terms obtained from the loss function to repeatedly update the set of weights for the neural network architecture.

[0082] After completing the training process of the neural network architecture, the training module 925 stores the first portion of the neural network architecture as a weight set for the feature extractor model and stores the second portion of the neural network architecture as a weight set for the feature classifier model. In one embodiment, the first portion and the second portion are selected so that the second portion includes weight sets for layers subsequent to the layers of the first portion of the neural network architecture. For example, the first portion may include weight sets for the first three layers of the neural network architecture, while the second portion may include weight sets for the last five layers of the neural network architecture.

[0083] The training module 925 then uses the training data set to retrain the weight set for the feature classifier model, the training data set including as input data x j∈Si The multiple components CF extracted from the training images j∈S and the corresponding label y j∈S =v j∈S , the label indicates that the feature classifier model is configured to predict the presence of a specific health condition in the training image. In one example, multiple components for the training image are extracted by applying the trained feature extractor model to the training image to generate an image tensor for the training image and performing ICA on the image tensor to generate multiple components. During the forward propagation step, the training module 925 extracts multiple components for the training image by applying the feature classifier model to the multiple components CF j∈S To generate the estimated output y' j∈S =v' j∈SDuring the back-propagation step, the training module 925 repeatedly updates the set of weights for the feature classifier model using the terms obtained from the loss function.

[0084] Training Module 925 Training Combination Figure 5 The training module 925 uses a training data set to train the weight set for the RNN architecture. The training data set includes as input data x j∈S The training image sequence and the corresponding label y j∈S =v j∈S , the label indicates the presence of a specific health condition that the RNN architecture is configured to predict. In one embodiment, when the training data for the RNN architecture includes a training image sequence presenting two or more health conditions, the label v j At each iteration of the sequence, y can be a one-hot encoded vector with non-zero values ​​at the corresponding health conditions present in the training images. During the forward propagation step, the training module 925 generates an estimated output y' by repeatedly applying the RNN architecture to the training image sequence. j∈S =v' j∈S In one embodiment, the tag v j can be reordered during the forward propagation step of the training process so that the differences, such as |v j∈S -v' j∈S |, can be reduced or minimized over all possible orderings or a subset thereof. The training module 925 determines a loss function by combining the differences between the estimated output and the label at each iteration of the sequence. During the back-propagation step, the training module 925 repeatedly updates the set of weights for the RNN architecture using the terms obtained from the loss function.

[0085] Training Module 925 Training Combination Figure 6 The training module 925 uses a training data set to train a weight set for the discriminator model. The training data set includes as input data x j∈S The training images and corresponding labels y j∈S =s j∈S , the label indicates the presence of an amorphous skin abnormality or the presence of a localized skin abnormality. In one example, the discriminator model is configured to generate the indication as a vector in which the first element corresponds to the likelihood that the image includes an amorphous skin abnormality and the second element corresponds to the likelihood that the image includes a localized skin abnormality. In this example, if the training image includes an amorphous skin abnormality, then the label y j∈S can be encoded so that the first element of the vector is a non-zero value (e.g., the value 1), or if the image does not include an amorphous skin abnormality, the label y j∈Scan be encoded such that the first element of the vector is zero. Similarly, if the training image includes a local skin abnormality, the second element of the vector is a non-zero value (e.g., a value of 1), or if the image does not include a local skin abnormality, the second element of the vector is zero. During the forward propagation step, the training module 925 generates an estimated output y' by applying the discriminator model to the training image. j∈S =s' j∈S , and during the back-propagation step the set of weights for the discriminator model is repeatedly updated with the terms obtained from the loss function.

[0086] The training module 925 uses a training data set to train a weight set for the amorphous anomaly model. The training data set includes as input data x j∈Si The training images and corresponding labels y j∈S =v j∈S , the label indicates the presence of a specific amorphous skin abnormality that the model is configured to predict. The training data for the amorphous abnormality model may include training images with a single amorphous skin abnormality or multiple amorphous skin abnormalities. During the forward propagation step, the training module 925 generates an estimated output y' by applying the amorphous abnormality model to the training images. j∈S =v' j∈D During the back-propagation step, the training module 925 repeatedly updates the set of weights for the amorphous anomaly model with the terms obtained from the loss function.

[0087] The training module 925 uses a training data set to train a weight set for a local anomaly model. The training data set includes as input data x j∈S The training images and corresponding labels y j∈S =v j∈S , the label indicates the presence of a specific local skin abnormality that the model is configured to predict. The training data for the local abnormality model may include training images with a single local skin abnormality or multiple local skin abnormalities. During the forward propagation step, the training module 925 generates an estimated output y' by applying the local abnormality model to the training images j∈S =v' j∈S During the back-propagation step, the training module 925 repeatedly updates the set of weights for the local anomaly model using the terms obtained from the loss function.

[0088] As further described below, the training module 925 further trains the combination Figure 8 The indicator model described above provides an indication of whether an image presents two or more health conditions. The training module 925 trains a set of weights for the indicator model using a training data set, the training data set including as input data x j∈Straining images of and corresponding labels y indicating whether the training images present two or more health conditions j∈S =e j∈S The training data may include training images with symptoms from a single health condition or multiple health conditions. In one example, the indicator model is configured to generate an indication indicating the likelihood that the image presents two or more health conditions. In such an example, if the training image presents two or more health conditions, then the label y j∈Si can be encoded as a non-zero value, or if the training image presents a single health condition, the label y j∈Si can be encoded as zero. During the forward propagation step, the training module 925 generates an estimated output y' by applying the indicator model to the training images j∈S =e' j∈S During the back-propagation step, the training module 925 repeatedly updates the set of weights for the indicator model with the terms obtained from the loss function.

[0089] The training module 925 can also train a diagnostic model in conjunction with the indicator functionality. The diagnostic model can be any model configured to generate a computerized prediction of one or more health conditions in the image, such as Figures 2 to 7 , or a diagnostic model configured to predict a single health condition. While diagnostic training data (e.g., training images and corresponding health condition labels) can be used to train diagnostic models in general, training module 925 further trains diagnostic models to incorporate indicator functionality by also including indicative training data for the diagnostic models in addition to the diagnostic training data.

[0090] In one embodiment, the indicative training data S a The training images including two or more health conditions and corresponding labels y indicating the presence of the two or more health conditions in the training images j∈Sa =v j∈Sa In one example, when the labels y used to diagnose the training data j∈S =v j∈S The label y for the indicative training data is a one-hot encoded vector j∈Sa =v j∈Sa It can be a vector in which all or most of the elements in the vector have non-zero values. For example, if the training images present two or more health conditions, the training data for a diagnostic model configured to predict a single health condition can include a label "

[11] " that includes non-zero values ​​for all elements. In this way, the diagnostic model is configured to generate an indication by predicting relatively similar or equal likelihoods for all or most of the health conditions.

[0091] In another example, for the label y of the indicative training dataj∈Sa =v j∈Sa Additional elements or categories can be included to designate images presenting two or more health conditions. For example, when the labels y of the diagnostic training data j∈S =v j∈S When is a one-hot encoded vector, in addition to elements for different health conditions, the label can be further configured to include a separate element for indication. If the training image presents two or more health conditions, the separate element is a non-zero value, and if the training image does not present two or more health conditions, the separate element is zero. For example, training data for a diagnostic model configured to predict rashes and / or nevi may include the label "

[111] ", where the first element indicates the presence of a rash in the corresponding training image, the second element indicates the presence of a nevus in the training image, and the third separate element indicates the presence of two or more health conditions in the training image. As another example, when the label y for diagnosis is j∈S =v j∈S When the labels are categories, the labels may be further configured to include separate categories for indications in addition to categories for different health conditions. For example, the training data for the example diagnostic model may include a label "1" indicating the presence of a rash in the training image, a label "2" indicating the presence of a mole in the training image, and a label "3" indicating the presence of two or more health conditions in the training image.

[0092] The training module 925 can store the trained machine learning models in the model data repository 965 so that they can be used during the inference process in combination with Figures 2 to 8 Deploy as described.

[0093] The prediction module 930 receives a request from the client device 110 to provide a computerized diagnosis of a health condition captured in an input image. In response to the request for a diagnosis, the prediction module 930 may select a diagnostic model set that may be used to generate a diagnosis for the input image using a diagnostic model trained by the diagnostic system 130. The prediction module 930 may apply the selected diagnostic model to generate a prediction of the health condition presented in the input image, similar to the combination of Figures 2 to 8 The example reasoning process described. In one example, if the predicted likelihood is above a threshold, the prediction module 930 determines a diagnosis of the health condition. For example, if the predicted likelihood is above 0.80 or 80%, the prediction module 930 can conclude that the corresponding health condition is definitely present in the input image. Otherwise, the prediction module 930 can conclude that the corresponding health condition is not present in the input image.

[0094] In some embodiments, the prediction module 930 determines that the prediction of the corresponding health condition is uncertain, and may generate information that a diagnosis request cannot be made to the client device 110, or alternatively, may output diagnoses for the remaining health conditions to the client device 110, which have a decisive prediction. For example, if the corresponding prediction has a probability between a first and a second threshold (e.g., 20%-80%), the prediction module 930 may determine that the diagnosis is uncertain. In another example, if the predicted probability is within a threshold ratio of other predicted probabilities generated by the diagnostic model, the prediction module 930 determines the diagnosis of the health condition.

[0095] In one embodiment, the prediction module 930 is programmed to comply with relevant government regulations or policies governing computerized diagnosis, and generates diagnostic results to comply with current regulations and policies. For example, regulations and policies may specify various prediction thresholds for determining whether a corresponding health condition is present in an input image, and these thresholds may vary between different health conditions. In one instance, an administrator of the diagnostic system 130 may review these regulations and policies and program the prediction module 930 to comply with them. In another instance, the prediction module 930 may collect information from a website or database of a relevant organization (e.g., FDA) and generate predictions based on rules parsed from the collected information. Therefore, the prediction module 930 may review the predictions generated by the model and generate diagnostic results for the request to comply with these regulations and policies. For example, according to the policy established by the FDA, if the predicted probability exceeds a threshold of 80%, the prediction module 930 may determine a diagnosis of acne, and only when the predicted probability exceeds a threshold of 95% may the prediction module 930 determine a diagnosis of rash.

[0096] In addition, in response to a request for a diagnosis, the prediction module 930 can select an indicator model or a diagnostic model with indicator functionality to generate an indication for the input image. The prediction module 930 can apply the selected model to generate the indication. In one example, if the likelihood of the indication is above a threshold, the prediction module 930 determines that the input image presents two or more health conditions. In another example, if the likelihood of the indication is within a threshold ratio of other outputs generated by the selected model, the prediction module 930 determines that the input image presents two or more health conditions. Figure 8 As described, the prediction module 930 can generate a result for the request based on the determination so that an appropriate diagnostic model can be selected to generate a diagnosis, or information that a diagnosis cannot be made can be output to the user of the client device 110, for example due to compliance with government regulations and policies regarding computerized diagnosis.

[0097] The treatment output module 935 provides potential treatment options in conjunction with the diagnosis provided by the prediction module 930. Given the diagnosis, it is advantageous for the user of the client device 110 to receive potential treatment options along with the diagnosis so that they can treat the health condition in a short period of time when needed. In one embodiment, the treatment output module 935 can generate treatment options by consulting human medical experts, reasoning with expert systems, or deriving options from previous systems. In one embodiment, the treatment output module 935 can suggest known treatment options to reduce and avoid side effects with respect to the health condition presented by the patient.

[0098] Approach to the diagnosis of overlapping skin abnormalities

[0099] Fig.11 A method of diagnosing overlapping skin abnormalities in an input image is illustrated. The diagnostic system 130 (e.g., via the network 120) receives 1102 a request from a client device to diagnose skin abnormalities in an input image. The input image includes overlapping skin abnormalities on the patient's skin. The diagnostic system 130 (e.g., using the prediction module 930 via the network 120) retrieves from a database (e.g., a model data repository 965, wherein the model 965 is trained using the training data 960 and may include a combination of Figures 2 to 8 The diagnostic model 130 accesses 1104 a set of machine learning models (models described in the examples). Each machine learning model includes a corresponding set of trained weights. The diagnostic model 130 performs the following steps by applying the set of machine learning models to the input image (e.g., in combination with Figures 2 to 8 The inference process described in the foregoing description) generates 1106 (e.g., using the prediction model 930) a corresponding prediction for each of the two or more skin abnormalities in the input image. The prediction indicates a likelihood that a corresponding skin abnormality in the two or more skin abnormalities is shown in the input image. The diagnostic system 130 generates 1108 a diagnosis for the overlapping skin abnormalities from the prediction (e.g., using the prediction module 930). The diagnostic system 130 provides 1110 the diagnosis to the client device (e.g., via the network 120).

[0100] Method of generating an indication of two or more skin abnormalities

[0101] Fig.12 The method of generating an indication of two or more skin abnormalities in an input image is illustrated. The diagnostic system 130 receives 1202 a request from a client device (e.g., via the network 120) to diagnose a skin abnormality in the input image. The diagnostic system 130 receives 1203 a request from a client device (e.g., via the network 120) to diagnose a skin abnormality in the input image. The diagnostic system 130 receives 1204 a request from a database (e.g., a model data repository 965, wherein the model 965 is trained using the training data 960 and may include a combination of Figure 8The machine learning model includes a corresponding set of trained weights. The diagnostic model 130 performs the following steps by applying the machine learning model to the input image (e.g., in combination with the prediction module 930). Figure 8 The diagnostic system 130 generates 1206 an indication (e.g., using the prediction module 930) based on determining the overlapping skin abnormalities from the predictions (e.g., using the prediction module 930). The diagnostic system 130 provides 1210 the result to the client device (e.g., via the network 120).

[0102] Summarize

[0103] The above description of the embodiments of the present invention is for illustrative purposes; it is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0104] Certain parts of this specification sheet describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are often used by those skilled in the art of data processing to effectively communicate the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. Moreover, without loss of generality, it is sometimes proven to be convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0105] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented by a computer program product including a computer readable medium, the computer readable medium containing computer program code, and the computer program code can be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0106] Embodiments of the present invention may also be directed to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-temporary, tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0107] Embodiments of the present invention may also relate to products produced by the computing process described herein. Such products may include information generated from the computing process, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0108] Finally, the language used in the specification is selected primarily for readability and didactic purposes, and may not be selected to delineate or limit the subject matter of the present invention. Therefore, the scope of the present invention is not intended to be limited by the specific embodiments, but rather by any claims issued based on the application herein. Therefore, the disclosure of the embodiments of the present invention is intended to illustrate, but not to limit, the scope of the present invention, which is set forth in the following claims.

Claims

1. A method for diagnosing overlapping skin abnormalities in an input image, the method comprising: receiving, from a client device, a request to diagnose a skin abnormality in an input image, the input image including an overlaid skin abnormality on the patient's skin; Accessing a set of machine learning models from a database, each machine learning model comprising a corresponding set of trained weights, wherein the corresponding set of trained weights is trained by a training module using a training data set comprising training images and corresponding labels, and the corresponding labels indicate the presence of overlapping skin abnormalities; generating a corresponding prediction for each of two or more skin abnormalities in the input image by applying the set of machine learning models to the input image, the prediction indicating a likelihood that a corresponding skin abnormality of the two or more skin abnormalities is shown in the input image; generating a diagnosis of the overlapping skin abnormality from the prediction of the input image; as well as The diagnostics are provided to the client device.

2. The method of claim 1, wherein the set of machine learning models is an integrated set of neural network models, and wherein generating the predictions for the two or more skin abnormalities further comprises: for each neural network model in the ensemble, generating one or more predictions from the neural network model by applying the neural network model to the input image, and The predictions for the ensemble of neural network models are combined to generate the predictions for the two or more skin abnormalities.

3. The method of claim 1, wherein the set of machine learning models comprises a second machine learning model and a third machine learning model, and wherein generating the predictions for the two or more skin abnormalities further comprises: generating an image tensor by applying the second machine learning model to the input image, the image tensor representing a plurality of spatial features in the input image, extracting a plurality of components from the image tensor, generating a corresponding tensor for each of the two or more skin abnormalities, and The prediction of the two or more skin abnormalities is generated by applying the third machine learning model to the corresponding tensor for each of the two or more skin abnormalities. 4 . The method of claim 3 , wherein the plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor.

5. The method of claim 3, wherein the trained weight sets for the second machine learning model and the third machine learning model are trained jointly.

6. The method of claim 1, wherein the set of machine learning models comprises a recurrent neural network model, and wherein generating the predictions for the two or more skin abnormalities further comprises: Repeatedly applying the recurrent neural network model to the input image to generate a corresponding prediction for a first skin abnormality among the two or more skin abnormalities at a first time, and to generate a corresponding prediction for a second skin abnormality among the two or more skin abnormalities at a second time after the first time.

7. The method of claim 1, wherein the set of machine learning models comprises a second machine learning model, a third machine learning model, and a fourth machine learning model, and wherein generating the predictions for the two or more skin abnormalities further comprises: generating a prediction of whether the input image includes an amorphous skin abnormality or a localized abnormality; as well as In response to determining that the input image includes an amorphous skin abnormality, generating a prediction of the amorphous skin abnormality by applying the amorphous abnormality model to the input image; In response to determining that the input image includes a local skin abnormality, a prediction of the local skin abnormality is generated by applying the local abnormality model to the input image.

8. The method of claim 1 , wherein at least one machine learning model in the set of machine learning models is configured as a neural network architecture comprising a set of node layers, each layer being connected to a previous layer via a subset of weights.

9. The method of claim 1, wherein the input image is one of: a radiological image, a computed tomography (CT) scan, a medical resonance imaging (MRI) scan, an X-ray image, an ultrasound or sonography image, a tactile image, or a thermographic image.

10. The method of claim 1, wherein the input image is an image captured by a user of the client device, wherein the client device is a smartphone.

11. A computer program product for diagnosing overlapping skin abnormalities in an input image, the computer program product comprising a computer readable storage medium containing computer program code for: receiving, from a client device, a request to diagnose a skin abnormality in an input image, the input image including an overlaid skin abnormality on the patient's skin; Accessing a set of machine learning models from a database, each machine learning model comprising a corresponding set of trained weights, wherein the corresponding set of trained weights is trained by a training module using a training data set comprising training images and corresponding labels, and the corresponding labels indicate the presence of overlapping skin abnormalities; generating a corresponding prediction for each of two or more skin abnormalities in the input image by applying the set of machine learning models to the input image, the prediction indicating a likelihood that a corresponding skin abnormality of the two or more skin abnormalities is shown in the input image; generating a diagnosis of the overlapping skin abnormality from the prediction of the input image; as well as The diagnostics are provided to the client device.

12. The computer program product of claim 11, wherein the set of machine learning models is an integrated set of neural network models, and wherein generating the predictions for the two or more skin abnormalities further comprises: for each neural network model in the ensemble, generating one or more predictions from the neural network model by applying the neural network model to the input image, and The predictions for the ensemble of neural network models are combined to generate the predictions for the two or more skin abnormalities.

13. The computer program product of claim 11, wherein the set of machine learning models comprises a second machine learning model and a third machine learning model, and wherein generating the predictions for the two or more skin abnormalities further comprises: generating an image tensor by applying the second machine learning model to the input image, the image tensor representing a plurality of spatial features in the input image, extracting a plurality of components from the image tensor, generating a corresponding tensor for each of the two or more skin abnormalities, and The prediction of the two or more skin abnormalities is generated by applying the third machine learning model to the corresponding tensor for each of the two or more skin abnormalities.

14. The computer program product of claim 13, wherein the plurality of components are extracted from the image tensor by performing independent component analysis (ICA) on the image tensor.

15. The computer program product of claim 13, wherein the sets of trained weights for the second machine learning model and the third machine learning model are trained jointly.

16. The computer program product of claim 11, wherein the set of machine learning models comprises a recurrent neural network model, and wherein generating the predictions for the two or more skin abnormalities further comprises: Repeatedly applying the recurrent neural network model to the input image to generate a corresponding prediction for a first skin abnormality among the two or more skin abnormalities at a first time, and to generate a corresponding prediction for a second skin abnormality among the two or more skin abnormalities at a second time after the first time.

17. The computer program product of claim 11, wherein the set of machine learning models comprises a second machine learning model, a third machine learning model, and a fourth machine learning model, and wherein generating the predictions for the two or more skin abnormalities further comprises: generating a prediction of whether the input image includes an amorphous skin abnormality or a localized abnormality; as well as In response to determining that the input image includes an amorphous skin abnormality, generating a prediction of the amorphous skin abnormality by applying the amorphous abnormality model to the input image; In response to determining that the input image includes the local skin abnormality, a prediction of the local skin abnormality is generated by applying the local abnormality model to the input image.

18. The computer program product of claim 11, wherein at least one machine learning model in the set of machine learning models is configured as a neural network architecture comprising a set of node layers, each layer connected to a previous layer via a subset of weights.

19. The computer program product of claim 11, wherein the input image is one of: a radiological image, a computed tomography (CT) scan, a medical resonance imaging (MRI) scan, an X-ray image, an ultrasound or sonography image, a tactile image, or a thermographic image.

20. The computer program product of claim 11, wherein the input image is an image captured by a user of the client device, wherein the client device is a smartphone.

21. A method of generating an indication of two or more skin abnormalities in an input image, the method comprising: receiving, from a client device, a request to diagnose a skin abnormality present in an input image; accessing a machine learning model from a database, the machine learning model comprising a corresponding set of trained weights, wherein the corresponding set of trained weights is trained by a training module using a training data set comprising training images and corresponding labels, and the corresponding labels indicate the presence of overlapping skin abnormalities; generating an indication by applying the machine learning model to the input image, the indication representing a likelihood that the input image shows two or more skin abnormalities; generating, from the indication generated for the input image, a determination whether the input image presents two or more skin abnormalities; as well as A result for the request is generated based on the determination, and the result is provided to the client device.

22. The method according to claim 21, further comprising: A care option for the input image is determined based on the determination, and the care option is provided to the client device as the result.

23. The method according to claim 21, further comprising: responsive to determining that the input image presents one skin abnormality but not two or more skin abnormalities, accessing from the database a diagnostic model configured to generate a prediction of a single skin abnormality; generating a prediction of the skin abnormality in the input image by applying the diagnostic model to the input image; as well as A diagnosis for the input image is generated from the prediction, and the diagnosis is provided as the result to the client device.

24. The method according to claim 21, further comprising: In response to determining that the input image presents two or more skin abnormalities, accessing a set of diagnostic models from the database, the set of diagnostic models configured to generate a respective prediction for each of the two or more skin abnormalities; generating predictions of the two or more skin abnormalities in the input image by applying the set of diagnostic models to the input image; as well as A diagnosis for the input image is generated from the prediction, and the diagnosis is provided as the result to the client device.

25. The method of claim 21, further comprising: In response to determining that the input image exhibits two or more skin abnormalities, providing information to the client device that the input image cannot be diagnosed.

26. The method of claim 21, further comprising: In response to determining that the input image exhibits two or more skin abnormalities, providing information about the indication to the client device without providing a prediction of the two or more skin abnormalities in the input image.

27. The method of claim 21, wherein the machine learning model is also a diagnostic model, the diagnostic model further configured to receive an image and generate a prediction of one or more skin abnormalities present in the image.

28. The method according to claim 27, The generating of the instruction further comprises: applying the diagnostic model to the input image to generate an output vector, the output vector comprising a set of elements, the prediction of the one or more skin abnormalities being represented by the values ​​of the elements in the output vector, and Wherein when the difference between the values ​​of the set of elements in the output vector is below a predetermined threshold, the input image is determined to present two or more skin abnormalities.

29. The method of claim 27, wherein generating the indication further comprises: The diagnostic model is applied to the input image to generate an output vector comprising a set of elements, wherein the prediction of the one or more skin abnormalities is represented by values ​​of a subset of the elements in the output vector, and wherein the indication is represented by values ​​of the remaining elements in the output vector.

30. The method according to claim 27, wherein said prediction of said one or more skin abnormalities is represented by one or more categories in a set of categories, The generating of the instruction further comprises: applying the diagnostic model to the input image to generate output values, and Wherein when the output values ​​are assigned to the remaining categories in the set of categories, the input image is determined to present two or more skin abnormalities.

31. A computer program product for generating an indication of two or more skin abnormalities in an input image, the computer program product comprising a computer readable storage medium containing computer program code for: receiving a request from a client device to diagnose a skin abnormality in an input image; accessing a machine learning model from a database, the machine learning model comprising a corresponding set of trained weights, wherein the corresponding set of trained weights is trained by a training module using a training data set comprising training images and corresponding labels, and the corresponding labels indicate the presence of overlapping skin abnormalities; generating an indication by applying the machine learning model to the input image, the indication representing a likelihood that the input image includes two or more skin abnormalities; generating, from the indication generated for the input image, a determination whether the input image presents two or more skin abnormalities; as well as A result for the request is generated based on the determination, and the result is provided to the client device.

32. The computer program product of claim 31, wherein the computer-readable storage medium further comprises computer program code for determining a care option for the person in the input image based on the determination and providing the care option as the result to the client device.

33. The computer program product of claim 31 , wherein the computer readable storage medium further comprises computer program code for: responsive to determining that the input image presents one skin abnormality but not two or more skin abnormalities, accessing from the database a diagnostic model configured to generate a prediction of a single skin abnormality; generating a prediction of the skin abnormality in the input image by applying the diagnostic model to the input image; and A diagnosis for the input image is generated from the prediction, and the diagnosis is provided as the result to the client device.

34. The computer program product of claim 31 , wherein the computer readable storage medium further comprises computer program code for: In response to determining that the input image presents two or more skin abnormalities, accessing a set of diagnostic models from the database, the set of diagnostic models configured to generate a respective prediction for each of the two or more skin abnormalities; generating predictions of the two or more skin abnormalities in the input image by applying the set of diagnostic models to the input image; as well as A diagnosis for the input image is generated from the prediction, and the diagnosis is provided as the result to the client device.

35. The computer program product of claim 31, wherein the computer-readable storage medium further comprises computer program code for: in response to determining that the input image presents two or more skin abnormalities, providing information to the client device that the input image cannot be diagnosed.

36. A computer program product according to claim 31, wherein the computer-readable storage medium further comprises computer program code for: in response to determining that the input image presents two or more skin abnormalities, providing information about the indication to the client device without providing a prediction of the two or more skin abnormalities in the input image.

37. The computer program product of claim 31 , wherein the machine learning model is also a diagnostic model, the diagnostic model further configured to receive an image and generate a prediction of one or more skin abnormalities present in the image.

38. The computer program product according to claim 37, The generating of the instruction further comprises: applying the diagnostic model to the input image to generate an output vector comprising a set of elements, the prediction of the one or more skin abnormalities being represented by the values ​​of the elements in the output vector, and Wherein when the difference between the values ​​of the set of elements in the output vector is below a predetermined threshold, the input image is determined to present two or more skin abnormalities.

39. The computer program product of claim 37, wherein generating the indication further comprises: The diagnostic model is applied to the input image to generate an output vector comprising a set of elements, wherein the prediction of the one or more skin abnormalities is represented by values ​​of a subset of the elements in the output vector, and wherein the indication is represented by values ​​of the remaining elements in the output vector.

40. The computer program product according to claim 37, wherein said prediction of said one or more skin abnormalities is represented by one or more categories in a set of categories, The generating of the instruction further comprises: applying the diagnostic model to the input image to generate output values, and Wherein when the output values ​​are assigned to the remaining categories in the set of categories, the input image is determined to present two or more skin abnormalities.

Images