Using set of machine learning diagnostic models to determine diagnosis based on skin tone of patient

Abstract

To provide use of a set of machine learning diagnostic models to determine a diagnosis based on a suitable skin tone of a patient.SOLUTION: Systems and methods are disclosed herein for determining a diagnosis based on a base skin tone of a patient. In an embodiment, the system receives a base skin tone image of a patient, generates a calibrated base skin tone image by calibrating the base skin tone image using a reference calibration profile, and determines a base skin tone of the patient based on the calibrated base skin tone image. The system receives a concerned image of a portion of the patient's skin, and selects a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the base skin tone of the patient, each of the sets of candidate machine learning diagnostic models is trained to receive the concerned image and output a diagnosis of a condition of the patient.SELECTED DRAWING: Figure 5

Description

[Background technology]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 862,844, filed June 18, 2019, which is incorporated by reference in its entirety.

[0002] Skin pigmentation, also referred to herein as skin tone or skin color, is highly variable across and within populations. Skin color is important from a clinical perspective because the incidence, prevalence, clinical manifestations, and treatment of skin diseases can vary based on skin type. However, bias against non-white skin types in today's dermatology practice and research significantly impacts the quality of treatment. The manifestations of skin diseases vary with skin tone and can significantly affect survival rates. As an example, skin cancers such as melanoma, the most deadly form of skin cancer, are diagnosed in people of color (i.e., non-white) at a later, less treatable stage, in part due to their manifestation in non-UV-exposed areas of the body, such as hands, nails, legs, and mucous membranes, which are not commonly examined. Thus, although a higher incidence of melanoma exists in white people, a higher mortality rate exists in skin of color.

[0003] One of the best known classification systems is the Fitzpatrick scale, which categorizes skin color into six skin types (I-VI) based on a self-reported questionnaire. Skin types I-IV were initially created in France in 1975 to categorize Caucasian skin based on clinical response to ultraviolet (UV) radiation, while types V (brown Asian and Latin American) and VI (dark African skin) were added only later to capture non-Caucasian skin tones based on constitutional pigmentation or ethnic origin. The Fitzpatrick scale is commonly used within skin cancer screening, however, reviews of the tool have noted errors associated with self-reporting and limited applicability to all skin types given the changing basis of ethnic origin, making the step of clustering an ethnic / racial group into one skin type difficult.

[0004] The use of systems such as the Fitzpatrick scale is deficient in terms of their ability to accurately define skin color, which in turn leads to inaccurate diagnosis or misdiagnosis of skin conditions through existing processes. Summary of the Invention [Means for solving the problem]

[0005] The present disclosure relates to an automated system and method for objectively analyzing and classifying skin tone using machine learning. Such analysis and classification may be used to enhance skin analysis, diagnosis, and treatment. In an embodiment, the system receives a basal skin tone image of a patient. The system generates a calibrated basal skin tone image by calibrating the basal skin tone image using a reference calibration profile. The system determines the basal skin tone of the patient based on the calibrated basal skin tone image. The system may then receive an image of concern of a portion of the patient's skin and may select a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the skin tone measurements of the patient, where each of the set of candidate machine learning diagnostic models may be trained to receive the image of concern and output a diagnosis of the patient's medical condition. The present invention provides, for example, the following: (Item 1) 1. A method for determining a diagnosis based on a patient's underlying skin tone, the method comprising: receiving a baseline skin tone image of a patient; generating a calibrated baseline skin tone image by calibrating said baseline skin tone image using a reference calibration profile; determining a baseline skin tone of the patient based on the calibrated baseline skin tone image; receiving an image of concern of a portion of the patient's skin; selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's underlying skin tone, each of the set of candidate machine learning diagnostic models being trained to receive the image of concern and to output a diagnosis of the patient's medical condition; A method comprising: (Item 2) 2. The method of claim 1, wherein generating the calibrated skin tone image is performed in response to determining that the baseline skin tone image meets a quality standard. (Item 3) determining a baseline skin tone of the patient based on the calibrated image, determining a numerical representation of each pixel of the calibrated baseline skin tone image; performing a statistical operation on each of the numerical representations to generate an aggregate representation; identifying a point in color space that corresponds to the aggregate representation; mapping the points in the color space to discrete values ​​in a classification system; The method according to item 1, comprising: (Item 4) The method further includes generating a plurality of calibrated baseline skin tone images, the plurality including the calibrated baseline skin tone images, and determining the patient's baseline skin tone based on the calibrated images further includes: inputting each of the plurality of calibrated baseline skin tone images into a measurement classifier; receiving the baseline skin tone as an output from the measurement classifier; The method according to item 1, comprising: (Item 5) The method further comprises: generating an adapted imaging profile based on the reference calibration profile and the baseline skin tone; calibrating the image of concern using the adapted imaging profile; and The method according to item 1, comprising: (Item 6) The method further includes determining a classification corresponding to the measured skin tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method further includes: inputting the image of concern into the selected diagnostic classifier; receiving, as output from the selected diagnostic classifier, information from which the diagnosis is derived; The method according to item 1, comprising: (Item 7) The method further comprises: inputting the baseline skin tone and the image of concern into at least one classifier of a selected set of the machine learning diagnostic models; receiving, as output from said at least one classifier, information from which said diagnosis is derived; The method according to item 1, comprising: (Item 8) Determining the method further comprises: inputting the image of concern into at least one classifier of a selected set of the machine learning diagnostic models; receiving, as output from the at least one classifier, information from which the diagnosis is derived; modifying the information based on the underlying skin tone; and determining the diagnosis based on the revised information; and The method according to item 1, comprising: (Item 9) The method further comprises: determining whether the image of concern is diagnosable based on the underlying skin tone; in response to determining that the image of concern is not diagnosable, outputting an indication that the image of concern cannot be diagnosed. The method according to item 1, comprising: (Item 10) 2. The method of claim 1, wherein the diagnosis of the condition comprises a probability that the patient has the condition. (Item 11) 1. A computer program product for determining a diagnosis based on a patient's underlying skin tone, the computer program product comprising: a computer readable storage medium; receiving a baseline skin tone image of a patient; generating a calibrated baseline skin tone image by calibrating said baseline skin tone image using a reference calibration profile; determining a baseline skin tone of the patient based on the calibrated baseline skin tone image; receiving an image of concern of a portion of the patient's skin; selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's underlying skin tone, each of the set of candidate machine learning diagnostic models being trained to receive the image of concern and to output a diagnosis of the patient's medical condition; A computer program product comprising computer program code for performing the steps of: (Item 12) Item 12. The computer program product of item 11, wherein generating the calibrated skin tone image is performed in response to determining that the baseline skin tone image meets a quality standard. (Item 13) determining a baseline skin tone of the patient based on the calibrated image, determining a numerical representation of each pixel of the calibrated baseline skin tone image; performing a statistical operation on each of the numerical representations to generate an aggregate representation; identifying a point in color space that corresponds to the aggregate representation; mapping points in said color space to discrete values ​​in a classification system; Item 12. The computer program product of item 11, comprising: (Item 14) The method further includes generating a plurality of calibrated baseline skin tone images, the plurality including the calibrated baseline skin tone images, and determining the patient's baseline skin tone based on the calibrated images further includes: inputting each of the plurality of calibrated baseline skin tone images into a measurement classifier; receiving the baseline skin tone as an output from the measurement classifier; Item 12. The computer program product of item 11, comprising: (Item 15) The method further includes determining a classification corresponding to the measured skin tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method further includes: inputting the image of concern into the selected diagnostic classifier; receiving, as output from the selected diagnostic classifier, information from which the diagnosis is derived; Item 12. The computer program product of item 11, comprising: (Item 16) The method further comprises: inputting the baseline skin tone and the image of concern into at least one classifier of a selected set of the machine learning diagnostic models; receiving, as output from said at least one classifier, information from which said diagnosis is derived; Item 12. The computer program product of item 11, comprising: (Item 17) Determining the method further comprises: inputting the image of concern into at least one classifier of a selected set of the machine learning diagnostic models; receiving, as output from the at least one classifier, information from which the diagnosis is derived; modifying the information based on the underlying skin tone; and determining the diagnosis based on the revised information; and Item 12. The computer program product of item 11, comprising: (Item 18) 12. The computer program product of claim 11, wherein the diagnosis of the medical condition comprises a probability that the patient has the medical condition. (Item 19) 1. A method for determining a diagnosis of a patient, the method comprising: Receiving biometric data of at least a portion of a patient; inputting the biometric data into a classifier trained to obtain attributes of the patient; receiving attributes of the patient as output from the classifier; selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on attributes of the patient, each of the set of candidate machine learning diagnostic models being trained to receive the attributes and to output a diagnosis of the medical condition of the patient; A method comprising: (Item 20) 20. The method of claim 19, wherein each of the set of candidate machine learning diagnostic models is trained to receive, in addition to the attributes, an image of concern depicting an organ of the patient, and to output a diagnosis of the patient's medical condition based on both the attributes and the image of concern. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an exemplary block diagram of system components in an environment for utilizing a skin image calibration tool, according to one embodiment.

[0007] [Diagram 2] FIG. 2 is an exemplary block diagram of modules and components of a skin image calibration tool, according to one embodiment.

[0008] [Diagram 3] FIG. 3 is an exemplary block diagram of modules and components of a treatment selection tool, according to one embodiment.

[0009] [Figure 4] FIG. 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).

[0010] [Diagram 5] FIG. 5 depicts an example image showing a baseline skin tone image and a skin concern image, according to one embodiment.

[0011] [Figure 6] FIG. 6 depicts an exemplary flowchart for determining a diagnosis based on a patient's underlying skin tone, according to one embodiment.

[0012] The figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the inventive principles described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (a) Environmental Overview FIG. 1 is an exemplary block diagram of system components in an environment for utilizing a skin image calibration tool, according to one embodiment. The environment 100 includes an imaging device 110, a network 120, a skin image calibration tool 130, a treatment selection tool 140, and a disease diagnosis tool 150. The imaging device 110 may be any device configured to capture an image of an affected area of ​​a patient. An example that pervades this disclosure is skin image calibration and skin disease diagnosis, however, the imaging, calibration, and diagnostic aspects disclosed herein may be applied to any other organ of the human body (e.g., the retina). The imaging device 110 may be specifically configured to capture an image of the patient's skin. Alternatively, the imaging device 110 may be a general client device, such as a smartphone, laptop, tablet, or other camera-equipped computing device, on which software (e.g., applications) for processing images in the manner disclosed herein is installed. In one embodiment, the imaging device 110 may be a general client device that captures images and transmits the images to a remote server (e.g., the skin image calibration tool 130), where image processing and calibration are performed by the remote server.

[0014] Network 120 may be any communications network, such as a local area network, a wide area network, the Internet, and the like. Alternatively, or in addition, network 120 may represent on-device activity (such as in a scenario where some or all of the functionality of skin image calibration tool 130, treatment selection tool 140, and / or disease diagnosis tool 150 are installed integral to imaging device 110).

[0015] The skin image calibration tool 130 receives a patient's basal skin tone image and determines the patient's basal skin tone from the basal skin tone image. As used herein, the term "basal skin tone image" may refer to a captured image that meets certain criteria, or an image captured based on instructions that the image should meet certain criteria. For example, the basal skin tone image is an image that should, in an embodiment, show healthy, untanned skin. As used herein, the term "basic skin tone" may refer to values ​​that represent the pigmentation (or tone, as used synonymously herein) of the patient's skin. Further details regarding the mechanics of how the skin image calibration tool 130 determines basal skin tone are described in more detail below with respect to Figures 2-6. Although depicted in Figure 1 as being an inverse network 120, the skin image calibration tool 130 may be installed partially or entirely on the imaging device 110 and / or in a device local to the imaging device 110.

[0016] Treatment selection tool 140 receives input (e.g., determined underlying skin tone) and outputs a treatment decision. For example, based on underlying skin tone determined by skin image calibration tool 130, treatment selection tool 140 may select a machine learning model that is best optimized to diagnose the image of concern. Disease diagnosis tool 150 performs a diagnosis based on its input. For example, disease diagnosis tool 150 may receive an image of concern as an input and output a diagnosis. Further details of how treatment selection tool 140 and disease diagnosis tool 150 perform these and other operations are disclosed in further detail below with respect to Figures 3-6. Wherever a classifier or machine learning diagnostic model is described herein, performance of the steps of inputting data into such a model and obtaining an output from the model may be performed by disease diagnosis tool 150. Like skin image calibration tool 130, which is depicted in Figure 1 as being in network 120 in opposition to each of the other elements depicted in Figure 1, treatment selection tool 140 and / or disease diagnosis tool 150 may be installed partially or wholly on imaging device 110 and / or in a device local to imaging device 110. The functionality of skin image calibration tool 130, treatment selection tool 140, and disease diagnosis tool 150 may be integrated onto a single server or a group of servers and / or distributed across many servers.

[0017] 2 is an exemplary block diagram of modules and components of a skin image calibration tool, according to one embodiment. As depicted in FIG. 2, skin image calibration tool 130 includes various modules, such as image capture module 210, basal skin tone determination module 220, and adapted imaging profile generation module 230. Skin image calibration tool 130 is also depicted as including various databases, such as reference calibration profile 250, skin tone images 260, and basal skin tone classifier 270. The depicted modules and databases are merely exemplary, and more or fewer modules and / or databases may be used to achieve the functionality disclosed herein.

[0018] The image capture module 210 captures a baseline skin tone image of the patient's skin for further processing. To capture an image, the image capture module 210 first obtains image data from the camera of the imaging device 110. The obtained image data may not be fully composed and thus not yet synthesized into a human-perceivable image. This raw image data often represents the photon intensity received per pixel in a Bayer pattern, which in effect requires further post-processing such as Bayer de-processing and color correction to be transformed into a calibrated image. The image capture module 210 may apply a reference calibration profile (e.g., from the reference calibration profile 250 or as read from the memory of the imaging device 110) to the obtained image data to capture the baseline skin tone image. The imaging device 110 may be calibrated to be within a set tolerance, including, but not limited to, colorimetric accuracy (conventionally described in terms of Delta E, which measures the change in visual perception of two given colors) and optical properties such as relative lens position. This calibration may be implemented in any number of ways, including, for example, camera settings (such as exposure or channel gain), color transformation profiles (such as an ICC profile), and / or transformations based on spatially related points (such as which pixels lie within the camera's field of view). These calibrations may be performed by the factory when producing imaging device 110 and / or may be commanded by imaging device 110. The resulting calibrations of imaging device 110 form a reference calibration profile.

[0019] In an embodiment, the image capture module 210 may cause the output of instructions (e.g., displayed and / or audio instructions) to be provided to an operator of the imaging device 110 prior to the step of obtaining the image data. The operator may be any human, such as a medical assistant, a minimally trained operator, or a patient. The instructions may include instructions to take a picture of skin having one or more predefined characteristics, such as skin on the inside of the wrist, healthy skin, untanned skin, skin that is consistent in tone without any artifacts such as lentigines, present skin, etc. The image capture module 210 may perform processing on the basal skin tone image to verify whether the photographed skin conforms to the predefined characteristics and whether the image itself is of sufficient quality. For example, pattern recognition may be performed, or the basal skin tone image may be input into a classifier that is trained to output an indication of whether the basal skin tone image conforms to the instructions.

[0020] The image capture module 210 may capture multiple basal skin tone images. The image capture module 210 may prompt the operator to capture several basal skin tone images. For example, a patient may not have a uniform skin tone across the patient's body, and thus multiple skin tone images may be used to obtain a more robust data set for determining the patient's basal skin tone. The image capture module 210 may store the basal skin tone images in a skin tone image database 260. The skin tone image database 260 may store basal skin tone images that are associated with the patient's electronic medical record, which includes other information about the patient, including biographic information, demographic information, and any other information that describes the patient.

[0021] The basal skin tone determination module 220 determines a basal skin tone for the patient based on one or more basal skin tone images. The basal skin tone determination module 220 may determine the basal skin tone in any of a variety of manners. In an embodiment, the basal skin tone determination module 220 may determine a numerical representation of each pixel of the calibrated basal skin tone image. For example, each color of each pixel may correspond to a numerical value on a scale. The basal skin tone determination module 220 may generate an aggregate representation by performing a statistical operation based on each numerical representation. For example, the basal skin tone determination module 220 may average the numerical representations. If multiple basal skin tone images are used, the basal skin tone determination module 220 may repeat this activity for each basal skin tone image and then perform a statistical operation on each of the aggregate representations to result in an aggregate representation for the entire set of basal skin tone images. The basal skin tone determination module 220 may then use scaling to map the aggregate representation to a point in color space and assign this point as the basal skin tone. Alternatively, rather than immediately assigning the point as the basis skin tone, the basis skin tone determination module 220 may map the point in color space to a discrete value in a classification system, such as the Fitzpatrick system or any other classification system. While this embodiment of basis skin tone determination may be sufficient in many cases, it may not take into account inconsistencies in pigmentation within a reference area, such as under- or hyper-pigmented skin patches, and so other measures may be implemented.

[0022] In an embodiment, the basal skin tone determination module 220 may input the basal skin tone images into the basal skin tone classifier 270. The basal skin tone classifier 270 may be a machine learning model that is trained to generate discrete values ​​from a set of images. For example, to train the basal skin tone classifier 270, an expert evaluator may assign a value to each image in the training set. The indicator may indicate the patient's precise skin tone. In an embodiment, the indicator may additionally or alternatively indicate an attribute of the skin tone (e.g., whether the skin is hypopigmented or hyperpigmented). Thus, the basal skin tone classifier 270 may output the patient's basal skin tone.

[0023] Alternatively or additionally, the basal skin tone classifier 270 may output an indicated type of skin (e.g., hypo- or hyperpigmented skin, or normal skin). The basal skin tone determination module 220 may determine based on the output of the basal skin tone classifier 270 that the aforementioned aggregate representation method is sufficient in response to receiving an output from the classifier that the skin is normal. In response to receiving an output from the classifier that the skin is of a certain type (e.g., hypo- or hyperpigmented), the basal skin tone determination module 220 may determine to use the measurements output by the basal skin tone classifier 270 as the basal skin tone. Exemplary classifier algorithms for the basal skin tone classifier 270 may include convolutional neural networks, support vector machines on extracted features, and XGBoost on extracted features. As another example, a segmentation algorithm may be trained to separate healthy, untanned skin from discolored skin, and then either the basic approach described above or a different classifier may be used on only the healthy patches of skin to determine the underlying skin tone. Exemplary segmentation algorithms include fully convolutional neural networks, Otsu's method, and color variance thresholding.

[0024] The adapted imaging profile generation module 230 generates an adapted imaging profile. As used herein, the term "adapted imaging profile" may refer to imaging parameters optimized to obtain skin concern images that are most useful for diagnosis versus skin concern images that would be obtained using only the reference calibration profile. The adapted imaging profile generation module 230 may generate the adapted imaging profile in several ways. In an embodiment, the adapted imaging profile generation module 230 may determine a set of predefined imaging parameters stored in the reference calibration profile database 250 based on a data structure such as a mapping table as a mapping to the determined baseline skin tone. In another embodiment, instead of conforming the imaging parameters to a data structure such as a mapping table, the adapted imaging profile generation module 230 will use the baseline skin tone and adjust the imaging parameters as a difference from the baseline value using a numerical representation of the skin tone. An example of this may include increasing the exposure in proportion to the darkness of the patient's skin to maximize detail. A more complex embodiment may involve interpolating between the values ​​of the known profiles to choose an intermediate value for skin tone that lies between the ranges of the two explicitly listed profiles. The adapted imaging profile generation module 230 may store the adapted imaging profile in the reference calibration profile database 250.

[0025] 3 is an exemplary block diagram of modules and components of a treatment selection tool, according to one embodiment. As depicted in FIG. 3, the treatment selection tool 140 includes various modules, such as an image of concern processing module 310, a compatibility module 320, and a classifier selection module 330. The treatment selection tool 140 also includes various databases, such as a diagnostic model inventory 350 and patient information 360. The modules and databases depicted with respect to FIG. 3 are merely exemplary, and more or fewer modules and / or databases may be used to achieve the functionality of the treatment selection tool 140 disclosed herein.

[0026] The image of concern processing module 310 may receive the skin image of concern and calibrate the skin image of concern using the adapted imaging profile. Calibration of the skin image of concern occurs in the same manner as the calibration of the baseline skin tone image, except that the adapted imaging profile is used instead of the reference calibration profile.

[0027] The suitability module 320 determines whether the skin concern image is suitable for diagnosis. Suitability may be based on any predefined criteria. For example, a regulatory body may indicate that some classifications (e.g., Fitzpatrick system classifications 1-3) are suitable for diagnosis using the systems and methods disclosed herein, while other classifications are not suitable for diagnosis using the systems and methods disclosed herein (e.g., darker skin tones may suffer higher error rates in the step of diagnosing whether the lentigines are benign). As another example, the skin concern image may fail a quality parameter that must be satisfied for processing (e.g., the skin concern image is overexposed, underexposed, does not show enough affected skin area, or any other predefined parameter). Additional exemplary validity criteria are not including color channel saturation across the image (channels that are at maximum values ​​and therefore not capable of providing accurate measurements would indicate failure), or passing a machine learning based quality checker. Image quality issues can be screened using a convolutional neural network classifier that is trained on examples of images with sufficient quality and images with poor quality. Additionally, image quality issues can be screened using features extracted from the images, which may include measurements of high frequency content of the image and color histograms.

[0028] In response to determining that the skin concern image is not suitable for diagnosis, the compatibility module 320 may output a prompt to the operator indicating that the skin concern image is not suitable for diagnosis. In embodiments where the skin concern image is not suitable for diagnosis due to quality issues, the compatibility module 320 may prompt the operator to capture another skin concern image that meets the quality parameters. Once the recaptured image is received, in an embodiment, the compatibility module 320 may instruct that the reference calibration profile is used to generate a recaptured skin concern image. In embodiments where the recaptured image still does not meet the quality parameters, the compatibility module 320 may prompt the operator to manually check the image for errors, manually check for signs of camera malfunction, and / or to recapture the skin concern image again (which may trigger regeneration of the adapted imaging profile using the recaptured baseline skin tone image, and the regenerated adapted imaging profile may be used to calibrate further skin concern images for the patient). In embodiments where skin concern images are not suitable due to patient characteristics, the compatibility module 320 may prompt the operator to instruct the patient that a diagnosis cannot be completed. Additional information may be prompted to the operator as well (e.g., instructions for the patient to obtain a diagnosis from a physician).

[0029] The classifier selection module 330 selects one or more classifiers, such as a set of machine learning diagnostic models, based on the patient's underlying skin tone, which are selected classifiers to be used to establish the patient's treatment. The set may include one or more machine learning models, as well as heuristics that together form a workflow for the patient's treatment. Multiple classifiers may be stored in the diagnostic model inventory 350. Each of these classifiers may be associated with a criterion that, if met, indicates to the classifier selection module 330 that the classifier should be used for the patient's diagnosis. The classifier selection module 330 may select a classifier based on information additional to the underlying skin tone, the additional information being retrieved from the patient information database 360. The additional information may include any information in the patient's electronic patient record, such as biographical information, demographic information, and / or any other information that describes the patient.

[0030] In an embodiment, the classifier selection module 330 may perform activities other than selecting one or more particular classifiers, or may perform activities related to obtaining further inputs for a particular classifier. For example, the classifier selection module 330 may determine that the underlying skin tone is a high-risk skin tone (e.g., as indicated in a database that maps the underlying skin tone to a risk category). For example, the underlying skin tone may correspond to a very light-skinned person having a higher incidence of skin cancer. In response, the classifier selection module 330 may prompt for input of answers to questions related to informing treatment (e.g., the patient may be prompted to answer questions about the patient's sun exposure behavior). As another example, the classifier selection module 330 may prompt for further skin concern images to be captured (e.g., palms and soles of the feet, where the patient has darker skin tones that make them more susceptible to developing dangerous growths in those locations).

[0031] In an embodiment, the classifier selection module 330 may have several classifiers available in the diagnostic model inventory 350, each available classifier corresponding to a different range of skin tones. The classifier selection module 330 may select a classifier based on the patient's underlying skin tone. The selected classifier may be trained to be optimized for its corresponding range of skin tones and may be less effective or ineffective if the patient has an underlying skin tone outside of that range. The selected classifier may receive as input the image of concern, with or without other information such as underlying skin tone and / or additional patient information, and may output a diagnosis.

[0032] In one embodiment, the classifier selection module 330 is not used as a gatekeeper for selecting a classifier. Instead, the treatment selection tool 140 inputs the underlying skin tone and image of concern (possibly along with additional patient information) into the classifier and outputs a diagnosis. As another exemplary embodiment in this spirit, a classifier may be used that receives an image of skin concern as input and outputs a probability of a diagnosis. The treatment selection tool 140 then weights the probability based on the underlying skin tone of the patient.

[0033] If the classifier outputs a diagnosis, the diagnosis may be a direct diagnosis (e.g., the image of concern indicates the absence of disease or a specific identification of a disease that is present). The diagnosis may alternatively be a number of probabilities, each probability corresponding to a different candidate diagnosis, and the probabilities indicating the likelihood that the disease corresponds to each candidate diagnosis. The classifier selection module 330 may output the probabilities directly to the operator. In an embodiment, the classifier selection module 330 may cause the candidate diagnosis with the highest probability to be displayed to the operator. The classifier selection module 330 may compare each probability to a threshold and may limit the candidate diagnoses displayed to the operator to those candidate diagnoses whose corresponding probability exceeds the threshold. Each different candidate diagnosis may have a different corresponding threshold that must be crossed for the candidate diagnosis to be displayed to the operator.

[0034] The treatment selection tool 140 may determine that at least two diseases cannot be ruled out as being depicted by the skin concern image based on two or more candidate diagnoses having probabilities exceeding a threshold output by the classifier. Thus, the treatment selection tool 140 may prompt the operator to instruct the patient to obtain a professional opinion (e.g., from a physician). The treatment selection tool 140 may withhold two or more candidate diagnoses or may have two or more candidate diagnoses presented.

[0035] While skin is a primary example, treatment selection tool 140 may apply treatment selection based on any patient attribute. In an embodiment, a model may be trained to classify patients based on any medical condition (e.g., the patient's skin tone, weight, sex, jaundice, diabetes, and / or any other medical condition). Based on the output of the model, treatment selection tool 140 may determine a further model to be used to determine a treatment for the patient (e.g., a refined model trained to determine whether a skin condition associated with diabetes is present in the patient).

[0036] In an embodiment, the treatment selection tool 140 may receive biometric data corresponding to the patient. The biometric data may include visual images, such as those discussed herein, and any other type of images (whether using a spectrum, such as a camera lens, or an ultrasound machine, capturing internal organs, etc.), and may be images in non-visible wavelengths (e.g., infrared, ultraviolet, x-ray, etc.). The biometric data may be any other data about the patient, or a combination of different types of data. Exemplary biometric data includes measurements derived from blood tests, cardiac activity (e.g., as measured by an electrocardiogram), pulse data, and any other data that describes the biometric activity of the patient. The treatment selection tool 140 may input the biometric data into a classifier, which is trained to obtain attributes of the patient. For example, blood test data may be input into the classifier, and the classifier may output whether the blood test indicates that the patient has one or more medical conditions, such as a disease (e.g., diabetes).

[0037] The treatment selection tool 140 may select a set of machine learning models from a plurality of sets of candidate machine learning diagnostic models based on the attributes of the patient, where each set of candidate machine learning diagnostic models is trained to receive the attributes and output a diagnosis of the patient's medical condition. For example, a set of machine learning models corresponding to a particular disease, such as diabetes, may include machine learning models that receive other inputs (e.g., retinal images, toe images) in addition to the disease itself and obtain outputs related to the disease (e.g., a determination of whether the patient has diabetic retinopathy based on the retinal images, a determination of whether the patient has a toe pathology associated with urinary disease, etc.). The operator may be prompted to instruct the patient to obtain additional information about the patient or to obtain other inputs by visiting another operator (e.g., a physician) and determining the additional information that may inform the selected set of machine learning models.

[0038] 4 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them in a processor (or controller). In particular, FIG. 4 shows a diagrammatic representation of a machine in the exemplary form of a computer system 400 in which program code (e.g., software) for causing the machine to perform any one or more of the methodologies discussed herein may be executed. The program code may consist of instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as an independent device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a server machine of a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0039] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions 124 (sequential or otherwise) that specify actions to be taken by the machine. Additionally, although only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that, individually or together, execute the instructions 124 and perform any one or more of the methodologies discussed herein.

[0040] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include software drivers that allow a user interface to be displayed on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, or the like. For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with a touch-enabled screen. Computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touch screen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a storage unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420, also configured to communicate over bus 408.

[0041] The storage unit 416 includes a machine-readable medium 422 on which are stored instructions 424 (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions 424 (e.g., software) may also reside, completely or at least partially, within the main memory 404 or within the processor 402 (e.g., within a processor's cache memory) during its execution by the computer system 400, and the main memory 404 and the processor 402 may also constitute machine-readable media. The instructions 424 (e.g., software) may be transmitted or received over a network 426 via the network interface device 420.

[0042] While the machine-readable medium 422 is shown in the exemplary embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" should also be taken to include any medium capable of storing instructions (e.g., instructions 424) for execution by a machine, causing the machine to perform any one or more of the methodologies disclosed herein. The term "machine-readable medium" includes data repositories in the form of, but not limited to, solid-state memories, optical media, and magnetic media.

[0043] 5 depicts an example image showing a baseline skin tone image and a skin concern image, according to one embodiment. The baseline skin tone image 510 is shown to be captured by an operator, not the patient, as depicted. The imaging device used is shown to be a smartphone camera. The imaging device is shown to display the patient's baseline skin tone.

[0044] A skin concern image 520 is shown being captured by an operator. The skin concern image 520 includes an image of a mole. The skin concern image 520 is calibrated using a fitted reference profile, which when input into a classifier for diagnosis, results in a skin concern image that optimizes the accuracy of the diagnosis outcome.

[0045] 6 depicts an exemplary flowchart for determining a diagnosis based on a patient's underlying skin tone, according to one embodiment. Process 600 may begin with skin image calibration tool 130 receiving a underlying skin tone image of the patient (602). The underlying skin tone image may be received by image capture module 210, which may be executed by processor 402. The underlying skin tone image may be underlying skin tone image 510. Skin image calibration tool 130 generates a calibrated skin tone image (604) (e.g., using image capture module 210) by calibrating the underlying skin tone image using a reference calibration profile (e.g., as retrieved from reference calibration profile database 250).

[0046] The skin image calibration tool 130 determines 606 the patient's underlying skin tone based on the calibrated underlying skin tone image (e.g., using the underlying skin tone determination module 220). The treatment selection tool 140 receives 608 the image of concern of the portion of the patient's skin (e.g., as depicted in the image of concern 520). The treatment selection tool 140 selects 610 a set of machine learning diagnostic models from the plurality of sets of candidate machine learning diagnostic models based on the patient's underlying skin tone (e.g., using the classifier selection module 330), where each of the set of candidate machine learning models is trained to receive the image of concern and output a diagnosis of the patient's medical condition.

[0047] (overview) The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will appreciate that numerous modifications and variations are possible in light of the above disclosure.

[0048] Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or theoretically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. In addition, it has also proven convenient at times, without loss of generality, to refer to arrangements of these operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0049] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code that may be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0050] Embodiments of the present invention may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design for increased computing power.

[0051] Embodiments of the invention may also relate to products produced by the computing processes described herein. Such products may comprise information resulting from the computing processes, which information may be stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of the computer program product or other data combination described herein.

[0052] Finally, the language used herein has been selected primarily for readability and instructional purposes, and may not have been selected to bound or delimit the subject matter of the invention. Accordingly, the scope of the invention is intended to be limited not by this detailed description, but rather by any claims issuing on an application based thereon. Thus, the disclosure of embodiments of the invention is intended to illustrate, but not limit, the scope of the invention, which is set forth in the following claims.

Claims

[Claim 1] The invention described in this specification.

Images