Use of a set of machine learning diagnostic models for determining a diagnosis based on a patient's skin color tone

Abstract

Disclosed herein are systems and methods for determining a diagnosis based on a patient's underlying skin tone. In one embodiment, the system receives a patient's underlying skin tone image, calibrates the underlying skin tone image using a reference calibration profile to generate a calibrated underlying skin tone image, and determines the patient's underlying skin tone based on the calibrated underlying skin tone image. The system receives an image of concern of a portion of the patient's skin, and selects a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's underlying skin tone, each of the set of candidate machine learning diagnostic models trained to receive the image of concern and output a diagnosis of the patient's medical condition.

Description

Background Art

[0001] (Cross - reference to related applications) This application claims the benefit of U.S. Provisional Application No. 62 / 862,844, filed on Jun. 18, 2019, which is incorporated herein by reference in its entirety.

[0002] Skin pigmentation, also referred to herein as skin tone or skin color, varies widely across and within populations. Skin color is clinically important because the incidence, prevalence, clinical findings, and treatment of skin diseases can vary based on skin type. However, bias against non - white skin types in today's dermatological practice and research significantly affects the quality of treatment. Skin disease findings vary by skin tone and can significantly affect survival rates. As an example, skin cancers such as melanoma, the most lethal form of skin cancer, are, in part, diagnosed in people of color (i.e., non - white) at a later, less treatable stage due to their findings in non - UV - exposed areas of the body such as the hands, nails, feet, and mucous membranes, which are generally not examined. Thus, while a higher incidence of melanoma exists in the white population, a higher mortality rate exists in people with colored skin.

[0003] One of the most well-known classification systems is the Fitzpatrick scale, which categorizes skin color into six skin types (I–VI) based on a self-report questionnaire. Skin types I–IV were first generated in France in 1975 to categorize Caucasian skin based on clinical responses to ultraviolet (UV) radiation, while types V (brown Asian and Latin American) and VI (dark brown African skin) were added later to capture non-Caucasian skin tones based on constitutive pigmentation or ethnic origin. The Fitzpatrick scale is commonly used within skin cancer examinations; however, reviews of the tool have noted errors associated with self-reporting and limited adaptability to all skin types based on a changing ethnic origin foundation, which poses difficulties in clustering a given ethnic / racial group into one skin type.

[0004] The use of systems such as the Fitzpatrick scale is inadequate in terms of their ability to accurately define skin color. This, in turn, results in an imprecise diagnosis or misdiagnosis of skin conditions through existing processes. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0005] The present disclosure relates to an automated system and method for objectively analyzing and classifying skin tones using machine learning. Such analysis and classification can be used to enhance the analysis, diagnosis, and treatment of the skin. In certain embodiments, the system receives a baseline skin tone image of a patient. The system generates a calibrated baseline skin tone image by calibrating the baseline skin tone image using a reference calibration profile. The system determines the patient's baseline skin tone based on the calibrated baseline skin tone image. The system may then receive a suspect image of a portion of the patient's skin and select a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's skin tone measurements, where each set of candidate machine learning diagnostic models is trained to receive the suspect image and output a diagnosis of the patient's medical condition. The present invention provides, for example, the following. (Item 1) A method for determining a diagnosis based on a patient's baseline skin tone, the method comprising: receiving a baseline skin tone image of a patient; generating a calibrated baseline skin tone image by calibrating the baseline skin tone image using a reference calibration profile; determining the patient's baseline skin tone based on the calibrated baseline skin tone image; receiving a suspect image of a portion of the patient's skin; selecting, based on the patient's baseline skin tone, a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models, wherein each set of candidate machine learning diagnostic models is trained to receive the suspect image and output a diagnosis of the patient's medical condition; comprising. (Item 2) The method according to item 1, wherein generating the calibrated skin tone image is performed in response to determining that the baseline skin tone image meets a quality criterion. (Item 3) Determining the patient's baseline skin color tone based on the calibrated image comprises: Determining a numerical representation of each pixel of the calibrated baseline skin color image; Generating an aggregated representation by performing a statistical operation based on each numerical representation; Identifying a point in the color space corresponding to the aggregated representation; Mapping the point in the color space to a discrete value in a classification system The method according to item 1, comprising. (Item 4) The method further comprises generating a plurality of calibrated baseline skin color images, the plurality including the calibrated baseline skin color image, and determining the patient's baseline skin color tone based on the calibrated image further comprises: Inputting each of the plurality of calibrated baseline skin color images into a measurement classifier; Receiving the baseline skin color tone as an output from the measurement classifier The method according to item 1, comprising. (Item 5) The method further comprises: Generating a calibrated imaging profile based on the reference calibration profile and the baseline skin color tone; Calibrating the image of concern using the calibrated imaging profile The method according to item 1, comprising. (Item 6) The method further comprises determining a classification corresponding to the measured skin color tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method further comprises: Inputting the image of concern into the selected diagnostic classifier; Receiving, as an output from the selected diagnostic classifier, information from which the diagnosis is derived The method according to item 1, comprising. (Item 7) The method further comprises: Inputting the basic skin tone and the image of concern into at least one classifier of the selected set of the machine learning diagnostic model and receiving, as an output from the at least one classifier, information from which the diagnosis is then derived The method according to item 1, comprising the above steps. (Item 8) Determining the method further comprises inputting the image of concern into at least one classifier of the selected set of the machine learning diagnostic model receiving, as an output from the at least one classifier, information from which the diagnosis is then derived modifying the information based on the basic skin tone determining the diagnosis based on the modified information The method according to item 1, comprising the above steps. (Item 9) The method further comprises determining whether the image of concern is diagnosable based on the basic skin tone outputting an indication that the image of concern cannot be diagnosed in response to determining that the image of concern is not diagnosable The method according to item 1, comprising the above steps. (Item 10) The diagnosis of the medical condition according to item 1 includes the probability that the patient has the medical condition. (Item 11) A computer program product for determining a diagnosis based on a patient's basic skin tone, the computer program product comprising a computer-readable storage medium, the computer-readable storage medium comprising receiving an image of the patient's basic skin tone generating a calibrated basic skin tone image by calibrating the basic skin tone image using a reference calibration profile determining the patient's basic skin tone based on the calibrated basic skin tone image​ Receiving a suspicious image of a part of the skin of the patient; Selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the basic skin color tone of the patient, wherein each set of candidate machine learning diagnostic models is trained to receive the suspicious image and output a diagnosis of the patient's medical condition; A computer program product containing computer program code for performing the above. (Item 12) The computer program product according to item 11, wherein generating the calibrated skin color tone image is performed in response to determining that the basic skin color tone image meets quality criteria. (Item 13) Determining the basic skin color tone of the patient based on the calibrated image includes: Determining a numerical representation of each pixel of the calibrated basic skin color tone image; Generating an aggregated representation by performing a statistical operation based on each numerical representation; Identifying a point in the color space corresponding to the aggregated representation; Mapping the point in the color space to a discrete value in a classification system. The computer program product according to item 11, including the above. (Item 14) The method further includes generating a plurality of calibrated basic skin color tone images, and determining the basic skin color tone of the patient based on the calibrated image further includes: Inputting each of the plurality of calibrated basic skin color tone images into a measurement classifier; Receiving the basic skin color tone as an output from the measurement classifier. The computer program product according to item 11, including the above. (Item 15) The method further includes determining a classification corresponding to the measured skin color tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method further includes inputting the image of concern into the selected diagnostic classifier; receiving, as an output from the selected diagnostic classifier, information from which the diagnosis is then derived; A computer program product according to item 11, including (Item 16) The method further includes inputting the basal skin color tone and the image of concern into at least one classifier of the selected set of machine learning diagnostic models; receiving, as an output from the at least one classifier, information from which the diagnosis is then derived; A computer program product according to item 11, including (Item 17) Determining the method further includes inputting the image of concern into at least one classifier of the selected set of machine learning diagnostic models; receiving, as an output from the at least one classifier, information from which the diagnosis is then derived; modifying the information based on the basal skin color tone; determining the diagnosis based on the modified information; A computer program product according to item 11, including (Item 18) The diagnosis of the medical condition includes a probability that the patient has the medical condition. A computer program product according to item 11. (Item 19) A method for determining a diagnosis of a patient, the method includes receiving at least a part of the biometric data of the patient; inputting the biometric data into a classifier trained to obtain an attribute of the patient; Receiving the attributes of the patient as an output from the classifier; Selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the attributes of the patient, wherein each set of candidate machine learning diagnostic models is trained to receive the attributes and output a diagnosis of the patient's medical condition; A method comprising: (Item 20) The method according to item 19, wherein each set of candidate machine learning diagnostic models is trained to receive a suspicious image depicting an organ of the patient in addition to the attributes and output a diagnosis of the patient's medical condition based on both the attributes and the suspicious image.

Brief Description of the Drawings

[0006]

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

[0012] The figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

DETAILED DESCRIPTION OF THE INVENTION

[0013] (a) Environmental Overview Figure 1 is an exemplary block diagram of system components within an environment for utilizing a skin image calibration tool according to one embodiment. Environment 100 includes an imaging device 110, a network 120, a skin image calibration tool 130, a treatment selection tool 140, and a disease diagnosis tool 150. The imaging device 110 may be any device configured to capture an image of the affected area of a patient. Examples extending to the present disclosure are skin image calibration and skin disease diagnosis. However, the imaging, calibration, and diagnostic aspects disclosed herein may be applied to any other organ of the human body (e.g., the retina). The imaging device 110 may be specially configured to capture an image of the patient's skin. Alternatively, the imaging device 110 may be a general client device such as a smartphone, laptop, tablet, or other camera-equipped computing device on which software (e.g., an application) for processing images in the manner disclosed herein is installed. In certain embodiments, the imaging device 110 may be a general client device that captures an image and transmits the image to a remote server (e.g., the skin image calibration tool 130), and the processing and calibration of the image are performed by the remote server.

[0014] Network 120 may be any communication network such as a local area network, wide area network, Internet, and the like. Alternatively, or in addition, Network 120 may represent on-device activity (such as in a scenario where some or all of the functionality of Skin Image Calibration Tool 130, Treatment Selection Tool 140, and / or Disease Diagnosis Tool 150 is installed such that it is built into Imaging Device 110).

[0015] Skin Image Calibration Tool 130 receives a patient's baseline skin tone image and determines the patient's baseline skin tone from the baseline skin tone image. As used herein, the term "baseline skin tone image" may refer to a captured image that meets a certain criterion, or an image captured based on an instruction that the image should meet a certain criterion. For example, in certain embodiments, the baseline skin tone image is an image that should show healthy, un-tanned skin. As used herein, the term "baseline skin tone" may refer to a value representing the pigmentation (or tone, as used synonymously herein) of the patient's skin. Further details regarding the mechanics of how Skin Image Calibration Tool 130 determines the baseline skin tone are described in more detail below with respect to FIGS. 2-6. Although depicted as being the exact opposite of Network 120 in FIG. 1, Skin Image Calibration Tool 130 may be installed, in part or in whole, on Imaging Device 110 and / or within a device local to Imaging Device 110.

[0016] Treatment Selection Tool 140 receives an input (e.g., the determined baseline skin tone) and outputs a treatment decision. For example, based on the baseline skin tone determined by Skin Image Calibration Tool 130, Treatment Selection Tool 140 may select a machine learning model that is optimally optimized for diagnosing a suspect image. Disease Diagnosis Tool 150 performs a diagnosis based on its input. For example, Disease Diagnosis Tool 150 receives a suspect image as an input ReceivedIt may also output a diagnosis. Further details of how the treatment selection tool 140 and the disease diagnosis tool 150 perform these and other operations are disclosed in more detail below with respect to FIGS. 3-6. In any case where a classifier or machine learning diagnosis model is described herein, the step of inputting data into such a model and obtaining an output from the model can be performed by the disease diagnosis tool 150. As with the skin image calibration tool 130, although depicted in FIG. 1 as a network 120 that is the exact opposite of each other element depicted in FIG. 1, the treatment selection tool 140 and / or the disease diagnosis tool 150 may be installed, in part or in whole, on the imaging device 110 and / or within a device local to the imaging device 110. The functionality of the skin image calibration tool 130, the treatment selection tool 140, and the disease diagnosis tool 150 may be integrated on a single server or a group of servers and / or may be distributed across multiple servers.

[0017] FIG. 2 is an exemplary block diagram of modules and components of a skin image calibration tool according to one embodiment. As depicted in FIG. 2, the skin image calibration tool 130 includes various modules such as an image capture module 210, a basic skin tone determination module 220, and an adapted imaging profile generation module 230. The skin image calibration tool 130 is also depicted as including various databases such as a reference calibration profile 250, a skin tone image 260, and a basic skin tone classifier 270. The modules and databases depicted are exemplary only, and more or fewer modules and / or databases may be used to achieve the functionality disclosed herein.

[0018] The image capture module 210 captures a baseline skin tone image of the patient's skin for further processing. To capture the image, the image capture module 210 first obtains image data from the camera of the imaging device 110. The obtained image data is in an incompletely configured state and thus may not yet be synthesized into an image perceptible by humans. This raw image data, which represents the photon intensity received pixel by pixel in a Bayer pattern, often requires further post-processing such as debayering and color correction to be converted into a virtually calibrated image. The image capture module 210 may apply a reference calibration profile to the obtained image data (e.g., as read from the reference calibration profile 250 or from the memory of the imaging device 110) to capture the baseline skin tone image. The imaging device 110 may be calibrated to be within a set tolerance, including but not limited to optical properties such as the accuracy of colorimetric analysis (traditionally described in terms of Delta E, which measures the perceptual change between two given colors) and the relative lens position. This calibration may be implemented in any number of ways, including, for example, camera settings (such as exposure or channel gain), color conversion profiles (such as ICC profiles), and / or spatially related point-based conversions (such as which pixels are located within the effective field of view of the camera). These calibrations may be performed by the factory when generating the imaging device 110 and / or may be commanded by the imaging device 110. The resulting calibration of the imaging device 110 forms the reference calibration profile.

[0019] In one embodiment, the image capture module 210 may cause an output of instructions (e.g., displayed instructions and / or audio instructions) that will be provided to an operator of the imaging device 110 prior to the step of obtaining image data. The operator may be any human, such as a medical assistant, a minimally trained operator, or a patient. The instructions may include instructions for taking a photograph of skin having one or more predefined characteristics, such as skin on the inner side of the wrist, healthy skin, un-tanned skin, skin with no artifacts such as moles, skin with a consistent color tone, skin that is present, etc. The image capture module 210 may perform processing on the baseline skin tone image to verify whether the photographed skin complies with the predefined characteristics and whether the image itself is of sufficiently high quality. For example, pattern recognition may be performed, or the baseline skin tone image may be input into a classifier that is trained to output a label as to whether the baseline skin tone image complies with the instructions.

[0020] The image capture module 210 may capture a plurality of baseline skin tone images. The image capture module 210 may prompt the operator to capture several baseline skin tone images. For example, a patient may not have a uniform skin tone across the patient's body, and thus, multiple skin tone images may be used to obtain a more robust dataset for determining the patient's baseline skin tone. The image capture module 210 may store the baseline skin tone images in the skin tone image database 260. The skin tone image database 260 may store baseline skin tone images associated with the patient's electronic chart, including other information about the patient, such as biometric information, demographic information, and any other information that describes the patient.

[0021] The base skin tone determination module 220 determines a base skin tone for a patient based on one or more base skin tone images. The base skin tone determination module 220 may determine the base skin tone in any of a variety of manners. In one embodiment, the base skin tone determination module 220 may determine a numerical representation of each pixel of the calibrated base skin tone image. For example, each color of each pixel may correspond to a numerical value on a scale. The base skin tone determination module 220 may generate an aggregated representation by performing statistical operations based on each numerical representation. For example, the base skin tone determination module 220 may average the numerical representations. If multiple base skin tone images are used, the base skin tone determination module 220 may repeat this activity for each base skin tone image and then perform statistical operations on each of the aggregated representations to yield an aggregated representation for the entire set of base skin tone images. The base skin tone determination module 220 may then map the aggregated representation to a point in color space using scaling and may assign this point as the base skin tone. Alternatively, rather than immediately assigning this point as the base skin tone, the base skin tone determination module 220 may map the point in color space to a discrete value within a classification system such as the Fitzpatrick system or any other classification system. This embodiment of base skin tone determination may often be sufficient but may not account for non-uniformities in pigmentation within a reference area such as hypopigmented or hyperpigmented skin patches, and thus, other measurement means may be implemented.

[0022] In one embodiment, the baseline skin tone determination module 220 may input the baseline skin tone image into the baseline skin tone classifier 270. The baseline skin tone classifier 270 may be a machine learning model trained to generate discrete values from a set of images. For example, to train the baseline skin tone classifier 270, a professional evaluator may assign values to each image in the training set. The label may indicate the patient's precise skin tone. In one embodiment, the label may additionally or alternatively indicate an attribute of the skin tone (e.g., whether the skin is hypopigmented or hyperpigmented). Thus, the baseline skin tone classifier 270 may output the patient's baseline skin tone.

[0023] Alternatively, or in addition, the baseline skin tone classifier 270 may output the type of skin shown (e.g., hypopigmented or hyperpigmented skin, or normal skin). The baseline skin tone determination module 220 may determine that the aforementioned aggregation representation method is sufficient in response to receiving an output from the classifier that the skin is normal. In response to receiving an output from the classifier that the skin is of a certain type (e.g., hypopigmented or hyperpigmented), the baseline skin tone determination module 220 may determine to use the measurement output by the baseline skin tone classifier 270 as the baseline skin tone. Exemplary classifier algorithms for the baseline skin tone classifier 270 may include a convolutional neural network, a support vector machine for the extracted features, and XGBoost for the extracted features. As another example, a segmentation algorithm may be trained to separate healthy un-tanned skin from discolored skin, and then either the basic approach described above or a different classifier may be used only on the healthy patches of skin to determine the baseline skin tone. Exemplary segmentation algorithms include a fully convolutional neural network, Otsu's method, and thresholding of color variance.

[0024] The adapted imaging profile generation module 230 generates an adapted imaging profile. As used herein, the term "adapted imaging profile" may refer to imaging parameters that are optimized to obtain a skin-concerning image that is most useful for the diagnosis of skin-concerning images, which would be obtained using only the reference calibration profile. The adapted imaging profile generation module 230 may generate an adapted imaging profile in several ways. In one embodiment, the adapted imaging profile generation module 230 may determine a set of predefined imaging parameters stored in the reference calibration profile database 250 as a mapping to the determined underlying skin tone, based on a data structure such as a mapping table. In another embodiment, instead of conforming the imaging parameters to a data structure such as a mapping table, the adapted imaging profile generation module 230 may use the underlying skin tone and a numerical representation of the skin tone to adjust the imaging parameters as a difference from a base value. This example may include increasing the exposure in proportion to the suntan of the patient's skin to maximize details. More complex examples may involve interpolating between the values of known profiles and selecting intermediate values for skin tones that are between the ranges of two explicitly listed profiles. The adapted imaging profile generation module 230 may store the adapted imaging profile in the reference calibration profile database 250.

[0025] Figure 3 is an exemplary block diagram of the modules and components of a treatment selection tool according to one embodiment. As depicted in Figure 3, the treatment selection tool 140 includes various modules such as a concern image processing module 310, a suitability module 320, and a classifier selection module 330. The treatment selection tool 140 also includes various databases such as a diagnostic model directory 350 and patient information 360. The modules and databases depicted with respect to Figure 3 are exemplary only, and more or fewer modules and / or databases may be used to achieve the functionality of the treatment selection tool 140 disclosed herein.

[0026] The Suspicion Image Processing Module 310 may receive a skin suspicion image and calibrate the skin suspicion image using the adapted imaging profile. The calibration of the skin suspicion image occurs in the same manner as the calibration of the basic skin tone image, except that the adapted imaging profile is used instead of the reference calibration profile.

[0027] The Suitability Module 320 determines whether the skin suspicion image is suitable for diagnosis. Suitability may be based on any predefined criteria. For example, a regulatory agency may show that some classifications (e.g., Fitzpatrick system classifications 1 - 3) are suitable for diagnosis using the systems and methods disclosed herein, while other classifications are not suitable for diagnosis using the systems and methods disclosed herein (e.g., darker skin tones may incur a higher error rate in the step of diagnosing whether a nevus is benign). As another example, the skin suspicion image may fail quality parameters that must be met for processing (e.g., the skin suspicion image is overexposed, underexposed, does not sufficiently show the affected skin area, or any other predetermined parameter). Additional exemplary justification criteria include not containing color channel saturation across the image (channels that are at maximum and thus cannot provide accurate measurements will indicate failure), or passing a machine learning-based quality checker. Image quality issues can be sorted using a convolutional neural network classifier trained on examples of images with sufficient quality and images with insufficient quality. Additionally, image quality issues can be sorted using features extracted from the image, which may include measurements of the high-frequency components of the image and the color histogram.

[0028] In response to the step of determining that the skin concern image is not suitable for diagnosis, the suitability module 320 may cause the operator to output a prompt indicating that the skin concern image is not suitable for diagnosis. In embodiments where the skin concern image is not suitable for diagnosis due to quality issues, the suitability module 320 may prompt the operator to capture another skin concern image that meets the quality parameters. If the re-captured image is Received such, in some embodiments, the suitability module 320 may instruct that a reference calibration profile be used to generate the re-captured skin concern image. In embodiments where the re-captured image still does not meet the quality parameters, the suitability module 320 may prompt the operator to manually check the image for errors, manually check for signs of malfunction of the camera, and again, re-take the skin concern image and / or re-take the baseline skin tone image (this may trigger the regeneration of the calibrated imaging profile using the re-taken baseline skin tone image, and the regenerated calibrated imaging profile may be used to calibrate further skin concern images of the patient). In embodiments where the skin concern image is not suitable due to patient characteristics, the suitability module 320 may prompt the operator to instruct the patient that the diagnosis cannot be completed. Additional information may also be prompted to the operator (e.g., the patient is instructed to obtain a diagnosis by a physician).

[0029] The classifier selection module 330 selects one or more classifiers, which are the selected classifiers to be used to establish the patient's treatment, such as a set of machine learning diagnostic models, based on the patient's baseline skin tone. The set may include one or more machine learning models, and in addition, may include discovery techniques that together form a workflow for the patient's treatment. A plurality of classifiers may be stored in the diagnostic model directory 350. Each of these classifiers may be associated with a criterion that, when satisfied, indicates to the classifier selection module 330 that the classifier should be used for the patient's diagnosis. The classifier selection module 330 may select a classifier based on information additional to the baseline skin tone, and the additional information is read from the patient information database 360. The additional information may include any information in the patient's electronic patient record, such as biographical information, demographic information, and / or any other information that describes the patient.

[0030] In certain embodiments, the classifier selection module 330 may perform activities other than the step of selecting one or more specific classifiers, or activities related to obtaining additional inputs for a specific classifier. For example, the classifier selection module 330 may determine that the baseline skin tone is a high-risk skin tone (e.g., as indicated in a database that maps the baseline skin tone to a risk category). For example, the baseline skin tone may correspond to a very fair-skinned person who has a higher incidence of skin cancer. In response, the classifier selection module 330 may prompt for an input to an answer to a question related to notifying the treatment (e.g., the patient may be prompted to answer a question about the patient's sunlight exposure behavior). As another example, the classifier selection module 330 may prompt for additional skin-concern images to be captured (e.g., the patient's palms and soles, which have a darker skin tone that makes the patient more prone to dangerous growths in those locations).

[0031] In one embodiment, the classifier selection module 330 may have several classifiers that are available within the diagnostic model directory 350, and the available classifiers each correspond to a different range of skin tones. The classifier selection module 330 may select a classifier based on the patient's baseline skin tone. The selected classifier may be trained to be optimized with respect to its corresponding range of skin tones, and if the patient has a baseline skin tone outside of that range, it may be less effective or ineffective. The selected classifier may use other information such as the baseline skin tone and / or additional patient information Together with, or without other information , along with the suspicious image as input Received and may output a diagnosis.

[0032] In one embodiment, the classifier selection module 330 is not used as a gatekeeper for selecting a classifier. Instead, the treatment selection tool 140 inputs the baseline skin tone and the suspicious image (optionally along with additional patient information) into the classifier and outputs a diagnosis. As another exemplary embodiment in this spirit, a classifier that takes a skin suspicious image as input Receiving and outputs the probability of a diagnosis may be used. The treatment selection tool 140 then weights the probabilities based on the patient's baseline skin tone.

[0033] When the classifier outputs a diagnosis, the diagnosis may be a direct diagnosis (e.g., the suspicious image indicates the absence of a disease or the identification of a specific disease present). The diagnosis may alternatively be a plurality of probabilities, each probability corresponding to a different candidate diagnosis, and the probabilities indicate the likelihood that the disease corresponds to each candidate diagnosis. The classifier selection module 330 may output the probabilities directly to the operator. In one embodiment, the classifier selection module 330 may display to the operator the candidate diagnosis with the highest probability. The classifier selection module 330 may compare each probability to a threshold and limit the candidate diagnoses displayed to the operator to those whose corresponding probabilities exceed the threshold. Each different candidate diagnosis may have a different corresponding threshold that must be crossed with respect to the candidate diagnosis to be displayed to the operator.

[0034] The treatment selection tool 140 may determine that at least two diseases cannot be excluded as being depicted by the skin concern image based on two or more candidate diagnoses having a probability of exceeding a threshold output by the classifier. Thus, the treatment selection tool 140 may prompt the operator to instruct the patient to obtain a professional opinion (e.g., from a physician). The treatment selection tool 140 may withhold two or more candidate diagnoses or may display two or more candidate diagnoses.

[0035] Although the skin is a primary example, the treatment selection tool 140 may apply treatment selection based on any patient attribute. In one embodiment, the model may be trained to classify patients based on any medical condition (e.g., the patient's skin tone, weight, gender, jaundice, diabetes, and / or any other medical condition). The treatment selection tool 140 may determine a further model (e.g., a refined model trained to determine whether a skin condition associated with diabetes is present within the patient) to be used to determine a treatment for the patient based on the output of the model.

[0036] In one embodiment, the treatment selection tool 140 may receive biometric data corresponding to a patient. The biometric data may include visual images such as the images discussed herein, and any other type of image (regardless of whether it uses a spectrum to capture internal organs such as a camera lens or an ultrasonic diagnostic device), and may be an image in a non-visible wavelength (e.g., infrared, ultraviolet, X-ray, etc.). The biometric data may be any other data about the patient, or a combination of different types of data. Exemplary biometric data includes measurements derived from blood tests, heart activity (such as measured by an electrocardiogram), pulse data, and any other data that describes the patient's biometric activities. The treatment selection tool 140 may input the biometric data into a classifier that is trained to obtain the patient's attributes. For example, blood test data may be input into the classifier, and the classifier may output whether the blood test indicates that the patient has one or more medical conditions such as a disease (e.g., diabetes).

[0037] Based on the patient's attributes, the treatment selection tool 140 may select a set of machine learning models from a plurality of sets of candidate machine learning diagnostic models, and each set of candidate machine learning diagnostic models is trained to receive attributes and output a diagnosis of the patient's medical condition. For example, a set of machine learning models corresponding to a specific disease such as diabetes may, in addition to the disease itself, receive other inputs (e.g., retinal images, toe images) Receiving and may include machine learning models that obtain outputs related to the disease (e.g., based on the retinal image, a determination of whether the patient has diabetic retinopathy, a determination of whether the patient has a toe condition associated with diabetes, etc.). The operator may be prompted to obtain additional information about the patient, or to visit another operator (e.g., a physician) and notify the selected set of machine learning models, and to obtain other inputs by determining additional information that may be used to instruct the patient to obtain other inputs.

[0038] FIG. 4 is a block diagram illustrating the components of an exemplary machine that can read instructions from a machine-readable medium and execute them within a processor (or controller). Specifically, FIG. 4 shows a schematic representation of a machine in an exemplary form of a computer system 400 in which program code (e.g., software) for causing the machine to implement any one or more of the methodologies discussed herein can be executed. The program code may consist of instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate within the capacity of a server machine or a server machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment.

[0039] The machine can be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions 424, which specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute the instructions 124 and implement any one or more of the methodologies discussed herein.

[0040] The exemplary computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include software drivers that enable the user interface to be displayed on a screen (or display). The visual interface may display the user interface directly (e.g., on the screen) or indirectly on a surface, window, or equivalent (e.g., via a visual projection unit). For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with a touch-responsive screen. The computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or a touch screen keyboard), a cursor control device 414 (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420, which are similarly configured to communicate via the bus 408.

[0041] Storage unit 416 includes a machine-readable medium 422 on which instructions 424 (e.g., software) are stored that embody any one or more of the methodologies or functions described herein. The instructions 424 (e.g., software) may also reside, fully or at least partially, within main memory 404 or within processor 402 (e.g., within a cache memory of the processor) during execution thereof by computer system 400, and main memory 404 and processor 402 may also constitute a machine-readable medium. The instructions 424 (e.g., software) may be transmitted or received via network 426 via network interface device 420.

[0042] Although machine-readable medium 422 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" should also be construed to include any medium capable of storing instructions (e.g., instructions 424) for machine execution that cause a machine to perform any one or more of the methodologies disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid state memories, optical media, and magnetic media.

[0043] FIG. 5 depicts an exemplary image showing a base skin tone image and a skin concern image, according to one embodiment. The base skin tone image 510 is shown as being captured by an operator who is not a patient, as depicted. The imaging device used is shown to be a smartphone camera. The imaging device is shown to display the patient's reference skin tone.

[0044] It is shown that the skin concern image 520 is captured by an operator. The skin concern image 520 includes an image of a nevus. The skin concern image 520 is calibrated using a calibrated reference profile, which results in a skin concern image that, when input into a classifier for diagnosis, optimizes the accuracy of the diagnosis result.

[0045] FIG. 6 depicts an exemplary flowchart for determining a diagnosis based on a patient's baseline skin tone according to one embodiment. Process 600 may begin with step (602) where the skin image calibration tool 130 receives an image of the patient's baseline skin tone. The image of the baseline skin tone may be received by the image capture module 210, which may be executed by the processor 402. The image of the baseline skin tone may be the baseline skin tone image 510. The skin image calibration tool 130 generates a calibrated skin tone image (604) by calibrating the image of the baseline skin tone using a reference calibration profile (such as read from the reference calibration profile database 250).

[0046] The skin image calibration tool 130 determines the patient's baseline skin tone based on the calibrated image of the baseline skin tone (606) (e.g., using the baseline skin tone determination module 220). The treatment selection tool 140 receives a concern image of a part of the patient's skin (608) (e.g., as depicted in the concern image 520). The treatment selection tool 140 selects a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic modules based on the patient's baseline skin tone (610) (e.g., using the classifier selection module 330), and each set of candidate machine learning models is trained to receive a concern image and output a diagnosis of the patient's condition.

[0047] (Summary) The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Those skilled in the art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0048] Some portions of this description describe embodiments of the invention from the perspective of algorithms of operations on information and symbolic representations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing field to effectively convey the substance of their work to other skilled artisans. These operations are described functionally, computationally, or theoretically, but are understood to be implemented by a computer program or equivalent electrical circuitry, microcode, or the like. Further, it has also been proven that it is expedient at any time to refer to an arrangement of these operations as a module without loss of generality. The operations described and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0049] Any of the steps, operations, or processes described herein can be implemented or carried out with one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0050] Embodiments of the present invention may also relate to an apparatus for performing the operations herein. The apparatus may in particular be constructed for the required purpose and / or comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored within a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Further, any computing system referred to herein may comprise a single processor or may be an architecture that employs multiple processor designs for increased computing capabilities.

[0051] Embodiments of the present invention may also relate to a product produced by the computing processes described herein. Such a product may comprise information resulting from the computing process, where the information is stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of the computer program product or other data combination described herein.

[0052] Finally, the language used herein has been principally selected for readability and illustrative purposes and may not have been selected to circumscribe or delimit the subject matter of the present invention. Accordingly, the scope of the present invention is intended to be limited not by the forms for carrying out the invention but rather by any claims that may arise on the basis of an application hereunder. Thus, the disclosure of embodiments of the present invention is intended to illustrate rather than limit the scope of the present invention as set forth in the following claims.

Claims

1. A method of operating a system for determining a diagnosis based on a patient's baseline skin tone, the system comprising one or more processors, the method of operation comprising: the one or more processors receiving a baseline skin tone image of a non-concerning part of the patient's skin; the one or more processors generating a calibrated baseline skin tone image by calibrating the baseline skin tone image using a reference calibration profile; the one or more processors determining the patient's baseline skin tone based on the calibrated baseline skin tone image; the one or more processors receiving a suspect image of a concerning part of the patient's skin; the one or more processors selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's baseline skin tone, each set of candidate machine learning diagnostic models being trained to receive the suspect image and output a diagnosis of the patient's medical condition; A method of operation comprising.

2. further comprising determining whether the suspect image meets a quality criterion and, in response to determining that the suspect image does not meet the quality criterion, outputting to an operator a prompt indicating that the suspect image is not suitable for diagnosis.

3. Determining the patient's baseline skin tone based on the calibrated image comprises: the one or more processors determining a numerical representation of each pixel of the calibrated baseline skin tone image; the one or more processors generating a summary representation by performing a statistical operation based on each numerical representation; the one or more processors identifying a point in a color space corresponding to the summary representation; The one or more processors map the point in the color space to a discrete value in the classification system The operating method according to claim 1, comprising: **Claim 4** The operating method further includes the one or more processors generating a plurality of calibrated basic skin tone images, the plurality of calibrated basic skin tone images including the calibrated basic skin tone image, and determining the basic skin tone of the patient based on the calibrated image is the one or more processors inputting each of the plurality of calibrated basic skin tone images into a measurement classifier the one or more processors receiving the basic skin tone as an output from the measurement classifier The operating method according to claim 1, further comprising: **Claim 5** An operating method of a system for determining a diagnosis based on a patient's basic skin tone, the system comprising one or more processors, the operating method comprising the one or more processors receiving a basic skin tone image of a patient the one or more processors generating a calibrated basic skin tone image by calibrating the basic skin tone image using a reference calibration profile the one or more processors determining the basic skin tone of the patient based on the calibrated basic skin tone image the one or more processors receiving a suspicious image of a part of the patient's skin the one or more processors selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's basic skin tone, each set of candidate machine learning diagnostic models being trained to receive the suspicious image and output a diagnosis of the patient's medical condition the one or more processors generating a fitted imaging profile based on the reference calibration profile and the basic skin tone The one or more processors calibrate the image of concern using the adapted imaging profile and a method of operation.

6. The method of operation further includes the one or more processors determining a classification corresponding to the determined underlying skin tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method of operation includes the one or more processors inputting the image of concern into the selected diagnostic classifier; the one or more processors receiving, as an output from the selected diagnostic classifier, information from which the diagnosis is then derived and further includes the method of operation according to claim 1.

7. The method of operation includes the one or more processors inputting the underlying skin tone and the image of concern into at least one classifier of the selected set of machine learning diagnostic models; the one or more processors receiving, as an output from the at least one classifier, information from which the diagnosis is then derived and further includes the method of operation according to claim 1.

8. A method of operating a system for determining a diagnosis based on a patient's underlying skin tone, the system comprising one or more processors, the method of operation comprising the one or more processors receiving an image of the patient's underlying skin tone; the one or more processors generating a calibrated image of the underlying skin tone by calibrating the image of the underlying skin tone using a reference calibration profile; the one or more processors determining the patient's underlying skin tone based on the calibrated image of the underlying skin tone; the one or more processors receiving an image of concern of a portion of the patient's skin The one or more processors select a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the patient's baseline skin tone, wherein each set of candidate machine learning diagnostic models is trained to receive the suspicious image and output a diagnosis of the patient's medical condition, and the one or more processors input the suspicious image into at least one classifier of the selected set of machine learning diagnostic models, and the one or more processors receive, as an output from the at least one classifier, information from which the diagnosis is derived, and the one or more processors modify the information based on the baseline skin tone, and the one or more processors determine the diagnosis based on the modified information A method of operation, comprising.

9. The method of operation further comprises the one or more processors determining whether the suspicious image is diagnosable based on the baseline skin tone, and in response to determining that the suspicious image is not diagnosable, the one or more processors output an indication that the suspicious image cannot be diagnosed The method of operation according to claim 1, further comprising.

10. The diagnosis of the medical condition according to claim 1, comprising the probability that the patient has the medical condition.

11. A computer-readable storage medium storing computer program code, which, when executed by a processor, causes the processor to perform a method for determining a diagnosis based on a patient's baseline skin tone, the method comprising receiving a baseline skin tone image of a non-suspicious part of the patient's skin, and Generating a calibrated baseline skin tone image by calibrating the baseline skin tone image using a reference calibration profile; Determining the baseline skin tone of the patient based on the calibrated baseline skin tone image; Receiving a suspicious image of a suspicious part of the patient's skin; Selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the baseline skin tone of the patient, wherein each set of candidate machine learning diagnostic models is trained to receive the suspicious image and output a diagnosis of the patient's condition; A computer-readable storage medium comprising: **Claim 12** The method further includes determining whether the suspicious image meets a quality standard and, in response to determining that the suspicious image does not meet the quality standard, outputting to an operator a prompt indicating that the suspicious image is not suitable for diagnosis. The computer-readable storage medium according to claim 11. **Claim 13** Determining the baseline skin tone of the patient based on the calibrated image comprises: Determining a numerical representation of each pixel of the calibrated baseline skin tone image; Generating an aggregated representation by performing a statistical operation based on each numerical representation; Identifying a point in a color space corresponding to the aggregated representation; Mapping the point in the color space to a discrete value in a classification system The computer-readable storage medium according to claim 11. **Claim 14** The method further includes generating a plurality of calibrated baseline skin tone images, the plurality of calibrated baseline skin tone images including the calibrated baseline skin tone image, and determining the baseline skin tone of the patient based on the calibrated image comprises: Inputting each of the plurality of calibrated basic skin tone images into a measurement classifier; Receiving the basic skin tone as an output from the measurement classifier; The computer-readable storage medium according to claim 11, further comprising: **Claim 15** The method further comprises determining a classification corresponding to the determined basic skin tone, and selecting the set of machine learning diagnostic models includes selecting a diagnostic classifier corresponding to the classification, and the method comprises: Inputting the image of concern into the selected diagnostic classifier; Receiving, as an output from the selected diagnostic classifier, information from which the diagnosis is derived; The computer-readable storage medium according to claim 11, further comprising: **Claim 16** The method comprises: Inputting the basic skin tone and the image of concern into at least one classifier of the selected set of machine learning diagnostic models; Receiving, as an output from the at least one classifier, information from which the diagnosis is derived; The computer-readable storage medium according to claim 11, further comprising: **Claim 17** A computer-readable storage medium storing computer program code, which, when executed by a processor, causes the processor to execute a method for determining a diagnosis based on a patient's basic skin tone, the method comprising: Receiving an image of the patient's basic skin tone; Generating a calibrated basic skin tone image by calibrating the basic skin tone image using a reference calibration profile; Determining the patient's basic skin tone based on the calibrated basic skin tone image; Receiving an image of concern of a part of the patient's skin; Selecting a set of machine learning diagnostic models from a plurality of sets of candidate machine learning diagnostic models based on the basic skin color tone of the patient, wherein each set of candidate machine learning diagnostic models is trained to receive the image of concern and output a diagnosis of the patient's medical condition; Inputting the image of concern into at least one classifier of the selected set of machine learning diagnostic models; Receiving, as an output from the at least one classifier, information from which the diagnosis is then derived; Modifying the information based on the basic skin color tone; Determining the diagnosis based on the modified information; A computer-readable storage medium comprising:

18. The computer-readable storage medium according to claim 11, wherein the diagnosis of the medical condition includes the probability that the patient has the medical condition.

Images