Collaborative long-term skin care monitoring

Abstract

A tool enabling professional-level monitoring at home can provide patients with better access to expert skin care advice. The tool can provide information about skin health to a patient's skin care specialist, with the most relevant information highlighted and presented in an expected manner. The device can include a processor configured to receive a first skin image at a first time and a second skin image at a second time. The first time and the second time can be separated by a duration associated with a skin event. Multiple skin characteristics can be determined from the first skin image and the second skin image. The processor can be configured to generate an analysis output and transmit the analysis output to one or more recipients. The analysis output can include a synoptic display of one or more of the multiple skin characteristics.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 411,267, filed September 29, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Proper skin care can be beneficial for reducing signs of aging, reducing acne, and improving overall health. Individuals have access to skin care information and tools, but generally lack the expertise of dermatologists. Dermatologists and other skin care specialists can provide the expertise and tools to improve skin health. However, access to dermatologists is limited (e.g., based on appointment availability, high costs, etc.). Furthermore, because dermatologists cannot continuously monitor patients, dermatologists' ability to assess a patient's skin health may be limited to the information available at the time of the appointment. Summary of the Invention [Means for solving the problem]

[0003] Devices and / or tools that enable skin health monitoring in a home or office environment can provide patients with better access to expert skin care advice. In some embodiments, monitoring can occur one time, occasionally, or continuously. In particular, devices and / or tools can enable information about skin health to be provided to a patient's skin care specialist, with the most relevant information highlighted and presented in an expected manner.

[0004] The device may include a processor configured to receive a first skin image at a first time and a second skin image at a second time, the first skin image and the second skin image may be associated with a user, and the first time and the second time may be separated by a duration associated with a skin event.

[0005] The processor may be configured to determine a plurality of skin characteristics from the first skin image and the second skin image, where one skin characteristic of the plurality of skin characteristics may represent at least a skin element and a score associated with the skin element.

[0006] The processor may be configured to generate an analysis output. The analysis output may be based on the analysis configuration and the plurality of skin characteristics. The analysis output may include a synoptic display of one or more of the plurality of skin characteristics.

[0007] The processor may be configured to transmit the analysis output to one or more recipients. For example, the one or more recipients may include a skin care professional. The overview display may also present a historical summary of one or more of the plurality of skin characteristics in a form suitable for the skin care professional. For example, the analysis output may include a historical summary of the plurality of skin characteristics representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is an example of a User Interface (UI) illustrating a technique for monitoring skin health or care over time. [Figure 2] 1 is an example of a User Interface (UI) illustrating a technique for monitoring skin health or care over time. [Figure 3] 1 is an example of a User Interface (UI) illustrating a technique for monitoring skin health or care over time. [Figure 4] 1 is an example of a User Interface (UI) illustrating a technique for monitoring skin health or care over time. [Figure 5] 1 is an exemplary timeline illustrating the collection of longitudinal information regarding multiple skin characteristics, the generation of an analysis output, and the transmission of the analysis output. [Figure 6] FIG. 1 is a flow diagram illustrating an exemplary computer-implemented method. [Figure 7] FIG. 1 is a block diagram illustrating the generation of an analytical output based on longitudinal information regarding one or more skin characteristics. [Figure 8A] 10 shows an exemplary analysis output. [Figure 8B] 10 shows an exemplary analysis output. [Figure 9] FIG. 1 is a block diagram illustrating an exemplary computing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Proper skin care can be beneficial for reducing signs of aging, reducing acne, and improving overall health. Individuals have access to skin care information and tools, but generally lack the expertise of dermatologists. Dermatologists and other skin care specialists can provide the expertise and tools to improve skin health. However, access to dermatologists is limited (e.g., based on appointment availability, high costs, etc.). Furthermore, because dermatologists cannot continuously monitor patients, dermatologists' ability to assess a patient's skin health may be limited to the information available at the time of the appointment.

[0010] Therefore, patients may benefit from devices or tools that allow for at-home skin health monitoring. Such devices may provide patients with better access to expert advice on skin care. Such devices may be able to provide longitudinal information (e.g., information collected over a period of time) to skin care professionals (e.g., dermatologists). The longitudinal information may enable skin care professionals to provide more insightful skin care recommendations to patients.

[0011] Because traditional dermatologist imaging is limited to in-office equipment, adequate interim imaging is unavailable. Traditional home imaging does not provide adequate analysis and information useful to dermatologists. For example, dermatologists who receive large volumes of interim images from users, typically in color digital format, lack the technical means to organize, map, compare, and / or analyze such images. The approach disclosed herein addresses this technical problem by analyzing the interim images and summarizing the results into a synoptic display of one or more of multiple skin characteristics associated with the user's area of ​​interest, for example, for transmission to the dermatologist.

[0012] The device may include a processor configured to receive the first skin image and the second skin image and determine a plurality of skin characteristics based on the first skin image and the second skin image. In some examples, one or more of the plurality of skin characteristics may be associated with a region of interest of the user. In some examples, the device may generate an analysis output including a synoptic display presenting a historical summary of one or more of the plurality of skin characteristics. The synoptic display may be in a form suitable for a skin care professional. For example, the synoptic display may be in a form that allows the skin care professional to quickly determine an appropriate treatment for the user.

[0013] 1 is an exemplary user interface (UI) illustrating a home screen of a mobile application for monitoring skin health or care over time. Those skilled in the art will understand that the UI described herein may be implemented in ways other than as a mobile application. For example, the UI described herein may be implemented as a desktop (e.g., computer) application, a web-based application, etc. As used herein, the term "application" may be used broadly to describe all of the possible ways in which the UI may be accessed by a user.

[0014] Using a digital camera, a user may first acquire a skin image of the user's face. In some embodiments, acquiring the skin image may include acquiring a series of skin images. For example, acquiring the skin images may include acquiring a front skin image, a right skin image, and a left skin image of the user's face. In some embodiments, the skin image may be acquired via photo capture. In some embodiments, the skin image may be acquired via video capture. For example, the video may be time-sliced ​​to acquire individual images from the video.

[0015] The application may include an Augmented Reality (AR) guide or other instructions to assist the user in obtaining a proper image. For example, the AR guide may include lines indicating where the user should position their face and eyes to obtain a proper image.

[0016] An application may then analyze the skin image. For example, the application may process the skin image into basic components to extract meaningful information. Image analysis may include tasks such as finding shapes, detecting edges, removing noise, counting objects, texture analysis, or calculating statistics for image quality.

[0017] Region analysis can be used to extract statistical data and interpret that data to determine a user's skin characteristics. For example, feature extraction can be used to extract / identify features from raw skin image data. An application can use classification techniques to identify a set of categories (e.g., acne, wrinkles, etc.) and assign the identified features to their respective categories.

[0018] For example, image processing algorithms that can be used to identify acne may include thresholding, blob detection, Hough transform, template matching, etc. Thresholding may include methods in which pixels with intensities above or below a certain threshold are classified as spots, e.g., acne. Blob detection algorithms such as Laplacian of Gaussian (LoG) or Difference of Gaussian (DoG) may identify spots by determining areas with high intensity variation. The Hough transform may be adapted to detect circular spots or ellipses within an image. Such closed loops within an image may be evaluated as acne, for example, if they fall below a threshold dimension. Template matching may include comparing one or more predefined acne templates to a region of the image. In embodiments in which a match is found, the match confidence may indicate the presence of acne.

[0019] For example, image processing algorithms that can be used to identify wrinkles may include the Hue transform, edge detection, Radon transform, line segment detector (LSD), etc. The Hue transform may include a technique for detecting lines in an image. The Hue transform can identify lines by converting lines into points in a parameter space, with intersecting lines corresponding to peaks. Edge detection algorithms such as the Canny edge detector can be used to find edges in an image and then link the edges to wrinkles and facial lines. The Radon transform may be used to detect lines, especially in skin images with complex pigmentation. The Radon transform may be used to calculate the sum of pixel values ​​along different angles to find wrinkles. The LSD is an algorithm specifically designed to detect line segments in an image and may be used to identify wrinkles that vary in thickness when imaged. By passing an image through one or more algorithms, the image may be reduced to several identified skin elements, each characterized by, for example, category, location, size, etc. The number and dimensions of the elements, normalized by the total area analyzed, can be used to rate the user's skin, which can be calculated as an overall score for the user's skin health.

[0020] As shown, the UI may display the user's overall skin health score (e.g., 7.4 out of 10) and daily insight report 102. The UI may display a user profile icon 108, which, when selected, may allow the user to customize their profile, set goals, control permissions and notifications, etc. Each user may have a personal profile / account associated only with that user. A user can switch between profiles / accounts by entering login credentials associated with the desired account. Thus, multiple users can have their own accounts with separate data (e.g., even if the users access the application via the same device).

[0021] The overall score may be shown using a graphical representation 104 that may have several sections 106 representing multiple skin characteristics. A skin characteristic may be a feature or quality that belongs to the user's skin. For example, the multiple skin characteristics may include skin clarity, under-eye bags, wrinkles, fine lines, age spots, redness, smoothness, etc.

[0022] The skin characteristics may represent at least one of a region of interest, a skin element, a degree, and / or a timestamp. For example, the region of interest may be any of the face, nose, chin, and / or cheek. The skin element may be a variable related to the skin characteristic. For example, the skin element may be a blackhead, a whitehead, a red spot, or a raised bump. The degree may include a density metric of the skin element within the region of interest, and the timestamp may represent the time the skin image was received.

[0023] The application may be configured to determine a score for the skin characteristics. In some examples, the overall score may be an average of the skin characteristic scores. In some examples, the overall score may be a weighted average of the skin characteristic scores.

[0024] The application may be configured to determine scores for variables associated with skin characteristics. The variables may be short-term variables. The variables may be linked to specific skin characteristics. That is, a combination of variables may be used to form a long-term skin characteristic score. For example, a skin clarity characteristic variable may include whether a user has blackheads, whiteheads, red bumps, raised bumps, and / or other such skin lesions. In one example, a user may have few blackheads but many whiteheads and red bumps on a given day. The user may begin a treatment routine to treat the whiteheads and red bumps. As a result, the skin clarity characteristic score may increase over time as the number of whiteheads and red bumps decreases. Artificial intelligence and / or machine learning (AI / ML) skin analysis may be used to count the number of skin lesions. The skin clarity characteristic may include information about pore dimensions.

[0025] As another example, the skin characteristic variables for age spots may include the number of moles and / or pigmentation grade over time. As another example, the skin characteristic variables for fine lines and wrinkles may include the number of fine lines / wrinkles and / or the severity of fine lines / wrinkles.

[0026] In some examples, the daily insight report 102 may include recommendations for skin care routines and / or topical medications or products based on current skin characteristics. The application may recommend products based on responses to a skin care questionnaire. As shown, the UI may include commerce functionality 110 that can be used to purchase products. Products may be recommended based on the user's current skin characteristic score. Products may be recommended based on the user's primary skin concerns or goals. In some examples, the user may use the device to capture images of products used in the routine. The UI may then provide the user with a link to order more product when the user runs out of the product. In some examples, the user may enter ingredients (e.g., rather than products) into the application. The application may list the potential benefits and / or risks of the ingredient. The application may list products that may contain the ingredient. In some examples, the user may be provided with time-dependent reports regarding the user's skin care routine, and the routine may be periodically adjusted based on observed changes to skin characteristics.

[0027] A user can provide a self-assessment of current skin characteristics and can define skin goals (e.g., desired changes in one or more skin characteristics). In some examples, a user can identify long-term skin appearance concerns. AI / ML image analysis can be used to determine scores for a subset of skin characteristics and / or variables. The application can then compare the AI / ML-derived scores to a user-defined subjective score and adjust the score based on the comparison.

[0028] In some examples, AI / ML image analysis may be used to identify a user's skin type (e.g., sensitive, oily, etc.). The user may provide a self-assessment of their skin type. The application may identify a list of ingredients that are potential allergens or irritants. The application may identify a list of ingredients based on the user's skin type.

[0029] In some examples, AI / ML image analysis can be used to determine skin behavior. For example, AI / ML image analysis may obtain a first skin image in which the user is not smiling. AI / ML image analysis may obtain a second skin image in which the user is smiling. AI / ML image analysis may determine, for example, how a product reduces the appearance of wrinkles and / or fine lines when the user is smiling.

[0030] The user may self-report lifestyle variables (e.g., amount of sleep, exercise, stress) that may correlate with skin characteristics. The UI may include instructions regarding ambient weather conditions that may affect the appearance of the user's skin. For example, the UI may indicate that it is very hot outside and that the user should focus on staying hydrated. As another example, the UI may indicate that the air quality is poor and that the user should avoid going outside.

[0031] 2, the UI may display skin images captured by the user along with an analysis overlay 202. The analysis overlay 202 may include markings (e.g., lines, circles, and / or dots) indicating the location of the user's skin features that contributed to each of the skin characteristic scores. For example, the line 204 may be a first color and may indicate wrinkles on the user's forehead. Similarly, the circle 206 may be a second color to indicate the presence of red bumps on the user's skin, and the dot 208 may be a third color to indicate the presence of blackheads.

[0032] As shown in FIG. 3 , the UI may display a report of progress made over a period of time (e.g., over a day, a week, or a month). The report may include a progress bar 302 for each of the skin characteristics. In some embodiments, the UI may display a progress bar for the user's primary goal (e.g., clearer skin, fewer wrinkles). The report may include a rating 304 of the skin characteristics indicating an improvement or decline in the user's skin health. For example, the rating 304 may state, "Your skin looks 5% clearer than it did last month," or "Your skin is 18% smoother than it did last month." The UI may display a button 306 that allows the user to take a new skin image. The new skin image may then be analyzed, and data from the analysis may be incorporated into the report.

[0033] 4, the UI may display a weekly progress graph 402 illustrating scores over time. For example, the weekly progress graph 402 may be a timeline graph, a bar graph, or any other graph suitable for illustrating data over time. The progress graph 402 may illustrate progress for a particular skin characteristic (e.g., smoothness), a subgrouping of skin characteristics, or a combination of skin characteristics (e.g., represented by an overall score). A report of the skin characteristic scores, a graphical representation of the skin characteristics, and an AR / ML image presentation of the image associated with the skin characteristic may be presented in chronological order.

[0034] The UI may display suggestions 404 for improving certain skin characteristics. For example, the UI may display, "Boost your collagen for smoother skin" or "Limit sun exposure to reduce the appearance of wrinkles."

[0035] A user may request that a report containing skin characteristic information collected over time be sent to one or more recipients. For example, a user may request that skin characteristics be sent to a skin care professional (e.g., a dermatologist or aesthetician). For example, as shown in FIG. 5 , a user may take a first skin image at a first time 502. An application may be configured to receive the first skin image and determine multiple skin characteristics 504 from the first skin image. After a duration, the user may take a second skin image at time 506, and the application may receive the second skin image and determine another multiple skin characteristics 508 from the second skin image.

[0036] The user may request that the application combine one or more of the plurality of skin characteristics 504 and 508 to generate an analysis output 510. The user may request that the application send the analysis output 510 to a skin care professional (e.g., a dermatologist or an aesthetician).

[0037] The content of the analysis output 510 may be based on an analysis configuration. The analysis configuration may be a default analysis configuration. The analysis configuration may depend on settings selected by a user. The analysis configuration may depend on settings selected by a skin care professional. For example, based on the analysis configuration, the analysis output 510 may include information associated with a particular subgrouping of skin characteristics (e.g., a report may include only information about clear skin and smoothness characteristics).

[0038] The analysis configuration may not affect how the analysis of the skin image is performed. The analysis configuration may define how the analysis output 510 looks (e.g., the content and / or layout of the analysis output 510). For example, the analysis configuration may indicate that a particular user prefers the analysis output to include tabular raw data, a graphical depiction of the analysis results, an AI overlay of identified skin characteristics, a skin score, etc.

[0039] In some cases, the application may be configured to receive user input identifying a skin care professional. The selected analysis configuration may be selected from multiple analysis configurations based on the user input. The user input may include an area of ​​interest for the user (e.g., a desire to reduce acne). The application may be configured to identify an appropriate analysis configuration and / or skin care professional based on the area of ​​interest. In some examples, the selected analysis configuration may be based on preferences of the skin care professional or the user. In some examples, the analysis output 510 may include images linked to the skin characteristics included in the analysis output 510.

[0040] The analysis output 510 may have a different appearance and / or layout than the report provided to the user (e.g., via a UI). For example, the report provided to the user may include a list of topics to discuss with a skin care professional (e.g., techniques for the user to better hydrate the user's skin). For example, the report provided to the user may include a score for the skin characteristics and suggestions for improving the skin characteristics. In some examples, the report provided to the user may include information about the underlying science and / or causes of a low skin score.

[0041] For a report provided to a skin care specialist, the analysis output 510 may be a historical summary of data collected over a sustained period of time, including more detailed information that the skin care specialist can use to diagnose and treat the user. In some examples, the historical summary may represent the difference between a first skin characteristic associated with a first time 502 and a second skin characteristic associated with a second time 506. In this case, the first skin characteristic and the second skin characteristic may correspond (e.g., overlap) in areas of interest and skin elements.

[0042] 6 is a flow diagram illustrating an exemplary computer-implemented method for monitoring skin over time. At 602, an application may receive a first skin image at a first time (e.g., first time 502) and a second skin image at a second time (e.g., second time 506). The first and second times may be separated by a duration associated with a skin event (e.g., skin treatment, sun exposure, sleep, and / or any other event that may affect the appearance of the user's skin). At 604, the application may determine multiple skin characteristics (e.g., skin characteristics 504 and 508) from the first and second skin images.

[0043] At 606, the application may be configured to generate an analysis output (e.g., analysis output 510). For example, the application may be configured to generate the analysis output based on the analysis configuration and a plurality of skin characteristics. For example, as described herein, the analysis configuration may determine which of the skin characteristics to include in the analysis output. The analysis output may include a synoptic display of one or more of the plurality of skin characteristics. The analysis configuration may determine how the synoptic display is presented to the user and / or skin care professional.

[0044] At 608, the application may be configured to send the analysis output to one or more recipients. For example, the application may be configured to send the analysis output to a skin care specialist or other recipient (e.g., a healthcare provider such as a primary care physician). The skin care specialist may be selected based on user input or based on a selected analysis configuration. As another example, the application may be configured to send the analysis output to memory. For example, the application may be configured to save the analysis output as an image file. In some examples, the application may be configured to upload the image file to a patient portal associated with the skin care specialist. In some examples, a user may send the analysis output directly to one or more recipients. In some examples, the analysis output may be sent to one or more recipients through a telehealth operator (e.g., an operator associated with the skin care specialist).

[0045] 6 ) may be computer-implemented by any suitable structure, such as, for example, a structure suitable for receiving, processing, and outputting data. The exemplary method may be performed by a device or tool, such as, for example, a mobile device, a smartphone, a tablet, a computer, a laptop, an application, a processor, or any other suitable equipment, hardware, firmware, and / or software capable of performing the techniques described herein.

[0046] 7 is a block diagram illustrating an example generation of an analysis output based on longitudinal information regarding one or more skin characteristics. At 702, a camera may be used to capture one or more skin images. The skin images may be associated with a user of an application. The skin images may then be sent to a processor (e.g., associated with the application).

[0047] At 704, the processor may send the skin image to an AI system. The AI ​​system may be a standalone system or may be part of an application. The AI ​​system may be a machine learning system (e.g., an AI / ML system). Machine learning is a branch of AI that attempts to build computer systems that can learn from data without human intervention. These techniques may rely on the creation of analytical models that can be trained to recognize patterns in datasets, such as data collections. These models can be deployed to apply these patterns to data, such as biomarkers, to improve performance without further guidance.

[0048] The AI ​​system can compare the skin image to data points on the skin image from a database. For example, the database of skin images may include image-score pairs, where each image is scored by a dermatologist. In some embodiments, the data points may be associated with skin characteristics of the skin image. Based on the comparison, the AI ​​system can generate a score for the plurality of skin characteristics associated with the skin image.

[0049] A score may be generated based on a comparison of the captured skin image with data points from a database. For example, the skin image may be compared to data points representing high and low scores for a particular skin characteristic, and a score for the skin image may be generated based on the data point to which the skin image most closely corresponds.

[0050] The AI / ML system may be trained. Machine learning may be supervised (e.g., supervised learning) or unsupervised (e.g., unsupervised learning). For example, the AI ​​system may be trained to give a lower score when a large number of undesirable skin characteristics (e.g., acne or wrinkles) are present. The AI ​​system may be further trained based on previous scores given to a user. For example, the AI ​​system may determine that a current score should be increased relative to a score of a previous skin image if the previous skin image has relatively fewer undesirable skin characteristics and / or if the severity of those undesirable skin characteristics has decreased.

[0051] A supervised learning algorithm may create a mathematical model from training a dataset (e.g., training data). The training data may consist of a set of training examples. The training examples may include one or more inputs and one or more labeled outputs. The labeled outputs may serve as supervised feedback. In a mathematical model, the training examples may be represented by an array or vector, sometimes called a feature vector. The training data may be represented by rows of the feature vector that make up a matrix. Through iterative optimization of an objective function (e.g., a cost function), a supervised learning algorithm may learn a function (e.g., a prediction function) that can be used to predict outputs associated with one or more new inputs. A successfully trained prediction function may determine outputs for one or more inputs that may not have been part of the training data. Exemplary algorithms may include linear regression, logistic regression, and neural networks. Exemplary problems that can be solved by supervised learning algorithms may include classification problems, regression problems, etc.

[0052] An unsupervised learning algorithm may be trained on a dataset that includes inputs. The unsupervised learning algorithm may find structure in the data. The structure in the data may resemble groupings or clusters of data points. Thus, the algorithm may learn from training data that may be unlabeled. Instead of responding to supervised feedback, the unsupervised learning algorithm may identify commonalities in the training data and take action based on the presence or absence of such commonalities in each training example. Exemplary algorithms may include the Apriori algorithm, K-means, K-Nearest Neighbor (KNN), K-Medians, etc. Exemplary problems that can be solved by unsupervised learning algorithms may include clustering problems, anomaly / outlier detection problems, etc.

[0053] Machine learning can include reinforcement learning. Reinforcement learning can be an area of ​​machine learning that concerns how a software agent takes actions in an environment to maximize some notion of cumulative reward. Reinforcement learning algorithms may not assume knowledge of an exact mathematical model of the environment (e.g., represented by a Markov Decision Process (MDP)) and may be used when an exact mathematical model may not be feasible. Reinforcement learning algorithms may be used, for example, in autonomous vehicles or in learning to play games against human opponents.

[0054] Machine learning may be part of a technology platform called Cognitive Computing (CC), which may comprise various fields such as computer science and cognitive science. CC systems may be able to learn extensively, reason purposefully, and interact naturally with humans. Self-learning algorithms, which may use data mining, visual recognition, and / or natural language processing, may enable CC systems to solve problems and optimize human processes.

[0055] The output of a machine learning training process may be a model for predicting outcomes for new data sets. For example, a linear regression learning algorithm may have a cost function that can minimize the prediction error of a linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. When the minimum error is reached, the linear prediction function with the adjusted coefficients may be considered trained and constitute the model produced by the training process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) may include a hypothesis function represented by a network of layers of nodes with assigned biases and interconnected by weighted connections. The hypothesis function may be a nonlinear function (e.g., a highly nonlinear function) that may include a linear function and a nested logistic function with an outermost layer consisting of one or more logistic functions. The NN algorithm may include a cost function to minimize the classification error (e.g., by adjusting the biases and weights through a process of feedforward propagation and backpropagation). If a global minimum is reached, the optimized hypothesis function with layers of adjusted biases and weights may be considered trained and constitute the model that the training process produced.

[0056] Data collection may be performed for machine learning as the first stage of a machine learning lifecycle. Data collection may include steps such as identifying various data sources, collecting data from the data sources, and integrating the data. For example, to train a machine learning model for predicting surgical complications and / or postoperative recovery rates, a data source may be identified that includes preoperative data such as a patient's medical condition and biomarker measurement data. Such a data source may be a patient's electronic medical record (EMR), a computing system that stores the patient's preoperative biomarker measurement data, and / or other similar data stores. Data from the data sources may be retrieved and stored in a central data repository for further processing in the machine learning lifecycle. Data from the data sources may be linked (e.g., logically linked). The data may be accessed as if centrally stored. Surgical and / or postoperative data may be similarly identified and / or collected. The collected data may be integrated (e.g., combined). For example, a patient's preoperative medical record data, preoperative biomarker measurement data, preoperative data, surgical data, and / or postoperative data may be combined into a patient record. The patient record may be an EMR.

[0057] Data preparation may be performed for machine learning as another stage of the machine learning lifecycle. Data preparation may include data preprocessing steps such as data formatting, data cleaning, and data sampling. For example, the collected data may not be in a suitable data format for training a model. In one example, a patient's integrated data record of preoperative EMR record data, biomarker measurement data, surgical data, and postoperative data may be in a rational database. Such data records may be converted to a flat file format for model training. In one example, a patient's preoperative EMR data may include textual medical data such as the patient's emphysema diagnosis, preoperative treatment (e.g., chemotherapy, radiation, blood thinners), etc. The data may be mapped to numerical values ​​for model training. For example, a patient's integrated data record may include personal identifiers or other information that may identify the patient (e.g., age, employer, Body Mass Index (BMI), demographic information, etc.). Such specific data may be removed before model training. For example, specific data may be removed for privacy reasons. As another example, data may be removed because there is more available data than can be used for model training. In this case, a subset of the available data may be randomly sampled and selected for model training, and the rest may be discarded.

[0058] Data preparation may include data transformation procedures (e.g., after preprocessing), such as scaling and aggregation. For example, the preprocessed data may include data values ​​at a mixture of scales. These values ​​may be scaled up or down, e.g., to be between 0 and 1, for model training. For example, the preprocessed data may include data values ​​that become more meaningful when aggregated. In one example, a patient may have undergone multiple previous colorectal procedures. The total number of previous colorectal procedures may be more meaningful for training a model to predict surgical complications due to adhesions. In such a case, the records of the previous colorectal procedures may be aggregated into a total number for model training purposes.

[0059] Model training may be another aspect of the machine learning life cycle. The model training process described herein may depend on the machine learning algorithm used. A model may be considered properly trained after it has been trained, cross-validated, and tested. Thus, a dataset from the data preparation stage (e.g., an input dataset) may be divided into a training dataset (e.g., 60% of the input dataset), a validation dataset (e.g., 20% of the input dataset), and a test dataset (e.g., 20% of the input dataset). After the model is trained on the training dataset, it may be run on the validation dataset to reduce overfitting. If the model's accuracy decreases when run on the validation dataset while its accuracy is increasing, this may indicate an overfitting problem. The test dataset may be used to test the accuracy of the final model to determine whether it is ready for deployment or whether more training is required.

[0060] Model deployment can be another aspect of the machine learning lifecycle. Models can be deployed as part of a standalone computer program. Models can be deployed as part of a larger computing system. Models can be deployed with model performance parameters. The performance parameters can monitor model accuracy as the model makes predictions based on a production dataset. For example, the performance parameters can track false positives and false negatives of a classification model. The performance parameters can store the false positives and false negatives for further processing to improve the accuracy of the model.

[0061] Post-deployment model updates may be another aspect of the machine learning cycle. For example, a deployed model may be updated when false positives and / or false negatives are predicted on production data. In one example, for an MLP model deployed for classification, when a false positive occurs, the deployed MLP model may be updated to raise the probability cutoff for predicting a positive to reduce the false positives. In one example, for an MLP model deployed for classification, when a false negative occurs, the deployed MLP model may be updated to lower the probability cutoff for predicting a positive to reduce the false negatives. In one example, for an MLP model deployed for classification of surgical complications, when false positives and false negatives occur, the deployed MLP model may be updated to lower the probability cutoff for predicting a positive to reduce the false negatives (e.g., because predicting a false positive may be less important than a false negative).

[0062] The deployed model may be updated as more live production data becomes available as training data. In such cases, the deployed model may be further trained, validated, and tested using additional live production data. In one embodiment, the updated biases and weights of the further trained MLP model may update the biases and weights of the deployed MLP model. Those skilled in the art will recognize that post-deployment model updates need not be a one-time occurrence, but may occur as frequently as is suitable to improve the accuracy of the deployed model.

[0063] The plurality of skin characteristics and associated scores (e.g., raw score data) may be stored in any suitable form of memory at 706. As shown, the raw data may be extensive and, therefore, may contain more information than is useful to a user and / or skin care professional.

[0064] In some examples, the processor may retrieve raw data that may be filtered using one or more analysis configurations at 708. For example, a first analysis configuration may be associated with a first skin care specialist, and a second analysis configuration may be associated with a second, different skin care specialist.

[0065] As an example, a first skin care specialist may specialize in treating acne, and a second skin care specialist may specialize in treating wrinkles and fine lines. Thus, a first analysis configuration may filter the raw data so that information sent to the first skin care specialist includes only information related to treating acne, while a second analysis configuration may filter the raw data so that information sent to the second skin care specialist includes only information related to treating wrinkles and fine lines.

[0066] The analysis configuration may determine how the analysis output is formatted. For example, some skin care practitioners may prefer that skin images (and / or skin images with overlays, such as overlay 302 in FIG. 3 ) be included in the analysis output, while other skin care practitioners may prefer that only data be included in the analysis output and / or that the data be organized in a particular way.

[0067] The processor may use the key to indicate which of the analysis configurations to use to generate the analysis output. For example, the processor may receive user input indicating that the user is interested in reducing the severity of wrinkles on the user's forehead. The processor may use the user input to generate a key that corresponds to (or indicates) an analysis configuration that generates an analysis output with information that may help a skin care professional advise patients on the most effective way to reduce the appearance of wrinkles.

[0068] One or more of the analysis configurations may be used to generate one or more outputs for the user who captured the skin image. As described herein, in some cases, the analysis output generated for the skin care professional may have a different appearance and / or layout than the output provided to the user.

[0069] As shown, at 710, an analysis output may be generated and sent to a processor. The processor may export the analysis output at 712. In some examples, exporting the analysis output may involve converting the analysis output to another format (e.g., an image file format, a Portable Document Format (PDF), etc.). In some examples, exporting the analysis output may involve uploading the analysis output (or a converted version of the analysis output) to a medical portal associated with a skin care specialist. The processor may be configured to convert the analysis output into a file format compatible with the file requirements of the selected medical portal.

[0070] 8A and 8B show exemplary analysis outputs. The analysis output may include a timeline 802 showing a user's progress. For example, as shown, the timeline 802 may show an overall score associated with skin images taken by the user over time. The analysis output may include skin images 804 associated with each of the overall scores. The analysis output may enable a skin care professional to easily perform follow-ups with a patient. For example, the timeline 802 and skin images 804 may enable a skin care professional to (e.g., easily and quickly) analyze a patient's progress over time (e.g., from previous reports, treatments, and / or appointments with the skin care professional).

[0071] The analysis output may include a score 806 for each of a plurality of skin characteristics. The skin characteristic scores 806 may enable a skin care specialist to better analyze a user's skin concerns. For example, as shown, a user's primary skin concern may be clear skin. Thus, a skin care specialist may focus on the skin clarity score to better analyze how a particular treatment has affected the user's skin clarity score over time.

[0072] The analysis output may include notes 808 regarding skin variables associated with primary skin concerns and / or other skin characteristics. For example, the analysis output may include a list of information regarding skin variables such as the number of clogged pores, raised bumps, red, painful bumps, etc. detected in a given skin image 804.

[0073] 9 is a system diagram illustrating an exemplary computing device 100 that may be used to monitor skin over time. Computing device 100 may be, for example, a mobile phone, a tablet, or other such device. As shown in FIG. 9, computing device 100 may include, among other things, a processor 118, a transceiver 121, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 131, removable memory 132, a power source 134, a Global Positioning System (GPS) chipset 136, peripherals 138, a camera 140, an operating system 144, and / or a database 146. It will be understood that computing device 100 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0074] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other function that enables the computing device 100 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 121, which may be coupled to a transmit / receive element 122. While FIG. 9 depicts the processor 118 and the transceiver 121 as separate components, those skilled in the art will understand that the processor 118 and the transceiver 121 may be integrated together in an electronic package or chip.

[0075] In some embodiments, the transceiver 121 and the transmit / receive element 122 may be used to transmit the analysis output to one or more skin care specialists. The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station via the air interface 116. For example, the transmit / receive element 122 may be an antenna configured to transmit and / or receive radio frequency (RF) signals. The transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive infrared (IR), ultraviolet (UV), or visible light signals. The transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. Those skilled in the art will understand that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0076] Processor 118 of computing device 100 may be coupled to and may receive user input data from speaker / microphone 124, keypad 126, and / or display / touchpad 128 (e.g., a Liquid Crystal Display (LCD) display unit or an Organic Light-Emitting Diode (OLED) display unit). Processor 118 may output user data to speaker / microphone 124, keypad 126, and / or display / touchpad 128. Processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 131 and / or removable memory 132. Non-removable memory 131 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a Subscriber Identity Module (SIM) card, a memory stick, a Secure Digital (SD) memory card, etc. As shown, user data 142 may be stored in non-removable memory 131 and / or removable memory 132. User data may include raw data associated with skin characteristics, data regarding user preferences (e.g., for use in determining an analysis configuration), etc. Processor 118 may access information from and store data in memory not physically located on computing device 100, such as a server or home computer (not shown).

[0077] Operating system 144, which may be single-tasking or multi-tasking, may manage the functions of processor 118. For example, operating system 144 may handle input / output to and from processor 118, schedule tasks performed by processor 118, and / or perform other such management functions. Operating system 144 may manage non-removable memory 131 and / or removable memory 132. For example, operating system 144 may determine what types of memory are used to store different data sets.

[0078] Processor 118 may receive power from power source 134 and may be configured to distribute and / or control power to other components within computing device 100. Power source 134 may be any suitable device for providing power to computing device 100. For example, power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0079] Processor 118 may be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of computing device 100. In addition to, or instead of, information from GPS chipset 136, computing device 100 may receive location information from a base station over air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. Those skilled in the art will understand that computing device 100 may obtain location information by any suitable location-determination method while remaining consistent with an embodiment.

[0080] The processor 118 may be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a Frequency Modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc.

[0081] The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a barometer, a gesture sensor, a biometric sensor, a humidity sensor, and the like.

[0082] Processor 118 may be coupled to camera 140. In some embodiments, camera 140 may be used to capture a skin image of a user. Camera 140 may transmit the image to processor 118 (or an AI system as discussed with respect to FIG. 7 ) to determine a plurality of skin characteristics from the first skin image and the second skin image. The skin characteristics and associated scores may be determined by comparing the captured skin image to skin images in a database, such as database 146. Processor 118 may be configured to generate a report (e.g., analysis output) based on one or more of the skin characteristics and display the report (e.g., via display / touchpad 128) on a mobile application UI such as those shown in FIGS. 1-4 .

[0083] [Embodiment] (1) A device having a processor, The processor: receiving a first skin image at a first time and a second skin image at a second time, the first skin image and the second skin image being associated with a user, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic representation of one or more of the plurality of skin characteristics associated with the region of interest of the user; and transmitting the analysis output to one or more recipients, including a skin care specialist, wherein the overview display presents a historical summary of the one or more of the plurality of skin characteristics in a format suitable for the skin care specialist. (2) The device of embodiment 1, wherein the processor is further configured to receive user input identifying the skin care specialist, and the analysis configuration is selected from a plurality of analysis configurations based on the user input. (3) A device as described in embodiment 1, wherein one skin characteristic of the plurality of skin characteristics represents at least a skin element and a score associated with the skin element. (4) The device of embodiment 1, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time. (5) A device as described in embodiment 4, wherein the first skin characteristic and the second skin characteristic correspond in the area of ​​interest and the skin element.

[0084] (6) the synoptic representation is a first synoptic representation, and the processor: generating a second analysis output based on the second analysis configuration and the plurality of skin characteristics, the second analysis output including a second synoptic representation of one or more of the plurality of skin characteristics, the second synoptic representation being different from the first synoptic representation; 2. The device of claim 1, further configured to: transmit the second analysis output to one or more other recipients based on input from the user. (7) The device of embodiment 6, wherein the first overview display includes an overview display of a first subset of the plurality of skin characteristics, and the second overview display includes an overview display of a second subset of the plurality of skin characteristics that is different from the first subset. (8) The processor: saving the analysis output as an image file; 2. The device of claim 1, further configured to upload the image file to a patient portal associated with the skin care specialist. (9) The device of embodiment 1, wherein the overview display includes at least one of the first skin image and the second skin image with an overlay of markings indicative of one or more of the plurality of skin characteristics. (10) The device of embodiment 1, wherein the duration associated with the skin event is equal to or greater than a minimum amount of time between images, and the processor is further configured to prevent the user from capturing the second skin image if the minimum amount of time between images has not elapsed since receipt of the first skin image.

[0085] (11) The device of embodiment 1, wherein the analysis output includes information regarding a subset of the plurality of skin characteristics, the subset being associated with one or more skin characteristics indicated by the user. (12) A method comprising: receiving a first skin image at a first time and a second skin image at a second time, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic representation of one or more of the plurality of skin characteristics; determining, based on the analysis configuration, that one or more recipients of the analysis output include a skin care specialist; and transmitting the analysis output to the one or more recipients, the analysis output including a historical summary of the one or more of the plurality of skin characteristics over the duration based on the skin care specialist's preferences. (13) The method of embodiment 12, wherein one of the plurality of skin characteristics represents at least a skin element and a degree of the skin element. (14) The method of embodiment 12, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time. (15) A non-transitory computer-readable medium containing computer-executable instructions, the computer-executable instructions, when executed on a smartphone, causing the smartphone to: receiving a first skin image at a first time and a second skin image at a second time, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; and generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic display of the plurality of skin characteristics, the synoptic display configured to be received by a telemedicine server for presentation to a skin care specialist.

[0086] (16) The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on a smartphone, further cause the smartphone to receive user input, the user input including a skin characteristic of interest, and the analysis output is based on the skin characteristic of interest. (17) The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on the smartphone, further cause the smartphone to receive user input, the user input including a skin characteristic of interest, and the skin care specialist is identified based on expertise regarding the skin characteristic of interest. (18) The computer-executable instructions, when executed on a smartphone, further cause the smartphone to: saving the analysis output as an image file; and uploading the image file to a patient portal of the telemedicine server. (19) The computer-readable medium of embodiment 15, wherein one of the plurality of skin characteristics represents at least a skin element and a degree of the skin element. (20) The computer-readable medium of embodiment 15, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

Claims

1. 1. A device comprising a processor, The processor: receiving a first skin image at a first time and a second skin image at a second time, the first skin image and the second skin image being associated with a user, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic representation of one or more of the plurality of skin characteristics associated with the region of interest of the user; and transmitting the analysis output to one or more recipients, including a skin care specialist, wherein the overview display presents a historical summary of the one or more of the plurality of skin characteristics in a format suitable for the skin care specialist.

2. The device of claim 1 , wherein the processor is further configured to receive user input identifying the skin care specialist, and the analysis configuration is selected from a plurality of analysis configurations based on the user input.

3. The device of claim 1 , wherein a skin characteristic of the plurality of skin characteristics represents at least a skin element and a score associated with the skin element.

4. 10. The device of claim 1, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

5. The device of claim 4 , wherein the first skin characteristic and the second skin characteristic correspond in a region of interest and a skin element.

6. The synoptic representation is a first synoptic representation, and the processor: generating a second analysis output based on the second analysis configuration and the plurality of skin characteristics, the second analysis output including a second synoptic representation of one or more of the plurality of skin characteristics, the second synoptic representation being different from the first synoptic representation; The device of claim 1 , further configured to: transmit the second analysis output to one or more other recipients based on input from the user.

7. 7. The device of claim 6, wherein the first overview representation includes a overview representation of a first subset of the plurality of skin characteristics, and the second overview representation includes a overview representation of a second subset of the plurality of skin characteristics that is different from the first subset.

8. The processor: saving the analysis output as an image file; The device of claim 1 , further configured to: upload the image file to a patient portal associated with the skin care specialist.

9. The device of claim 1 , wherein the overview display includes at least one of the first skin image and the second skin image with an overlay of markings indicative of one or more of the plurality of skin characteristics.

10. 2. The device of claim 1, wherein the duration associated with the skin event is greater than or equal to a minimum amount of time between images, and the processor is further configured to prevent the user from capturing the second skin image if the minimum amount of time between images has not elapsed since receipt of the first skin image.

11. The device of claim 1 , wherein the analysis output includes information regarding a subset of the plurality of skin characteristics, the subset being associated with one or more skin characteristics indicated by the user.

12. 1. A method comprising: receiving a first skin image at a first time and a second skin image at a second time, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic representation of one or more of the plurality of skin characteristics; determining, based on the analysis configuration, that one or more recipients of the analysis output include a skin care specialist; and transmitting the analysis output to the one or more recipients, the analysis output including a historical summary of the one or more of the plurality of skin characteristics over the duration based on the skin care specialist's preferences.

13. The method of claim 12 , wherein one skin characteristic of the plurality of skin characteristics represents at least a skin element and a degree of the skin element.

14. 13. The method of claim 12, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

15. A non-transitory computer-readable medium containing computer-executable instructions that, when executed on a smartphone, cause the smartphone to: receiving a first skin image at a first time and a second skin image at a second time, the first time and the second time being separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; and generating an analysis output based on the analysis configuration and the plurality of skin characteristics, the analysis output including a synoptic display of the plurality of skin characteristics, the synoptic display configured to be received by a telemedicine server for presentation to a skin care specialist.

16. 16. The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on a smartphone, further cause the smartphone to receive user input, the user input including a skin characteristic of interest, and the analysis output is based on the skin characteristic of interest.

17. 16. The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on the smartphone, further cause the smartphone to receive user input, the user input including a skin characteristic of interest, and the skin care specialist is identified based on expertise regarding the skin characteristic of interest.

18. The computer-executable instructions, when executed on a smartphone, further cause the smartphone to: saving the analysis output as an image file; and uploading the image file to a patient portal of the telemedicine server.

19. The computer-readable medium of claim 15 , wherein one skin characteristic of the plurality of skin characteristics represents at least a skin element and a degree of the skin element.

20. 16. The computer-readable medium of claim 15, wherein the analysis output includes a historical summary of the plurality of skin characteristics over the duration, the historical summary representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

Images