Synergistic longitudinal skin care monitoring

Abstract

A tool enabling professional level monitoring at home may provide a patient with a better way to obtain expert suggestions about skin care. The tool may enable information about skin health to be provided to a skin care expert of the patient such that the most relevant information is emphasized and presented in an expected manner. An apparatus may include a processor configured to receive a first skin image at a first time and a second skin image at a second time. The first time and the second time may be separated by a duration associated with a skin event. A plurality of skin characteristics may be determined from the first skin image and the second skin image. The processor may be configured to generate an analysis output and send the analysis output to one or more recipients. The analysis output may include a summary representation of one or more skin characteristics of the plurality of skin characteristics.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 411,267, filed on September 29, 2022, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Proper skin care can help reduce signs of aging, reduce acne, and improve overall health. Although individuals have access to skin care information and tools, they often lack the expertise of a dermatologist. Dermatologists and other skin care specialists can provide expertise and tools for improving skin health. However, access to dermatologists is limited (e.g., based on appointment availability, high costs, etc.). In addition, because dermatologists cannot monitor patients continuously, the dermatologist's ability to assess the health of a patient's skin may be limited to the information available at the time of the appointment. Summary of the invention

[0004] Devices and / or tools that enable skin health monitoring at home or in an office environment can provide patients with better access to expert advice on skin care. In some embodiments, monitoring can occur once, occasionally, or continuously. In particular, the device and / or tool can enable information about skin health to be provided to a patient's skin care expert so that the most relevant information is emphasized and presented in a desired manner.

[0005] The device may include a processor. The processor may be configured to receive a first skin image at a first time and receive a second skin image at a second time. The first skin image and the second skin image may be associated with a user. And the first time and the second time may be separated by a duration associated with a skin event.

[0006] The processor may be configured to determine a plurality of skin characteristics from the first skin image and the second skin image. For example, a skin characteristic in the plurality of skin characteristics may at least represent a skin element and a score associated with the skin element.

[0007] The processor may be configured to generate an analysis output. The analysis output may be based on the analysis configuration and on the plurality of skin characteristics. The analysis output may include a summary representation of one or more skin characteristics of the plurality of skin characteristics.

[0008] The processor may be configured to send the analysis output to one or more recipients. For example, the one or more recipients may include a skin care professional. And the summary representation may present a historical summary of one or more skin characteristics in a form suitable for the skin care professional. For example, the analysis output may include a historical summary of the plurality of skin characteristics representing a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figures 1 to 4 is a user interface (UI) example illustrating techniques for monitoring skin health or care over time.

[0010] Figure 5 is an exemplary timeline illustrating the collection of longitudinal information regarding a plurality of skin characteristics, the generation of analysis output, and the transmission of the analysis output.

[0011] Figure 6 is a flow chart illustrating an exemplary computer-implemented method.

[0012] Figure 7 is a block diagram illustrating generation of analysis output based on longitudinal information regarding one or more skin characteristics.

[0013] Fig. 8A and Figure 8B An exemplary analysis output is shown.

[0014] Fig. 9 is a block diagram illustrating an exemplary computing device. DETAILED DESCRIPTION

[0015] Proper skin care can help reduce signs of aging, reduce acne, and improve overall health. Although individuals have access to skin care information and tools, they often lack the expertise of a dermatologist. Dermatologists and other skin care specialists can provide expertise and tools for improving skin health. However, access to dermatologists is limited (e.g., based on appointment availability, high costs, etc.). In addition, because dermatologists cannot monitor patients continuously, the dermatologist's ability to assess the health of a patient's skin may be limited to the information available at the time of the appointment.

[0016] Thus, patients may benefit from a device or tool that enables skin health monitoring at home. Such a device may provide patients with better access to expert advice on skin care. Such a device may be able to provide longitudinal information (e.g., information collected over a period of time) to a skin care expert (e.g., a dermatologist). Longitudinal information may enable a skin care expert to provide more insightful skin care advice to a patient.

[0017] Because conventional dermatologist imaging is limited to on-site devices, appropriate intermediate imaging is not possible. Conventional at-home imaging does not provide sufficient analysis and information useful to the dermatologist. For example, a dermatologist who receives a large number of intermediate images in typical color, digital form from a user does not have the technical means to organize, map, compare and / or analyze such images. The method disclosed herein solves this technical problem by analyzing the intermediate images and compressing the results into a summary representation of one or more skin characteristics of a plurality of skin characteristics associated with the user's area of ​​interest (e.g., for transmission to a dermatologist).

[0018] The device may include a processor configured to receive a first skin image and a second skin image and determine a plurality of skin characteristics based on the first skin image and the second skin image. In some examples, one or more of the plurality of skin characteristics may be associated with an area of ​​interest of a user. In some examples, the device may generate an analysis output including a summary representation presenting a historical summary of one or more of the plurality of skin characteristics. The summary representation may be in a form suitable for a skin care professional. For example, the summary representation may be in a form that will allow a skin care professional to quickly determine an appropriate treatment for a user.

[0019] Figure 1 is an exemplary user interface (UI) showing the home screen of a mobile application for monitoring skin health or care over time. One of ordinary skill in the art will appreciate that the UI described herein may be implemented in ways other than a mobile application. For example, the UI described herein may be implemented as a desktop (e.g., computer) application, a web-based application, etc. As used herein, the term "application" may be used broadly to describe all possible ways in which a UI may be accessed by a user.

[0020] Using a digital camera, a user may first acquire a skin image of the user's face. In some examples, acquiring the skin image may include acquiring a series of skin images. For example, acquiring the skin image may include acquiring a front skin image, a right skin image, and a left skin image of the user's face. In some examples, the skin image may be acquired via picture capture. In some examples, the skin image may be acquired via video capture. For example, a video may be time sliced ​​to acquire separate images from the video.

[0021] The application may include augmented reality (AR) guidance or other instructions to assist the user in proper image acquisition. For example, the AR guidance may include lines indicating where the user should position their face and eyes to obtain a proper image.

[0022] The application can then analyze the skin image. For example, the application can process the skin image into basic components to extract meaningful information. Image analysis can include tasks such as finding shapes, detecting edges, removing noise, counting objects, calculating statistics for texture analysis or image quality, etc.

[0023] Regional analysis can be used to extract statistics and interpret the data to determine the user's skin characteristics. For example, feature extraction can be used to extract / identify features from raw data of skin images. The application can use classification techniques to identify a set of categories (e.g., acne, wrinkles, etc.) and assign the identified features to their corresponding categories.

[0024] For example, image processing algorithms that can be used to identify acne may include thresholding, blob detection, Hough transform, template matching, and the like. Thresholding may include a method in which pixels with intensities above or below a certain threshold are classified as blob, such as acne. Blob detection algorithms, such as Laplacian of Gaussian (LoG) or Difference of Gaussian (DoG), can identify blob by determining areas with high intensity variations. The Hough transform may be suitable for detecting circular blobs or ellipses in an image. For example, such closed loops in an image may be evaluated as acne if they are below a threshold size. Template matching may include comparing one or more predefined acne templates to an area of ​​an image. In an example where a match is found, the confidence level of the match may indicate the presence of acne.

[0025] For example, image processing algorithms that can be used to identify wrinkles may include Hough transforms, edge detection, Radon transforms, line segment detectors (LSD), and the like. Hough transforms may include techniques for detecting lines in an image. Hough transforms may identify lines by converting them into points in a parameter space, where intersecting lines correspond to peaks. Edge detection algorithms such as the Canny edge detector may be used to find edges in an image, which may then be associated with wrinkles and facial lines. Radon transforms may be used to detect lines, particularly in skin images with complex pigmentation. Radon transforms may be used to calculate the sum of pixel values ​​along different angles to find wrinkles. LSD is an algorithm specifically designed to detect line segments in an image, and it may be used to identify wrinkles that vary in thickness when imaged. By passing an image through one or more of the algorithms, the image may be reduced to a plurality of identified skin elements, each characterized by, for example, category, location, size, and the like. The number and size of the elements normalized by the total analysis area may be used to assess the user's skin. The assessment may be calculated as an overall score for the user's skin health.

[0026] As shown, the UI may display an overall rating of the user's skin health (e.g., 7.4 out of 10) and a daily insights report 102. The UI may display a user profile icon 108, which, when selected, may allow the user to customize their profile, set goals, control permissions and notifications, etc. Each user may have a personal profile / account associated only with that user. The user may switch between profiles / accounts by entering the login credentials associated with the desired account. Thus, multiple users may have their own accounts with separate data (e.g., even if the users access the application via the same device).

[0027] The overall score may be depicted using a graphical representation 104, which may have a number of sections 106 representing a plurality of skin characteristics. A skin characteristic may be a feature or quality pertaining to the user's skin. For example, the plurality of skin characteristics may include clear skin, dark circles, wrinkles, fine lines, dark spots, redness, smoothness, etc.

[0028] The skin characteristic may represent at least one of a region of interest, a skin element, a magnitude, and / or a timestamp. For example, the region of interest may be any of a face, a nose, a chin, and / or a cheek. The skin element may be a variable associated with the skin characteristic. For example, a skin element may be a blackhead, a whitehead, a red area, or a raised bump. The magnitude may include a density measure of the skin element within the region of interest, and the timestamp may represent the time at which the skin image was received.

[0029] The application may be configured to determine a score for the skin property. In some examples, the total score may be an average of the skin property scores. In some examples, the total score may be a weighted average of the skin property scores.

[0030] The application may be configured to determine scores for variables associated with skin characteristics. These variables may be short-term variables. Variables may be associated with specific skin characteristics. That is, a combination of variables may be used to form a long-term skin characteristic score. For example, variables for a clean skin characteristic may include whether a user has blackheads, whiteheads, redness, raised bumps, and / or other such skin damage. In one example, a user may have very few blackheads on a given day, but many whiteheads and redness. The user may start a treatment routine to treat the whiteheads and redness. As a result, the clean skin characteristic score may increase over time as the number of whiteheads and redness decreases. Artificial intelligence and / or machine learning (AI / ML) skin analysis may be used to count the number of skin damage. The clean skin characteristic may include information related to pore size.

[0031] As another example, a variable for a dark spot skin property may include the number of moles and / or a rating of pigmentation over time. As another example, a variable for a fine line and wrinkle skin property may include the number of fine lines / wrinkles and / or the severity of fine lines / wrinkles.

[0032] In some examples, the daily insights report 102 may include recommendations for skin care routines and / or topical medications or products based on current skin properties. The application may recommend products based on responses to skin care questionnaires. As shown, the UI may include a business function 110 that can be used to purchase products. Products may be recommended based on the user's current skin property score. Products may be recommended based on the user's primary skin concerns or goals. In some examples, the user may use the device to capture images of products used in the routine. Then, when the user runs out, the UI may provide the user with a link to order more products. In some examples, the user may enter an ingredient (e.g., instead of a product) into the application. The application may list the potential benefits and / or risks of the ingredient. The application may list the products where the ingredient can be found. In some examples, the user may provide a time-dependent report on the user's skin care routine, and the routine may be periodically adjusted based on observed changes in skin properties.

[0033] The user can provide a self-assessment of current skin characteristics and can define skin goals (e.g., desired changes in one or more skin characteristics). In some examples, the user can identify long-term skin appearance concerns. AI / ML image analysis can be used to determine scores for subsets of skin characteristics and / or variables. The application can then compare the AI / ML derived scores with the user-defined subjective scores and adjust the scores based on the comparison.

[0034] In some examples, AI / ML image analysis can be used to identify a user's skin type (e.g., sensitive, oily, etc.). The user can provide a self-assessment of their skin type. The application can identify a list of ingredients that are potential allergens or irritants. The application can identify the list of ingredients based on the user's skin type.

[0035] In some examples, AI / ML image analysis can be used to determine skin dynamics. For example, AI / ML image analysis can obtain a first skin image in which the user is not smiling. AI / ML image analysis can obtain a second skin image in which the user is smiling. AI / ML image analysis can determine, for example, how a product reduces the appearance of wrinkles and / or fine lines when the user smiles.

[0036] The user may self-report lifestyle variables (e.g., amount of sleep, exercise, stress) that may be associated with skin characteristics. The UI may include indications of ambient weather conditions that may affect the appearance of the user's skin. For example, the UI may indicate that it is very hot outside and the user should focus on staying hydrated. As another example, the UI may indicate that the air quality is poor and the user should avoid going out.

[0037] like Figure 2 As shown, the UI may display a skin image captured by a user with an analysis overlay 202. The analysis overlay 202 may include markers (e.g., lines, circles, and / or dots) indicating the locations of features of the user's skin that contribute to each of the skin characteristic scores. For example, line 204 may be a first color and may indicate wrinkles on the user's forehead. Similarly, circle 206 may be a second color to indicate the presence of redness on the user's skin, and dot 208 may be a third color to indicate the presence of blackheads.

[0038] like Figure 3 As shown, the UI may display a report of progress made over a period of time (e.g., within a day, week, or month). The report may include a progress bar 302 for each skin characteristic. In some examples, the UI may display a progress bar for the user's primary goal (e.g., clearer skin, fewer wrinkles). The report may include an assessment 304 of the skin characteristic, which indicates an improvement or decrease in the user's skin health. For example, the assessment 304 may state "Your skin looks 5% clearer than last month," or "Your skin is 18% smoother than last month." The UI may display a button 306 to allow the user to take a new skin image. The new skin image may then be analyzed, and the data from the analysis may be incorporated into the report.

[0039] like Figure 4 As shown, the UI may display a weekly progress graph 402 showing the score over time. For example, the weekly progress graph 402 may be a timeline graph, a bar graph, or any other graph suitable for illustrating data over time. The progress graph 402 may show progress for a specific skin property (e.g., smoothness), for a subgrouping of skin properties, or for a combination of skin properties (e.g., represented by an overall score). Reports of skin property scores, graphical representations of skin properties, and AR / ML image presentations of images associated with skin properties may be presented in a time series.

[0040] The UI may display tips for improving specific skin characteristics 404. For example, the UI may display "For smoother skin, boost your collagen" or "To reduce the appearance of wrinkles, limit exposure to the sun."

[0041] A user may request that a report containing skin characteristic information collected over time be sent to one or more recipients. For example, a user may request that skin characteristics be sent to a skin care specialist (e.g., a dermatologist or beautician). Figure 5 As shown, a user may take a first skin image at a first time 502. The application may be configured to receive the first skin image and determine a plurality of skin characteristics from the first skin image 504. Some time later, the user may take a second skin image at time 506, and the application may receive the second skin image and determine an additional plurality of skin characteristics 508 from the second skin image.

[0042] The user may request the application to combine one or more of the plurality of skin characteristics 504 and 508 to generate analysis output 510. The user may request the application to send analysis output 510 to a skin care professional (eg, a dermatologist or esthetician).

[0043] The content of the analysis output 510 may be based on an analysis configuration. The analysis configuration may be a default analysis configuration. The analysis configuration may depend on settings selected by a user. The analysis configuration may depend on settings selected by a skin care professional. For example, based on the analysis configuration, the analysis output 510 may include information associated with a particular subgrouping of skin properties (e.g., a report may only include information related to clear skin and smoothness properties).

[0044] The analysis configuration may not affect how the analysis of the skin image is performed. The analysis configuration may specify what the analysis output 510 looks like (e.g., the content and / or layout of the analysis output 510). For example, the analysis configuration may indicate that a particular user prefers that the analysis output include raw data in a tabular format, a graphical depiction of the analysis results, an AI overlay of identified skin characteristics, a skin score, etc.

[0045] In some cases, the application may be configured to receive user input identifying a skin care specialist. The selected analysis configuration may be selected from a plurality of analysis configurations based on the user input. The user input may include an area of ​​concern to the user (e.g., a desire to reduce acne). The application may be configured to identify an appropriate analysis configuration and / or skin care specialist based on the area of ​​concern. In some examples, the selected analysis configuration may be based on a skin care specialist or a user's preference. In some examples, the analysis output 510 may include an image linked to the skin characteristics included in the analysis output 510.

[0046] The analysis output 510 may have a different appearance and / or layout than a report provided to the user (e.g., via the UI). For example, the report provided to the user may include a list of topics for which the user consulted a skin care expert (e.g., techniques for the user to better hydrate the user's skin). For example, the report provided to the user may include a score of skin characteristics and tips for improving the skin characteristics. In some examples, the report provided to the user may include information about the underlying science and / or reasons for a low skin score.

[0047] For a report provided to a skin care professional, the analysis output 510 can be a historical summary of the data collected over a duration of time, which includes more detailed information that the skin care professional can use to diagnose and treat the user. In some examples, the historical summary can represent the difference between a first skin characteristic associated with a first time 502 and a second skin characteristic associated with a second time 506. In this case, the first skin characteristic and the second skin characteristic can correspond (e.g., overlay) in the region of interest and the skin element.

[0048] Figure 6 is a flow chart illustrating an exemplary computer-implemented method for monitoring skin over time. At 602, an application may receive a first skin image at a first time (e.g., first time 502) and a second skin image at a second time (e.g., second time 506). The first time and the second time may be separated by a duration associated with a skin event (e.g., skin treatment, exposure to sunlight, sleep, and / or any other event that may affect the appearance of a user's skin). At 604, the application may determine a plurality of skin characteristics (e.g., skin characteristics 504 and 508) from the first skin image and the second skin image.

[0049] At 606, the application can be configured to generate an analysis output (e.g., analysis output 510). For example, the application can be configured to generate the analysis output based on the analysis configuration and the plurality of skin characteristics. For example, as explained herein, the analysis configuration can determine which skin characteristics are to be included in the analysis output. The analysis output can include summary representations of one or more skin characteristics of the plurality of skin characteristics. The analysis configuration can determine how the summary representations are presented to the user and / or the skin care professional.

[0050] At 608, the application may be configured to send the analysis output to one or more recipients. For example, the application may be configured to send the analysis output to a skin care specialist or other recipient (e.g., a health care provider, such as a primary care physician). The skin care specialist may be selected based on user input or based on the selected analysis configuration. As another example, the application may be configured to send the analysis output to a memory. For example, the application may be configured to save the analysis output as an image file. In some examples, the application may be configured to upload the image file to a patient portal associated with the skin care specialist. In some examples, the user may send the analysis output directly to one or more recipients. In some examples, the analysis output may be sent to one or more recipients via a telemedicine operator (e.g., an operator associated with a skin care specialist).

[0051] Exemplary methods (e.g., such as Figure 6 The exemplary method may be performed by a device or tool, such as a mobile device, a smart phone, a tablet, a computer, a laptop, an application, a processor, or any other suitable device, hardware, firmware, and / or software capable of performing the techniques described herein.

[0052] Figure 7 is a block diagram illustrating an exemplary generation of an analysis output based on longitudinal information about one or more skin characteristics. At 702, a camera can be used to capture one or more skin images. The skin images can be associated with a user of an application. The skin images can then be transmitted to a processor (e.g., associated with an application).

[0053] At 704, the processor may transmit the skin image to an AI system. The AI ​​system may be a standalone system or may be part of an application. The AI ​​system may be a machine learning system (e.g., an AI / ML system). Machine learning is a branch of artificial intelligence that seeks to build computer systems that can learn from data without human intervention. These techniques may rely on the creation of analytical models that can be trained to recognize patterns within data sets (such as data sets). These models may be deployed to apply these patterns to data, such as biomarkers, to improve performance without further guidance.

[0054] The AI ​​system may compare the skin image to data points from a database of skin images. For example, the database of skin images may include image-score pairs, where each image has been scored by a dermatologist. In some examples, the data points may be associated with skin characteristics of the skin image. Based on the comparison, the AI ​​system may generate scores for a plurality of skin characteristics associated with the skin image.

[0055] The score may be generated based on a comparison of the captured skin image with data points from a database. For example, a skin image may be compared with high and low scoring data points representing certain skin characteristics, and a score for the skin image may be generated based on the data points to which the skin image most closely corresponds.

[0056] The AI / ML system may be trained. Machine learning may be supervised (e.g., supervised learning) or unsupervised (e.g., unsupervised learning). For example, if there are a large number of undesirable skin characteristics (e.g., acne or wrinkles), the AI ​​system may be trained to give a lower score. The AI ​​system may be further trained based on scores previously given to the user. For example, if there are relatively fewer undesirable skin characteristics than in previous skin images and / or if the severity of those undesirable skin characteristics has been reduced, the AI ​​system may determine that the current score should be increased relative to the score of the previous skin image.

[0057] Supervised learning algorithms can create mathematical models from training data sets (e.g., training data). Training data can be composed of a set of training examples. Training examples can include one or more inputs and one or more labeled outputs. The labeled outputs can be used as supervised feedback. In the mathematical model, the training examples can be represented by arrays or vectors (sometimes referred to as eigenvectors). The training data can be represented by the rows of the eigenvectors that constitute the matrix. Through iterative optimization of the objective function (e.g., cost function), the supervised learning algorithm can learn a function (e.g., prediction function) that can be used to predict the output associated with one or more new inputs. The appropriately trained prediction function can determine the output of one or more inputs that may not be part of the training data. Exemplary algorithms may include linear regression, logistic regression, and neural networks. Exemplary problems that can be solved by supervised learning algorithms may include classification, regression problems, etc.

[0058] An unsupervised learning algorithm may be trained on a data set including an input. An unsupervised learning algorithm may find structures in the data. The structures in the data may be similar to groupings or clusterings of data points. In this way, the algorithm can learn from training data that may not be labeled. Unresponsive to supervised feedback, an unsupervised learning algorithm may identify commonalities in the training data and may react based on the presence or absence of such commonalities in each training example. Exemplary algorithms may include a priori algorithms, K-means, K-nearest neighbors (KNN), K-median, and the like. Exemplary problems that may be solved by unsupervised learning algorithms may include clustering problems, anomaly / outlier detection problems, and the like.

[0059] Machine learning may include reinforcement learning. Reinforcement learning may be a field of machine learning that deals with the concept of how a software agent takes actions in an environment to maximize a cumulative reward. Reinforcement learning algorithms may not assume knowledge of an exact mathematical model of the environment (e.g., represented by a Markov decision process (MDP)) and may be used when an exact model is not feasible. For example, reinforcement learning algorithms may be used for self-driving vehicles or for learning to play games with human opponents.

[0060] Machine learning can be part of a technology platform called cognitive computing (CC), which can comprise various disciplines such as computer science and cognitive science. CC systems are capable of learning at scale, reasoning purposefully, and interacting naturally with humans. Through self-teaching algorithms that can use data mining, visual recognition, and / or natural language processing, CC systems are able to solve problems and optimize human processing.

[0061] The output of the training process of machine learning can be a model for predicting the results of a new data set. For example, a linear regression learning algorithm can be a cost function that can minimize the prediction error of a linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. If the minimum error is reached, the linear prediction function with the adjusted coefficients can be considered to be trained and constitutes a model that has generated a training process. For example, a neural network (NN) algorithm (e.g., a multilayer perceptron (MLP)) for classification may include a hypothesis function represented by a node layer network, which is assigned a bias and interconnected with a weight connection. The hypothesis function may be a nonlinear function (e.g., a highly nonlinear function), which may include a linear function and a logic function nested together with the outermost layer consisting of one or more logic functions. The NN algorithm may include a cost function to minimize the classification error (e.g., by adjusting the bias and weight via a process of forward propagation and backward propagation). If the global minimum is reached, the optimized hypothesis function of the layer with its adjusted bias and weight can be considered to be trained and constitutes a model that has generated a training process.

[0062] As the first stage of the machine learning life cycle, data collection can be performed for machine learning. Data collection may include steps such as identifying various data sources, collecting data from data sources, integrating data, etc. For example, in order to train a machine learning model for predicting surgical complications and / or postoperative recovery rates, a data source containing preoperative data (such as a patient's medical condition and biomarker measurement data) may be identified. Such data sources may be a patient's electronic medical record (EMR), a computing system storing a patient's preoperative biomarker measurement data, and / or other similar data storage. Data from the data source may be retrieved and stored in a central location for further processing in the machine learning life cycle. Data from the data source may be linked (e.g., logically linked). Data may be accessed as if stored centrally. Surgical data and / or postoperative data may be similarly identified and / or collected. The collected data may be integrated (e.g., combined). For example, a patient's preoperative medical record data, preoperative biomarker measurement data, preoperative data, surgical data, and / or postoperative data may be combined into a patient's record. A patient's record may be an EMR.

[0063] As another stage of the machine learning life cycle, data preparation can be performed for machine learning. Data preparation may include data preprocessing steps, such as data formatting, data cleaning, and data sampling. For example, the collected data may not be in a data format suitable for training the model. In one example, the integrated data record of the patient's preoperative EMR record data and biomarker measurement data, surgical data, and postoperative data may be in a reasonable database. Such data records can be converted into a flat file format for model training. In one example, the patient's preoperative EMR data may include medical data in text format, such as the patient's emphysema diagnosis, preoperative treatment (e.g., chemotherapy, radiotherapy, blood dilution), etc. The data can be mapped to a numerical value for model training. For example, the patient's integrated data record may include personal identification information or other information that can identify the patient (e.g., age, work unit, body mass index (BMI), demographic information, etc.). Such identification data can be removed before model training. For example, identification data can be removed for privacy reasons. As another example, data can be removed because there is more available data than data available for model training. In this case, a subset of the available data can be randomly sampled and selected for model training, and the remaining data can be discarded.

[0064] Data preparation may include data transformation processes (e.g., after preprocessing), such as scaling and aggregation. For example, the preprocessed data may include data values ​​in a scaled mixture. These values ​​may be scaled up or down, such as between 0 and 1, for model training. For example, the preprocessed data may include data values ​​that carry more meaning when aggregated. In one example, there may be multiple previous colorectal surgeries that a patient has undergone. For training a model to predict surgical complications due to adhesions, the total number of previous colorectal surgeries may be more meaningful. In this case, for the purpose of model training, the records of previous colorectal surgeries may be aggregated into a total number.

[0065] Model training can be another aspect of the machine learning life cycle. The model training process as described herein may depend on the machine learning algorithm used. After the model has been trained, cross-validated, and tested, the model can be considered to be properly trained. Therefore, the data set (e.g., input data set) from the data preparation phase can be divided into a training data set (e.g., 60% of the input data set), a validation data set (e.g., 20% of the input data set), and a test data set (e.g., 20% of the input data set). After training the model on the training data set, the model can be run on the validation data set to reduce overfitting. If the accuracy of the model decreases when running on the validation data set while the accuracy of the model has been increasing, this may indicate an overfitting problem. The test data set can be used to test the accuracy of the final model to determine whether it is ready for deployment or whether more training is needed.

[0066] Model deployment can be another aspect of the machine learning lifecycle. Models can be deployed as part of a standalone computer program. Models can be deployed as part of a larger computing system. Models can be deployed using model performance parameters. Performance parameters can monitor model accuracy when the model makes predictions based on a dataset in production. For example, performance parameters can track false positives and false negatives for a classification model. Performance parameters can store false positives and false negatives for further processing to improve the accuracy of the model.

[0067] Post-deployment model updates can be another aspect of the machine learning cycle. For example, when false positives and / or false negatives are predicted on the generated data, the deployed model can be updated. In one example, for the deployed MLP model for classification, when a false positive occurs, the deployed MLP model can be updated to increase the probability cutoff for predicting positives, thereby reducing false positives. In one example, for the deployed MLP model for classification, when a false negative occurs, the deployed MLP model can be updated to reduce the probability cutoff for predicting positives, thereby reducing false negatives. In one example, for the deployed MLP model for surgical complication classification, when false positives and false negatives occur, the deployed MLP model can be updated to reduce the probability cutoff for predicting positives, thereby reducing false negatives (e.g., because predicting false positives is less critical than predicting false negatives).

[0068] As more field-generated data becomes available as training data, the deployed model may be updated. In such cases, the deployed model may be further trained, validated, and tested using the additional field-generated data. In one example, the updated biases and weights of the further trained MLP model may update the biases and weights of the deployed MLP model. Those skilled in the art will recognize that post-deployment model updates may not occur all at once, and may occur at a frequency suitable to improve the accuracy of the deployed model.

[0069] At 706, a plurality of skin characteristics and associated scores (e.g., raw score data) may be stored in any suitable form of memory. As shown, the raw data may be extensive and thus may include more information than is useful to the user and / or skin care professional.

[0070] In some examples, the processor may retrieve raw data, which may be filtered using one or more analysis configurations at 708. For example, a first analysis configuration may be associated with a first skin care specialist, and a second analysis configuration may be associated with a second, different skin care specialist.

[0071] For example, a first skin care specialist may specialize in treating acne, while a second skin care specialist may specialize in treating wrinkles and fine lines. Thus, a first analysis configuration may filter the raw data such that the information transmitted to the first skin care specialist includes only information related to treating acne, while a second analysis configuration may filter the raw data such that the information transmitted to the second skin care specialist includes only information related to treating wrinkles and fine lines.

[0072] The analysis configuration determines how the analysis output is formatted. For example, some skin care experts may prefer to format skin images (and / or images with features such as Figure 3In some embodiments, the skin care professional may include only the skin image of the overlay layer 302 in the analysis output, while other skin care professionals may prefer to include only the data in the analysis output and / or organize the data in a particular manner.

[0073] The processor may use the keys to indicate which analysis configuration to use to generate the analysis output. For example, the processor may receive user input indicating that the user is interested in reducing the severity of wrinkles on the user's forehead. The processor may use the user input to generate a key corresponding to (or indicating) an analysis configuration that will generate an analysis output with information that can help a skin care professional advise a patient on the most effective way to reduce the appearance of wrinkles.

[0074] One or more of the analysis configurations may be used to generate one or more outputs for a user capturing a skin image. As explained herein, in some cases, the analysis output generated for a skin care professional may have a different appearance and / or layout than the output provided to the user.

[0075] As shown, at 710, an analysis output may be generated and transmitted to a processor. At 712, the processor may export the analysis output. In some examples, exporting the analysis output may involve converting the analysis output to another format (e.g., an image file format, a portable document format (PDF), etc.). In some examples, exporting the analysis output may involve uploading the analysis output (or a converted version of the analysis output) to a health care portal associated with the skin care professional. The processor may be configured to convert the analysis output to a file format compatible with the file requirements of the selected health care portal.

[0076] Fig. 8A and Figure 8B An exemplary analysis output is shown. The analysis output may include a timeline 802 showing the user's progress. For example, as shown, the timeline 802 may show total scores associated with skin images taken by the user over time. The analysis output may include skin images 804 associated with each total score. The analysis output may allow a skin care professional to easily follow up with a patient. For example, the timeline 802 and skin images 804 may allow a skin care professional to analyze (e.g., easily and quickly) a patient's progress over time (e.g., since a previous report, treatment, and / or appointment with a skin care professional).

[0077] The analysis output may include a score 806 for each of the plurality of skin characteristics. The skin characteristic score 806 may allow the skin care professional to better analyze the user's skin concerns. For example, as shown, the user's primary skin concern may be clear skin. Therefore, the skin care professional may focus on the clear skin score to better analyze how a particular treatment affects the user's clear skin score over time.

[0078] The analysis output may include annotations 808 about skin variables associated with primary skin concerns and / or other skin characteristics. For example, the analysis output may include a list of information about skin variables, such as the number of blocked pores, raised bumps, red painful bumps, etc. detected in a given skin image 804.

[0079] Fig. 9 is a system diagram illustrating an exemplary computing device 100 that can be used to monitor skin over time. The computing device 100 can be, for example, a cellular phone, a tablet computer, or other such device. Fig. 9 As shown, computing device 100 may include processor 118, transceiver 121, transmit / receive element 122, speaker / microphone 124, keyboard 126, display / touchpad 128, non-removable memory 131, removable memory 132, power supply 134, global positioning system (GPS) chipset 136, peripherals 138, camera 140, operating system 144 and / or database 146, etc. It should be understood that computing device 100 may include any sub-combination of the foregoing elements while remaining consistent with the embodiment.

[0080] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other function that enables the computing device 100 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 121, which may be coupled to the transmit / receive element 122. Although Fig. 9 The processor 118 and the transceiver 121 are depicted as separate components, but one of ordinary skill in the art will appreciate that the processor 118 and the transceiver 121 may be integrated together in an electronic package or chip.

[0081] In some examples, transceiver 121 and send / receive element 122 can be used to send the analysis output to one or more skin care experts. Send / receive element 122 can be configured to send a signal to the base station or receive a signal from the base station through air interface 116. For example, send / receive element 122 can be an antenna configured to send and / or receive a radio frequency (RF) signal. Send / receive element 122 can be a transmitter / detector configured to, for example, send and / or receive infrared (IR), ultraviolet (UV) or visible light signals. Send / receive element 122 can be configured to send and / or receive both RF and optical signals. Those of ordinary skill in the art will appreciate that send / receive element 122 can be configured to send and / or receive any combination of wireless signals.

[0082] The processor 118 of the computing device 100 may be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from them. The processor 118 may output user data to the speaker / microphone 124, the keyboard 126, and / or the display / touchpad 128. The processor 118 may access information from any type of suitable memory (such as a non-removable memory 131 and / or a removable memory 132) and store data therein. The non-removable memory 131 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. As shown, the user data 142 may be stored in the non-removable memory 131 and / or the removable memory 132. The user data may include raw data associated with skin characteristics, data about the user's preferences (e.g., for determining an analysis configuration), etc. Processor 118 may access information from, and store data in, memory that is not physically located on computing device 100 , such as on a server or a home computer (not shown).

[0083] The operating system 144 may be single-tasking or multi-tasking and may manage the functions of the processor 118. For example, the operating system 144 may process input and output to and from the processor 118, schedule tasks to be performed by the processor 118, and / or perform other such management functions. The operating system 144 may manage the non-removable memory 131 and / or the removable memory 132. For example, the operating system 144 may determine which type of memory will be used to store different data sets.

[0084] The processor 118 may receive power from the power supply 134 and may be configured to distribute the power to and / or control the power to other components in the computing device 100. The power supply 134 may be any suitable device for powering the computing device 100. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0085] Processor 118 may be coupled to GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of computing device 100. In addition to or in lieu of the information from GPS chipset 136, computing device 100 may receive location information from a base station over air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. Those of ordinary skill in the art will appreciate that computing device 100 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0086] The processor 118 may be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, frequency modulation (FM) radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc.

[0087] Peripheral device 1 38 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a barometer, a posture sensor, a biometric sensor, a humidity sensor, etc.

[0088] Processor 118 may be coupled to camera 140. In some examples, camera 140 may be used to capture an image of the user's skin. Camera 140 may transmit the image to processor 118 (or other device such as a computer). Figure 7The AI ​​system discussed herein) can be used to determine a plurality of skin characteristics from the first skin image and the second skin image. The skin characteristics and associated scores can be determined by comparing the captured skin image with skin images in a database (such as database 146). The processor 118 can be configured to generate a report (e.g., analysis output) based on one or more of the skin characteristics and display the report on the mobile application UI (e.g., via the display / touchpad 128), such as on a mobile application UI. Figures 1 to 4 Those shown in .

Claims

1. A device comprising: A processor, the processor being configured to: receiving a first skin image at a first time and receiving a second skin image at a second time, wherein the first skin image and the second skin image are associated with a user, and wherein the first time and the second time are separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and based on the plurality of skin characteristics, the analysis output comprising a summary representation of one or more skin characteristics of the plurality of skin characteristics associated with the area of ​​interest of the user; as well as The analysis output is sent to one or more recipients including a skin care professional, wherein the summary representation presents a historical summary of the one or more skin characteristics of the plurality of skin characteristics in a form suitable for the skin care professional.

2. The apparatus of claim 1, wherein the processor is further configured to receive user input identifying the skin care professional, and wherein the analysis configuration is selected from a plurality of analysis configurations based on the user input. 3 . The apparatus according to claim 1 , wherein a skin property among the plurality of skin properties represents at least a skin element and a score associated with the skin element.

4. The apparatus of claim 1, wherein the analysis output comprises a historical summary of the plurality of skin characteristics over the duration, wherein the historical summary represents a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time. The apparatus according to claim 4 , wherein the first skin property and the second skin property correspond in a region of interest and a skin element.

6. The apparatus of claim 1 , wherein the summary representation is a first summary representation, and wherein the processor is further configured to: generating a second analysis output based on the second analysis configuration and based on the plurality of skin characteristics, the second analysis output comprising a second summary representation of one or more of the plurality of skin characteristics that is different from the first summary representation; and The second analysis output is sent to one or more other recipients based on input from the user.

7. The apparatus of claim 6, wherein the first summary representation comprises a summary representation of a first subset of the plurality of skin properties, and wherein the second summary representation comprises a summary representation of a second subset of the plurality of skin properties different from the first subset.

8. The apparatus of claim 1, wherein the processor is further configured to: Saving the analysis output as an image file; and The image file is uploaded to a patient portal associated with the skin care specialist.

9. An apparatus according to claim 1, wherein the summary representation includes at least one of the first skin image and the second skin image, and at least one of the first skin image and the second skin image has an overlay layer with a mark showing one or more skin characteristics of the plurality of skin characteristics.

10. An apparatus according to claim 1, wherein the duration associated with the skin event is greater than or equal to a minimum amount of time between images, and wherein the processor is further configured to prevent the user from capturing the second skin image if the minimum amount of time between images has not passed since the first skin image was received.

11. The apparatus of claim 1, wherein the analysis output includes information about a subset of the plurality of skin characteristics, the subset being associated with one or more skin characteristics indicated by the user.

12. A method comprising: receiving a first skin image at a first time and receiving a second skin image at a second time, wherein the first time and the second time are separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; generating an analysis output based on the analysis configuration and based on the plurality of skin characteristics, the analysis output comprising a summary representation of one or more skin characteristics of the plurality of skin characteristics; determining, based on the analysis configuration, that the one or more recipients of the analysis output include a skin care professional; as well as The analysis output is sent to the one or more recipients, wherein the analysis output includes a historical summary of the one or more skin characteristics of the plurality of skin characteristics based on the skin care professional's preferences over the duration.

13. The method according to claim 12, wherein the skin properties of the plurality of skin properties represent at least a skin element and a magnitude of the skin element.

14. The method of claim 12, wherein the analysis output comprises a historical summary of the plurality of skin characteristics over the duration, wherein the historical summary represents a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

15. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed on a smartphone, cause the smartphone to: receiving a first skin image at a first time and receiving a second skin image at a second time, wherein the first time and the second time are separated by a duration associated with a skin event; determining a plurality of skin characteristics from the first skin image and the second skin image; as well as An analysis output including a summary representation of the plurality of skin characteristics is generated based on the analysis configuration and based on the plurality of skin characteristics, wherein the summary representation is configured to be received by the telemedicine server for presentation to a skin care professional.

16. The computer-readable medium of claim 15, wherein the computer-executable instructions when executed on a smart phone further cause the smart phone to receive user input, wherein the user input includes a skin characteristic of interest, and wherein the analysis output is based on the skin characteristic of interest.

17. The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on a smart phone, further cause the smart phone to receive user input, wherein the user input includes a skin characteristic of concern, and wherein the skin care expert is identified based on expertise regarding the skin characteristic of concern.

18. The computer-readable medium of claim 15, wherein the computer-executable instructions, when executed on a smartphone, further cause the smartphone to: Saving the analysis output as an image file; and The image file is uploaded to a patient portal of the telemedicine server.

19. The computer-readable medium of claim 15, wherein a skin property of the plurality of skin properties represents at least a skin element and a magnitude of the skin element.

20. The computer-readable medium of claim 15, wherein the analysis output comprises a historical summary of the plurality of skin characteristics over the duration, wherein the historical summary represents a difference between a first skin characteristic associated with a first time and a second skin characteristic associated with a second time.

Images