Skin diagnosis system and method based on image analysis using deep learning

The skin diagnosis system uses deep learning and anonymization techniques to provide detailed, quantitative skin analysis across diverse imaging devices, addressing accuracy and privacy issues in conventional methods.

JP2024540858APending Publication Date: 2024-11-06AMOREPACIFIC CORP

Patent Information

Application Number
JP2024521878
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-10-13
Filing Date
2022-10-14
Publication Date
2024-11-06

AI Technical Summary

Technical Problem

Conventional skin diagnosis methods lack accuracy in measuring various skin items, fail to provide detailed quantitative data, and do not adequately protect personal information, while being limited to specific imaging devices and environments.

Method used

A skin diagnosis system using deep learning that includes a face recognition model for anonymization and multiple neural network models to analyze symptoms like wrinkles, pigmentation, pores, and erythema, providing quantitative data and visualizing results while protecting personal information.

Benefits of technology

Enables accurate, quantitative analysis of skin conditions with personalized product recommendations, while ensuring data privacy and flexibility across various imaging devices, suitable for long-term skin research and cosmetic evaluations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024540858000001_ABST
    Figure 2024540858000001_ABST
Patent Text Reader

Abstract

The embodiment relates to a skin diagnosis system and method based on image analysis using deep learning, which includes a face recognition model that derives shape or position information of a facial structure by recognizing feature points that can identify an individual from an acquired face image, an anonymization model that anonymizes the face image based on the shape or position information of the facial structure so that personal information of the analysis target cannot be identified, and a plurality of artificial neural network models for each of at least one of the diagnosis items of wrinkles, pigmentation, pores, erythema, and aging.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present application relates to a skin diagnosis system and method based on image analysis using deep learning.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority based on Korean Patent Application No. 10-2021-0136568 filed on October 14, 2021, and Korean Patent Application No. 10-2022-0131817 filed on October 13, 2022, the entire specifications of which are incorporated by reference into this application. [Background technology]

[0003] Skin diagnosis is actively used not only in medical applications but also in the cosmetics industry and related research fields. With the development of online markets and personalized services, there is a growing demand for methods that can perform skin diagnosis more easily and accurately. On the other hand, various conventional skin diagnosis and measurement methods are highly accurate, but the measurement items are limited and require expensive equipment. Conventional skin diagnosis methods based on image analysis are relatively inexpensive and can measure various items with a single image, but they have low accuracy and it is difficult or expensive to obtain quantitative detailed data.

[0004] Furthermore, in order to avoid the difficulty of achieving accuracy and the problem of data labeling, conventional skin diagnosis methods either do not provide the detection location and quantitative data of symptoms, or are implemented by simply grading the symptoms, thereby not providing the actual location and quantitative information of symptoms for each item.

[0005] In addition, conventional prior art does not include a procedure for removing personal identification information contained in facial images used for skin diagnosis, and has the problem of insufficient procedures for managing and protecting personal information during analysis processing and data collection.

[0006] Other prior art technologies also only define the concept of skin diagnosis using artificial intelligence (AI) with the aid of images, without including technical definitions of actual detailed operations or diagnosis, or limit image collection and diagnosis equipment to smartphones and smart devices, without taking into account the possibility of combining collection environments in various image devices with separate diagnosis service configurations, thereby limiting the scope of application. Summary of the Invention [Problem to be solved by the invention]

[0007] The present application is intended to solve the above-mentioned problems, and provides a diagnostic technology capable of recognizing facial skin aging and performing integrated analysis by applying artificial intelligence (AI) technology to skin aging evaluation and skin lesion evaluation. Specifically, an artificial neural network model is created by deep learning, symptoms corresponding to each item in a skin image are diagnosed and analyzed, and the position, shape, and deepening degree of each detailed individual of the actual skin symptoms can be quantitatively derived. It is possible to analyze the number, average size, shape, overall area, and intensity of each symptom, and the obtained information can be used to provide visualization information that emphasizes the diagnosis content on the skin image. [Means for solving the problem]

[0008] A skin diagnosis method based on image analysis using deep learning performed by a processor according to one aspect of the present application, comprising the steps of: photographing a target skin to obtain a facial image of a subject; deriving shape or position information of a facial structure by recognizing feature points that can identify an individual from the obtained facial image; anonymizing the facial image based on the shape or position information of the facial structure so that personal information of the subject to be analyzed cannot be identified; and inputting the anonymized facial images into a plurality of artificial neural network models, respectively, and visualizing and providing skin diagnosis results for items corresponding to the relevant artificial neural network models and symptom locations for each item, wherein the items may include at least one of wrinkles, pigmentation, pores, erythema, and aging diagnoses.

[0009] In one embodiment, the step of anonymizing the facial image may include a step of separating feature points and a background from the facial image and removing skin shapes including the feature points, a step of dividing the facial image from which the feature points have been removed into a plurality of patches, and a step of randomizing and rearranging the plurality of patches.

[0010] In one embodiment, the feature points may be at least one of eyebrows, eyes, nose, and lips.

[0011] In one embodiment, each of the plurality of artificial neural network models is trained using a plurality of training samples as training data, the plurality of training samples may include samples transformed using data augmentation techniques.

[0012] In one embodiment, the data augmentation techniques may include at least one of a random crop, a blur, and a flip operation.

[0013] In one embodiment, the artificial neural network model for diagnosing wrinkles may be trained by adjusting parameters including a total number of individuals with detected wrinkles, an intensity (depth) estimate for the individuals with detected wrinkles relative to surrounding undetected areas, a total area with detected wrinkles, and length and width of the detected wrinkles, and may output analysis results for at least one of wrinkle number, wrinkle intensity (depth), wrinkle area, wrinkle length, wrinkle width, distribution of wrinkle intensity, area, length and width, and wrinkle score.

[0014] In one embodiment, the artificial neural network model for diagnosing pigmentation may be trained by adjusting parameters including a total number of individuals with detected pigmentation, intensity estimates for surrounding undetected areas in individuals with detected pigmentation, and a total area of ​​detected pigmentation, and may output analysis results for at least one of the number of pigmentations, pigmentation intensity, pigmentation area, distribution of pigmentation intensity, area, length, width, and pigmentation score.

[0015] In one embodiment, the artificial neural network model for diagnosing pores may be trained by adjusting parameters including a total number of individuals with detected pores, an estimate of the strength (depth) of the individuals with detected pores relative to surrounding undetected areas, a total area of ​​detected pores, a pore length, and a pore width, and may output analysis results for at least one of the number of pores, pore strength (depth), pore size, pore area, pore length, pore width, pore sagging (ratio of length to width), pore strength, area, length, width, distribution by sagging, and a pore score.

[0016] In one embodiment, an artificial neural network model for diagnosing erythema may be trained by adjusting parameters including a total number of individuals with detected erythema, an intensity estimate for the surrounding undetected areas of individuals with detected erythema, and a total area of ​​detected erythema, and may output analysis results for at least one of number of erythema, erythema intensity, erythema area, distribution of erythema intensity and area, and erythema score.

[0017] In one embodiment, the artificial neural network model for diagnosing aging may receive as input at least one of an anonymized facial image, an output result of a single artificial neural network model, and a combined value of output results of multiple artificial neural network models, and predict age based on facial aging or facial skin aging estimated from the facial image.

[0018] In one embodiment, each of the plurality of artificial neural network models comprises U -It may also be an encoder-decoder structural model based on the net model.

[0019] In one embodiment, each of the multiple artificial neural network models may be trained on imageNet pre-trained weights based on ResNet.

[0020] In one embodiment, the method may further include a step of evaluating at least one of the antioxidant effect and the whitening effect of a particular product based on the skin diagnosis result for the item.

[0021] In one embodiment, the method further includes a step of obtaining information regarding the subject's skin concerns and lifestyle through a medical interview before inputting the anonymized facial images into a plurality of artificial neural network models, respectively, and the skin diagnosis result may be a result related to the subject's skin concerns and lifestyle.

[0022] In one embodiment, the method may further include a step of recommending a specific product or providing a beauty diet tailored to the subject's skin concerns and lifestyle based on the skin diagnosis results.

[0023] According to another aspect of the present application, a skin diagnosis system based on image analysis using deep learning may include: an imaging unit that captures a facial image by photographing a target skin; a facial recognition model that derives shape or position information of a facial structure by recognizing feature points that can identify an individual from the captured facial image; an anonymization model that anonymizes the facial image based on the shape or position information of the facial structure so that personal information of the analysis target cannot be identified; and a plurality of artificial neural network models for at least one of the diagnosis items of wrinkles, pigmentation, pore erythema, and aging, each of which receives an anonymized facial image and provides a skin diagnosis result for the item corresponding to the corresponding artificial neural network model and a visualized location of symptoms for each item. Effect of the Invention

[0024] The skin diagnosis system and method based on image analysis using deep learning according to one embodiment of the present application is composed of a face recognition model for determining the position of the face, an anonymization model for randomly arranging the face image while leaving only the skin area in order to anonymize personal information, and a model for each diagnostic symptom including diagnosis of wrinkles, pigmentation, pores, erythema, and aging, etc. Therefore, optimization and function separation and adjustment for each characteristic are possible, and redundant diagnosis and efficient system management and operation are possible.

[0025] Skin diagnosis is possible even with anonymized skin images taken from acquired face images, and the diagnostic system is designed not to use, store and / or process personal information, so leakage and management of personal information can be addressed in advance in the system operation and data collection parts.

[0026] The images used by this system to diagnose skin conditions are not limited to images taken with specialized imaging equipment, general cameras, smartphones, etc., and are designed to derive the characteristics of each skin diagnosis item according to the image conditions, minimizing the impact of image type and format and allowing the system to be used for a variety of purposes.

[0027] It is possible to quantitatively evaluate skin changes and cosmetic efficacy in long-term skin aging research and efficacy evaluation of anti-aging products that require long-term follow-up observation and analysis, such as anti-aging research, and efficacy evaluation of whitening cosmetics that require analysis of minute changes in skin pigmentation. Therefore, it is possible to perform various skin diagnosis items quickly and inexpensively without a separate skin measurement device or analysis preparation, providing advantageous characteristics in fields such as long-term follow-up observation and large-scale skin research.

[0028] The effects of the present application are not limited to the effects described above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the accompanying claims. [Brief description of the drawings]

[0029] In order to more clearly describe the technical solutions of the embodiments of the present application or the prior art, the drawings necessary for describing the embodiments are briefly introduced below. It should be understood that the following drawings are only for describing the embodiments of the present specification, and are not intended to be limiting. In addition, for clarity of description, the following drawings may show some elements to which various modifications such as exaggeration, omission, etc. have been applied.

[0030] [Figure 1] FIG. 1 is a block diagram of a skin diagnosis system based on image analysis according to an embodiment of the present application. [Diagram 2] 4 is a flowchart of a method for skin diagnosis based on image analysis according to another embodiment of the present application. [Diagram 3] FIG. 2 is a schematic diagram illustrating the input and output process of multiple artificial neural network models according to an embodiment of the present application. [Figure 4] 1 is a diagram illustrating a process in which a face recognition model according to an embodiment of the present application extracts and positions face feature points, and extracts only the skin area through an anonymization model. [Diagram 5] FIG. 2 illustrates a process of anonymizing a face image in an anonymization model according to an embodiment of the present application. [Figure 6A]FIG. 1 illustrates a process in which multiple training samples are labeled by an expert to train each of multiple artificial intelligence neural network models according to one embodiment of the present application. [Figure 6B] FIG. 1 illustrates a process in which multiple training samples are labeled by an expert to train each of multiple artificial intelligence neural network models according to one embodiment of the present application. [Figure 6C] FIG. 1 illustrates a process in which multiple training samples are labeled by an expert to train each of multiple artificial intelligence neural network models according to one embodiment of the present application. [Figure 6D] FIG. 1 illustrates a process in which multiple training samples are labeled by an expert to train each of multiple artificial intelligence neural network models according to one embodiment of the present application. [Figure 7] FIG. 1 illustrates an artificial intelligence neural network model based on Unet, according to an embodiment of the present application. [Figure 8] This is a table summarizing the main items that each of multiple artificial intelligence neural network models diagnoses for skin according to one embodiment of the present application. [Figure 9A] This is an image in which the diagnostic results for each item of an anonymization model and multiple artificial intelligence neural network models according to one embodiment of the present application are reconstructed into an image of a pre-anonymization state. [Figure 9B] This is an image in which the diagnostic results for each item of an anonymization model and multiple artificial intelligence neural network models according to one embodiment of the present application are reconstructed into an image of a pre-anonymization state. [Figure 9C] This is an image in which the diagnostic results for each item of an anonymization model and multiple artificial intelligence neural network models according to one embodiment of the present application are reconstructed into an image of a pre-anonymization state. [Figure 9D] This is an image in which the diagnostic results for each item of an anonymization model and multiple artificial intelligence neural network models according to one embodiment of the present application are reconstructed into an image of a pre-anonymization state. [Figure 10A]11 is a diagnostic image for each item of multiple artificial intelligence neural network models when anonymization processing is performed using an anonymization model according to one embodiment of the present application. [Figure 10B] 11 is a diagnostic image for each item of multiple artificial intelligence neural network models when anonymization processing is performed using an anonymization model according to one embodiment of the present application. [Figure 10C] 11 is a diagnostic image for each item of multiple artificial intelligence neural network models when anonymization processing is performed using an anonymization model according to one embodiment of the present application. [Figure 10D] 11 is a diagnostic image for each item of multiple artificial intelligence neural network models when anonymization processing is performed using an anonymization model according to one embodiment of the present application. [Figure 11A] 1 is a graph showing the correlation of diagnostic results for each item with age according to one embodiment of the present application. [Figure 11B] 1 is a graph showing the correlation of diagnostic results for each item with age according to one embodiment of the present application. [Figure 11C] 1 is a graph showing the correlation of diagnostic results for each item with age according to one embodiment of the present application. [Figure 11D] 1 is a graph showing the correlation of diagnostic results for each item with age according to one embodiment of the present application. [Figure 12A] 1 is a graph analyzing the results of an evaluation study of an anti-aging product using the present application, according to one embodiment of the present application. [Figure 12B] 1 is a graph analyzing the results of an evaluation study of an anti-aging product using the present application, according to one embodiment of the present application. [Figure 13A] 1 is a graph analyzing the results of an evaluation study of a skin lightening product using the present application, according to one embodiment of the present application. [Figure 13B] 1 is a graph analyzing the results of an evaluation study of a skin lightening product using the present application, according to one embodiment of the present application. [Figure 14] FIG. 2 illustrates a process of obtaining information about a subject's skin concerns and lifestyle through a medical interview, according to one embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The terminology used herein is merely for the purpose of referring to particular embodiments and is not intended to limit the present application. As used herein, the singular form includes the plural form unless the description clearly indicates otherwise. As used herein, the meaning of "comprises" embodies certain properties, regions, integers, steps, operations, items and / or components and does not exclude the presence or addition of other properties, regions, integers, steps, operations, items and / or components.

[0032] Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by a person of ordinary skill in the art to which this application belongs. Commonly used predefined terms are further interpreted as having a meaning consistent with the relevant technical literature and the presently disclosed content, and should not be interpreted as an ideal or very formal meaning unless otherwise defined.

[0033] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present application will be described in detail with reference to the drawings.

[0034] FIG. 1 is a block diagram of a skin diagnosis system 1 based on image analysis using deep learning (hereinafter, referred to as "skin diagnosis system") according to an embodiment of the present application.

[0035] Referring to FIG. 1, the skin diagnosis system 1 may include a photographing unit 10 that photographs the skin of a subject to obtain a facial image, a facial recognition model 11 that derives shape or position information of a facial structure by recognizing feature points that can identify an individual from the obtained facial image, an anonymization model 12 that performs an anonymization process on the facial image so that personal information of the subject to be analyzed cannot be identified based on the shape or position information of the facial structure, and a plurality of artificial neural network models 13, 14, 15, and 16 for at least one of wrinkles, pigmentation, pores, and erythema, each of which receives an input of an anonymized facial image and visualizes and provides a skin diagnosis result for the item corresponding to the corresponding artificial neural network model and the location of symptoms of each item. The items may include at least one of wrinkles, pigmentation, pores, and erythema. The skin diagnosis system 1 may further include an aging diagnosis model 17 that receives as an input a value obtained by integrating output results of a single or multiple artificial neural network models including an anonymization model, and predicts an age estimated from a facial image.

[0036] In this specification, the artificial neural network model may include a deep learning model, and the deep learning model may be a form in which artificial neural networks are stacked in multiple layers. The deep learning model is a form in which a deep neural network consisting of a multi-layer network automatically learns the features of each image by learning a large amount of data, and thereby the network is trained in a manner that minimizes the objective function, i.e., the error in prediction accuracy.

[0037] In this specification, the deep learning model may be, for example, a Convolutional Neural Network (CNN), a Deep Hierachical Network (DHN), a Convolutional Deep Belief Network (CDBN), a Deconvolutional Deep Network (DDN), or the like, but may also be any of various current or future deep learning models.

[0038] The skin diagnosis system 1 according to the embodiment may be entirely hardware, entirely software, or may have an aspect of being partially hardware and partially software. For example, a system may generally refer to hardware having data processing capabilities and operating software for driving the hardware. Terms such as "unit," "system," and "apparatus" in this specification are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device including a central processing unit (CPU), a graphic processing unit (GPU), or other processor. In addition, software may refer to a running process, an object, an executable, a thread of execution, a program, and the like.

[0039] For example, the concept of the skin diagnostic system 1 (e.g., the operations of components 10-17) may be incorporated into various computing devices (e.g., smart devices including smartphones, computers, etc.) as an image system for skin analysis (such as the VISIA-CR® system from Canfield, Inc., Fairfield, NJ, USA) or computer software, or a customer app. The software may be designed to process captured images of the target skin, diagnose and evaluate the target skin in the images, and even provide the operation results to a user.

[0040] The image capture unit 10 is a component that captures the skin of a target, and may be implemented in various video capture devices available in the art. In one embodiment, the image capture unit 10 is configured to capture the skin image of the target and obtain the face image. For example, the image capture unit 10 may be implemented in a VISIA-CR® from Canfield, Inc. (Fairfield, NJ, USA).

[0041] The image capturing unit 10 may provide the target skin image data to the other components 11 to 17. The image capturing unit 10 may be further configured to perform an operation of extracting a partial region of the target skin image, for example, a cropping process.

[0042] It will be apparent to those skilled in the art that the skin diagnostic system 1 may include other components not explicitly described herein. For example, the skin diagnostic system may include other hardware elements necessary for the operations described herein, including a network interface, an input device for data entry, and an output device for display, printing or other display of data, and storage devices such as memory.

[0043] FIG. 2 is a flowchart of a skin diagnosis method based on image analysis according to another embodiment of the present application.

[0044] 2, a skin diagnosis method based on image analysis (hereinafter, "skin diagnosis method") is performed by a computing device including a processor. The computing device including the processor may be performed by, for example, the skin diagnosis system 1 or some of its components (e.g., the image capture unit 10, the face recognition model 11, the anonymization model 12, and / or a plurality of artificial neural network models 13-17), or may be performed by another computing device. Hereinafter, for clarity of explanation, the present application will be described in more detail by taking an embodiment in which the skin diagnosis method is performed by the skin diagnosis system 1.

[0045] Referring to FIG. 2, the skin diagnosis method includes a step (S20) of acquiring information on the subject's skin concerns and lifestyle through a medical interview; a step (S21) of acquiring a facial image of the subject by photographing the target skin (for example, by the photographing unit 10); a step (S22) of deriving shape or position information of the facial structure by recognizing feature points that can identify an individual from the acquired facial image (for example, by the facial recognition model 11); a step (S23) of anonymizing the facial image based on the shape or position information of the facial structure so that personal information of the analysis target cannot be identified (for example, by the anonymization model 112); For example, the system includes a step (S24) of inputting anonymized facial images (by a plurality of artificial neural network models 13 to 17) into a plurality of artificial neural network models, respectively, and visualizing and providing skin diagnosis results for items corresponding to the artificial neural network models and the location of symptoms for each item, a step (S25) of recommending a specific product customized to the subject's skin concerns and lifestyle or providing a beauty diet based on the skin diagnosis results, and a step (S26) of evaluating at least one of the antioxidant effect and the whitening effect of the specific product.

[0046] FIG. 3 is a schematic diagram illustrating the input and output process of multiple artificial neural network models according to an embodiment of the present application.

[0047] 3, the skin diagnosis system 1 is made up of a total of six types of models, including a face detection model for determining the position of a face, an anonymization model for removing personal information from an image, and diagnostic models for each item for diagnosing pigmentation, pores, erythema, and wrinkles, and is configured to enable characteristic optimization and function separation for each model. Each model is trained to accurately diagnose symptoms for each item.

[0048] A wrinkle model for diagnosing wrinkles may receive an anonymized face image as an input, and output an analysis result for at least one of the number of wrinkles, wrinkle strength (depth), wrinkle area, wrinkle length, wrinkle width, wrinkle strength or area, length, and width distribution, and wrinkle score. A pigmentation model for diagnosing pigmentation may output an analysis result for at least one of the number of pigmentations, pigmentation strength, pigmentation area, pigmentation strength or area distribution, and pigmentation score. A pore model for diagnosing pores may output an analysis result for at least one of the number of pores, pore strength (depth), pore size, pore area, pore length, pore width, pore sagging (length to width ratio), pore strength or area, length, width, and width distribution, and pore score. The erythema model for diagnosing erythema may output at least one of the analysis results of the number of erythema, the intensity of erythema, the area of ​​erythema, the distribution of the intensity and area of ​​erythema, and the erythema score. Each of the wrinkle score, pigmentation score, pore score, and erythema score means a scored value for the skin condition of each skin item based on data collected in advance, and is calculated by considering a single or multiple combinations of data such as the intensity and total area of ​​each of wrinkles, pigmentation, pores, and erythema. The analysis results output by the wrinkle model, pigmentation model, pore model, and erythema model may be visualized and displayed in the area corresponding to the symptom of the corresponding item. In addition, the aging diagnosis model for diagnosing facial aging may receive an anonymous or non-anonymized face image as an input, and output an analysis result for the aging state of the face and the aging state of the facial skin.

[0049] FIG. 4 is a diagram illustrating a process in which a face recognition model according to an embodiment of the present application extracts and positions face feature points, and extracts only a skin region through an anonymization model.

[0050] Referring to FIG. 4, the face recognition model may derive shape or position information of a face structure by recognizing feature points that can identify an individual from an acquired face image. The feature points may be at least one of eyebrows, eyes, nose, and lips, and may be recognized separately from the background. In one embodiment, the face recognition model may determine whether the image is suitable for skin diagnosis, and may proceed with skin analysis only if the image is determined to be suitable. Furthermore, the face recognition model may output a face size, which is the size of the skin area detected after performing an anonymization process on the face image.

[0051] FIG. 5 is a diagram illustrating a process of anonymizing a face image in an anonymization model according to an embodiment of the present application.

[0052] Referring to FIG. 5, the anonymization model may include a step of separating feature points and background from a facial image and removing skin features including the feature points, a step of dividing the facial image from which the feature points have been removed into a plurality of patches, and a step of randomizing and rearranging the plurality of patches.

[0053] Therefore, skin diagnosis is possible even with some skin images that have been anonymized from facial images acquired using an anonymization model, and since the diagnostic system is designed not to utilize, store and / or process personal information, it is possible to proactively address the leakage and management of personal information in the system operation and data collection parts.

[0054] 6A-6D are diagrams illustrating a process in which a plurality of training samples are labeled by an expert for training each of a plurality of artificial intelligence neural network models according to one embodiment of the present application.

[0055] Referring to FIG. 6A to FIG. 6D, each of the plurality of artificial neural network models may be trained using a plurality of training samples as training data. In one embodiment, 13,596 images of Korean men and women taken by a high-resolution anterior eye imaging device (VISIA-CR, VISIA, etc.) for our own human body application research were selected and processed, and 50,358 image patches were used for learning the artificial neural network model, and 1,150 image patches were used for verifying the model learning results. In the case of data labeling, the position and shape of each item that can be visually confirmed from the skin image were segmented, and the work data was inspected and produced by more than 50 workers and more than 3 clinical experts with more than 10 years of experience. Data labeling is a process of displaying and processing the final result or information in the original data so that the artificial intelligence can learn according to the development goal. As an example, the exact position of the wrinkles is displayed in the skin image that the wrinkle model is to diagnose for wrinkle diagnosis learning. Segmentation is a process of classifying and displaying the type and position of a specific target area when labeling image data or analyzing images.

[0056] The plurality of training samples may be transformed using a data augmentation technique, which converts a dataset for AI learning into various realistic forms to ensure data diversity and quantity greater than that of actually collected data, thereby enhancing adaptability to various data environments and development performance.

[0057] In one embodiment, the data augmentation technique may include at least one of a random crop, a blur, and a flip process. The data augmentation technique can provide general and consistent results across a variety of environments, thereby enabling the system to operate meaningfully on untrained devices and environments.

[0058] FIG. 7 is a diagram illustrating an artificial intelligence neural network model based on Unet according to one embodiment of the present application.

[0059] Referring to FIG. 7, each of the anonymization model and the multiple artificial neural network models may be an encoder-decoder structure model based on the U-net model. The U-net model is one of the representative artificial neural network models developed for the purpose of image segmentation in the biomedical field. The U-net model has a U-shaped process of rearranging various processes of a target process by one neural network. The encoder-decoder structure model is an artificial intelligence model designed to be able to separate and associate input and output data processing, thereby improving processing efficiency and quality. Each of the anonymization model and the multiple artificial neural network models may be trained in the form of imageNet pre-trained weight based on ResNet. As a learning criterion, mIoU (mean intersection over union) may be used, and in addition to the commonly used Cross-Entropy loss, Dice loss, Focal loss, etc. may be additionally used.

[0060] The ResNET model is a high-performance multi-stage artificial neural network model developed by Microsoft. It solves the problem of accuracy degradation that occurs when forming a multi-stage neural network to improve performance by using residual learning, a method that utilizes the difference between stepwise input and output for learning. ImageNet is a large-scale database for developing software to recognize objects in images, and pre-trained weight refers to the initial artificial intelligence model conditions that are learned to be weighted for specific conditions based on this data. mIoU (mean intersection over union) is one of the methodologies for evaluating digital image analysis models, and is a method to quantify the match rate of the results by dividing the number of common pixels between the target and the predicted result by the total number of pixels. Cross-Entropy loss is one of the methodologies for evaluating digital analysis models, and evaluates the difference between the probability distribution of the target data and the probability distribution of the predicted model result data using an entropy-based loss function. Focal loss is one of the methodologies for evaluating digital analysis models, and is a method of evaluation using weighting parameters to reduce the impact of sample data that is easy to analyze on the overall prediction model learning and to focus on samples that are difficult to analyze.

[0061] Due to the characteristics of skin, in most situations the environment is imbalanced data with many general skin areas and few lesion areas, so for smooth learning, we use a learning method that measures the frequency of each label and reflects it when calculating loss. Since diagnostic models are separated for each item, individual characteristics are optimized and duplicate diagnosis for each item is possible. Furthermore, by applying a separate face image anonymization model, it is possible to address security issues such as protecting personal information on diagnostic information, such as personal identification data that is difficult to remove using face recognition models alone.

[0062] FIG. 8 is a table summarizing the main items that each of the multiple artificial neural network models diagnoses for skin, according to one embodiment of the present application.

[0063] Referring to FIG. 8, a wrinkle model for diagnosing wrinkles may learn by adjusting parameters including the total number of individuals with detected wrinkles, an estimate of the strength (depth) of the individuals with detected wrinkles relative to surrounding undetected areas, the total area of ​​detected wrinkles, and the length and width of the detected wrinkles, and may output analysis results of the number of wrinkles, wrinkle strength (depth), wrinkle area, wrinkle length and width, distribution state for each wrinkle strength, area, length and width, and wrinkle score.

[0064] A pigmentation model for diagnosing pigmentation may be trained by adjusting parameters including the total number of individuals with detected pigmentation, an intensity estimate for undetected areas surrounding the individuals with detected pigmentation, and the total area in which pigmentation is detected, and output analysis results for the number of pigmentations, the intensity of pigmentation, the area of ​​pigmentation, the distribution of pigmentation intensity and area, and the pigmentation score.

[0065] The pore model for diagnosing pores is trained by adjusting parameters including the total number of pores detected, an estimate of the strength (depth) of the pores detected relative to the surrounding undetected areas, the total area of ​​the pores detected, the pore length, and the pore width, and may output the analysis results of the number of pores, the pore depth, the pore area, the pore length and width, the pore sagging (the ratio of length to width), the pore strength, area, length, width, distribution state by sagging, and the pore score.

[0066] The erythema model for diagnosing erythema may learn by adjusting parameters including the total number of individuals in which erythema is detected, an intensity estimate for undetected areas surrounding individuals in which erythema is detected, and the total area in which erythema is detected, and output analysis results for the number of erythema, erythema intensity, erythema area, distribution of erythema intensity and area, and erythema score.

[0067] The aging diagnostic model may output an analysis result regarding facial aging or facial skin age estimated from a face image.

[0068] Additionally, the face recognition model may output a face size, which is the size of the skin area detected after performing an anonymization process on the face image.

[0069] 9A to 9D are images in which the diagnostic results for each item of an anonymized model and a plurality of artificial intelligence neural network models are reconstructed into images in a pre-anonymized state according to an embodiment of the present application.

[0070] 9A to 9D, skin diagnosis system 1 can diagnose and analyze each symptom in a skin image, and quantitatively derive the position, shape, and degree of deepening of each detailed individual symptom of an actual skin condition, and can analyze the number, average size, shape, overall area, and intensity of each symptom, and can utilize the obtained information to provide visualization information that emphasizes the diagnostic content on the skin image again.

[0071] In addition, it is composed of separate models such as personal information processing anonymization model and each diagnostic symptom model including wrinkles, pigmentation, pores, and erythema, and it is possible to optimize and separate / adjust functions for each characteristic, which allows for overlapping diagnoses and efficient system management / operation.

[0072] 10A to 10D are diagnostic images for each item of a plurality of artificial intelligence neural network models when anonymization processing is performed using an anonymization model according to an embodiment of the present application.

[0073] 10A to 10D are images in which the originals of the respective figures of Figures 9A to 9D are anonymized using an anonymization model, and then visualized in the corresponding areas using a pigmentation model, a pore model, a wrinkle model, and an erythema model. Referring to Figures 10A to 10D, the anonymization model may include a step of separating feature points and a background from the original face image, removing the shape of the skin including the feature points, and dividing the face image from which the feature points have been removed into a plurality of patches, and a step of randomizing and rearranging the plurality of patches.

[0074] 11A to 11D are graphs showing correlations of diagnosis results for each skin diagnosis item with aging according to one embodiment of the present application.

[0075] 11A to 11D, in order to check the consistency and analytical applicability of the present application for Korean people of various ages and skin conditions and to advance new skin characteristic analysis and insight derivation, facial skin images of various ages were used to diagnose and analyze skin characteristics by age. In one embodiment, facial images of 120 Korean women aged 14 to 69, 20 people per age group, were used. Referring to FIG. 11A, when the images of 120 women in the normal skin group were analyzed, the number of wrinkles was correlated with increasing age in all cases, and the overall appearance of wrinkles accelerating in the 30s and 50s change sections.

[0076] Referring to FIG. 11B, in the case of pigmentation, the total area of ​​pigmentation showed a correlation with increasing age, increasing significantly in the 30s, stagnating in the 40s, and then showing a further increase in occurrence in the 50s.

[0077] Referring to FIG. 11C, in the case of erythema (trouble), the average size and total area show a correlation that decreases with age, and overall it is highly prevalent at a young age such as in the teens and twenties, decreases in the thirties and forties, and then shows a tendency to increase somewhat, suggesting that it is largely due to the influence of temporary trouble at a young age rather than aging.

[0078] In the case of pores, the number / average size / total area of ​​pores increased with age, while the intensity (depth) decreased. Referring to FIG 11D, the number of pores increased overall, but showed signs of congestion in the 50s.

[0079] 12A and 12B are graphs analyzing the results of an evaluation study of an anti-aging product using the present application, according to one embodiment of the present application.

[0080] 12A and 12B, the skin diagnosis system 1 may evaluate at least one of the antioxidant effect and the whitening effect of a particular product based on the skin diagnosis results for the above items. Unlike expert evaluation, the skin diagnosis system 1 can handle minute changes and a large number of repeated follow-up observations using the same criteria, and can quantitatively evaluate and visualize various items using only images, so it can be used to study changes in the skin and evaluate the efficacy of cosmetics, and should be useful for customer after-sales service and the creation of more reliable customer communication content.

[0081] In one embodiment, skin diagnosis was performed using images from a long-term skin follow-up study of a retinol product with proven anti-aging efficacy, and it was found that pigmentation and wrinkle counts increased or remained the same in the group that did not use the product during the four-year follow-up observation period, while the group that used the product showed a significant skin improvement effect. This is the same as the results of separate clinical measurement evaluations, and it was possible to provide more detailed quantified evaluation results.

[0082] 13A and 13B are graphs analyzing the results of a skin lightening product evaluation study using the present application, according to one embodiment of the present application.

[0083] 13A and 13B, when the skin diagnostic system 1 was applied to the efficacy evaluation of a whitening product, it was confirmed that the size and area of ​​pigmentation of the test participants who used a product containing 3,4,5-trimethoxycinnamatethymolester (TCTE) was significantly reduced after 4 weeks, which was consistent with the results of an actual human application test. Therefore, it was determined that skin diagnosis using deep learning can be suitably used for the efficacy evaluation of cosmetics, which requires evaluation of changes in various items in a more subtle way than pharmaceutical evaluations, even with just image capture.

[0084] FIG. 14 is a diagram illustrating a process of obtaining information about a subject's skin concerns and lifestyle through a medical interview according to one embodiment of the present application.

[0085] 14, the skin diagnosis system 1 may obtain information on the subject's skin concerns and lifestyle through a medical interview, in addition to inputting the anonymized facial images into a plurality of artificial neural network models. The skin diagnosis result may be related to the subject's skin concerns and lifestyle. The skin diagnosis system 1 may provide a diagnosis result by focusing on and specifying the subject's skin concerns using the information obtained through the medical interview.

[0086] Based on the skin diagnosis result, the skin diagnosis system 1 may recommend a specific product or provide a beauty diet suited to the subject's skin concerns and lifestyle.

[0087] In one embodiment, the skin diagnosis system 1 provides a skin condition analysis result of the subject for each of the categories of pigmentation, pores, erythema, and wrinkles. In addition, the system provides comments on the subject's skin concerns using information obtained by the interview, and provides advice related to cleansing or lifestyle. Cosmetics suitable for each skin condition may be recommended along with a customized skin care method according to the subject's skin condition.

[0088] These skin diagnosis systems and methods based on image analysis using deep learning are configured separately into a face position and feature point recognition model, a personal information processing anonymization model, and a model for each diagnostic symptom including diagnosis of wrinkles, pigmentation, pores, erythema, and aging, and can be optimized and functions separated and adjusted for each characteristic, allowing for overlapping diagnoses and efficient system management and operation.

[0089] Skin diagnosis is possible even with anonymized skin images taken from acquired face images, and the diagnostic system is designed not to use, store and / or process personal information, so leakage and management of personal information can be addressed in advance in the system operation and data collection parts.

[0090] The images used by this system to diagnose skin conditions are not limited to images taken with specialized imaging equipment, general cameras, smartphones, etc., and are designed to derive the characteristics of each skin diagnosis item according to the image conditions, minimizing the impact of image type and format and allowing the system to be used for a variety of purposes.

[0091] It is possible to quantitatively evaluate skin changes and cosmetic efficacy in long-term skin aging research and efficacy evaluation of anti-aging products that require long-term follow-up observation and analysis, such as anti-aging research, and efficacy evaluation of whitening cosmetics that require analysis of minute changes in skin pigmentation. Therefore, it is possible to perform various skin diagnosis items quickly and inexpensively without a separate skin measurement device or analysis preparation, providing advantageous characteristics in fields such as long-term follow-up observation and large-scale skin research.

[0092] The operations of the image analysis based skin diagnosis system and method according to the above-mentioned embodiments may be implemented at least in part in a computer program and stored in a computer readable storage medium, for example, implemented with a program product consisting of a computer readable medium including program code, which may be executed by a processor to perform any or all of the steps, operations, or processes described above.

[0093] The computer-readable storage medium includes any kind of recording device in which data that can be read by a computer is stored. Examples of the computer-readable storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. The computer-readable storage medium may also be distributed in computer systems connected via a network, and computer-readable code may be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present embodiment will be easily understood by those of ordinary skill in the art to which the present embodiment pertains.

[0094] The present application has been described above with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. However, such modifications should be considered to be within the technical scope of protection of the present application. Therefore, the true technical scope of protection of the present application should be determined by the technical ideas of the appended claims. [Industrial Applicability]

[0095] The skin diagnosis system and method based on image analysis using deep learning according to one embodiment of the present application is composed of a face recognition model for determining the position of the face, an anonymization model for randomly arranging the face image while leaving only the skin area in order to anonymize personal information, and a model for each diagnostic symptom including diagnosis of wrinkles, pigmentation, pores, erythema, and aging, etc. Therefore, optimization and function separation and adjustment for each characteristic are possible, and redundant diagnosis and efficient system management and operation are possible.

Claims

1. A skin diagnosis method based on image analysis using deep learning performed by a processor, comprising: acquiring a facial image of the subject by photographing the skin of interest; deriving shape or position information of a face structure by recognizing feature points capable of identifying an individual from the acquired face image; A step of anonymizing the face image based on the shape or position information of the face structure so that the personal information to be analyzed cannot be identified; inputting the anonymized facial image into a plurality of artificial neural network models, respectively, and visualizing and providing skin diagnosis results for items corresponding to the artificial neural network models and symptom locations for each item; A skin diagnosis method based on image analysis using deep learning, wherein the items include at least one of diagnosis of wrinkles, pigmentation, pores, erythema, and aging.

2. The step of anonymizing the face image includes: Separating feature points and background from a face image and removing skin shapes including the feature points; Dividing the facial image from which the feature points have been removed into a plurality of patches; The method for skin diagnosis based on image analysis using deep learning according to claim 1 , further comprising: a step of randomizing and rearranging the plurality of patches.

3. The method for skin diagnosis based on image analysis using deep learning according to claim 1 , wherein the feature points are at least one of eyebrows, eyes, nose, and lips.

4. each of the plurality of artificial neural network models is trained using a plurality of training samples as training data; The method for skin diagnosis based on image analysis using deep learning according to claim 1 , wherein the plurality of training samples include those transformed using a data augmentation technique.

5. The skin diagnosis method based on image analysis using deep learning, wherein the data augmentation technique includes at least one of a random crop, a blur, and a flip process.

6. an artificial neural network model for diagnosing wrinkles is trained by adjusting parameters including a total number of individuals with detected wrinkles, an intensity (depth) estimate of the individuals with detected wrinkles relative to surrounding undetected areas, a total area with detected wrinkles, and lengths and widths of the detected wrinkles; 2. The skin diagnosis method based on image analysis using deep learning according to claim 1, characterized in that it outputs at least one analysis result from among the number of wrinkles, wrinkle strength (depth), wrinkle area, wrinkle length, wrinkle width, distribution of wrinkle strength, area, length and width, and wrinkle score.

7. An artificial neural network model for diagnosing pigmentation is trained by adjusting parameters including a total number of individuals with detected pigmentation, intensity estimates for surrounding undetected areas in individuals with detected pigmentation, and a total area of ​​detected pigmentation; The skin diagnosis method based on image analysis using deep learning according to claim 1, characterized in that it outputs at least one analysis result from among the number of pigmentation spots, the intensity of pigmentation, the area of ​​pigmentation, the distribution by pigmentation intensity, area, length, and width, and the pigmentation score.

8. An artificial neural network model for diagnosing pores is trained by adjusting parameters including a total number of individuals with detected pores, an intensity (depth) estimate of the individuals with detected pores relative to surrounding undetected areas, a total area of ​​detected pores, a pore length, and a pore width; 2. The skin diagnosis method based on image analysis using deep learning according to claim 1, characterized in that it outputs an analysis result for at least one of the number of pores, pore strength (depth), pore size, pore area, pore length, pore width, pore sagging (ratio of length to width), pore strength, area, length, width, distribution of pore sagging, and pore score.

9. an artificial neural network model for diagnosing erythema is trained by adjusting parameters including a total number of individuals with erythema detected, an intensity estimate for surrounding undetected areas of individuals with erythema detected, and a total area of ​​erythema detected; The skin diagnosis method based on image analysis using deep learning according to claim 1, characterized in that it outputs analysis results for at least one of the number of erythema, the intensity of erythema, the area of ​​erythema, the distribution of erythema intensity and area, and the erythema score.

10. 2. The skin diagnosis method based on image analysis using deep learning according to claim 1, characterized in that the artificial neural network model for diagnosing aging receives as input at least one of an anonymized facial image, an output result of a single artificial neural network model, and a value obtained by integrating output results of a plurality of artificial neural network models, and predicts facial aging estimated from the facial image or age based on facial skin aging.

11. The skin diagnosis method based on image analysis using deep learning according to claim 1, wherein each of the plurality of artificial neural network models is an encoder-decoder structure model based on a U-net model.

12. The method for skin diagnosis based on image analysis using deep learning according to claim 1, characterized in that each of the plurality of artificial neural network models is trained in the form of an imageNet pre-trained weight based on ResNet.

13. The skin diagnosis method based on image analysis using deep learning as described in claim 1, further comprising a step of evaluating at least one of the antioxidant effect and the whitening effect of a specific product based on the skin diagnosis results for the items.

14. and acquiring information about the subject's skin concerns and lifestyle through a medical interview before inputting the anonymized facial images into the plurality of artificial neural network models, respectively. The skin diagnosis method based on image analysis using deep learning according to claim 1 , wherein the skin diagnosis result is related to the subject's skin concerns and lifestyle.

15. The skin diagnosis method based on image analysis using deep learning according to claim 14, further comprising a step of recommending a specific product or providing a beauty diet tailored to the subject's skin concerns and lifestyle based on the skin diagnosis results.

16. an imaging unit that captures a facial image by photographing the target skin; A face recognition model that derives shape or position information of a face structure by recognizing feature points that can identify an individual from the acquired face image; An anonymization model that anonymizes a face image based on the shape or position information of a face structure so that personal information of a target for analysis cannot be identified; A skin diagnosis system based on image analysis using deep learning, comprising: a plurality of artificial neural network models for each of at least one of the diagnosis items of wrinkles, pigmentation, pore erythema, and aging, the plurality of artificial neural network models receiving anonymized facial images and visualizing and providing skin diagnosis results for the items corresponding to the corresponding artificial neural network models and the location of symptoms for each item.

Citation Information

Patent Citations

  • A face skin detection system based on computer vision and an implementation method thereof

    CN109948476A

  • Feature extraction in beauty analysis

    JP2003187251A

  • Image acquisition method

    JP2004000428A

  • Curvilinear object segmentation with noise priors

    JP2020098588A

  • System and method for determining apparent skin age

    JP2020522810A

Cited By

  • Image diagnosis system, image diagnosis method, and image diagnosis program

    JP7824698B1