Skin condition analysis device and system

Abstract

To propose a new device that analyzes and diagnoses a state of the skin of a face and whether or not having a skin disease.SOLUTION: A smart mirror 2 for acquiring a face image and a smartphone 3 for acquiring a skin image are connected to a skin information processing apparatus 4, and thereby both images are recorded with the skin information processing apparatus 4. A recorded past face image is contrasted with the face image acquired recently and skin conditions are determined. Also a state whether or not having a skin disease is analyzed together. Determination / analysis results are notified to users to thereby contribute to convenience of beauty promotion and health maintenance.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus for analyzing the condition of a user's facial skin and providing the user with useful beauty and medical information. [Background technology]

[0002] 2. Description of the Related Art Support systems that provide users with information aimed at promoting beauty are known. This support system uses image processing technology and is configured to, for example, evaluate the condition of the user's skin and scalp from photographic data of the user and notify them of information regarding beauty promotion (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-97899 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, with regard to the application of systems using the above-mentioned image processing technology to medical use, they have not yet been used to diagnose skin diseases, such as skin cancer (melanoma), where it is particularly difficult to visually distinguish between diseased areas and normal areas, such as blemishes and moles. It is said that approximately 320,000 people worldwide are affected by skin cancer each year. The mortality rate for melanoma is high, at 18%, compared to 5% for non-melanoma. While it is difficult to realize that you have skin cancer, the disease progresses very quickly.

[0005] The present invention has been made in consideration of the above-mentioned situation, and the problem to be solved is to propose a new device and system for analyzing and diagnosing skin conditions, particularly as a preventive measure for skin cancer, from the standpoint of achieving both diagnostic accuracy and early detection of diseased areas. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention provides a skin condition analysis device to which a device for acquiring a face image of a user and a device for acquiring a skin image of the user are connected, an image recording means for recording the face image; a means for comparing the acquired face image with a past face image recorded in the image recording means to determine a skin condition of the entire face; a means for generating beauty information such as a makeup method suitable for the user based on the result of the judgment and notifying the user of the information; A specific skin disease analysis means for analyzing the possibility of occurrence of a specific skin disease based on the result of the determination; a means for requesting a user to acquire a skin image of a local area of ​​the face based on the analysis result; a skin condition diagnosis means for diagnosing a skin image acquired by a user in response to the request; and means for notifying a user of the diagnosis result by the skin condition diagnosis means. In addition to the above-mentioned means, the skin condition diagnostic device is characterized by comprising a means for notifying a medical department of the diagnosis result by the skin condition diagnostic means and requesting a detailed diagnosis, and a means for notifying a user of the diagnosis result by the medical department. The device is also characterized in that it is a device for diagnosing skin cancer as the specific skin disease.

[0007] According to the skin condition analysis device of the present invention, an image of the user's bare face before makeup is obtained and recorded daily as a face image, and by comparing the obtained face image with the recorded past face image, the user's current skin condition can be determined. From the results of this determination, beauty information such as the user's preferred makeup method can be provided to the user, facilitating daily beauty promotion. Furthermore, based on the judgment results, the possibility that the user has a specific skin disease such as skin cancer is analyzed, and if there is a possibility, the user is prompted to take a skin image of a local area on the face, and a detailed skin condition is diagnosed from the taken skin image, making it possible to determine whether or not the user has a skin disease before he or she even realizes it. In this case, the diagnosis result by the skin condition diagnosis means is notified to the medical department, and a detailed diagnosis is made by a specialist, so that an accurate diagnosis result can be obtained. If a user is diagnosed with a disease, the user can be notified and early treatment can be provided, thereby contributing to the maintenance of the user's health.

[0008] In the skin condition analysis device having the above configuration, it is preferable that the beauty information generation means and the specific skin disease analysis means include a model generation means of deep learning using AI. In other words, it is preferable that the beauty information generation means has a model generation means that uses actual measurements of a person's facial contour and the position and shape of the eyes, nose, mouth, and eyebrows as training data and generates an estimated model through deep learning using AI that will improve the appearance of the user's face when made up based on the skin condition of the user's entire face. In addition, it is preferable that the specific skin disease analysis means has a model generation means that uses actual measured values ​​such as the size, shape, color, and location of the affected area obtained from an image of the skin disease as training data and generates an estimated model of a skin image that is likely to be a skin disease through deep learning using AI. Since the beauty information generation means and the specific skin disease analysis means have a model generation means using AI deep learning, it is possible to automatically generate beauty information such as makeup methods suitable for the user and the selection of makeup to use. In addition, it is possible to automatically perform the process of determining whether or not the user is suffering from a specific skin disease with high accuracy. By notifying the user of the generated beauty information and the diagnosis results of whether or not the user is suffering from a skin disease, it is possible to contribute to promoting the user's beauty and maintaining their health.

[0009] In the skin condition analyzing device having the above-described configuration, the face image acquiring means can use a smart mirror. It is preferable to use a smart mirror that is large enough to capture a life-size image of the entire face or the entire body. The skin image acquisition means may be a smartphone.

[0010] One of the purposes of the skin condition analysis device configured as above is to provide an environment in which the skin condition can be analyzed and diagnostic results obtained in the course of daily life without the user having to install new equipment, and the device can be configured to be usable not only in the user's private home, but also in commercial facilities such as beauty salons, fitness clubs, homes, cosmetics sales corners and restrooms in department stores, etc. In this case, the smart mirror is installed in any of the above facilities, and the smart mirror is configured to acquire a facial image of the user, and the data is transmitted to a processing device connected to the smart mirror via a communication line for processing. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of a skin condition analyzing system according to an embodiment of the present invention. [Diagram 2] 2 is a diagram for explaining a schematic configuration of a data processing unit of the skin condition analyzing device. FIG. [Diagram 3] The figures show face images captured by a smart mirror, where (A) is a face image captured and recorded in the past, and (B) is a recently captured face image. [Figure 4] FIG. 13 is a diagram showing a skin image acquired by a smartphone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A preferred embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0013] The present invention is intended to obtain an image of a user's face, analyze the condition of the facial skin based on the image, and further diagnose the occurrence of skin diseases, thereby providing the user with useful beauty information and disease diagnosis information. As shown in Fig. 1, a skin condition analysis system according to one embodiment of the present invention (hereinafter also referred to as "this system") includes a smart mirror 2, a smartphone 3 with a camera function, and a skin condition analysis device 4 which is a data processing server, and is configured by connecting the smart mirror 2 and smartphone 3 to the skin condition analysis device 4 via a communication line 5. In the figure, reference numeral 6 denotes an information processing terminal used by a specialist doctor in the medical department.

[0014] The smart mirror 2 is a means for acquiring a facial image of the user, and is equipped with a display unit having a function of displaying electronic information on the mirror surface reflecting the user, a camera unit for photographing the user facing the display unit, and a communication unit for connecting to a communication line 5. The camera unit photographs the user's face to acquire a facial image, which is then transmitted to the skin condition analysis device 4. The smart mirror 2 is also configured to be able to display and output information notified from the skin condition analysis device 4 as electronic information on the display unit. The smart mirror 5 may be equipped with a microphone, a speaker, and a voice recognition means, and information processing such as acquisition of a face image may be performed through voice recognition processing by conversation between the smart mirror 5 and the skin condition analyzer 4.

[0015] The smartphone 3 is a means for acquiring an image of the user's facial skin, and is configured to transmit the skin image captured by the built-in camera to the skin condition analysis device 4 via a communication line 5, and also to receive information notified from the skin condition analysis device 4 and display it on the display. Communication between the smartphone 3 and the skin condition analysis device 4 may be performed by installing a dedicated application related to skin condition analysis processing on the smartphone 3 and executing this application, or by launching a web browser on the smartphone 3 to access the skin condition analysis device 4 via the communication line 5 and send and receive data.

[0016] The skin condition analysis device 4 is a computer having the functions of receiving image data sent from the smart mirror 2 and smartphone 3 connected via a communication line 5, generating information related to the user's skin condition by recording and analyzing the received image data, and outputting and providing the generated information, files, and other data to the recipient via the communication line 5, and is equipped with various processing units, such as a data input unit such as a keyboard, a display unit such as a monitor that displays data, a communication unit that sends and receives data via the communication line 5, a memory unit that stores and keeps various data, files, and processing programs, an external storage medium reading unit that reads data, files, and programs stored in an external storage medium and inputs them into the storage unit, and a control unit that controls each of the above units in accordance with the processing program stored in the storage unit. The skin condition analyzer 4 has a function as a web server that transmits web information to the communication line 5, and is configured to be able to send and receive files, data, etc. between a smartphone 3 with a web browsing function and the information processing terminal 6 in the medical department via a website set up on the communication line 5, or by sending and receiving data via e-mail or other text communication apps, or by sending and receiving information via a calling app. The skin condition analysis device 4 is not limited to being a standalone computer or server, but can be configured as a collection of multiple distributed computers or servers, or multiple servers installed on a cloud, and can be configured so that each computer or server works together to perform processing to analyze the user's skin condition.

[0017] As shown in FIG. 2, the present system 1 constitutes a platform for beauty information processing and a platform for medical information processing, and is configured to process the information by controlling each processing unit of the skin condition analyzer 4, such as the data input / output unit, memory unit, and analysis unit, using a processing program.

[0018] In detail, the hardware and software equipped in the skin condition analysis device 4 constitute an information processing means, specifically, at least an image recording means for recording a face image acquired by the smart mirror 2 and a skin image acquired by the smartphone 3, a means for judging the skin condition of the entire face by comparing the acquired face image with a past face image recorded in the image recording means, a means for generating beauty information such as a makeup method suitable for the user based on the judgment result and notifying the user, a specific skin disease analysis means for analyzing the possibility of the occurrence of a specific skin disease based on the judgment result, a means for requesting the user to acquire skin images of localized areas of the face based on the analysis result, a skin condition diagnosis means for diagnosing the skin image acquired by the user upon request, a means for notifying a medical department of the diagnosis result by the skin condition diagnosis means and requesting a detailed diagnosis, and a means for notifying the user of the skin condition diagnosis result or the medical department's diagnosis result, and these are used to process information related to beauty and medicine.

[0019] The service of providing beauty and medical information to users using this system 1 goes through the following procedures: acquiring a face image, determining skin condition, analyzing skin disease, acquiring a skin image, and reviewing the information in the medical department. In the following, a case will be described in which the smart mirror 2 is installed in a sports club that has authorized its members to use the system, and the members use the smart mirror 2.

[0020] Acquiring face images First, a smart mirror 2 installed in a sports club is connected to a skin condition analyzer 4, and the bare face of a member (user) before makeup is reflected in the smart mirror 2, photographed, and transmitted as a face image to the skin condition analyzer 4. The skin condition analyzer 4 records the received face image by linking it to the user's registration information, information on the photograph date and time, etc.

[0021] <Skin condition assessment> The skin condition of the member is judged by comparing face images acquired in the past and recorded in the skin condition analyzer 4 with a recently acquired face image. The judgment process is performed automatically according to a request from a member who operates the information input interface of the smart mirror 2 when the face image is acquired, or according to a set period such as once a week. The skin condition analyzer 4 reads out a face image 2P1 acquired by the smart mirror 2 in the past as shown in FIG. 3(A) from the image recording means, and compares it with a face image 2P2 acquired by the smart mirror 2 as shown in FIG. 3(B) to judge the skin condition of the entire face. Both images are processed to make blemishes, wrinkles, pores, skin color, etc. on the skin surface clear, and judges how different the current skin is compared to the past skin. The judgment is made by visualizing the change in the skin condition, for example, by expressing it as an image or quantifying it as a score, and recording it in the recording means as the judgment result linked to the member's registration information, and displaying and outputting it on the smart mirror 2 to notify the member who has just acquired the face image. Additionally, beauty information such as makeup techniques suitable for the member is generated based on the judgment results, and is displayed and output on the smart mirror 2 to notify the member. In addition, the beauty information generation means has a model generation means that generates an estimated model using deep learning with AI to generate an estimate of how the member's face will look better when made up based on the member's overall facial skin condition, and beauty information is generated based on this estimate. The judgment results and beauty information may be notified to a smartphone 3 carried by the member.

[0022] Analysis of skin diseases When the skin condition is judged to be poor or abnormal, a specific skin disease analyzing means analyzes whether or not a specific skin disease may occur based on the judgment. For example, as shown in Figure 3(B), if a large area of ​​pigmentation that was not previously present is found above the eyebrows on a member's face (depicted by a dashed rectangular frame in the figure), this area will be analyzed to determine whether it is suffering from a disease such as skin cancer. This specific skin disease analysis means has a model generation means that uses AI deep learning to generate an estimated model of skin images that are likely to be skin diseases based on images of skin diseases registered in an external medical database, etc., and analyzes whether areas such as the pigmentation are areas of skin disease based on this model. If the analysis reveals that the member is not suffering from a skin disease and that there is no abnormality, a notification to that effect is sent to the member's smartphone 3 or the like.

[0023] Acquiring skin images In the analysis of the specific skin disease, the analysis is performed based on the face image 2P2 in Figure 3 (B), but when it is necessary to perform the analysis using a high-precision image because the image of the relevant area is unclear, the member is requested to obtain a magnified image of the area such as the pigmentation. The request is made by a notification from the skin condition analysis device 4 to the member's smartphone 3, and upon receiving the notification, the member operates the smartphone 3 to take a picture of the area above the eyebrows on their face, as shown in Figure 4, and sends this to the skin condition analysis device 4 as a skin image 3P. The skin condition analyzer 4 records the received skin image 3P in the image recording means in association with the member's registration information, and diagnoses whether or not the member has a specific skin disease using the skin condition diagnosis means. The skin condition diagnosis means is also configured in the same manner as the specific skin disease analysis means. If the diagnosis indicates that the member does not have a skin disease, the member's smartphone 3 is notified of this, and if the diagnosis indicates that the member is highly likely to have a skin disease, the member is encouraged to see a doctor as soon as possible.

[0024] <Considerations in the medical sector> The diagnosis of whether or not the member has a specific skin disease may be made by having the member undergo a diagnosis by a specialist in the medical department based on the diagnosis result of the skin condition diagnosis means, or upon request of a member who desires a more accurate diagnosis. In this case, the skin condition diagnosis device 4 sends a notification to the medical department's information processing terminal 6 requesting a diagnosis by a specialist, and the specialist diagnoses whether or not the member has a skin disease based on the image of the member's skin condition sent from the skin condition diagnosis device 4 to the information processing terminal 6 and the AI ​​diagnosis results from the judgment, analysis and diagnosis means. The diagnostic results are notified from the information processing terminal 6 to the skin condition diagnosis device 4, which then links them to the member's registration information, registers them, and notifies the member's smartphone 4. The member can then use the notified diagnostic results for subsequent treatment and disease prevention measures.

[0025] In the embodiment, a smart mirror is used as the face image acquisition means, and a smartphone is used as the skin image acquisition means, but other information processing terminals having the function of taking and recording images and transmitting and receiving them as digital signals may be used. In addition, this system may be used for diagnosing diseases other than specific skin diseases such as skin cancer. [Explanation of symbols]

[0026] 1 Skin condition diagnosis system (this system), 2 Smart mirror, 2P1, 2P2 file image, 3 Smartphone, 3P skin image, 4 Skin condition diagnosis device, 5 Communication line, 6 Information processing terminal in medical department

Claims

1. A skin condition analysis device in which a face image acquisition means for acquiring a face image of a user and a skin image acquisition means for acquiring a skin image of the user are connected, an image recording means for recording the face image; a means for comparing the acquired face image with a past face image recorded in the image recording means to determine a skin condition of the entire face; a beauty information generating means for generating beauty information for the convenience of the user in promoting beauty based on the result of the judgment; A means for notifying a user of the beauty information; A specific skin disease analysis means for analyzing the possibility of occurrence of a specific skin disease based on the result of the determination; a means for requesting a user to acquire a skin image of a local area of ​​the face based on the analysis result; a skin condition diagnosis means for diagnosing a skin image acquired by a user in response to the request; and means for notifying a user of a diagnosis result by said skin condition diagnosis means.

2. A means for notifying a medical department of a diagnosis result by the skin condition diagnosis means and requesting a detailed diagnosis; The skin condition analyzing device according to claim 1 , further comprising a means for notifying a user of the diagnosis results from the medical department.

3. 3. The skin condition analyzing device according to claim 1, wherein the specific skin disease is diagnosed as skin cancer.

4. The skin condition analyzing device according to any one of claims 1 to 3, wherein the beauty information generating means and the specific skin disease analyzing means include a model generating means of deep learning using AI.

5. 5. The skin condition analyzing device according to claim 1, wherein the face image acquiring means is a smart mirror.

6. The skin condition analyzing device according to claim 5 , wherein the smart mirror is sized to be capable of capturing a life-size image of the entire face or entire body.

7. The skin condition analyzing device according to any one of claims 1 to 6, wherein the skin image acquiring means is a smartphone.

8. 8. The skin condition analyzing device according to claim 1, which can be used in beauty salons, fitness clubs, homes, cosmetics sales corners in department stores, and commercial facilities equipped with restrooms.

9. A skin condition analysis system comprising: the skin condition analysis device according to any one of claims 1 to 8; a smart mirror as a face image acquisition means; and a smartphone as a skin image acquisition means.

Images