Information processing method, processing device, and portable terminal
The method synchronizes spectral and image capture to tailor skin condition determination to individual users, addressing environmental variability and predicting future skin conditions for personalized recommendations and device control.
Patent Information
- Application Number
- JP2024164850
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-03-18
- Filing Date
- 2024-09-24
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing skin condition determination methods fail to account for individual user variations, leading to inaccurate assessments when environmental conditions differ from those used to set uniform threshold values.
A method and system that uses a spectral camera to measure skin moisture content and hidden blemishes in synchronization with face image capture, leveraging past measurement results to determine skin condition tailored to the individual user, adjusting for environmental factors and predicting future skin conditions.
Accurately determines skin condition based on user-specific data, providing personalized recommendations and controlling indoor devices to improve user convenience and skin health.
Smart Images

Figure 2024169620000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing method, a processing device, and a mobile terminal. [Background technology]
[0002] Conventionally, skin moisture content has been measured to determine skin conditions (e.g., whether the skin is dry, etc.). Skin moisture content can vary depending on weather conditions such as temperature and humidity. For this reason, Patent Document 1 discloses determining the correlation between skin moisture content and weather conditions. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2012-95825 A Summary of the Invention [Problem to be solved by the invention]
[0004] Incidentally, when determining a user's skin condition based on the skin moisture content or the like, it is desirable to determine a skin condition that is appropriate for the user.
[0005] In this regard, the present disclosure relates to an information processing method, a processing device, and a mobile terminal capable of determining a skin condition according to a user. [Means for solving the problem]
[0006] An information processing method according to one embodiment of the present disclosure is an information processing method executed by at least one processor, and includes a first acquisition step of acquiring a first measurement result including data obtained by a spectral camera measuring a user's skin at a first time point, and a first determination step of determining a skin condition of the user based on the first measurement result, wherein the first measurement result includes data obtained by measuring the user's skin by the spectral camera operating in synchronization with an operation of an image camera capturing a first facial image of the user for facial authentication.
[0007] A processing device according to one embodiment of the present disclosure includes at least one processor, and the at least one processor executes a first acquisition step of acquiring a first measurement result including data obtained by a spectral camera measuring a user's skin at a first time point, and executes a first determination step of determining a skin condition of the user based on the first measurement result, the first measurement result including data obtained by measuring the user's skin by the spectral camera operating in synchronization with an operation of an image camera capturing a first facial image of the user for facial authentication.
[0008] A mobile terminal according to one embodiment of the present disclosure includes a spectral camera, an image camera, and at least one processor, wherein the at least one processor executes a first acquisition step of acquiring a first measurement result including data obtained by the spectral camera measuring a user's skin at a first time point, and executes a first determination step of determining a skin condition of the user based on the first measurement result, wherein the first measurement result includes data obtained by measuring the user's skin by the spectral camera operating in synchronization with an operation of an image camera capturing a first facial image of the user for facial authentication. Effect of the Invention
[0009] According to the information processing method and the like according to one aspect of the present disclosure, it is possible to determine a skin condition according to a user. [Brief description of the drawings]
[0010] [Figure 1]FIG. 1 is a schematic diagram showing a schematic configuration of a skin condition determining system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the skin condition determining system according to the embodiment. [Diagram 3] FIG. 3 is a sequence diagram showing the operation of the skin condition determining system according to the embodiment. [Figure 4A] FIG. 4A is a diagram illustrating a first example of status information displayed on an information terminal according to an embodiment. [Figure 4B] FIG. 4B is a diagram showing a second example of the status information displayed on the information terminal according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing the operation of the information terminal according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the operation of the server device according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing a first example of the operation of determining the skin condition of the user shown in FIG. [Figure 8] FIG. 8 is a flowchart showing a second example of the operation of determining the skin condition of the user shown in FIG. [Figure 9] FIG. 9 is a flowchart showing a third example of the operation of determining the skin condition of the user shown in FIG. [Figure 10] FIG. 10 is a flowchart showing a fourth example of the operation of determining the skin condition of the user shown in FIG. [Figure 11] FIG. 11 is a flowchart showing an operation of the server device according to the embodiment to control indoor devices. [Figure 12] FIG. 12 is a diagram showing a configuration of an information terminal according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] (Summary of the Disclosure) As described in Patent Document 1, by obtaining the correlation between skin moisture content and weather conditions, for example, it is possible to correct the skin moisture content measured under different weather conditions to the skin moisture content measured under the same weather conditions. In other words, it is possible to treat the two skin moisture content as being measured under the same weather conditions. When a threshold value of skin moisture content is set for determining the skin condition under a specified weather condition, it is possible to determine the skin condition based on the threshold value by correcting the skin moisture content measured under weather conditions other than the specified weather condition based on the correlation. This allows the skin condition to be accurately determined even if the skin moisture content is measured under weather conditions different from the specified weather condition.
[0012] However, normal skin conditions vary from user to user. Therefore, if a uniform threshold is used to determine the skin condition, it may not be possible to perform a determination appropriate for the user. Therefore, the inventors of the present application have conducted extensive research into a skin condition determination method that can determine the skin condition appropriate for the user, and have devised the skin condition determination method described below.
[0013] A skin condition determination method according to one aspect of the present disclosure includes a first acquisition step of acquiring a first measurement result including data obtained by measuring a user's skin at a first time point with a spectral camera; a first estimation step of estimating a first estimation result including at least one of the user's skin moisture content and hidden blemish status at the first time point based on the first measurement result; and a first determination step of determining the user's skin condition based on the first estimation result and a plurality of second estimation results including at least one of the user's skin moisture content and hidden blemish status based on second measurement results including data obtained by measuring the user's skin with the spectral camera at each of a plurality of different second time points prior to the first time point.
[0014] This allows the user's skin condition to be determined based on a plurality of past second estimation results of the user. The plurality of second estimation results includes, for example, at least one of the skin moisture content and the state of hidden blemishes of the user U under normal circumstances, and is a result that differs for each user. Therefore, by using such a plurality of second estimation results, it is possible to determine the skin condition according to the user, compared to the case where a fixed threshold is used.
[0015] For example, in the first judgment step, two or more second estimation results having a measurement condition log that matches or is similar to a measurement condition log that indicates the conditions at the time of measurement at the first time point may be extracted as the multiple second estimation results, and the judgment may be made based on the extracted two or more second estimation results.
[0016] This makes it possible to reduce the influence of differences in the measurement condition log on the estimation result when such differences affect the estimation result, and thus makes it possible to determine the skin condition according to the measurement condition log at the time of measurement at the first time point.
[0017] For example, the first estimation result may include at least the condition of the hidden blemishes, and may further include a second estimation step of estimating the amount of ultraviolet rays at the first point in time based on the first estimation result, and in the first judgment step, the condition of the hidden blemishes of the user at a third point in time after the first point in time may be estimated based on the condition of the hidden blemishes of the user based on the first estimation result and the amount of ultraviolet rays, and the judgment may be made at the third point in time.
[0018] This allows the user to know the future state of hidden blemishes. For example, the degree of change in skin condition (degree of change in hidden blemishes) due to the amount of ultraviolet light varies from user to user. When the skin condition of the user at the third time point is determined based on the correlation between the amount of ultraviolet light and the degree of change in the skin condition of the user, the skin condition of the user can be determined at the third time point in the future from the present.
[0019] For example, the method may further include a second acquisition step of acquiring image data of the user's skin by an image camera, and a first output step of outputting status information reflecting the judgment result of the user's skin condition in the image data.
[0020] This allows the skin condition to be presented to the user in an easily understandable manner.
[0021] For example, the status information may include information indicating a recommended action for the user based on the determination result.
[0022] This allows the user to know the recommended action by checking the status information, thereby improving convenience for the user.
[0023] For example, the spectral camera and the image camera may operate synchronously.
[0024] This allows the image data and the first measurement to be acquired at the same time.
[0025] For example, the method may further include a second determination step of determining whether or not the user is indoors at the first time point, and a control step of controlling operation of a device indoors in accordance with the skin condition of the user if it is determined in the second determination step that the user is indoors.
[0026] This makes it possible to automatically realize a spatial environment that matches the user's skin condition, thereby further improving convenience for the user.
[0027] For example, the first measurement result may include data of sunlight measured by the spectral camera at the first time point, and the first estimation step may involve taking a difference between data of the user's skin and data of the sunlight.
[0028] This makes it possible to eliminate the effects of ambient light, and therefore to estimate the moisture content of the user's skin with greater accuracy.
[0029] A skin condition determination method according to one aspect of the present disclosure includes an acquisition step of acquiring a first estimation result including at least one of the user's skin moisture content and the status of hidden blemishes at a first time point based on a first measurement result including data obtained by measuring the user's skin with a spectral camera at the first time point, and a determination step of determining the user's skin condition based on the first estimation result and a plurality of second estimation results including at least one of the user's skin moisture content and the status of hidden blemishes based on second measurement results including data obtained by measuring the user's skin with the spectral camera at each of a plurality of different second time points prior to the first time point.
[0030] This makes it possible to determine a skin condition appropriate for the user using the acquired first estimation result.
[0031] A skin condition determination method according to one aspect of the present disclosure includes a first acquisition step of acquiring a first measurement result including data obtained by a spectral camera measuring a user's skin at a first time point; a second acquisition step of acquiring a second measurement result including data obtained by the spectral camera measuring the user's skin at each of a plurality of second time points prior to the first time point, the second measurement result being different from one another; and a determination step of acquiring a determination result of the user's skin condition at the first time point by inputting the first measurement result into a trained model trained to use the plurality of second measurement results as input and output a determination result of the user's skin condition.
[0032] This makes it possible to obtain a judgment result of the user's skin condition without estimating the user's skin condition at the first time point. Therefore, the amount of processing required to judge the user's skin condition can be reduced. For example, the processing for estimating the skin condition at the first time point can be omitted.
[0033] The trained model is trained to output a judgment result of the user's skin condition using as input a plurality of second measurement results including a measurement condition log at each of the plurality of second time points, and in the judgment step, a judgment result of the user's skin condition at the first time point may be obtained by inputting the first measurement result including the measurement condition log of the user at the first time point.
[0034] This makes it possible to reduce the influence of differences in the measurement condition log on the estimation result when such differences affect the estimation result, and thus makes it possible to determine the skin condition according to the measurement condition log at the time of measurement at the first time point.
[0035] A skin condition determination system according to one embodiment of the present disclosure includes an acquisition unit that acquires a first measurement result including data obtained by a spectral camera measuring a user's skin at a first time point; an estimation unit that estimates a first estimation result including at least one of the user's skin moisture content and the status of hidden blemishes at the first time point based on the first measurement result; and a determination unit that determines the user's skin condition based on the first estimation result and a plurality of second estimation results including at least one of the user's skin moisture content and the status of hidden blemishes based on second measurement results including data obtained by measuring the user's skin by the spectral camera at each of a plurality of different second time points prior to the first time point.
[0036] This provides the same effects as the above-mentioned skin condition determining method.
[0037] These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, or the recording medium. The program may be stored in the recording medium in advance, or may be supplied to the recording medium via a wide area communication network including the Internet.
[0038] Hereinafter, the embodiment will be specifically described with reference to the drawings.
[0039] The embodiments and modifications described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments and modifications are merely examples and are not intended to limit the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims are described as optional components.
[0040] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, substantially the same components are given the same reference numerals, and duplicated explanations may be omitted or simplified.
[0041] In this specification, numerical values are not expressions that express only a strict meaning, but are expressions that include a substantially equivalent range, for example, a difference of about a few percent.
[0042] (Embodiment) Hereinafter, the skin condition determining system according to the present embodiment will be described with reference to FIGS.
[0043] [1. Skin condition assessment system configuration] First, the configuration of a skin condition determining system according to the present embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a schematic diagram showing a schematic configuration of a skin condition determining system 1 according to the present embodiment. Fig. 2 is a block diagram showing a functional configuration of the skin condition determining system 1 according to the present embodiment.
[0044] 1 and 2, the skin condition determining system 1 includes an information terminal 10 carried by a user U, and a server device 20. The information terminal 10 and the server device 20 are connected via a network N so as to be able to communicate with each other.
[0045] The information terminal 10 is a mobile terminal carried by a user U, such as a smartphone or a tablet terminal. The information terminal 10 measures the spectrum of the skin to determine the skin condition of the user U. The information terminal 10 includes, for example, an image camera 11, a hyperspectral camera 12, a control unit 13, a communication unit 14, a display unit 15, an input unit 16, and a storage unit 17. The hyperspectral camera 12 may also be referred to as an HSC (Hyper Spectral Camera).
[0046] The image camera 11 captures image data. The image camera 11 captures, for example, face image data for face authentication. For example, the image camera 11 may capture face image data of a user U when the user U is looking at the display unit 15. The face image data may be used when displaying the skin condition judgment result of the user U in the server device 20, which will be described later, on the display unit 15 of the information terminal 10. The face image data is an example of image data.
[0047] The image camera 11 includes a lens, a light receiving element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and a signal processing unit. The image camera 11 generates one image (color image) by superimposing three types of images of red, green, and blue obtained by focusing an image on a light receiving sensor such as a CCD or a CMOS via a lens. Note that the image camera 11 may be a monochrome camera that generates a monochrome image.
[0048] The hyperspectral camera 12 is a camera capable of measuring the luminance for each wavelength band of light, and measures the spectrum of the skin of the user U and the spectrum of the ambient light around the user U. The measurement data (optical spectrum data) of the hyperspectral camera 12 is used to determine the skin condition of the user U. The ambient light around the user U is the light that the user U is exposed to, such as sunlight. The spectrum of the ambient light is acquired, for example, by directly measuring sunlight, but is not limited to this.
[0049] The hyperspectral camera 12 is configured to be able to measure light with a wavelength of, for example, 280 nm to 2500 nm, but may be configured to be able to measure light with a wavelength of 400 nm to 2500 nm. In other words, the hyperspectral camera 12 may be capable of measuring light from the ultraviolet range to the short-wavelength infrared range, or may be capable of measuring light from the visible range to the short-wavelength infrared range.
[0050] The hyperspectral camera 12 is a camera capable of acquiring spectra of tens of bands (for example, 10 bands) or more. The hyperspectral camera 12, for example, separates incident light into tens of bands or more and acquires an image for each band.
[0051] The hyperspectral camera 12 includes, for example, an optical system for dispersing incident light, a hyperspectral sensor capable of simultaneously acquiring spectra of several tens of bands or more, and a signal processor for generating two-dimensional images for each band based on electrical signals from the hyperspectral sensor. The optical system includes a slit, a diffraction grating, a lens, etc.
[0052] For example, but not limited to, the hyperspectral camera 12 measures the spectrum of the skin at each position while scanning the face of the user U. The hyperspectral camera 12 is an example of a spectral camera.
[0053] 1, the image camera 11 and the hyperspectral camera 12 are provided on the same surface of the information terminal 10, for example, on the same surface as the display unit 15. This allows the image camera 11 and the hyperspectral camera 12 to capture or measure the face of the user U when the user U is looking at the display unit 15. The image camera 11 and the hyperspectral camera 12 may be disposed close to each other on the surface. For example, the image camera 11 and the hyperspectral camera 12 may be disposed side by side.
[0054] The control unit 13 is a control device that controls each component of the information terminal 10. The control unit 13 controls the image camera 11 to capture face image data of the user U. The control unit 13 controls the hyperspectral camera 12 to measure at least the spectrum of the skin of the user U. For example, the control unit 13 may control the operation of the hyperspectral camera 12 in association with the operation of the image camera 11. For example, the control unit 13 may control the image camera 11 and the hyperspectral camera 12 to operate synchronously. For example, when the image camera 11 is activated, the control unit 13 may activate the hyperspectral camera 12 and cause the hyperspectral camera 12 to perform measurement. For example, the control unit 13 may activate the hyperspectral camera 12 when the image camera 11 is activated. For example, the control unit 13 may stop the hyperspectral camera 12 when the image camera 11 is stopped. The control unit 13 functions as an acquisition unit that acquires measurement data obtained by measuring the skin of the user U from the hyperspectral camera 12.
[0055] The control unit 13 performs various processes based on the face image data acquired from the image camera 11 and the measurement data acquired from the hyperspectral camera 12. The control unit 13 has an analysis unit 13a and a matching unit 13b.
[0056] The analysis unit 13a estimates at least one of the current skin moisture content and the state of hidden spots of the user U based on the first measurement result including the measurement data acquired by the hyperspectral camera 12. When estimating the state of hidden spots, the analysis unit 13a may further estimate the amount of ultraviolet rays to which the user U is exposed. It can also be said that the analysis unit 13a performs a spectrum analysis of the skin. The present is an example of a first point in time.
[0057] The hidden spots are spots that are hidden under the surface of the skin and cannot be seen by the eye, and are so-called potential spots. If a user U has hidden spots and is exposed to a lot of ultraviolet rays, the hidden spots will gradually surface and become visible spots. The amount of ultraviolet rays is the amount of ultraviolet rays that the user U is estimated to have been exposed to. The analysis unit 13a is an example of an estimation unit.
[0058] The matching unit 13b compares the facial image data of the user U captured by the image camera 11 with facial image data for matching stored in the memory unit 17 to determine whether the user U is a pre-registered user.
[0059] The control unit 13 outputs, for example, the estimation result estimated by the analysis unit 13a to the server device 20 via the communication unit 14. The estimation result includes, for example, identification information unique to the information terminal 10 or the user U.
[0060] The communication unit 14 is a communication circuit (communication module) for the information terminal 10 to communicate with the server device 20 via a network N such as the Internet. The communication method between the information terminal 10 and the server device 20 is not particularly limited.
[0061] The display unit 15 displays various information to the user U under the control of the control unit 13. For example, the display unit 15 displays information indicating the judgment result of the skin condition of the user U. The display unit 15 is realized by a liquid crystal panel, but may be realized by other display panels such as an organic EL panel. The display unit 15 may have a backlight.
[0062] Note that the information terminal 10 may present the user U with information indicating the determination result by voice instead of the display unit 15 or in addition to the display unit 15.
[0063] The input unit 16 is a user interface that accepts input from a user. The input unit 16 is realized by a hardware key (hardware button), a slide switch, a touch panel, etc. The input unit 16 may accept input from the user U by, for example, voice, gesture, etc.
[0064] The storage unit 17 is a storage device that stores various information for processing by the control unit 13. The storage unit 17 may store, for example, face image data used for matching by the matching unit 13b. The storage unit 17 may store, for example, at least one of identification information unique to the information terminal 10 and identification information unique to the user U. The storage unit 17 may store at least one of face image data captured by the image camera 11 and measurement data measured by the hyperspectral camera 12. In other words, the storage unit 17 may store data (raw data) acquired from the image camera 11 and the hyperspectral camera 12.
[0065] The information terminal 10 may further include various sensors such as a temperature and humidity sensor for measuring temperature and humidity, a gyro sensor for measuring the rotation and orientation of the information terminal 10, and a GPS sensor for acquiring the position of the information terminal 10. At least one of these sensors may be connected to the information terminal 10 as an external sensor via a connection terminal.
[0066] The server device 20 judges the skin condition of the user U based on the information acquired from the information terminal 10. The server device 20 includes a communication unit 21, a control unit 22, and a storage unit .
[0067] The communication unit 21 is a communication circuit (communication module) for the server device 20 to communicate with the information terminal 10 via a network N such as the Internet.
[0068] The control unit 22 is a control device that controls each component of the server device 20. The control unit 22 stores in the storage unit 23 the estimation results of the user U's current skin moisture amount, the state of hidden blemishes, and the amount of ultraviolet light to which the user U is exposed, which are acquired from the information terminal 10. That is, the control unit 22 accumulates the estimation results of the user U's current skin moisture amount, the state of hidden blemishes, and the amount of ultraviolet light. The control unit 22 accumulates these estimation results in the storage unit 23 in association with the identification information of the information terminal 10 or the identification information of the user U. As a result, the storage unit 23 stores time-series data of the estimation results associated with the information terminal 10 or the user U.
[0069] The control unit 22 includes a determination unit 22a. The determination unit 22a determines the skin condition of the user U. The determination unit 22a determines the skin condition of the user U, for example, based on the currently acquired estimation result (first estimation result) and time series data of the estimation results acquired at each of a plurality of different time points before the present (a plurality of second estimation results). The determination unit 22a determines the current skin condition of the user U, for example, by determining the currently acquired estimation result based on the time series data of the estimation results acquired at each of a plurality of different time points before the present. In other words, the determination unit 22a does not determine the skin condition of the user U using a fixed threshold. Note that the plurality of different time points before the present are an example of a plurality of second time points.
[0070] The determination unit 22a may determine a recommended action to be recommended to the user U based on the determination result. For example, when the determination result includes a lack of skin moisture, the determination unit 22a may recommend hydration. For example, the determination unit 22a may determine the recommended action based on a table in which the determination result and the recommended action are associated with each other.
[0071] The storage unit 23 is a storage device that stores various information for processing by the control unit 22. The storage unit 23 stores time-series data of the estimation result. The storage unit 23 stores the estimation result of the user U acquired from each of the multiple users U for each information terminal 10 or for each user U. The storage unit 23 may store information indicating a correlation between at least one of the temperature and humidity around the user U and the skin moisture content of the user U. The storage unit 23 may store information about a device corresponding to the user U and installed indoors. The device corresponding to the user U is, for example, an electrical device installed in the home or office of the user U, such as an air conditioner, a humidifier, a dehumidifier, a heater, or an electric curtain. The storage unit 23 is realized, for example, by a semiconductor memory.
[0072] As described above, the information terminal 10 has the hyperspectral camera 12 for measuring the spectrum of the skin of the user U and the environmental light around the user U. This allows the information terminal 10 to estimate the skin moisture content of the user U and the like.
[0073] As described above, the server device 20 does not judge the current estimation result of the user based on a fixed threshold value, but judges the result based on time series data of the estimation result of the user in the past. This allows the server device 20 to judge the skin condition of the user U based on the skin moisture amount, the state of hidden spots, and the amount of ultraviolet rays under normal circumstances for each user U, and therefore makes it possible to judge the skin condition according to the user U.
[0074] [2. Operation of skin condition judgment system] Next, the operation of the above-mentioned skin condition determining system 1 will be described with reference to Fig. 3 to Fig. 11. Fig. 3 is a sequence diagram showing the operation of the skin condition determining system 1 according to the present embodiment.
[0075] As shown in Fig. 3, the control unit 13 of the information terminal 10 activates the hyperspectral camera 12 (HSC) to measure the skin and sunlight of the user U (S11). This makes it possible to obtain the spectrum of the skin of the user U and the spectrum of the sunlight. The measurement of the skin and the measurement of the sunlight may be performed simultaneously. In other words, the reflected light from the skin and the sunlight may be measured in one measurement. For example, when the reflected light from the skin and the sunlight can be measured simultaneously depending on the time of measurement, the positions of the user U and the sun, the direction of the hyperspectral camera 12 based on the attitude of the information terminal 10, etc., the spectrum of the skin of the user U and the spectrum of the sunlight are measured simultaneously.
[0076] The skin measurement and the sunlight measurement may be performed at different times. For example, after performing the skin measurement, the control unit 13 may display information indicating that the hyperspectral camera 12 should be directed toward the sun via the display unit 15, and when it detects that the hyperspectral camera 12 has been directed toward the sun, the control unit 13 may control the hyperspectral camera 12 to measure the sunlight.
[0077] When the hyperspectral camera 12 scans the face of the user U, the control unit 13 may calculate the area of the face of the user U relative to the information terminal 10 based on the face image data captured by the image camera 11, and scan the calculated face area. This makes it possible to efficiently acquire the spectrum of the skin of the face of the user U. The control unit 13 may measure sunlight by calculating the area of the face of the user U relative to the information terminal 10 based on the face image data captured by the image camera 11, and scanning areas other than the calculated face area.
[0078] It should be noted that step S11 is an example of a first obtaining step.
[0079] Next, the analysis unit 13a of the control unit 13 estimates the skin moisture content, the state of hidden blemishes, and the amount of ultraviolet light based on the measurement data, and outputs the estimation results including the estimated skin moisture content, the state of hidden blemishes, and the amount of ultraviolet light to the server device 20 (S12). The estimation of the skin moisture content, the state of hidden blemishes, and the amount of ultraviolet light will be described later.
[0080] In addition, the analysis unit 13a may estimate at least one of the skin moisture content and the state of hidden spots in step S12. When the analysis unit 13a estimates the state of hidden spots in step S12, the analysis unit 13a may further estimate the amount of ultraviolet rays. Step S12 is an example of the first estimation step and the second estimation step.
[0081] Next, the server device 20 acquires the estimation result from the information terminal 10, and stores the acquired estimation result in the storage unit 23 in association with the user U (S21).
[0082] Next, the server device 20 judges the skin condition of the user U based on the current and past estimation results (S22). The server device 20 uses the past estimation results to judge the skin condition of the user U. The judgment of the skin condition will be described later. Step S22 is an example of a first judgment step. Note that the past estimation results are estimated based on measurement data obtained by measuring the user U with the hyperspectral camera 12, similar to the current estimation result.
[0083] Next, the control unit 22 outputs information (condition information) based on the determination result indicating the skin condition of the user U determined by the determination unit 22a to the information terminal 10 (S23).
[0084] Next, when the control unit 13 of the information terminal 10 acquires the information from the server device 20, the control unit 13 displays the information via the display unit 15 (S13). Here, the information displayed on the display unit 15 will be described with reference to Figs. 4A and 4B. Fig. 4A is a diagram showing a first example of status information displayed on the information terminal 10 according to the present embodiment. Fig. 4B is a diagram showing a second example of status information displayed on the information terminal 10 according to the present embodiment.
[0085] FIG. 4A shows an example in which the display unit 15 displays information based on the judgment result of the skin moisture content of the user U, and for example, information indicating that the skin is dry is displayed. For example, information indicating the dry areas of the user U's face (dashed frame) may be displayed. At this time, the control unit 13 may, for example, superimpose the information indicating the dry areas on the facial image data of the user U captured by the image camera 11. Furthermore, a recommended action for the judgment result may be displayed. For example, information indicating that it is recommended to hydrate around 2 p.m. may be displayed. The dry areas and the recommended action are included in the information based on the judgment result of the skin moisture content.
[0086] FIG. 4B shows an example in which the display unit 15 displays information based on the judgment result of hidden blemishes on the user U's skin. For example, information indicating that the amount of ultraviolet rays the user has been exposed to is greater than usual and that there is a risk of hidden blemishes increasing may be displayed. For example, information indicating an area where hidden blemishes are present on the user U's face (dashed frame) may be displayed. At this time, the control unit 13 may display, for example, information indicating an area where hidden blemishes are present superimposed on the face image data of the user U captured by the image camera 11. Furthermore, a recommended action for the judgment result may be displayed. For example, information indicating that it is recommended to go indoors may be displayed. The area where hidden blemishes are present and the recommended action are included in the information based on the judgment result of the state of hidden blemishes.
[0087] The user U can know the current skin condition and the recommended action by checking the display shown in FIG. 4A or FIG. 4B. The displays shown in FIG. 4A and FIG. 4B may be displayed simultaneously. The information displayed on the display unit 15 needs to include at least the judgment result of the current skin condition. Specifically, in the example of FIG. 4A, it is sufficient to display information indicating that the skin is dry. In the example of FIG. 4B, it is sufficient to display information indicating that there are hidden blemishes.
[0088] Next, the operation of the information terminal 10 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the operation of the information terminal 10 according to the present embodiment.
[0089] As shown in Fig. 5, the control unit 13 of the information terminal 10 activates the image camera 11 (S111). For example, when the control unit 13 detects that a predetermined operation has been performed on the information terminal 10 by the user U, the control unit 13 activates the image camera 11. The predetermined operation is, for example, lifting the information terminal 10 in a sleep state or tapping the display unit 15, but is not limited thereto. The predetermined operation may be, for example, an operation by voice. Step S111 is an example of a second acquisition step.
[0090] When the image camera 11 is activated, it captures an image of the face of the user U and generates face image data.
[0091] Next, the control unit 13 starts up the hyperspectral camera 12 (HSC) in conjunction with the start-up of the image camera 11 (S112). The control unit 13 may start up the hyperspectral camera 12, for example, when it detects that the above-mentioned predetermined operation has been performed on the information terminal 10.
[0092] When the hyperspectral camera 12 is activated, it measures at least the spectrum of the skin of the user U. In this embodiment, when the hyperspectral camera 12 is activated, it measures the spectrum of the skin of the user U and the spectrum of sunlight. This makes it possible to obtain measurement data of the current measurement of the skin of the user U. Step S112 is an example of a first acquisition step. The measurement data is an example of a first measurement result.
[0093] It should be noted that step S112 corresponds to step S11 shown in FIG.
[0094] Next, when the matching unit 13b acquires the face image data from the image camera 11, the matching unit 13b matches the face image data with the face image data stored in the storage unit 17 (S113). The matching unit 13b matches the two sets of face image data to determine whether the user U is a user who has been registered in advance. Note that the matching method used by the matching unit 13b is not particularly limited and any known technology may be used.
[0095] Next, the analysis unit 13a estimates the skin moisture content of the user U based on the measurement data measured by the hyperspectral camera 12 (S114). The analysis unit 13a may estimate the skin moisture content of the user U based on, for example, the spectrum of a band including the absorption wavelength of water among the multiple bands. The absorption wavelength of water may be a near-infrared or mid-infrared wavelength, for example, 750 nm, 970 nm, 1450 nm, etc.
[0096] The analysis unit 13a may estimate the moisture content of the skin of the user U based on the spectrum of a band including a wavelength absorbed by water and the spectrum of a band including a wavelength with low water absorption among the multiple bands. The wavelength with low water absorption may be a near-infrared or mid-infrared wavelength, such as 810 nm, 1160 nm, or 1300 nm.
[0097] The analysis unit 13a may estimate the skin moisture content of the user U based on the spectrum of a band including the absorption wavelength of water in the measurement data obtained by measuring the skin and the spectrum of a band including the absorption wavelength of water in the measurement data obtained by measuring the sunlight. That is, the analysis unit 13a may estimate the skin moisture content of the user U based on the absorbance of the skin at the absorption wavelength of water. In this case, the absorption amount due to the skin moisture of the user U may be calculated by obtaining the difference between the spectrum of the band including the absorption wavelength of water in the measurement data obtained by measuring the skin and the spectrum of the band including the absorption wavelength of water in the measurement data obtained by measuring the sunlight. This makes it possible to eliminate the influence of environmental light and to estimate the skin moisture content of the user U with higher accuracy. The analysis unit 13a may estimate the skin moisture content of the user U using any known method for estimating the skin moisture content using the light spectrum.
[0098] Next, the analysis unit 13a estimates the state of hidden blemishes on the skin of the user U based on the measurement data measured by the hyperspectral camera 12 (S115). The analysis unit 13a may estimate the state of hidden blemishes on the skin of the user U based on the spectrum of a band including the absorption wavelength of melanin that causes hidden blemishes, among multiple bands. The absorption wavelength of melanin is, for example, 660 nm, 880 nm, etc.
[0099] The analysis unit 13a may estimate the state of hidden spots on the skin of the user U based on the spectrum of a band including the absorption wavelength of melanin and the spectrum of a band including wavelengths where melanin has low absorption among the multiple bands. The analysis unit 13a may estimate the state of hidden spots on the skin of the user U based on the spectrum of a band including the absorption wavelength of melanin in the measurement data obtained by measuring the skin and the spectrum of a band including the absorption wavelength of melanin in the measurement data obtained by measuring sunlight. In this case, the amount of absorption by melanin on the skin of the user U may be calculated by obtaining the difference between the spectrum of the band including the absorption wavelength of melanin in the measurement data obtained by measuring the skin and the spectrum of the band including the absorption wavelength of melanin in the measurement data obtained by measuring sunlight. This makes it possible to eliminate the influence of environmental light, and to estimate the state of hidden spots of the user U with higher accuracy. The analysis unit 13a may estimate the state of hidden spots of the user U using any known method for estimating the state of hidden spots using light spectrum. The state of hidden spots may be the presence or absence of hidden spots, the position of hidden spots, or the number of hidden spots.
[0100] At least one of steps S114 and S115 may be performed. At least one of steps S114 and S115 is an example of a first estimation step.
[0101] Next, the analysis unit 13a estimates the amount of ultraviolet rays based on the measurement data measured by the hyperspectral camera 12 (S116). The analysis unit 13a may estimate the amount of ultraviolet rays that the user U is presumed to be exposed to based on, for example, the spectrum of a band including wavelengths in the ultraviolet region among the multiple bands. The ultraviolet region is, for example, 280 nm to 400 nm. The analysis unit 13a may estimate the amount of ultraviolet rays that is presumed to be absorbed by the skin of the user U based on the spectrum of a band including wavelengths in the ultraviolet region in the measurement data obtained by measuring the skin, and the spectrum of a band including wavelengths in the ultraviolet region in the measurement data obtained by measuring sunlight. The analysis unit 13a may estimate the amount of ultraviolet rays that the user U is exposed to or absorbed by using any known method for estimating the amount of ultraviolet rays that a person is exposed to or absorbed by using a light spectrum.
[0102] It should be noted that step S116 is executed when step S115 is performed. Step S116 is an example of a second estimation step.
[0103] Next, the control unit 13 outputs the estimation result including the estimated skin moisture content, the state of hidden blemishes, and the amount of ultraviolet rays to the server device 20 via the communication unit 14 (S117). The estimation result may include the identification information of the user U or the identification information of the information terminal 10. In this embodiment, the estimation result includes the identification information of the user U. The estimation result may further include information on the surrounding environment of the user U when the hyperspectral camera 12 measured the skin of the user U. The surrounding environment includes at least one of temperature and humidity, maximum and minimum temperatures, weather information, whether the user U is indoors or outdoors, and the setting conditions of devices such as an air conditioner and a humidifier when the user U is indoors. If the information terminal 10 has a temperature and humidity sensor, the temperature and humidity may be the measurement result of the temperature and humidity sensor, or may be acquired via a dedicated application. The maximum and minimum temperatures, weather information, and the like are acquired, for example, from a server device that manages each of them. In the case where the information terminal 10 has a GPS sensor, whether the user U is indoors or outdoors may be obtained from the GPS sensor or may be obtained from the user U via the input unit 16. The setting conditions of the device may be obtained from a control device that controls the device in an integrated manner, may be obtained directly from the device by communication, or may be obtained from the user U via the input unit 16. The estimation result may include information on biological information such as body temperature, heart rate, and blood pressure, as well as lifestyle information such as the amount of exercise, diet (ingested nutrients), and sleep time. The biological information may be obtained from a smartphone or wearable terminal equipped with a biological sensor. The lifestyle information may be obtained from an app included in the smartphone. The conditions at the time of measurement included in the estimation result, such as information indicating the surrounding environment, the biological information of the user U, and lifestyle information, are collectively referred to as a "measurement condition log." By using this measurement condition log, it is possible to make a judgment according to an individual in consideration of correlation with past measurement results at multiple points in time, and the accuracy of the judgment and the learning efficiency can be improved to improve the accuracy of the estimation result.
[0104] It should be noted that steps S114 to S117 correspond to step S12 shown in FIG.
[0105] Next, the information terminal 10 judges whether or not the server device 20 has acquired status information based on the judgment result of the skin condition of the user U, which is determined by the server device 20 using the estimation result (S118). When the information terminal 10 has acquired status information from the server device 20 (Yes in S118), the information terminal 10 outputs the status information (S119). The control unit 13 may superimpose information based on the judgment result of the skin condition included in the acquired status information (e.g., a dashed frame) on the face image data, and output the status information including the superimposed face image data. Thereby, the control unit 13 causes the display unit 15 to perform a display as shown in FIG. 4A or FIG. 4B. Note that the status information including the face image data on which the information based on the judgment result is superimposed on the face image data is an example of status information reflecting the judgment result of the skin condition. Reflecting includes, for example, superimposing.
[0106] The control unit 13 may display, for example, information showing the change over time of the estimated estimation result in a predetermined period. The control unit 13 may display, for example, the change over time of the estimated skin moisture content in one day in the form of a line graph or the like. The control unit 13 may display information showing the change over time of the skin condition judgment result of the judgment unit 22a of the server device 20. The control unit 13 may display, for example, information showing whether the current skin moisture content is higher than the average value of skin moisture content estimated in the past. The control unit 13 can display information showing the change over time of the judgment result by sequentially storing the status information acquired from the server device 20 in the storage unit 17.
[0107] If the information terminal 10 has not acquired the state information from the server device 20 (No in S118), the information terminal 10 returns to step S118 and waits until the state information is acquired.
[0108] It should be noted that steps S118 and S119 correspond to step S13 shown in FIG.
[0109] The process from step S114 onwards may be executed when the collation of face image data by the collation unit 13b determines that the user U is a pre-registered user. That is, the process from step S114 onwards may be executed when the user U who currently owns the information terminal 10 is a legitimate user of the information terminal 10. The control unit 13 may determine whether or not to execute the process from step S114 onwards based on the collation result of step S113.
[0110] Next, the operation of the server device 20 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the operation of the server device 20 according to the present embodiment.
[0111] 6, the control unit 22 of the server device 20 acquires an estimation result including the skin moisture content, the state of hidden blemishes, and the amount of ultraviolet rays via the communication unit 21 (S121), and stores the acquired estimation result in the storage unit 23 (S122). The control unit 22 stores the estimation result corresponding to the user U in the storage unit 23 based on the identification information of the user U included in the estimation result. The control unit 22 stores the estimation result in the storage unit 23 every time it acquires it. Thus, the storage unit 23 stores time-series data of the past estimation results of the user U.
[0112] It should be noted that steps S121 and S122 correspond to step S21 shown in FIG.
[0113] Next, the determination unit 22a of the control unit 22 determines the skin condition of the user U based on the current and past estimation results (S123). The method of determining the skin condition will be described in detail later.
[0114] It should be noted that step S123 corresponds to step S22 shown in FIG.
[0115] Next, the determination unit 22a determines an action recommended for the user U based on the determination result (S124). The determination unit 22a determines an action recommended for the user U, for example, by using a table in which the determination result and the recommended action are associated with each other. For example, when the determination result includes that the user U's skin is dry, the determination unit 22a determines a recommended action such as hydrating or applying a moisturizing cream. For example, when the determination result includes that the user U's skin has hidden blemishes, the determination unit 22a determines a recommended action such as going indoors or applying sunscreen.
[0116] Next, the control unit 22 outputs state information including information indicating the determination result and the recommended action to the information terminal 10 (S125). The state information is an example of information based on the determination result.
[0117] It should be noted that step S125 corresponds to step S23 shown in FIG.
[0118] Here, an example of a method for determining a skin condition will be described with reference to Fig. 7 to Fig. 10. Fig. 7 is a flow chart showing a first example of the operation (S123) for determining a skin condition of a user shown in Fig. 6.
[0119] As shown in Fig. 7, the determination unit 22a acquires the current estimation result (first estimation result) and a plurality of past estimation results (second estimation results) by reading them from the storage unit 23 (S131). The determination unit 22a may acquire, for example, all past second estimation results, may acquire second estimation results for the same period, or may acquire second estimation results acquired between the present and a time a predetermined time before the present. The same period may be, for example, the same season (for example, spring, summer, autumn, or winter), the same month, or a predetermined period before or after the present day. The predetermined period may be, for example, one week or one month.
[0120] Next, the determination unit 22a calculates the average value of the read out second estimation results (S132). If the estimation result is skin moisture content, the determination unit 22a calculates the average value of the skin moisture content in step S132, for example. If the estimation result is the state of hidden blemishes, the determination unit 22a calculates the average value of the number of hidden blemishes in step S132, for example. The average value calculated here is a different value for each user.
[0121] The determination unit 22a may calculate an average value of the multiple second estimation results by weighted averaging. The determination unit 22a may set weights for the multiple second estimation results so that the weight of the second estimation result acquired closer to the present time is higher. This makes it easier to detect a recent large change in the skin condition.
[0122] Next, the determination unit 22a determines the current skin condition of the user U based on the first estimation result and the average value (S133). The determination unit 22a may determine the current skin condition of the user U by comparing the first estimation result with the average value. When the estimation result is skin moisture content, the determination unit 22a determines that the skin is currently dry when the first estimation result is lower than the average value. When the estimation result is the state of hidden blemishes, the determination unit 22a determines that hidden blemishes are currently occurring or increasing when the first estimation result is higher than the average value.
[0123] The method of determination by the determination unit 22a is not limited to the above, and may be another method as long as it uses a plurality of past second estimation results. The determination unit 22a may, for example, calculate a skin moisture threshold (<average value) for determining that the skin is dry based on a plurality of second estimation results, and determine that the skin is dry when the first estimation result is lower than the calculated threshold. The determination unit 22a may, for example, calculate a skin moisture threshold (>average value) for determining that hidden blemishes are increasing based on a plurality of second estimation results, and determine that hidden blemishes are increasing when the first estimation result is higher than the calculated threshold. The threshold may, for example, be calculated by multiplying the average value by a predetermined coefficient, or may be calculated taking into consideration the variation of the plurality of second estimation results.
[0124] The determining unit 22a may determine the skin condition based on the position of the first estimation result relative to the distribution of the multiple second estimation results. When the estimation result is skin moisture content, the determining unit 22a may determine that the skin moisture content is appropriate if the first estimation result is located between the top 20% of high skin moisture content and the bottom 20% of low skin moisture content among the multiple second measurement results, and may determine that the skin moisture content is inappropriate if the first estimation result is outside of this range.
[0125] Next, another example of the method for determining the skin condition will be described. Fig. 8 is a flow chart showing a second example of the operation (S123) for determining the skin condition of the user shown in Fig. 6. Fig. 8 shows the process when the estimation result includes information on the surrounding environment.
[0126] 8, the determination unit 22a acquires the current estimation result (first estimation result) and a plurality of past estimation results (second estimation results) by reading them from the storage unit 23 (S141). Step S141 is similar to step S131 shown in FIG. 7, and therefore a description thereof will be omitted.
[0127] Next, the determination unit 22a extracts two or more second estimation results including an ambient environment that matches or is similar to the ambient environment of the current user U from the multiple second estimation results (S142). It can also be said that the determination unit 22a extracts two or more second estimation results when the ambient environment matches or is similar to the ambient environment of the current user U from the multiple second estimation results. The determination unit 22a extracts two or more second estimation results whose second ambient environment information matches or is similar to the first ambient environment information from the multiple second estimation results based on information indicating the ambient environment (first ambient environment information) included in the first estimation result and information indicating the ambient environment (second ambient environment information) included in each of the multiple second estimation results. Note that similarity indicates, for example, that the difference between the first ambient environment information and the second ambient environment information satisfies a predetermined condition. For example, in a case where the ambient environment is temperature, the predetermined condition may be determined based on the correlation between the skin moisture content and temperature of the user U, and may be, for example, that the temperature difference is within 5 degrees. The predetermined condition may be set for each user U. In step S142, the second estimation result may be extracted based on at least one correlation of the measurement condition log, which is the conditions at the time of measurement included in the estimation result, such as biometric information, lifestyle information, etc. of user U, not limited to information indicating the surrounding environment.
[0128] Next, the determination unit 22a determines the skin condition of the user U based on the first estimation result and the two or more extracted second estimation results. For example, the determination unit 22a calculates an average value of the two or more extracted second estimation results (S143). Then, the determination unit 22a determines the current skin condition of the user U based on the first estimation result and the average value (S144).
[0129] Incidentally, step S144 is similar to step S133 shown in FIG. 7, and therefore the description thereof will be omitted.
[0130] Next, another example of the method for determining the skin condition will be described. FIG. 9 is a flowchart showing a third example of the operation (S123) for determining the skin condition of the user shown in FIG. 6. FIG. 9 shows a process for correcting the estimation result according to the surrounding environment. FIG. 9 shows another example of the process for the case where the estimation result includes information on the surrounding environment. Specifically, an example of the process for the case where the estimation result includes at least one of the temperature and humidity around the current user U will be shown. Below, the case where the estimation result includes the temperature around the current user U will be described. In the following, the estimation result is skin moisture content.
[0131] 9, the determination unit 22a acquires the current estimation result (first estimation result) and a plurality of past estimation results (second estimation results) by reading them from the storage unit 23 (S151). Step S151 is similar to step S131 shown in FIG. 7, and therefore a description thereof will be omitted.
[0132] Next, the determination unit 22a acquires the temperature (ambient temperature) at the time of measurement for the first estimation result of the hyperspectral camera 12 (S152). The determination unit 22a acquires, for example, the temperature included in the first estimation result as the temperature at the time of measurement.
[0133] Next, the determination unit 22a corrects the first estimation result based on the acquired temperature (S153). For example, when the temperature is different from a reference temperature, the determination unit 22a corrects the first estimation result based on the temperature and the reference temperature. For example, the determination unit 22a calculates the skin moisture content at the reference temperature based on information indicating the correlation between the skin moisture content and temperature of the user U stored in advance in the storage unit 23 and the skin moisture content at the temperature. The information indicating the correlation may be, for example, a first-order approximation line (regression line) indicating a linear regression model based on multiple temperatures and the skin moisture content of the user U at each of the multiple temperatures. This allows the skin moisture content to be corrected based on the first-order approximation line corresponding to the user U himself / herself, so that the correction can be performed with higher accuracy.
[0134] Next, the determination unit 22a determines the current skin condition of the user U based on the corrected first estimation result and the multiple second estimation results (S154). The determination unit 22a determines the current skin condition of the user U based on, for example, the corrected first estimation result and an average value of the multiple second correction results. The determination unit 22a may determine the current skin condition of the user U by comparing the corrected first estimation result with the average value. Note that the multiple second estimation results here are values that have been corrected in advance to the moisture content at the reference temperature based on the above correlation.
[0135] Next, a further example of the skin condition determining method will be described. Fig. 10 is a flow chart showing a fourth example of the operation (S123) for determining the user's skin condition shown in Fig. 6. Fig. 10 shows a process for correcting the estimation result according to the amount of ultraviolet rays. Fig. 9 shows a process for including the state of hidden blemishes and the amount of ultraviolet rays in the estimation result. Specifically, Fig. 9 shows a process for predicting the future state of hidden blemishes based on the amount of ultraviolet rays.
[0136] 10, the determination unit 22a acquires the current estimation result (first estimation result) and a plurality of past estimation results (second estimation results) by reading them from the storage unit 23 (S161). Step S161 is similar to step S131 shown in FIG. 7, and therefore a description thereof will be omitted.
[0137] Next, the determination unit 22a acquires the amount of ultraviolet rays at the time of measurement for the first estimation result of the hyperspectral camera 12 (S162). The determination unit 22a acquires, for example, the amount of ultraviolet rays included in the first estimation result as the amount of ultraviolet rays at the time of measurement.
[0138] Next, the judgment unit 22a predicts the future state of hidden spots based on the current state of hidden spots and the amount of ultraviolet rays (S163). The judgment unit 22a may predict the future state of hidden spots using a table in which the current state of hidden spots and the amount of ultraviolet rays are associated with the future state of hidden spots, or may predict using a trained model that inputs the current state of hidden spots and the amount of ultraviolet rays and outputs the future state of hidden spots. The table is generated, for example, based on a plurality of past second estimation results of the user and the degree of change (e.g., the amount of increase or decrease) of the user's hidden spots in the future at that time. The trained model is generated using machine learning with the plurality of second estimation results of the user and the amount of ultraviolet rays as training data and the amount of change of the user's hidden spots in the future at that time as correct answer data.
[0139] In this way, the judgment unit 22a judges the skin condition of the user U in the first estimation result, for example, using a table or a trained model generated based on a plurality of second estimation results. The amount of ultraviolet light and the degree of change in hidden spots are different for each user U. Therefore, the above table or trained model may be generated for each user U or for each user U with similar attributes. In this case, it can be said that the judgment unit 22a predicts the future state of hidden spots based on the correlation between the current state of hidden spots of the user and the amount of ultraviolet light and the degree of change in hidden spots of the user U. Note that similar attributes include, for example, being of the same age, having the same sex, and living in the same area. Information indicating the correlation between the amount of ultraviolet light and the degree of change in hidden spots of the user U is stored in advance in the storage unit 23.
[0140] The future state of hidden spots may be, for example, a 20% increase in hidden spots after five days, etc. Five days later is, for example, an example of a third time point.
[0141] Next, the determination unit 22a determines the future skin condition of the user U based on the current state of the hidden blemish and the future state of the hidden blemish (S164). Here, the determination unit 22a determines how the future state of the hidden blemish will change relative to the current state of the hidden blemish. That is, the determination unit 22a determines the skin condition of the user U at the third time point.
[0142] However, some of the above steps may not be executed. For example, when machine learning is used, the spectrum measuring the user's skin condition may be analyzed using a trained model to determine the skin condition without executing the step of calculating an estimated result from the spectrum. The determination unit 22a determines the skin condition of the user U in the first measurement result, for example, using a trained model generated based on a plurality of second measurement results. The amount of ultraviolet light and the degree of change in hidden spots differ for each user U. Therefore, the trained model may be generated for each user U or for each user U with similar attributes. In this case, it can be said that the determination unit 22a predicts the future state of hidden spots based on the correlation between the current state of hidden spots of the user U and the amount of ultraviolet light and the degree of change in hidden spots of the user U.
[0143] In this manner, the determination unit 22a may obtain a judgment result of the current skin condition of the user U by inputting the first measurement result into a trained model that has been trained to use a plurality of second measurement results as input and output a judgment result of the skin condition of the user U. Obtaining the judgment result of the skin condition by the determination unit 22a is an example of a judgment step.
[0144] Such a trained model is generated by machine learning using the measurement results (e.g., spectrum) measured by the hyperspectral camera 12 as training data and the skin condition at the time of the measurement results as correct answer data. The trained model is generated by deep learning using, for example, a neural network.
[0145] The trained model may be trained to input a plurality of second measurement results including a measurement condition log at each of a plurality of second time points and output a judgment result of the skin condition of the user U. The judgment unit 22a may obtain a judgment result of the skin condition of the user at the first time point by inputting a first measurement result including the measurement condition log of the user at the first time point.
[0146] Such a trained model is generated using machine learning with multiple measurement results (e.g., multiple second measurement results) including a measurement condition log as training data and the skin condition at the time of the measurement results as correct answer data.
[0147] Next, the control of indoor devices in the server device 20 when it is determined that the skin condition is deteriorating, such as the skin being dry or the number of hidden spots increasing, will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the operation of the server device 20 according to the present embodiment to control indoor devices. Fig. 11 is executed, for example, after the determination in step S123 in Fig. 6.
[0148] 11, the control unit 22 determines whether or not the user U is currently indoors (S201). For example, when the determining unit 22a determines that the skin condition is worsening, such as the skin being dry or the number of hidden blemishes is increasing, the control unit 22 executes the determination of step S201. The control unit 22 determines whether or not the user U is indoors, for example, based on the position information (e.g., latitude, longitude, altitude, etc.) of the user U acquired from the information terminal 10. Step S201 is an example of a second determination step.
[0149] Next, when the user U is indoors (Yes in S201), the control unit 22 acquires device information of the indoor device (S202), and controls the operation of the indoor device based on the acquired device information and the skin condition of the user U (S203). Step S203 is an example of a control step.
[0150] The device is an electric device installed indoors, such as an air conditioner, a humidifier, a dehumidifier, a heater, or an electric curtain. For example, when the device is a humidifier and the user U has dry skin, the control unit 22 acquires device information including the current set humidity from the humidifier. Then, the control unit 22 determines whether the set humidity is a value suitable for the current skin condition of the user U, and when the set humidity is not a value suitable for the skin condition of the user U, performs control to increase the set humidity of the humidifier. When the humidifier is not currently operating, the control unit 22 starts the operation of the humidifier. Note that the control unit 22 may control the set humidity of the humidifier based on the current humidity of the space where the user U is, instead of the set humidity. The device is connected to the server device 20 via the network N so as to be able to communicate with each other.
[0151] As described above, the determination unit 22a according to this embodiment determines the current skin condition of the user U based on the spectrum measured by the hyperspectral camera 12, using the current estimation result and a plurality of past estimation results. This allows a determination according to the nature of the user U's skin, so that the skin condition of the user U can be determined with higher accuracy.
[0152] In the above embodiment, the user's skin condition includes at least one of the skin moisture content and the state of hidden blemishes, but is not limited thereto. The user's skin condition may be at least one of the following: the amount of resident bacteria, makeup or sunscreen removal, cosmetic cleansing state, salt content (e.g., salt content in sweat), teeth or tongue condition, inflammation such as acne, dark circles under the eyes, and grooming such as protruding nose hair.
[0153] The amount of resident bacteria is estimated based on, for example, a spectrum of a band including the absorption wavelength of the resident bacteria. The resident bacteria is a skin resident bacteria, such as Propionibacterium acnes. The absorption wavelength of the resident bacteria is, for example, 540 nm. The server device 20 can determine whether the resident bacteria is increasing based on the current amount of resident bacteria and the amount of resident bacteria at multiple past times (multiple points in time). As a recommended action, it can recommend whether or not to wash the face.
[0154] The server device 20 can determine the makeup or sunscreen removal and the cleansing state of cosmetics based on the current makeup or sunscreen state and the past makeup or sunscreen state (for example, the state immediately after applying makeup). As recommended actions, it can recommend the timing of touching up makeup, reapplication of sunscreen, re-doing cleansing, etc.
[0155] The server device 20 may estimate the health condition of the user based on the condition of the teeth, the areas left unbrushed, the tongue, etc. The server device 20 may determine the state of inflammation or swelling such as acne. The server device 20 may determine the sleep state, nutritional state, etc. according to the type of bear. When the server device 20 detects nose hair, it may determine that nose hair is sticking out.
[0156] (Other embodiments) Although the skin condition assessment method and the like according to one or more aspects of the present disclosure have been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art to the embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects of the present disclosure.
[0157] For example, in the above embodiment, an example has been described in which the information terminal has a hyperspectral camera, but the present invention is not limited thereto. The information terminal may have a multispectral camera or an ultraspectral camera instead of the hyperspectral camera. Here, the multispectral camera is a spectral camera capable of acquiring a spectrum of about 10 bands at most, and the ultraspectral camera is a spectral camera capable of acquiring a spectrum of 1000 or more bands. Even if the information terminal has a multispectral camera or an ultraspectral camera, the effect of the present application can be achieved without any change. The multispectral camera and the ultraspectral camera are examples of a spectral camera.
[0158] In the above embodiment and the like, an example has been described in which the control unit of the information terminal activates the hyperspectral camera in conjunction with activation of the image camera, but this is not limiting. The hyperspectral camera may be controlled regardless of activation of the image camera. For example, the control unit may activate the hyperspectral camera at a predetermined time interval or at a predetermined time, or may activate the hyperspectral camera in response to an input to the input unit by a user.
[0159] In the above embodiment, the case where the moisture content and the state of hidden blemishes are obtained by measuring the user's face has been described, but the present invention is not limited to this. The moisture content and the state of hidden blemishes may be obtained by measuring skin other than the user's face. The moisture content and the state of hidden blemishes may be obtained by measuring, for example, the user's neck, hands, etc.
[0160] The information terminal in the above embodiment may have a light source for measuring skin moisture content. The light source may irradiate the skin with light having a near-infrared or mid-infrared wavelength that is absorbed by water, and the hyperspectral camera may measure the amount of light reflected by the skin and returned.
[0161] In the above embodiment, the information terminal estimates the state of hidden blemishes using light with a wavelength of visible light or near-infrared light, but the present invention is not limited to this. The information terminal may estimate the state of hidden blemishes using light with a wavelength of near-ultraviolet light.
[0162] In the above embodiment, an example in which the hyperspectral camera is provided on the same surface as the image camera of the information terminal has been described, but a hyperspectral camera may also be provided on the surface opposite to the surface. FIG. 12 is a diagram showing a configuration of an information terminal 110 according to another embodiment. As shown in FIG. 12, the information terminal 110 has, for example, an image camera 11 and hyperspectral cameras 12 and 113. The hyperspectral camera 113 is provided on the surface opposite to the surface of the information terminal 10 on which the image camera 11 and the hyperspectral camera 12 are provided. Thereby, for example, the spectrum of the skin of the user U can be measured by the hyperspectral camera 12, and the spectrum of sunlight can be measured by the hyperspectral camera 113. The hyperspectral camera 113 may be activated when sunlight can be measured, for example, depending on the orientation of the information terminal 10 and the position of the sun.
[0163] The order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Some of the steps may be performed simultaneously (in parallel) with other steps, or some of the steps may not be performed.
[0164] The division of functional blocks in the block diagram is one example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be moved to other functional blocks. Functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.
[0165] The skin condition determination system may be realized by one device. Although an example in which each of the information terminal and the server device is realized by a single device has been described, each of the information terminal and the server device may be realized by a plurality of devices. For example, when the information terminal is realized by a plurality of devices, each component of the information terminal may be distributed in any manner among the plurality of devices. For example, when the server device is realized by a plurality of devices, each component of the server device may be distributed in any manner among the plurality of devices. Each component of the skin condition determination system may be distributed in any manner among the devices of the skin condition determination system. For example, at least one of the components of the information terminal may be included in the server device. For example, the server device may have an analysis unit.
[0166] The communication method between the devices included in the skin condition determining system in the above embodiment is not particularly limited, and may be wireless communication or wired communication. Wireless communication and wired communication may be combined between the devices.
[0167] Some or all of the components included in the skin condition determining system in the above-described embodiment may be configured as one system LSI (Large Scale Integration).
[0168] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and specifically, is a computer system that includes a microprocessor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.
[0169] One aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the skin condition determination method. For example, the program may be a program to be executed by a computer. One aspect of the present disclosure may be a computer-readable non-transitory recording medium in which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby making it possible to cause the device to perform each of the above-mentioned processes. [Industrial Applicability]
[0170] The present disclosure is widely applicable to devices and the like that determine a user's skin condition. [Explanation of symbols]
[0171] 1. Skin condition assessment system 10, 110 Information terminal 11 Image Camera 12, 113 Hyperspectral camera 13 Control unit (acquisition unit) 13a Analysis section (estimation section) 13b Matching section 14 Communications Department 15 Display section 16 Input section 17, 23 Memory section 20 Server device 21 Communications Department 22 Control section 22a Judgment part N Network U User
Claims
1. 1. An information processing method executed by at least one processor, comprising: A first acquisition step of acquiring a first measurement result including data obtained by a spectral camera measuring the user's skin at a first time point; a first determination step of determining a skin condition of the user based on the first measurement result, The first measurement result includes data obtained by measuring the skin of the user by operating the spectral camera in synchronization with an operation of an image camera capturing a first face image for face authentication of the user. Information processing methods.
2. The method further includes a second acquisition step of acquiring a second measurement result including data obtained by the spectral camera measuring the skin of the user at a second time point before the first time point; the second measurement result includes data obtained by measuring the skin of the user by operating the spectral camera in synchronization with an operation of the image camera to capture a second face image for face authentication of the user; In the first determination step, the skin condition is determined based on the first measurement result and the second measurement result. The information processing method according to claim 1 .
3. a first estimation step of estimating a first estimation result of a skin condition of the user at the first time point based on the first measurement result; and a second estimation step of estimating a second estimation result of the skin condition of the user at the second time point based on the second measurement result, In the first determination step, the skin condition is determined based on the first estimation result and the second estimation result. The information processing method according to claim 2 .
4. Each of the first estimation result and the second estimation result includes at least one of a skin moisture content and a state of hidden blemishes. The information processing method according to claim 3.
5. the first determination step includes, when the first biological information of the user at the first time point and the second biological information of the user at the second time point match or are similar to each other, determining the skin condition based on the first measurement result and the second measurement result. The information processing method according to any one of claims 2 to 4.
6. the first determination step includes, when first surrounding environment information of the user at the first time point and second surrounding environment information of the user at the second time point match or are similar to each other, determining the skin condition based on the first measurement result and the second measurement result; Each of the first surrounding environment information and the second surrounding environment information includes at least one information selected from the group consisting of temperature and humidity, maximum and minimum temperatures, weather information, whether the user is indoors or outdoors, and setting conditions of devices such as an air conditioner and a humidifier when the user is indoors. The information processing method according to any one of claims 2 to 4.
7. The spectral camera and the image camera are provided on a mobile terminal of the user. The information processing method according to any one of claims 1 to 4.
8. At least one processor; The at least one processor: A first acquisition step is performed to acquire a first measurement result including data obtained by measuring the user's skin at a first time point using a spectral camera; A first determination step is carried out to determine a skin condition of the user based on the first measurement result; The first measurement result includes data obtained by measuring the skin of the user by operating the spectral camera in synchronization with an operation of an image camera capturing a first face image for face authentication of the user. Processing unit.
9. A spectral camera; An image camera; At least one processor; Equipped with The at least one processor: A first acquisition step is performed to acquire a first measurement result including data obtained by measuring the user's skin at a first time point using a spectral camera; A first determination step is carried out to determine a skin condition of the user based on the first measurement result; The first measurement result includes data obtained by measuring the skin of the user by operating the spectral camera in synchronization with an operation of an image camera capturing a first face image for face authentication of the user. Mobile device.
Citation Information
Patent Citations
Skin quality classification method and cosmetics selection method
JP2012095825A
