Information processing device, processing system, and program

Abstract

This disclosure proposes a technique for determining whether the skin of a subject in an captured image is bare skin. [Solution] An information processing device comprising: a first image acquisition unit that acquires a first image including the skin of a subject; and a skin determination unit that determines whether the skin is bare skin or the degree to which makeup has been applied to the skin based on the first image.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a processing system, and a program.

Background Art

[0002] Patent Document 1 discloses a method for estimating the amount of makeup applied to a made-up face based on the change in the spread of light irradiated on the bare skin before applying makeup and the made-up face after applying makeup.

[0003] Patent Document 2 discloses a skin image analysis method that executes a process of obtaining index values of skin attributes or skin conditions for each skin patch image by inputting each normalized skin patch image into a discrimination model.

[0004] Patent Document 3 discloses an operating method of an apparatus including a control unit for controlling privacy settings of a camera and / or a microphone that monitors one or more subjects. Patent Document 3 discloses that the control unit detects the state of the one or more subjects in the acquired signal, and the state of the subject includes a naked level.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] Devices, programs, and apps are known for diagnosing the skin condition of subjects. To accurately diagnose skin condition, it is desirable to measure bare skin.

[0007] This disclosure proposes a technique for determining whether the skin of a subject in an captured image is bare skin. [Means for solving the problem]

[0008] This disclosure provides an information processing device comprising: a first image acquisition unit that acquires a first image including the skin of a subject; and a skin determination unit that determines, based on the first image, whether the skin is bare skin or the degree to which makeup has been applied to the skin. [Effects of the Invention]

[0009] According to this disclosure, it is possible to determine whether the skin of a subject in an captured image is bare skin. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an overview of a skin diagnostic device equipped with an information processing device according to the first embodiment. [Figure 2] Figure 2 is a diagram illustrating the hardware configuration of the information processing device according to the first embodiment. [Figure 3] Figure 3 is a diagram illustrating the functional configuration of a skin diagnostic device equipped with an information processing device according to the first embodiment. [Figure 4] Figure 4 is a diagram illustrating the functional configuration and processing of the information processing apparatus according to the first embodiment. [Figure 5] Figure 5 illustrates a first example of the processing performed by the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 6] Figure 6 illustrates the frequency distribution of color information in the information processing apparatus according to the first embodiment. [Figure 7] Figure 7 illustrates a second example of the processing performed by the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 8] FIG. 8 is a diagram for explaining a first modification example of a second example of the processing of the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 9] FIG. 9 is a diagram for explaining the variation in color information about the lips in the second example of the processing of the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the variation in color information about the cheeks in the second example of the processing of the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining the variation in the thickness of the eyebrows in the second example of the processing of the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining a second modification example of a second example of the processing of the skin determination unit in the information processing apparatus according to the first embodiment. [Figure 13] FIG. 13 is a diagram for explaining the functional configuration of a skin diagnosis apparatus including the information processing apparatus according to the second embodiment. [Figure 14] FIG. 14 is a diagram for explaining the processing of the motion estimation unit in the information processing apparatus according to the second embodiment. [Figure 15] FIG. 15 is a diagram for explaining the processing of the motion estimation unit in the information processing apparatus according to the second embodiment. [Figure 16] FIG. 16 is a diagram for explaining the processing of the image acquisition unit in the information processing apparatus according to the second embodiment. [Figure 17] FIG. 17 is a diagram for explaining the processing of the motion estimation unit in the information processing apparatus according to the second embodiment. [Figure 18] FIG. 18 is a diagram for explaining the functional configuration of a skin diagnosis apparatus including the information processing apparatus according to the third embodiment. [Figure 19] FIG. 19 is a flowchart for explaining the processing in a skin diagnosis apparatus including the information processing apparatus according to the third embodiment. [Figure 20] FIG. 20 is a flowchart for explaining the processing in a skin diagnosis apparatus including the information processing apparatus according to the third embodiment. [Figure 21]FIG. 21 is a diagram for explaining a functional configuration of a skin diagnosis apparatus including an information processing apparatus according to the fourth embodiment. [Figure 22] FIG. 22 is a flowchart for explaining a process in a skin diagnosis apparatus including an information processing apparatus according to the fourth embodiment. [Figure 23] FIG. 23 is a diagram for explaining a functional configuration of a skin diagnosis apparatus including an information processing apparatus according to the fifth embodiment. [Figure 24] FIG. 24 is a flowchart for explaining a process in a skin diagnosis apparatus including an information processing apparatus according to the fifth embodiment. [Figure 25] FIG. 25 is a flowchart for explaining a process in a skin diagnosis apparatus including an information processing apparatus according to the fifth embodiment. [Figure 26] FIG. 26 is a diagram for explaining a functional configuration of a skin diagnosis apparatus including an information processing apparatus according to the sixth embodiment. [Figure 27] FIG. 27 is a flowchart for explaining a process in a skin diagnosis apparatus including an information processing apparatus according to the sixth embodiment. [Figure 28] FIG. 28 is a flowchart for explaining a process in a skin diagnosis apparatus including an information processing apparatus according to the sixth embodiment.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, each embodiment of the present disclosure will be described with reference to the accompanying drawings. Note that, regarding the description in the specification and drawings according to each embodiment, components having substantially the same or corresponding functional configurations may be denoted by the same reference numerals, and redundant descriptions may be omitted.

[0012] ≪First Embodiment≫ An information processing apparatus according to the first embodiment will be described. The information processing apparatus according to the first embodiment includes a first image acquisition unit that acquires a first image including the skin of a subject, and a skin determination unit that determines whether the skin is bare skin or the degree of makeup applied to the skin based on the first image.

[0013] The first image acquired by the first image acquisition unit in the information processing apparatus according to the first embodiment is, for example, a visible image captured by a camera or the like. The first image is not limited to a visible image; it may also be, for example, an infrared image, a distance image, or a depth image. Furthermore, the first image may be point cloud data. The first image may be a moving image or a still image.

[0014] Furthermore, the first image may also contain sound information. For example, behavioral analysis may be performed using the sound information contained in the first image. For instance, if the sound information contained in the first image includes humming, it may be analyzed that the subject is in a good mood. Also, if the sound information contained in the first image includes talking to oneself or having a conversation, the mood may be determined based on the tone of voice in the talking to oneself or having a conversation, according to the content of the talking to oneself or having a conversation.

[0015] An information processing device according to the first embodiment will be described with reference to the drawings. Figure 1 is a diagram showing an overview of a skin diagnostic device 1 equipped with an information processing device 20, which is an example of an information processing device according to the first embodiment.

[0016] The skin diagnostic device 1 photographs the subject's OBJ. The skin diagnostic device 1 may photograph the subject's OBJ if the subject or the user or administrator of the skin diagnostic device 1 has given prior consent to the photograph. For example, the skin diagnostic device 1 checks for consent by referring to a storage device that stores information indicating that the subject or the user or administrator of the skin diagnostic device 1 has given consent to the photograph. By photographing the subject's OBJ, the skin diagnostic device 1 acquires an image IMG including the skin of the subject's OBJ. The skin diagnostic device 1 may acquire an image IMG if the subject or the user or administrator of the skin diagnostic device 1 has given prior consent to the acquisition of the image. For example, the skin diagnostic device 1 checks for consent by referring to a storage device that stores information indicating that the subject or the user or administrator of the skin diagnostic device 1 has given consent to the acquisition of the image.

[0017] In one embodiment, the skin diagnostic device 1 determines whether the skin is bare or not based on an image IMG including the skin of the acquired subject OBJ. This determination process may be referred to as a determination process related to bare skin. In another embodiment, the skin diagnostic device 1 determines the degree of makeup applied to the skin based on an image IMG including the skin of the acquired subject OBJ. Whether or not the skin is bare may be just one example of the degree of makeup applied to the skin. This determination process may be referred to as a determination process related to the degree of makeup.

[0018] The skin diagnostic device 1 uses an image IMG containing the skin of the acquired subject OBJ to diagnose the skin of the subject OBJ. In one embodiment, the skin diagnostic device 1 estimates the condition of the subject OBJ's skin using the image IMG used in the determination process regarding the degree of bare skin or makeup.

[0019] In another embodiment, the skin diagnostic device 1 estimates the skin condition of the subject OBJ using an image IMG (sometimes referred to as the second IMG) different from the image IMG (sometimes referred to as the first IMG) used in the determination process regarding the degree of bare skin or makeup. The second IMG may be an image that includes the skin of the subject OBJ. The second IMG may be an image with a higher resolution than the first IMG. Skin may be an example of skin. Skin condition may be an example of the subject's condition.

[0020] The skin diagnostic device 1 is, for example, a so-called smart mirror. The skin diagnostic device 1 comprises a main unit 10, an information processing device 20, an imaging device 30, and a display device 40. Each component of the skin diagnostic device 1 will be described below.

[0021] In the example of the skin diagnostic device 1 equipped with an information processing device 20, which is an example of an information processing device according to the first embodiment, a smart mirror was described. However, the processing system equipped with an imaging device (first imaging unit) and an information processing device according to the first embodiment is not limited to a smart mirror. The processing system equipped with an imaging device and an information processing device according to the first embodiment may be, for example, an information processing terminal or a server. The information processing terminal may be a mobile terminal such as a smartphone, a laptop computer, a tablet computer, or a wearable device. The mobile terminal may have communication functions. The above information processing system may also include an information processing terminal and a server. The above processing system may be a building or a mobile object. Furthermore, the location where the processing system equipped with an imaging device and an information processing device according to the first embodiment is used may be a building or a mobile object (vehicle, ship, aircraft). Examples of buildings may be residences, commercial facilities (department stores, beauty salons), nursing homes, or hospitals. The same applies to the following embodiments.

[0022] [Main body 10] The main body 10 comprises an upper housing 11 and a lower housing 12. The upper housing 11 has a mirror section 13 on its front surface. The mirror section 13 is a light-transmitting mirror, for example, a half-mirror. The skin diagnostic device 1 has a display device 40 on the back of the mirror section 13 in the upper housing 11. Since the mirror section 13 is light-transmitting, the subject OBJ can see the display device 40 provided on the back of the mirror section 13. The skin diagnostic device 1 has an imaging device 30 further behind the display device 40 in the upper housing 11. The imaging device 30 images the subject OBJ. The skin diagnostic device 1 has an information processing device 20 in the lower housing 12.

[0023] [Imaging device 30] The imaging device 30 images the subject OBJ.

[0024] [Display device 40] The display device 40 displays information.

[0025] [Information processing device 20] The information processing device 20 controls the entire skin diagnostic device 1.

[0026] The hardware configuration of the information processing device 20 will now be described. Figure 2 is a diagram illustrating the hardware configuration of an information processing device 20, which is an example of an information processing device according to the first embodiment.

[0027] The information processing device 20 is, for example, a computer. The information processing device 20 includes a CPU (Central Processing Unit) 21, RAM (Random Access Memory) 22, and ROM (Read Only Memory) 23. The information processing device 20 also includes a storage interface 24i, a communication interface 25i, an external interface 26i, and a display interface 27i. The CPU 21, RAM 22, ROM 23, storage interface 24i, communication interface 25i, external interface 26i, and display interface 27i are each connected to a bus. For example, a storage medium 24 is connected to the storage interface 24i. An imaging device 30 is connected to the external interface 26i. A display device 40 is connected to the display interface 27i.

[0028] The CPU 21 is an arithmetic unit that reads programs (applications) from storage devices such as ROM 23 and storage media 24 onto RAM 22 and executes processing.

[0029] RAM22 is a volatile semiconductor memory that temporarily holds programs (applications), for example.

[0030] ROM23 is a non-volatile semiconductor memory that can retain programs (applications), etc., even when the power is turned off. ROM23 stores programs such as the BIOS that are executed at startup, as well as various settings such as OS settings and network settings.

[0031] The storage interface 24i is an interface to an external storage device such as a storage medium 24. The information processing device 20 uses the storage interface 24i to read from and write to the storage medium 24.

[0032] The storage medium 24 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0033] The Communication I / F25i is an interface for connecting to external networks such as the Internet. The Communication I / F25i connects to external networks using, for example, a wired communication method.

[0034] External I / F26i is an interface for connecting the information processing device 20 to external devices. External I / F26i is, for example, USB (Universal Serial Bus) or GP-IB (General Purpose Interface Bus). The imaging device 30 is connected to external I / F27i.

[0035] The display I / F 27i is an interface in the information processing device 20 that connects to an external display device. The display I / F 27i is, for example, VGA (Video Graphics Array), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or DisplayPort. The display device 40 is connected to the display I / F 27i.

[0036] The information processing device 20 performs various processes described later by executing a program (application) in the hardware configuration described above. Details of the processing performed by the information processing device 20 will be described later.

[0037] Next, the functional configuration of the information processing device 20 will be described. Figure 3 is a diagram illustrating the functional configuration of the skin diagnostic device 1 equipped with the information processing device 20, which is an example of the information processing device according to the first embodiment. Figure 4 is a diagram illustrating the functional configuration and processing of the information processing device 20, which is an example of the information processing device according to the first embodiment. The following description will explain the processing of the program that executes the processing in each part of the information processing device 20.

[0038] The information processing device 20 includes an image acquisition unit 20A, a skin determination unit 20B, an estimation unit 20M, and an output unit 20D.

[0039] [Image acquisition unit 20A] The image acquisition unit 20A controls the imaging device 30 to acquire an image of the subject OBJ including the skin. The image acquired by the image acquisition unit 20A is an image that includes, for example, the face, as part of the body part where the skin is located. The image acquired by the image acquisition unit 20A may also include, for example, the hands, chest, abdomen, back, and feet, as an example of the body part where the skin is located.

[0040] The image acquisition unit 20A transmits control commands to the imaging device 30. The imaging device 30 operates based on the control commands from the image acquisition unit 20A. The image acquisition unit 20A controls the imaging device 30 by transmitting control commands to the imaging device 30.

[0041] The image acquisition unit 20A transmits a control command to the imaging device 30 to take an image. The image acquisition unit 20A then transmits a control command to the imaging device 30 to transmit the image it has captured to the information processing device 20. The image acquisition unit 20A acquires the image transmitted from the imaging device 30.

[0042] The image acquisition unit 20A outputs the image IMG acquired from the imaging device 30 to the skin detection unit 20B.

[0043] The image acquisition unit 20A may, for example, output an image as an image IMG after performing preprocessing such as noise reduction on an image acquired from the imaging device 30.

[0044] [Skin determination section 20B] The skin determination unit 20B determines, based on the image IMG acquired from the image acquisition unit 20A, whether the skin included in the image IMG is bare skin or the degree to which makeup has been applied to the skin included in the image IMG.

[0045] The following describes the processing performed by the skin detection unit 20B. Here, we will explain the case where skin is detected on a face included in the image IMG.

[0046] (First example of skin detection processing) The following describes the process in the skin determination unit 20B for determining whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG, based on color information or shape at specific parts of the face included in the image IMG. Figure 5 is a diagram illustrating a first example of the processing of the skin determination unit 20B in the information processing device 20, which is an example of an information processing device according to the first embodiment.

[0047] The skin detection unit 20B detects faces included in the image IMG, for example. The skin detection unit 20B then extracts specific parts of the faces included in the image IMG, such as lips, cheeks, and eyebrows. Figure 5 shows the numerical results of the evaluation of lip color information, cheek color information, and eyebrow size in seven images.

[0048] Figure 5 schematically illustrates the difference between bare skin and skin with makeup. Images 1, 2, 6, and 7 each show a face with bare skin, while images 3, 4, and 5 each show a face with makeup, i.e., a face with makeup on.

[0049] The skin determination unit 20B determines, for example, whether the face included in the image IMG is bare skin or the degree of makeup applied to the skin of the face included in the image IMG, based on lip color information, for example, hue or saturation. The skin determination unit 20B may also determine, for example, whether the face included in the image IMG is bare skin or the degree of makeup applied to the skin of the face included in the image IMG, based on cheek color information, for example, hue or saturation. Furthermore, the skin determination unit 20B may determine, for example, whether the face included in the image IMG is bare skin or the degree of makeup applied to the skin of the face included in the image IMG, based on eyebrow shape, for example, eyebrow size.

[0050] As shown in Figure 5, lip color information, cheek color information, and eyebrow size differ between bare skin and made-up skin. The skin determination unit 20B may, for example, perform thresholding using a predetermined threshold to determine whether the face included in the image IMG is bare skin or not. Alternatively, the skin determination unit 20B may use the lip color information, cheek color information, and eyebrow size to determine the degree of makeup applied to the skin of the face included in the image IMG.

[0051] (Second example of skin detection processing) In the skin determination unit 20B, the process of determining whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG is described based on the frequency distribution of color information in specific areas of the face included in the image IMG. Figure 6 is a diagram illustrating the frequency distribution of color information in the information processing device 20, which is an example of an information processing device according to the first embodiment.

[0052] Figure 6 schematically shows the frequency distribution of lip color information for multiple images (IMG), including both bare skin and made-up skin, with respect to the number of images. As shown in Figure 6, the distribution for bare skin and the distribution for made-up skin have different peaks and spreads. As shown in Figure 6, when comparing the distribution for bare skin and the distribution for made-up skin, the distribution for bare skin has less variability.

[0053] For example, the appearance of makeup after application can vary from day to day due to factors such as makeup breakdown. Therefore, the variance of each indicator in the image will be greater for skin with makeup on.

[0054] The skin determination unit 20B uses the difference in distribution between bare skin and made-up skin, as shown in Figure 6, to determine whether the skin included in the image IMG is bare skin or the degree to which makeup has been applied to the skin included in the image IMG.

[0055] The specific processing steps will now be explained. Figure 7 is a diagram illustrating a second example of the processing of the skin determination unit 20B in the information processing device 20, which is an example of an information processing device according to the first embodiment. The skin determination unit 20B assumes that the frequency distribution shown in Figure 7 is a mixture distribution of two normal distributions and determines a threshold using statistical parameters. Based on the determined threshold, the skin determination unit 20B determines whether the skin contained in the image IMG is bare skin or the degree to which makeup has been applied to the skin contained in the image IMG.

[0056] For example, the skin detection unit 20B derives two normal distributions using the Expectation-Maximization (EM) algorithm of a mixture of normal distributions. Then, for example, the skin detection unit 20B estimates the mean values ​​of each of the two normal distributions. The skin detection unit 20B may then use the midpoint of the estimated mean values ​​as a threshold.

[0057] Furthermore, for example, the skin determination unit 20B estimates the standard deviations of two normal distributions (for example, standard deviation σ1 and standard deviation σ2). Then, the standard deviation σ1 of the normal distribution with the smaller mean may be used as the threshold, and the midpoint between the positive side of the standard deviation σ1 of the mean and the negative side of the standard deviation σ2 of the normal distribution with the larger mean may be used as the threshold.

[0058] Next, we will describe another example of processing. Figure 8 is a diagram illustrating a modified example of the processing of the skin determination unit 20B in the information processing device 20, which is an example of the information processing device according to the first embodiment. The skin determination unit 20B may determine a threshold using two peak values ​​in the frequency distribution shown in Figure 8. For example, the skin determination unit 20B may use the average value of the two peak values ​​as the threshold. Based on the determined threshold, the skin determination unit 20B determines whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG.

[0059] Next, we will explain the threshold values ​​for each body part. Figure 9 is a diagram illustrating the variation in color information for the lips in a second example of processing by the skin determination unit 20B in the information processing device 20, which is an example of the information processing device according to the first embodiment. Figure 10 is a diagram illustrating the variation in color information for the cheeks in a second example of processing by the skin determination unit 20B in the information processing device 20, which is an example of the information processing device according to the first embodiment.

[0060] As described above, the skin determination unit 20B determines, based on a threshold, whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG. On the other hand, the skin determination unit 20B needs to determine whether a value greater than the threshold indicates bare skin, or whether a value less than the threshold indicates bare skin.

[0061] When comparing bare skin and skin with makeup, as mentioned above, bare skin shows less variation compared to skin with makeup. For example, as shown in Figure 9, when looking at the frequency distribution of saturation as color information for lips, there are distributions with small variation and distributions with large variation. The skin determination unit 20B determines that the distribution with small variation is bare skin. In other words, the skin determination unit 20B determines that the distribution with lower values ​​is bare skin. Therefore, the skin determination unit 20B determines that the skin is bare if the value is smaller than the threshold.

[0062] Furthermore, as shown in Figure 10, for example, when we look at the frequency distribution of saturation as cheek color information, there are distributions with small variations and distributions with large variations. The skin determination unit 20B determines that the distribution with small variations is bare skin. In other words, the skin determination unit 20B determines that the distribution with large values ​​is bare skin. Therefore, the skin determination unit 20B determines that it is bare skin if the value is greater than the threshold.

[0063] In the example described above, color information was used, but the determination may also be made using shape. Figure 11 is a diagram illustrating the variation in eyebrow thickness in a second example of the processing of the skin determination unit 20B in the information processing device 20, which is an example of an information processing device according to the first embodiment.

[0064] Regarding eyebrow thickness, when comparing bare skin and skin with makeup, as mentioned above, bare skin shows less variation compared to skin with makeup. Therefore, the skin determination unit 20B determines that the distribution with less variation is bare skin. In other words, the skin determination unit 20B determines that the distribution with lower values ​​is bare skin. Therefore, the skin determination unit 20B determines that the skin is bare if the value is smaller than the threshold.

[0065] In the example described above, two distributions were explained, but for example, the skin determination unit 20B may use three or more distributions to determine whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG. Figure 12 is a diagram illustrating a second modification of a second example of the processing of the skin determination unit 20B in the information processing device 20, which is an example of the information processing device according to the first embodiment.

[0066] In the example shown in Figure 12, the skin determination unit 20B makes a determination between bare skin, skin with natural-looking makeup, and skin with heavy makeup. As shown in Figure 12, the frequency distributions differ for each of the three cases. Therefore, the skin determination unit 20B calculates thresholds between each of the three distributions to determine whether the skin included in the image IMG is bare skin or the degree of makeup applied to the skin included in the image IMG.

[0067] Furthermore, the skin determination unit 20B may use artificial intelligence-related technology to input an image IMG into an estimation model to determine whether the skin of the subject OBJ is bare or the degree of makeup applied to the skin. Alternatively, an image IMG in which the subject OBJ's skin is determined to be bare may be used as training data for the estimation model to learn.

[0068] [Estimation section 20M] The estimation unit 20M estimates the state of the subject's OBJ. The estimation unit 20M may determine the state of the subject's OBJ using, for example, an image IMG that the skin determination unit 20B has determined to include bare skin. The state of the subject's OBJ may include, for example, the physical state or mental state. The physical state may include the skin state, the state of blood vessels, the state of blood flow and / or blood pressure, and the state of the autonomic nervous system. The estimation unit 20M may also estimate the degree of wrinkles, blemishes, shine, dullness, uneven color, pore size, number of pores, or surface density of the subject's OBJ. The above degrees may be examples of indicators showing the state of the skin. Examples of the mental state include psychological or emotional states. The emotional state may be expressed, for example, by the degree of pleasure or displeasure, and / or the degree of arousal or non-arousal. The emotional state may also be a state represented by Russell's cyclic structure model. The process of estimating the state of the subject's OBJ may be referred to as diagnosis.

[0069] [Output section 20D] The output unit 20D outputs information regarding the state of the subject OBJ estimated by the estimation unit 20M.

[0070] ≪Second Embodiment≫ The information processing apparatus according to the second embodiment further includes a motion estimation unit that estimates the movements of the subject, in addition to the information processing apparatus according to the first embodiment. The skin diagnosis unit in the information processing apparatus according to the second embodiment determines, based on the estimation results of the motion estimation unit, whether or not cosmetics have been applied to the subject's skin, or whether or not cosmetics have been removed from the subject's skin. Then, based on the result of this determination, the skin diagnosis unit in the information processing apparatus according to the second embodiment determines whether or not the skin is bare skin or the degree to which cosmetics have been applied to the skin.

[0071] The configuration of the information processing device according to the second embodiment will be omitted here, as it will be explained by referring to the description of the information processing device according to the first embodiment. The functional configuration of the information processing device according to the second embodiment will be described in detail with reference to the drawings. Figure 13 is a diagram illustrating the functional configuration of a skin diagnostic device 2 equipped with an information processing device 120, which is an example of an information processing device according to the second embodiment.

[0072] The skin diagnostic device 2 comprises a main unit 10, an information processing device 20, an imaging device 30, a detection device 35, and a display device 40. The individual components of the skin diagnostic device 1 will now be described.

[0073] The information processing device 120 includes an image acquisition unit 120A, a skin determination unit 120B, a motion estimation unit 120C, an estimation unit 120M, and an output unit 120D.

[0074] [Image acquisition unit 120A] The image acquisition unit 120A includes the functions of the image acquisition unit 20A.

[0075] [Motion estimation unit 120C] The motion estimation unit 120C estimates the motion of the subject object object (OBJ) based on the image IMG of the OBJ and the output of the detection device 35 that detects the motion of the OBJ. The detection device 35 can be any type of detection device that can detect information for estimating the motion of the OBJ. For example, the detection device 35 may be an imaging device or a human presence sensor. Alternatively, the detection device 35 may be a mobile device carried by the OBJ, such as a smartwatch or smartphone.

[0076] The processing of the motion estimation unit 120C will now be explained. Figure 14 is a diagram illustrating the processing of the motion estimation unit 120C in an information processing device 120, which is an example of an information processing device according to the second embodiment.

[0077] As shown in Figure 14 as an example, let's assume that the subject OBJ wakes up in the morning, brushes its teeth, puts on makeup, eats, exercises, removes its makeup, takes a bath, does skincare, brushes its teeth, and goes to bed.

[0078] The motion estimation unit 120C analyzes the behavior of the subject object (OBJ). More specifically, the motion estimation unit 120C estimates the behavior of the subject object (OBJ) based, for example, on the image IMG acquired by the image acquisition unit 120A and / or the output of the detection device 35. Specific examples of methods for estimating the behavior of an object included in an image include various known methods and various methods to be developed in the future. The motion estimation unit 120C may determine what kind of behavior the subject object (OBJ) is performing from the image IMG using image recognition or the like, and then estimate the behavior. The motion estimation unit 120C may also estimate the behavior by detecting the position of the subject object (OBJ) from the detection device 35. The motion estimation unit 120C may also detect the position of the subject object (OBJ) from the output of a human presence sensor, which is a detection device 35 provided at various locations. The motion estimation unit 120C may also detect the position of the subject object (OBJ) from the output of a portable terminal, which is a detection device 35 carried by the subject object (OBJ).

[0079] The motion estimation unit 120C determines key actions such as waking up, brushing teeth, applying makeup, eating, exercising, removing makeup, bathing, skincare, and going to sleep, as shown in Figure 14.

[0080] The motion estimation unit 120C estimates, in particular, key actions such as applying makeup and removing makeup. For example, when the image IMG recognizes that the subject OBJ is applying makeup in front of the skin diagnostic device 2, which is a smart mirror, the motion estimation unit 120C estimates that the subject is applying makeup. Also, for example, when the image IMG recognizes that the subject OBJ is removing makeup in front of the skin diagnostic device 2, which is a smart mirror, the motion estimation unit 120C estimates that the subject is removing makeup. Note that the location where the subject OBJ performs the makeup and / or makeup removal actions is not limited to within the field of view of the imaging device installed in the skin diagnostic device 2. In other embodiments, the subject OBJ may perform the makeup and / or makeup removal actions within the field of view of an imaging device different from the imaging device installed in the skin diagnostic device 2. Even in such cases, the motion estimation unit 120C can acquire an image of the subject OBJ performing the makeup or makeup removal action and determine whether the subject OBJ is performing the makeup or makeup removal action based on the image.

[0081] The motion estimation unit 120C determines that the subject OBJ's skin is made up (makeup-covered) from the time the makeup action is performed until the makeup removal action is performed. The motion estimation unit 120C also determines that the subject OBJ's skin is bare from the time the makeup removal action is performed until the makeup action is performed.

[0082] The motion estimation unit 120C may, for example, determine that the skin of the subject OBJ is bare skin when conditions are met that combine multiple actions. Figure 15 is a diagram illustrating the processing of the motion estimation unit 120C in an information processing device 120, which is an example of an information processing device according to the second embodiment.

[0083] The motion estimation unit 120C may, for example, determine that the skin changes from made-up skin to bare skin when three actions (movements) are combined, such as the makeup removal action, the bathing action, and the towel drying action, as shown in Figure 15. The motion estimation unit 120C may also consider it undefined whether the skin is made-up or bare between the makeup removal action, the bathing action, and the towel drying action.

[0084] Furthermore, the motion estimation unit 120C may assign points to multiple key actions and determine that the person is bare-skinned when the total score of the determined actions exceeds a threshold. For example, points of 50, 80, 80, 80, 80, 50, and 10 could be assigned to makeup removal, face washing, bathing, towel drying, hair drying, lotion / emulsion, and brushing teeth before bed, respectively. The motion estimation unit 120C may then add the scores of the detected key actions and determine that the person is bare-skinned if the total score exceeds a threshold. In addition, for example, the motion estimation unit 120C may add further points to the summed score as time passes.

[0085] Furthermore, the image acquisition unit 120A may determine whether an image IMG is suitable for skin diagnosis and select an image suitable for skin diagnosis. Figure 16 is a diagram illustrating the processing of the image acquisition unit 120A in an information processing device 120, which is an example of an information processing device according to the second embodiment.

[0086] The image acquisition unit 120A may discard an image if, for example, the orientation or gaze direction of the subject OBJ in the image IMG is inappropriate, the brightness in the image IMG is insufficient, or the image IMG is unclear. In other words, the image acquisition unit may include a condition determination unit that determines whether or not the image IMG meets predetermined conditions or the degree to which it meets them. For example, the condition determination unit may determine whether or not the brightness or clarity in the image IMG meets predetermined conditions or the degree to which it meets them. Alternatively, for example, the condition determination unit may determine whether or not the orientation of the face or the direction of the gaze in the face meets predetermined conditions or the degree to which it meets them. The image acquisition unit 120A may then select only the image IMGs that it deems suitable for skin diagnosis.

[0087] Furthermore, if the motion estimation unit 120C determines that the subject OBJ's face, for example, is wet, it may exclude it as an image to be used for skin diagnosis, even if it determines that the skin is bare. In other words, the motion estimation unit 120C may determine whether the subject OBJ's skin is dry or not. Figure 17 is a diagram illustrating the processing of the motion estimation unit 120C in an information processing device 120, which is an example of an information processing device according to the second embodiment.

[0088] For example, as shown in Figure 17, after removing makeup or after bathing, the skin is considered bare, but there is a high probability that the skin is wet. Therefore, the motion estimation unit 120C may choose not to use an image of bare skin when there is a high probability that the skin is wet.

[0089] The motion estimation unit 120C may determine that the skin is not bare during the period after the application determination until a predetermined event occurs. The predetermined event may be, for example, the passage of a predetermined period of time, the arrival of a predetermined time (such as a change of date), or the next removal determination.

[0090] [Estimation part 120M] The estimation unit 120M estimates the state of the subject OBJ. The estimation unit 120M performs the same processing as the estimation unit 20M. In addition, the estimation unit 120M may further estimate the state of the subject OBJ based on the results of the action analysis of the action estimation unit 120C. As described above, examples of the state of the subject OBJ include the physical state and the mental state.

[0091] The estimation unit 120M may use the estimation results from the motion estimation unit 120C as an auxiliary tool to estimate the state of the subject OBJ, or it may use the estimation results from the motion estimation unit 120C to perform preprocessing for the estimation of the state of the subject OBJ. For example, based on the estimation results from the motion estimation unit 120C, the estimation unit 120M determines one or more images from among multiple images in which the subject OBJ was captured to be used for the estimation process in the estimation unit 120M. Based on the estimation results from the motion estimation unit 120C, the estimation unit 120M may determine which estimation model from among multiple estimation models will be used for the estimation process in the estimation unit 120M. Based on the estimation results from the motion estimation unit 120C, the estimation unit 120M may adjust the parameters of the estimation model used for the estimation process in the estimation unit 120M.

[0092] In other embodiments, the estimation unit 120M may derive the physical activity level of the subject OBJ based on the estimation results in the motion estimation unit 120C. Examples of physical activity levels include the degree of rest or activity. The estimation unit 120M may perform the various processes described above according to the physical activity level of the subject OBJ. For example, the estimation unit 120M estimates the state of the subject OBJ based on an image taken under appropriate imaging conditions, where the subject OBJ's skin is bare or the degree of makeup applied to the skin meets predetermined criteria, the subject OBJ is in a resting state, and the image is captured. When the subject OBJ is in a resting state, a more reliable diagnostic result can be obtained compared to when the subject OBJ is actively moving. Examples of the above imaging conditions include conditions related to resolution, conditions related to clarity, and / or conditions related to illumination or brightness. Examples of brightness conditions include conditions related to F-number, shutter speed, and / or ISO sensitivity.

[0093] Physical activity levels can vary depending on the state of food intake. Examples of physical activity levels include the physical state after, before, immediately before, immediately after, or between meals. For example, the complexion may differ between immediately after a meal and after an appropriate amount of time has passed since the meal. Therefore, the estimation unit 120M may perform the various processes described above according to the state of food intake of the subject OBJ.

[0094] [Output section 120D] The output unit 120D outputs information regarding the state of the subject OBJ estimated by the estimation unit 120M.

[0095] <Image acquisition in information processing equipment> This section describes the method for acquiring images in the information processing device according to this embodiment. When performing skin diagnosis, it is desirable to perform the diagnosis using high-resolution images for skin diagnosis. Therefore, this section will explain how to acquire high-resolution images.

[0096] [When it is possible to determine whether the skin is bare or made up based solely on the image, and high-resolution images can be captured at all times.] This section describes a case where the information processing device can determine whether the skin is bare or made up based solely on the image, and can continuously capture high-resolution images. Figure 18 is a diagram illustrating the functional configuration of a skin diagnostic device 3, which is an example of a skin diagnostic device equipped with the information processing device according to the third embodiment. Figures 19 and 20 are flowcharts illustrating the processing in the skin diagnostic device 3, which is an example of a skin diagnostic device equipped with the information processing device according to the third embodiment.

[0097] The skin diagnostic device 3 comprises a bare skin determination unit, a skin diagnostic image acquisition unit, a diagnostic image selection unit, an image storage unit, a diagnostic unit, and a display unit.

[0098] This section explains the processing performed by the skin diagnostic device 3. First, we will explain the process of acquiring and storing images.

[0099] (Step S10) The skin diagnostic device 3 determines whether the subject is within the area where skin diagnostic images can be acquired. If the subject is within the area where skin diagnostic images can be acquired (YES in step S10), the skin diagnostic device 3 proceeds to step S20. If the subject is not within the area where skin diagnostic images can be acquired (NO in step S10), the skin diagnostic device 3 proceeds to step S40.

[0100] (Step S20) The skin diagnostic device 3 acquires images from the skin diagnostic image acquisition unit.

[0101] (Step S30) The skin diagnostic device 3 stores the acquired images in its image storage unit.

[0102] (Step S40) The skin diagnostic device 3 determines whether to continue the process. If the process is to continue (YES in step S40), the skin diagnostic device 3 returns to step S10 and repeats the process. If the process is not to continue (NO in step S40), the skin diagnostic device 3 terminates the process.

[0103] Next, we will explain the process of diagnosing the images. The skin diagnostic device 3 performs the process of diagnosing the images in parallel with the process of acquiring and storing the images.

[0104] (Step S110) The skin diagnostic device 3 selects an image from the image storage unit.

[0105] (Step S120) Next, the skin diagnostic device 3 determines whether the selected image is suitable for diagnosis. If the selected image is suitable for diagnosis (YES in step S120), the skin diagnostic device 3 proceeds to step S130. If the selected image is not suitable for diagnosis (NO in step S120), the skin diagnostic device 3 proceeds to step S150.

[0106] (Step S130) Next, the skin diagnostic device 3 determines whether the selected image includes bare skin. If the selected image includes bare skin (YES in step S130), the skin diagnostic device 3 proceeds to step S130. If the selected image does not include bare skin (NO in step S130), the skin diagnostic device 3 proceeds to step S150.

[0107] (Step S140) The diagnostic department uses the selected images to perform a skin diagnosis.

[0108] (Step S150) The skin diagnostic device 3 determines whether to continue the process. If the process is to continue (YES in step S150), the skin diagnostic device 3 returns to step S110 and repeats the process. If the process is not to continue (NO in step S150), the skin diagnostic device 3 terminates the process.

[0109] [When it is possible to determine from the image alone whether the skin is bare or made up, and when capturing high-resolution images after determining if they are in an appropriate state.]

[0110] This section describes a case where an information processing device can determine whether the skin is bare or made up based solely on the image, and then determines whether a high-resolution image is in an appropriate state before capturing it. Figure 21 is a diagram illustrating the functional configuration of a skin diagnostic device 4, which is an example of a skin diagnostic device equipped with an information processing device according to the fourth embodiment. Figure 22 is a flowchart illustrating the processing in a skin diagnostic device 4, which is an example of a skin diagnostic device equipped with an information processing device according to the fourth embodiment.

[0111] The skin diagnostic device 4 comprises a bare skin determination unit, a skin diagnostic image acquisition unit, a diagnostic image selection unit, an image storage unit, a diagnostic unit, and a display unit.

[0112] (Step S210) The skin diagnostic device 4 determines whether the subject, posture / position, and shooting environment are suitable for skin diagnosis. If the subject, posture / position, and shooting environment are suitable for skin diagnosis (YES in step S210), the skin diagnostic device 4 proceeds to step S220. If the subject, posture / position, and shooting environment are not suitable for skin diagnosis (NO in step S210), the skin diagnostic device 4 proceeds to step S260.

[0113] (Step S220) Next, the skin diagnostic device 4 determines whether the selected image includes bare skin. If the selected image includes bare skin (YES in step S220), the skin diagnostic device 4 proceeds to step S230. If the selected image does not include bare skin (NO in step S220), the skin diagnostic device 4 proceeds to step S260.

[0114] (Step S230) The skin diagnostic device 4 acquires images using the skin diagnostic image acquisition unit.

[0115] (Step S240) The skin diagnostic device 4 stores the acquired images in the image storage unit.

[0116] (Step S250)

[0117] The diagnostic department uses the acquired images to perform a skin diagnosis.

[0118] (Step S260) The skin diagnostic device 4 determines whether to continue the process. If the process is to continue (YES in step S260), the skin diagnostic device 4 returns to step S210 and repeats the process. If the process is not to continue (NO in step S260), the skin diagnostic device 4 terminates the process.

[0119] [When it is not possible to determine from the image alone whether the skin is bare or made up, and high-resolution images can be taken at all times.] This section describes a case where the information processing device cannot determine from the image alone whether the skin is bare or made up, and where high-resolution images can be continuously captured. Figure 23 is a diagram illustrating the functional configuration of a skin diagnostic device 5, which is an example of a skin diagnostic device equipped with the information processing device according to the fifth embodiment. Figures 24 and 25 are flowcharts illustrating the processing in the skin diagnostic device 5, which is an example of a skin diagnostic device equipped with the information processing device according to the fifth embodiment.

[0120] The skin diagnostic device 5 comprises a bare skin determination unit, a skin diagnostic image acquisition unit, a diagnostic image selection unit, an image storage unit, a diagnostic unit, a display unit, a communication unit, a time acquisition unit, an activity acquisition unit, and an activity recording unit.

[0121] This section will explain the processing performed by the skin diagnostic device 5. First, we will explain the process of acquiring and storing images.

[0122] (Step S310) The skin diagnostic device 5 determines whether the subject is within the area where skin diagnostic images can be acquired. If the subject is within the area where skin diagnostic images can be acquired (YES in step S310), the skin diagnostic device 5 proceeds to step S320. If the subject is not within the area where skin diagnostic images can be acquired (NO in step S310), the skin diagnostic device 5 proceeds to step S350.

[0123] (Step S320) The skin diagnostic device 5 acquires images from the skin diagnostic image acquisition unit.

[0124] (Step S330) The skin diagnostic device 5 links images with time. In other words, images and time are linked.

[0125] (Step S340) The skin diagnostic device 5 stores the acquired images in its image storage unit.

[0126] (Step S350) The skin diagnostic device 5 determines whether to continue the process. If the process is to continue (YES in step S350), the skin diagnostic device 5 returns to step S310 and repeats the process. If the process is not to continue (NO in step S350), the skin diagnostic device 5 terminates the process.

[0127] Next, we will explain the process of diagnosing the images. The skin diagnostic device 5 performs the process of diagnosing the images in parallel with the process of acquiring and storing the images.

[0128] (Step S410) The skin diagnostic device 5 selects an image from the image storage unit.

[0129] (Step S420) Next, the skin diagnostic device 5 determines whether the selected image is suitable for diagnosis. If the selected image is suitable for diagnosis (YES in step S420), the skin diagnostic device 5 proceeds to step S430. If the selected image is not suitable for diagnosis (NO in step S420), the skin diagnostic device 5 proceeds to step S450.

[0130] (Step S430) Next, the skin diagnostic device 5 determines whether the image selected from the time of activity includes bare skin. If the selected image includes bare skin (YES in step S430), the skin diagnostic device 5 proceeds to step S440. If the selected image does not include bare skin (NO in step S430), the skin diagnostic device 5 proceeds to step S450.

[0131] (Step S440) The diagnostic department uses the selected images to perform a skin diagnosis.

[0132] (Step S450) The skin diagnostic device 5 determines whether to continue the process. If the process is to continue (YES in step S450), the skin diagnostic device 5 returns to step S410 and repeats the process. If the process is not to continue (NO in step S450), the skin diagnostic device 5 terminates the process.

[0133] [When it is not possible to determine from the image alone whether the skin is bare or made up, and a high-resolution image is used to determine if it is in an appropriate state before capturing.] This section describes a case where the information processing device cannot determine from the image alone whether the skin is bare or made up, and instead determines whether a high-resolution image is in an appropriate state before capturing it. Figure 26 is a diagram illustrating the functional configuration of a skin diagnostic device 6, which is an example of a skin diagnostic device equipped with the information processing device according to the sixth embodiment. Figures 27 and 28 are flowcharts illustrating the processing in the skin diagnostic device 6, which is an example of a skin diagnostic device equipped with the information processing device according to the sixth embodiment.

[0134] The skin diagnostic device 6 comprises a bare skin determination unit, a skin diagnostic image acquisition unit, a diagnostic image selection unit, an image storage unit, a diagnostic unit, a display unit, a communication unit, a time acquisition unit, an activity acquisition unit, and an activity recording unit.

[0135] This section will explain the processing performed by the skin diagnostic device 6. First, we will explain the process of acquiring and storing images.

[0136] (Step S510) The skin diagnostic device 6 determines whether the subject, posture / position, and shooting environment are suitable for skin diagnosis. If the subject, posture / position, and shooting environment are suitable for skin diagnosis (YES in step S510), the skin diagnostic device 6 proceeds to step S520. If the subject, posture / position, and shooting environment are not suitable for skin diagnosis (NO in step S510), the skin diagnostic device 6 proceeds to step S550.

[0137] (Step S520) The skin diagnostic device 6 acquires images from the skin diagnostic image acquisition unit.

[0138] (Step S530) The skin diagnostic device 6 links images with time. In other words, images and time are linked.

[0139] (Step S540) The skin diagnostic device 6 stores the acquired images in its image storage unit.

[0140] (Step S550) The skin diagnostic device 6 determines whether to continue the process. If the process is to continue (YES in step S550), the skin diagnostic device 6 returns to step S510 and repeats the process. If the process is not to continue (NO in step S550), the skin diagnostic device 6 terminates the process.

[0141] Next, the process of diagnosing the images will be explained. The skin diagnostic device 6 performs the process of diagnosing the images in parallel with the process of acquiring and storing the images.

[0142] (Step S610) The skin diagnostic device 6 selects an image from the image storage unit.

[0143] (Step S620) Next, the skin diagnostic device 6 determines whether the selected image includes bare skin based on the time of the action. If the selected image includes bare skin (YES in step S620), the skin diagnostic device 6 proceeds to step S630. If the selected image does not include bare skin (NO in step S620), the skin diagnostic device 6 proceeds to step S640.

[0144] (Step S630) The diagnostic department uses the selected images to perform a skin diagnosis.

[0145] (Step S640) The skin diagnostic device 6 determines whether to continue the process. If the process is to continue (YES in step S640), the skin diagnostic device 6 returns to step S610 and repeats the process. If the process is not to continue (NO in step S640), the skin diagnostic device 6 terminates the process.

[0146] The information processing apparatus according to the third to sixth embodiments may include a first information acquisition unit that acquires first information associated with the estimation result of the motion estimation unit and first time information, which is the time or period during which the first image was captured. Furthermore, the information processing apparatus according to the third to sixth embodiments may capture a second image, which includes the skin of a subject, using a second imaging device different from the first imaging device that captured the first image. Furthermore, the information processing apparatus according to the third to sixth embodiments may include a second information acquisition unit that acquires second information associated with second time information, which is the time or period during which the second image was captured.

[0147] Furthermore, the second imaging device may differ from the first imaging device in at least one of its resolution, field of view, imaging interval, and frame rate. In one embodiment, the resolution of the image captured by the second imaging device is greater than the resolution of the image captured by the first imaging device. In another embodiment, the resolution of the image captured by the first imaging device is greater than the resolution of the image captured by the second imaging device. In yet another embodiment, the imaging interval of the image captured by the second imaging device may be smaller than the imaging interval of the image captured by the first imaging device, and the frame rate of the image captured by the second imaging device may be greater than the frame rate of the image captured by the first imaging device. In yet another embodiment, the imaging interval of the image captured by the first imaging device may be smaller than the imaging interval of the image captured by the second imaging device, and the frame rate of the image captured by the first imaging device may be greater than the frame rate of the image captured by the second imaging device.

[0148] Furthermore, based on the behavioral analysis (estimated processing by the motion estimation unit) of the first image, a period in which the degree of bare skin / makeup matches predetermined conditions may be identified. From one or more second images, a second image captured within the above period may be extracted. From the extracted second images, a second image whose imaging conditions match predetermined conditions may be extracted. Note that this step may or may not be performed, and may be performed before the bare skin determination. Based on the extracted second image, the subject's condition may be estimated, i.e., diagnosed.

[0149] The information processing apparatus according to the third to sixth embodiments may include a second image acquisition unit that acquires a second image, which is an image including the skin of a subject, and which is captured by a second imaging device different from the first imaging device that captured the first image. Furthermore, the information processing apparatus according to the third to sixth embodiments may include a first information generation unit that associates the estimation result of the motion estimation unit with first time information, which is the time or period during which the first image was captured. Moreover, the information processing apparatus according to the third to sixth embodiments may include a second information generation unit that associates the second image with second time information, which is the time during which the second image was captured.

[0150] The second image is, for example, a visible image captured by a camera or the like. The second image is not limited to a visible image; it may also be an infrared image, a distance image, or a depth image. Furthermore, the second image may be point cloud data. The second image may be a moving image or a still image. The second image may also contain sound information.

[0151] Furthermore, the information processing device according to the third to sixth embodiments may perform bare skin determination based on the time of the action and the time of imaging.

[0152] According to one embodiment, the privacy of the subject may be considered during image capture / image acquisition. The level of privacy may also be determined based on image analysis and / or equipment usage. If the level of privacy exceeds a predetermined standard, image capture may be restricted. Furthermore, the level of privacy may be determined based on image analysis and / or equipment usage, and a physical shutter may be activated accordingly.

[0153] In other embodiments, the information processing device may be configured to be attachable to mirrors, furniture, home appliances, gimbals, drones, etc. In other words, the information processing device may be retrofitted to existing objects. A mirror or half-mirror may be placed on the surface of the information processing device. The information processing device may include an imaging device and a communication device.

[0154] For example, if an information processing device is attached to a gimbal, the gimbal may include a holding portion for holding other terminals, a mounting portion for being attached to, placed on, or held by another object, a connecting portion for connecting the holding portion and the mounting portion and configured to adjust the position and / or orientation of the holding portion relative to the mounting portion, an imaging portion for imaging the area around the holding portion or the mounting portion, and a control portion for controlling the operation of the connecting portion.

[0155] When the information processing device is mounted on a gimbal or drone, it may have a function to automatically track a subject. Specifically, the control unit may include a detection unit that detects the movement of a subject in an image captured by the imaging unit, and a tracking control unit that controls the operation of the connecting unit so that the orientation of a specific part of the holding unit tracks the subject. The control unit may also further include a recognition unit that recognizes the subject to be detected by the detection unit from among a plurality of objects included in the image.

[0156] This disclosure is not limited to the examples given above, but is intended to include all modifications within the meaning and scope of the claims, as indicated by the claims. For example, to the extent that it is not technically contradictory, matters described for a particular embodiment may also apply to other embodiments. Furthermore, embodiments with such modifications or improvements may fall within the technical scope of the present invention. [Explanation of symbols]

[0157] 1, 2, 3, 4, 5, 6 Skin diagnostic device 10 Main body 11 Upper cabinet 12 Lower cabinet 13 Mirror part 20, 120 Information Processing Devices 20A, 120A Image Acquisition Unit 20B, 120B Skin judgment section 30 Imaging device 35 Detection device 40 Display device 120C motion estimation section IMG image OBJ Subject

Claims

1. A first image acquisition unit that acquires a first image including the subject's skin, A skin determination unit that determines whether the skin is bare skin or the degree to which makeup has been applied to the skin based on the first image, Equipped with, Information processing device.

2. The system further includes a motion estimation unit that estimates the motion of the subject based on the output of a detection device that detects the motion of the subject, The skin determination unit is, Based on the estimation result of the motion estimation unit, it is determined whether or not the cosmetic has been applied to the skin of the subject, or whether or not the cosmetic has been removed from the skin of the subject. Based on the result of the said determination, it is determined whether the skin is bare or the degree to which makeup has been applied to the skin. The information processing apparatus according to claim 1.

3. If the motion estimation unit determines that a cosmetic has been applied to the subject's skin, the skin determination unit determines that the skin is not bare skin. The information processing apparatus according to claim 2.

4. If the motion estimation unit determines that the cosmetic has been removed from the subject's skin, the skin determination unit determines that the skin is bare skin. The information processing apparatus according to claim 2.

5. A first information acquisition unit acquires first information which is an association between the estimation result of the motion estimation unit and first time information which is the time or period during which the first image was captured. A second information acquisition unit acquires second information, which is an image of the subject including the skin, a second image taken by a second imaging device different from the first imaging device that took the first image, and second time information which is the time or period during which the second image was taken. It also has, The information processing apparatus according to claim 2.

6. The second imaging device differs from the first imaging device in that at least one of the resolution, field of view, imaging interval, and frame rate. The information processing apparatus according to claim 5.

7. A second image acquisition unit acquires a second image, which is an image including the skin of the subject, and is captured by a second imaging device different from the first imaging device that captured the first image. A first information generation unit associates the estimation result of the motion estimation unit with first time information, which is the time or period during which the first image was captured. A second information generation unit associates the second image with second time information, which is the time the second image was captured. It also has, The information processing apparatus according to claim 2.

8. The skin determination unit determines whether the skin is bare or the degree to which makeup has been applied to the skin, based on the color information at a predetermined position on the skin of the subject in the first image. The information processing apparatus according to claim 1.

9. The skin determination unit determines, based on the distribution of color information, whether the skin is bare or the degree to which makeup has been applied to the skin. The information processing apparatus according to claim 8.

10. The skin determination unit determines a threshold based on the distribution of color information, and determines whether the skin is bare skin or the degree to which makeup has been applied to the skin based on the threshold. The information processing apparatus according to claim 9.

11. The skin determination unit determines whether the skin is bare or the degree to which makeup has been applied to the skin, based on the shape of the skin of the subject in the first image at a predetermined position. The information processing apparatus according to claim 1.

12. The skin determination unit has an estimation model that takes the first image as input and determines whether the skin is bare skin or the degree to which makeup has been applied to the skin. The aforementioned estimation model learns from images that have been determined to be bare skin, The information processing apparatus according to claim 1.

13. The system further includes a motion estimation unit that estimates the motion of the subject based on the output of a detection device that detects the motion of the subject, The skin determination unit is, Based on the estimation result of the motion estimation unit and the first image, it is determined whether or not the cosmetic has been applied to the skin of the subject, or whether or not the cosmetic has been removed from the skin of the subject. Based on the result of the said determination, it is determined whether the skin is bare or the degree to which makeup has been applied to the skin. The information processing apparatus according to claim 8.

14. The first image further comprises a condition determination unit that determines whether or not it meets predetermined conditions or the degree to which it meets them. The information processing apparatus according to claim 1.

15. The condition determination unit determines whether the brightness or clarity in the first image matches a predetermined condition, or the degree to which it matches. The information processing apparatus according to claim 14.

16. The condition determination unit determines whether the orientation of the skin or the direction of the line of sight of the eyes in the first image matches a predetermined condition, or the degree to which it matches. The information processing apparatus according to claim 14.

17. The condition determination unit determines that the skin in the first image is dry, and that the predetermined conditions are met. The information processing apparatus according to claim 14.

18. The first image acquisition unit acquires the first image when the condition determination unit determines that the predetermined conditions are met. The information processing apparatus according to claim 14.

19. The skin determination unit determines, based on the first image, whether the skin is bare skin or the degree to which makeup has been applied to the skin, when the condition determination unit determines that the first image meets predetermined conditions. The information processing apparatus according to claim 14.

20. An estimation unit for estimating the state of the subject, An output unit that outputs information regarding the state of the subject estimated by the estimation unit, It also has, The information processing apparatus according to claim 1.

21. A first imaging unit that captures the first image, An information processing device according to any one of claims 1 to 20, Equipped with, Processing system.

22. A program that causes a computer provided in the information processing device to execute the processing in each part of the information processing device described in any one of claims 1 to 20.

Images