Image processing system, image processing method, and image processing program

Abstract

This system provides an image processing system that allows users to check the appearance of their own face while evaluating changes in facial shape. [Solution] The device includes a mirror device equipped with a display unit on the mirror surface and an imaging unit for imaging the user's face, and an image analysis device for analyzing and evaluating image data of the user's face, and includes a reference point setting unit for setting the midpoint of a line segment located at the bridge of the user's nose and connecting the user's eyes, a first reference point located on the skin directly above the zygomatic arch of the face, and a second reference point located on the skin directly above the mandibular angle, based on a frontal image of the user's face, an angle calculation unit for calculating a first angle formed by the line segment, the midpoint and the line segment connecting the line segment and the first reference point, and a second angle formed by the line segment, the midpoint and the second reference point, and an angle evaluation unit for evaluating changes in the proportions of the user's face from the time-series changes in the first and second angles each time the face is imaged.

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an image processing method, and an image processing program. [Background Art]

[0002] Conventionally, systems that use image processing technology to encourage beauty care for users are known. For example, Patent Document 1 discloses a system that uses images captured before and after a facial beauty treatment such as facial massage to obtain a strain distribution of facial skin, and quantitatively analyzes the effect of the beauty treatment using this strain distribution as an evaluation index. However, with conventional systems, while evaluating the shape change of the face, the user could not look at their own face and check its condition. [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2017-9598 [Patent Document 2] Japanese Patent No. 7442171 [Patent Document 3] Japanese Patent No. 7442172 [Patent Document 4] Japanese Patent No. 7442173 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In order to improve conventional systems, an image processing system has been proposed that allows a user to check the condition of their own face while evaluating changes in facial shape (Patent Documents 2, 3, 4).

[0005] Accordingly, an object of the present invention is to provide an image processing system that allows a user to check the condition of their own face while identifying and evaluating a site where chronological comparison of changes in the shape of the user's face becomes clearer. [Means for solving the problem]

[0006] To solve the above problems, the image processing system according to the present invention comprises a mirror device that serves as a mirror surface that reflects the user's face, a display unit provided on the mirror surface, and an imaging unit positioned on the mirror surface to image the user's face, and an image analysis device that analyzes and evaluates the image data of the user's face captured by the imaging unit, the image analysis device uses the frontal image of the user's face captured by the imaging unit to determine the midpoint (QM) of the line segment (L0) located at the base of the user's nose and connecting both of the user's eyes, a first reference point (Q1) located on the skin directly above the zygomatic arch of the user's face, and the skin directly above the mandibular angle of the user's face The system is characterized by comprising: a reference point setting unit that extracts a second reference point (Q2) that exists in the system and sets the first reference point (Q1) and the second reference point (Q2); an angle calculation unit that calculates a first angle (θ1) formed by a line segment (L0), a midpoint (QM), and a line segment connecting the first reference point (Q1), and a second angle (θ2) formed by a line segment (L0), a midpoint (QM), and a line segment connecting the second reference point (Q2); and an angle evaluation unit that evaluates changes in the proportions of the user's face from the time-series changes in the first angle (θ1) and the second angle (θ2) each time the imaging unit captures the user's face.

[0007] Furthermore, in the image processing system, the reference point setting unit may extract a third reference point (Q3) located on the eyebrow of the user's face and set the third reference point (Q3), the angle calculation unit may calculate a third angle (θ3) formed by a line segment (L0) and a line segment connecting the midpoint (QM) and the third reference point (Q3), and the angle evaluation unit may evaluate the change in the proportions of the user's face from the time-series changes of the first angle (θ1), second angle (θ2), and third angle (θ3) each time the imaging unit captures an image of the user's face.

[0008] Furthermore, in the image processing system, the first angle, second angle, and third angle may be set on both the left and right sides, with respect to the midline of the face.

[0009] Furthermore, in an image processing system, the display unit may display the analysis results from an image analysis device.

[0010] Furthermore, in the image processing system, the display unit may display the outline of previously captured image data of the user's face so that the user, who is positioned in front of the imaging unit, can align their face when the imaging unit captures the user's face.

[0011] Furthermore, in the image processing system, the display unit may be designed to allow adjustment between the area for displaying data and the area that acts as a mirror.

[0012] Furthermore, in the image processing system, the angle evaluation unit may analyze and evaluate image data of the user's face.

[0013] Furthermore, in an image processing system, the image analysis device may include a storage unit that stores the evaluation results performed by the angle evaluation unit, and a future prediction unit that uses the user's face shape evaluation stored in the storage unit to predict the future shape of the user's face.

[0014] Furthermore, in the image processing system, the future prediction unit may perform future predictions of the user's face by using machine learning to evaluate the shape of the user's face stored in the memory unit. [Effects of the Invention]

[0015] The present invention provides an image processing system comprising a mirror device that serves as a mirror surface reflecting the user's face, a display unit provided on the mirror surface, and an imaging unit positioned on the mirror surface to image the user's face, and an image analysis device that analyzes and evaluates the image data of the user's face captured by the imaging unit, wherein the image analysis device extracts from the frontal image of the user's face captured by the imaging unit the midpoint (QM) of the line segment (L0) located at the base of the user's nose and connecting the user's eyes, a first reference point (Q1) located on the skin directly above the zygomatic arch of the user's face, and a second reference point (Q2) located on the skin directly above the mandibular angle of the user's face, and the first reference point (Q1) and the second reference point (Q 2) The system includes a reference point setting unit for setting a reference point, an angle calculation unit for calculating a first angle (θ1) formed by a line segment (L0) and a line segment connecting the midpoint (QM) and the first reference point (Q1), and a second angle (θ2) formed by a line segment (L0) and a line segment connecting the midpoint (QM) and the second reference point (Q2), and an angle evaluation unit for evaluating changes in the proportions of the user's face from the time-series changes in the first angle (θ1) and the second angle (θ2) each time the imaging unit captures the user's face. As a result, the system allows the user to check the appearance of their own face while identifying and evaluating areas where changes in the shape of the user's face over time are clearer.

[0016] In addition, it becomes possible to evaluate changes in facial proportions based on the positional relationship of the user's bones, muscles, and fat, making it easy to judge the quality of the changes before and after, and the effectiveness of the treatment.

[0017] Of course, the same effects can be obtained with image processing methods and image processing programs as with image processing systems. [Brief explanation of the drawing]

[0018] [Figure 1] This is a schematic diagram showing an example configuration of an image processing system according to one embodiment of the present invention. [Figure 2] This diagram shows the mirror device shown in Figure 1 in use by a user. [Figure 3] Figure 1 is a block diagram showing an example configuration of an image processing system. [Figure 4] It is a diagram explaining the overall processing flow of the image processing system. [Figure 5] It is a diagram explaining evaluation items that can be evaluated by the skin condition evaluation unit. [Figure 6] It is a block diagram showing a configuration example of the skin condition evaluation unit. [Figure 7] It is a diagram showing an example of an evaluation table serving as an evaluation criterion when detecting spots by the skin condition evaluation unit. [Figure 8] (a) is a diagram showing an example of a processing result in the image processing system, and (b) is a partially enlarged view of FIG. 8(a). [Figure 9] It is a block diagram showing a configuration example of the mobile terminal shown in FIG. 1. [Figure 10] It is a diagram showing a processing flow in the skin condition evaluation unit. [Figure 11] It is a diagram showing another example of a processing result in the image processing system. [Figure 12] It is a diagram showing an example of an evaluation table serving as an evaluation criterion when detecting pores by the skin condition evaluation unit. [Figure 13] It is a diagram showing an example of an evaluation table serving as an evaluation criterion when detecting dark circles, redness, and spots by the skin condition evaluation unit. [Figure 14] It is a diagram showing an example of an evaluation table serving as an evaluation criterion when detecting texture, fine lines, pores, and wrinkles by the skin condition evaluation unit. [Figure 15] It is a block diagram showing a first configuration example of the face shape evaluation unit shown in FIG. 1. [Figure 16] It is a diagram showing an example of each vertex recognized by the vertex recognition unit shown in FIG. 13, which is (a) a front view and (b) a side view of imaged data. [Figure 17] It is a diagram showing a first processing flow in the face shape evaluation unit. [Figure 18] It is a schematic diagram illustrating a process in which the vertex recognition unit recognizes a vertex on the cheek. [Figure 19] It is a diagram showing an example of display content by the display processing unit. [Figure 20]This figure shows another example of the content displayed by the display processing unit. [Figure 21] This block diagram shows a second example of the facial shape evaluation unit shown in Figure 1. [Figure 22] This is a schematic diagram showing reference points and line segments on the face. [Figure 23] This diagram shows the second processing flow in the facial shape evaluation unit. [Figure 24] This figure shows yet another example of the content displayed by the display processing unit. [Figure 25] This is a schematic diagram illustrating the processing of machine learning. [Figure 26] (a) External view of the beauty device used in the massage step, (b) Cross-sectional view. [Figure 27] This figure shows each step (a) to (d) of the first step in the first example of the massage method. [Figure 28] This figure shows each step (a) to (d) of the second step in the first example of the massage method. [Figure 29] This figure shows each step (a) to (c) of the third step in the first example of the massage method. [Figure 30] This figure shows each step (a) to (d) of the fourth step in the first example of the massage method. [Figure 31] This figure shows each step (a) to (d) of the fifth step in the first example of the massage method. [Figure 32] This figure shows each step (a) to (c) of the sixth step in the first example of the massage method. [Figure 33] This figure shows each step (a) to (d) of the first step in the second example of the massage method. [Figure 34] This figure shows each step (a) to (d) of the second step in the second example of the massage method. [Figure 35] This figure shows each step (a) to (d) of the third step in the second example of the massage method. [Figure 36] This figure shows each step (a) to (b) of the first step in the third example of the massage method. [Figure 37] This figure shows each step (a) to (b) of the second step in the third example of the massage method. [Figure 38] This figure shows each step (a) to (b) of the third step in the third example of the massage method. [Figure 39] This figure shows each step (a) to (b) of the fourth step in the third example of the massage method. [Figure 40] This figure shows each step (a) to (b) of the first step in the fourth example of the massage method. [Modes for carrying out the invention]

[0019] (Overall structure) The image processing system 100 according to this embodiment will be described with reference to the drawings. Figure 1 is a schematic diagram showing an example of the configuration of the image processing system 100 according to this embodiment. Figure 2 is a diagram showing the state in which user 5 is using the mirror device 2 shown in Figure 1.

[0020] As shown in Figure 1, the image processing system 100 is a system that provides advice on promoting beauty by performing image processing on image data of user 5's face and displaying the results to user 5, in order to help promote the beauty of user 5. When user 5 actually uses the system, as shown in Figure 2, user 5 sits in front of the mirror device 2, and various analyses described later are performed on user 5's face using image data captured by the imaging unit 21.

[0021] As shown in Figure 1, the image processing system 100 includes mirror devices 2 and an image analysis device 1 that are connected to each other via a network 3. In the illustrated example, multiple mirror devices 2 are provided. The mirror devices 2 include a store terminal 2A and a personal terminal 2B.

[0022] The store terminal 2A is a terminal used in a store that provides beauty promotion guidance to user 5, and can be used by user 5 when they visit the store. The personal terminal 2B is a terminal mainly intended for use by user 5 at home, and can be used by the user in their daily life, for example, when grooming themselves or before going to bed. In the illustrated example, user 5's mobile terminal 4 is connected to a network 3 such as an internet connection. The mobile terminal 4 is connected to the network 3 via wireless communication.

[0023] For example, the image processing system 100 of this embodiment uses a store terminal 2A installed in a store providing beauty-related services such as an aesthetic salon, cosmetics store, beauty equipment store, or sports club to capture an image of the user's face 5 and display the evaluation results. The image processing system 100 may also be used, for example, to suggest measures that the user 5 should take in the future to improve the condition of their facial skin.

[0024] Furthermore, the image processing system 100 can also display the results of routinely capturing and evaluating the user 5's face using a personal terminal 2B installed at the user 5's home. In other words, the use of the image processing system 100 may be performed under the operation of the store operator 6, or it may be performed by the user 5 themselves.

[0025] Figure 3 is a block diagram showing an example configuration of the image processing system 100. As shown in Figure 3, the mirror device 2 includes a display unit 20, an imaging unit 21, and a communication unit 22. The display unit 20 is provided on the surface of the mirror device 2 and is a display that serves as both a mirror surface and a display capable of displaying data. In other words, the display unit 20 is a display device such as a liquid crystal television and displays various analysis results from the image analysis device 1, which will be described later.

[0026] The display unit 20 allows adjustment between the area for displaying data and the area that acts as a mirror reflecting the user's face. That is, for example, the entire surface of the display unit 20 may be a mirror, or the entire surface of the display unit 20 may display data. Alternatively, data may be displayed on the mirror surface of the display unit 20, or half of the display unit 20 may be a mirror and the other half may display data. In the display unit 20, the division (ratio of area between the data display area and the mirror surface) is changed as appropriate depending on the content to be displayed.

[0027] The imaging unit 21 captures an image of the area in front of the display unit 20. The imaging unit 21 is not particularly limited as long as it is a device that can capture an image of the user 5's face and acquire image data when the user 5 is positioned in front of it. The imaging unit 21 may have an image sensor such as a CMOS image sensor or a CCD camera. The communication unit 22 transmits the image data captured by the imaging unit 21 to the communication unit 23 of the image analysis device 1.

[0028] The display unit 20 displays the outline of previously captured image data of the user 5's face when the imaging unit 21 captures the user 5's face (see outline 24 in Figure 23). In other words, when imaging is performed habitually, it is preferable that the position of the user 5's face relative to the imaging unit 21 does not change significantly. For this reason, the display unit 20 displays the outline of previously captured image data of the user 5's face so that the user 5, who is positioned in front of the imaging unit 21, can align their face with the imaging unit 21. At this time, the image of the user 5's face captured by the imaging unit 21 is displayed on the display surface, and the imaging unit 21 acquires the image data after the user 5 has aligned their face.

[0029] The image data acquired by the mirror device 2 may be 2D data or 3D data. In this embodiment, a configuration in which the imaging unit 21 acquires image data as 2D data will be described. The mirror device 2 may be, for example, a 3D camera in which multiple imaging units 21 are arranged at intervals, or it may be a configuration comprising one imaging unit 21 and a distance sensor.

[0030] The communication unit 22 of the mirror device 2 communicates with the communication unit 23 of the image analysis device 1 via the network 3 shown in Figure 1. Network 3 is a network for interconnecting the mirror device 2, the image analysis device 1, and the mobile terminal 4, and can be, for example, a wireless network or a wired network.

[0031] Specifically, Network 3 includes wireless LAN (WLAN), wide area network (WAN), ISDNs (integrated service digital networks), wireless LANs, LTE (long term evolution), LTE-Advanced, 4G, 5G, CDMA (code division multiple access), WCDMA (registered trademark), Ethernet (registered trademark), etc.

[0032] Furthermore, Network 3 is not limited to these examples, and may include, for example, a Public Switched Telephone Network (PSTN), Bluetooth (registered trademark), Bluetooth Low Energy, fiber optic lines, ADSL (Asymmetric Digital Subscriber Line) lines, satellite communication networks, or any other network.

[0033] Network 3 may also be, for example, NB-IoT (Narrow Band IoT) or eMTC (enhanced Machine Type Communication). Note that NB-IoT and eMTC are wireless communication methods for IoT, and are low-cost, low-power networks that enable long-distance communication.

[0034] Network 3 may also be a combination of these. Network 3 may also include multiple different networks that combine these examples. For example, Network 3 may include a wireless network using LTE and a wired network such as an intranet, which is a closed network.

[0035] (Image analysis device) As shown in Figure 3, the image analysis device 1 includes a skin condition evaluation unit 30, a face shape evaluation unit 50, a communication unit 23, a future prediction unit 60, an evaluation result provision unit 61, a confirmation content reporting unit 62, a storage unit 63, and a user identification unit 64. The image analysis device 1 analyzes the image data of the user 5's face captured by the imaging unit 21.

[0036] The skin condition evaluation unit 30 is a functional unit that evaluates the skin condition of user 5. Details of the configuration of the skin condition evaluation unit 30 will be described later with reference to Figure 6. The face shape evaluation unit 50 is a functional unit that evaluates changes in the proportions of user 5's face. Details of the configuration of the face shape evaluation unit 50 will be described later with reference to Figures 15 and 21.

[0037] The future prediction unit 60 uses either the skin health status of user 5 stored in the skin condition evaluation unit 30, or the facial shape evaluation of user 5 stored in the facial shape evaluation unit 50, or both, to make a future prediction for user 5's face. The future prediction unit 60 generates composite data of user 5's face, showing what effects can be expected if the proposed measures are continued in the future, while referring to past history, and displays it on the display unit 20 of the mirror device 2. As described later, the accuracy of the prediction may be improved by machine learning on either the skin health status of user 5 stored in the skin condition evaluation unit 30, or the facial shape evaluation of user 5 stored in the facial shape evaluation unit 50, or both.

[0038] The evaluation result provision unit 61 analyzes the degree of beauty enhancement of user 5 from the image data and provides the analysis results to the personal terminal 2B. The analysis results referred to here refer to the analysis results performed by the skin condition evaluation unit 30 and the face shape evaluation unit 50, which will be described later. Here, the degree of beauty enhancement refers to the progress made in the process of aiming to improve beauty items such as skin condition and facial shape from the current state towards the future.

[0039] The confirmation report unit 62 reports confirmation history information to the store terminal 2A, based on the analysis results provided to user 5, specifically the confirmation history that user 5 confirmed using their personal terminal 2B. The confirmation history information can be confirmed by obtaining the data log, which is a history of the usage status of the personal terminal 2A.

[0040] The confirmation content reporting unit 62 aggregates the number of times the user 5 has checked each analysis result performed by the skin condition evaluation unit 30 and the face shape evaluation unit 50 within a predetermined period, and reports this to the store terminal 2A. The same information may also be reported to the personal terminal 2B. In addition, the confirmation content reporting unit 62 reports at least one of the maintenance method and maintenance product displayed by the display unit 20 to the store terminal 2A. Details of the maintenance method and maintenance product will be described later.

[0041] The memory unit 63 stores the information that user 5 confirmed using the store terminal 2A when visiting the store, along with the confirmation history information. In other words, the information confirmed by user 5 is aggregated from the information confirmed at home using the personal terminal 2B and the information confirmed at the store using the store terminal 2A, and stored in the memory unit 63. The memory unit 63 also stores the facial data of each user 5 and user 5's ID.

[0042] The user identification unit 64 identifies the user 5 to be used. The user identification unit 64 may, for example, accept input of the user 5's ID from the touch panel keyboard displayed on the display unit 20, or it may identify the user 5 by, for example, referring to the captured image data of the user 5's face in the storage unit 63.

[0043] Here, the overall processing flow of the image processing system 100 will be explained using Figure 4. Figure 4 is a diagram showing the overall processing flow (image processing method) of the image processing system 100. As shown in Figure 4, the user first performs user authentication (S10: user authentication step). User authentication may be performed by the user 5 entering a user ID using the touch panel keyboard displayed on the display unit 20, or by the user identification unit 64 referring to the user 5 information stored in the storage unit 63 from the facial image data captured by the imaging unit 21.

[0044] Next, the skin condition evaluation unit 30 evaluates the skin condition of user 5 (S11: skin condition evaluation step). This will be explained later. Next, the face shape evaluation unit 50 evaluates the changes in the facial proportions of user 5 (S12: face shape evaluation step). This will be explained later.

[0045] Next, the evaluation result provision unit 61 provides the evaluation results to the user 5 by transmitting each analysis result to the personal terminal 2B (S13: evaluation result provision step). The contents of each analysis result will be described later. Next, the confirmation content reporting unit 62 reports the contents confirmed by the user 5 to the store terminal 2A (S14: confirmation content reporting step). This allows store staff to confirm what aspects of beauty the user 5 is interested in.

[0046] (Skin condition evaluation department) Next, the configuration of the skin condition evaluation unit 30 will be described in detail. The skin condition evaluation unit 30 evaluates the health status of user 5's skin based on the user 5's skin color from the image data. Based on the user 5's skin color, the skin condition evaluation unit 30 detects areas where abnormalities have occurred on user 5's skin as abnormal areas and displays the number of abnormal areas along with past history. The number of abnormal areas can be determined, for example, by counting the amount of areas that exceed a preset threshold for the hue of the skin color.

[0047] Figure 5 illustrates the evaluation items that can be evaluated by the skin condition evaluation unit 30. In Figure 5, the skin condition evaluation unit 30 shows the facial regions in the image captured by the imaging unit 21 that are used for detection of each abnormality item. As shown in Figure 5, the skin condition evaluation unit 30 has the function of detecting abnormalities in the skin condition. Skin abnormalities that the skin condition evaluation unit 30 can detect include fine lines, wrinkles, age spots, enlarged pores, rough skin (texture), redness, and dark circles. The skin condition evaluation unit 30 sets each region shown in Figure 5 and performs detection processing on each of these regions as a detection region corresponding to each abnormality item to be detected.

[0048] Next, the configuration of the skin condition evaluation unit 30 will be described in detail using Figure 6. Figure 6 is a block diagram of the skin condition evaluation unit 30. In this explanation, the evaluation of blemishes will be used as an example among the multiple evaluation functions of the skin condition evaluation unit 30. As shown in Figure 6, the skin condition evaluation unit 30 comprises a device-side communication unit 31, a data storage unit 32, a data processing unit 33, and a device-side display unit 34. The skin condition evaluation unit 30 is an information processing device that analyzes the skin condition of the user's face from image data captured from the user's face.

[0049] The device-side communication unit 31 is a communication interface that sends and receives various types of data via the network 3. These types of data include image data, processing data, and improvement data. In other words, the device-side communication unit 31 functions as a receiver that receives image data transmitted from the communication unit 22 of the mirror device 2.

[0050] Here, image data refers to data captured by the mirror device 2 of the user 5's face. Processing data refers to data obtained by the evaluation result display unit 33C (described later) that identifies and marks the location of blemishes on the image data. Improvement data refers to data obtained by the improvement data generation unit 33F (described later) that assumes an improved state of pigmentation in the pigment abnormality area and changes the hue of the pigment abnormality area to display to the user 5.

[0051] The data storage unit 32 has the function of storing various control programs necessary for the operation of the data processing unit 33, and various data received from the outside by the device-side communication unit 31. The data storage unit 32 also has an evaluation table that serves as a reference when the skin color evaluation unit 33A, which will be described later, evaluates the skin color of the user 5. The data storage unit 32 can be implemented using various storage media such as an HDD, SSD, or flash memory.

[0052] The data processing unit 33 executes the control program stored in the data storage unit 32 to realize each function that the image processing system 100 should achieve. These functions include skin color evaluation, pigmentation identification, evaluation result display, depth estimation, policy proposal, and improvement data generation functions. The device-side display unit 34 is a monitor device that displays the operation details and processing results of the image analysis device 1.

[0053] The data processing unit 33 is a computer that controls each part of the image analysis device 1, and may be, for example, a central processing unit (CPU, GPU), a microprocessor, an ASIC, or an FPGA. However, the data processing unit 33 is not limited to these examples and may be any computer that controls each part of the image analysis device 1.

[0054] The data processing unit 33 includes a skin color evaluation unit 33A, a skin abnormality identification unit 33B, an evaluation result display unit 33C, a depth estimation unit 33D, a measure proposal unit 33E, and an improvement data generation unit 33F. When detecting blemishes, the skin color evaluation unit 33A uses image data of the user 5's skin to divide any part of the user 5's skin into multiple stages.

[0055] Multiple stages refer to pre-defined categories used to classify the skin condition of user 5. For example, they might be represented as stages 1 through 4. A higher stage number indicates that the pigment abnormality is located deeper in the skin and the symptoms of the pigment abnormality are more severe. Separate stage groups may be prepared for different races with varying skin tones.

[0056] The skin tone evaluation unit 33A categorizes the user 5's skin tone into several stages based on the hue value of the skin (for example, RGB value). This is explained in detail using Figure 7. Figure 7 is an example of an evaluation table that serves as the evaluation criteria for the skin tone evaluation unit 33A when detecting blemishes. Note that the hue value is not limited to RGB values, but may also be CMYK values ​​or other index values.

[0057] In the example evaluation table shown in Figure 7, the classification of hue and stage is described for each type of pigment spot within the pigment abnormality area. Hue can be represented, for example, by RGB values. That is, although hue is represented by color in this figure, data for the corresponding RGB values ​​may also be provided.

[0058] For example, as shown in Figure 7, in the case of senile lentigines, the pigmented area tends to occur on the cheekbone, and if it is brownish-red to brown, it is judged as Stage 1. If the pigmented area is light brownish-red to light brown, it is judged as Stage 2. If the pigmented area is pale brownish-red to pale brown, it is judged as Stage 3.

[0059] Next, in the case of post-inflammatory hyperpigmentation, it tends to occur all over the face, and if the pigmented area is brownish-red to brown, it is judged as Stage 1. If the pigmented area is light brownish-red to light brown, it is judged as Stage 2. If it is light brownish-red to light brown, it is judged as Stage 3. If it is blue to gray, it is judged as Stage 4. Here, the type of pigmented spot is determined by Operator 6 based on the location and appearance of the pigmented area.

[0060] Note that this evaluation table is merely an example, and a different evaluation table can be used to evaluate blemishes. Furthermore, the skin tone evaluation unit 33A changes the evaluation table it references depending on the type of skin abnormality (fine lines, wrinkles, blemishes, enlarged pores, rough skin, redness, or dark circles) (see Figures 13 and 14). This point will be explained later.

[0061] The skin abnormality identification unit 33B identifies areas on user 5's skin where pigment abnormalities, including spots caused by hyperpigmentation, have occurred, based on the stages classified by the skin color evaluation unit 33A. Here, hyperpigmentation refers to skin pigment abnormalities caused by the accumulation of black melanin in the epidermis.

[0062] To explain the mechanism by which pigment abnormalities such as age spots (pigmented spots) occur on the skin, when the skin is exposed to stimuli such as ultraviolet rays, melanocytes (melanin-forming cells), which are part of the skin's internal tissue, produce black melanin. This black melanin plays a role in protecting the skin, but in healthy skin, it is expelled from the body over time.

[0063] On the other hand, if the skin's metabolic cycle is disrupted or if there is an excessive amount of black melanin production, some of the black melanin is not excreted from the body and remains and accumulates in the epidermis. Among these pigment abnormalities, those caused by inflammation or ultraviolet radiation are called hyperpigmentation or pigment spots (age spots).

[0064] The evaluation result display unit 33C displays the location of each area in the image data according to the stages categorized by the skin color evaluation unit 33A. The evaluation result display unit 33C also marks the pigment abnormalities identified by the skin abnormality identification unit 33B. This generates processed data in which the location of the blemishes is displayed and marked in the image data. The marking is done by placing a mark of a color set for each stage on the part of the image data whose hue corresponds to one of stages 1 to 4. Furthermore, the evaluation result display unit 33C can also display the location and marking for each stage for multiple image data captured by the same user 5 at different times and display them side by side.

[0065] Here, an example of processed data will be explained using Figure 8. Figure 8(a) is a diagram showing an example of the processing result in the image processing system 100, and (b) is a magnified view of a part of Figure 8(a). As shown in Figures 8(a) and 8(b), the locations of multiple blemishes are identified at arbitrary locations on the skin of user 5's face. At each location, the pigment abnormality is marked and represented. Some of these markings are not visible to the naked eye. In Figures 8(a) and 8(b), the lightest mark M1 indicates stage 1, and the darkest mark M3 indicates stage 3. The mark M2, which is of intermediate intensity, indicates stage 2. In this diagram, the mark for stage 4 is not visible.

[0066] The depth estimation unit 33D estimates the depth from the skin surface of the pigment abnormality area on user 5's skin based on the hue value of the pigment abnormality area identified by the skin abnormality identification unit 33B. It is generally known that the depth from the skin surface of the location where the pigment abnormality area occurs differs depending on the hue of the blemish or pigment spot.

[0067] For example, in the pigmentation evaluation table for blemishes (pigmented spots) shown in Figure 7, pigmentation abnormalities corresponding to Stage 1 are determined to occur in the upper layer of the epidermis, and pigmentation abnormalities corresponding to Stage 2 are determined to occur in the middle layer of the epidermis. Furthermore, pigmentation abnormalities corresponding to Stage 3 are determined to occur in the lower layer of the epidermis, and pigmentation abnormalities corresponding to Stage 4 are determined to occur from the lower layer of the epidermis to the dermis. Here, "upper side" refers to the side of the skin that faces the surface, and "lower side" refers to the side of the skin that faces into the body. Such criteria for judging color and depth can be set arbitrarily.

[0068] The policy proposal unit 33E proposes measures to promote improvement of pigmentation based on the depth of the pigment abnormality estimated by the depth estimation unit 33D. These measures may include the use of a beauty device 70, the use of a serum, the use of a carbonated face mask, and UV care. The appropriateness of these measures will be determined based on the depth of the pigment abnormality. For pigment abnormalities formed in deeper layers, the unit may also suggest seeking medical attention.

[0069] The improvement data generation unit 33F, based on the image data, assumes that the pigmentation in the pigment abnormality area has been improved, and changes the hue of the pigment abnormality area for display to the user 5. In other words, it has a function to visually represent what kind of effect can be obtained when the proposed measures are implemented for a certain period of time.

[0070] The improvement data generation unit 33F uses, for example, historical data to estimate how the pigment abnormality would change if the measure were implemented for a certain period of time from a similar state. Based on this estimation, the improvement data generation unit 33F generates improvement data from the image data.

[0071] Next, the configuration of the mobile terminal 4 will be explained using Figure 9. Figure 9 is a block diagram showing an example of the configuration of the mobile terminal 4. The mobile terminal 4 includes a terminal-side communication unit 41, a terminal storage unit 42, a terminal processing unit 43, a camera 44, and a terminal-side display unit 45.

[0072] The terminal-side communication unit 41 is a communication interface that sends and receives various types of data via the network 3. These types of data include image data and data showing comparison results. In other words, the terminal-side communication unit 41 receives various types of information from the image analysis device 1.

[0073] The terminal storage unit 42 has the function of storing various control programs and data necessary for the operation of the terminal processing unit 43. The terminal storage unit 42 can be implemented using various storage media such as an HDD, SSD, or flash memory. By executing the control programs stored in the terminal storage unit 42, the terminal processing unit 43 may implement at least some of the functions that the image processing system 100 should implement.

[0074] The terminal processing unit 43 is a computer that controls various parts of the mobile terminal 4, and may be, for example, a central processing unit (CPU, GPU), a microprocessor, an ASIC, or an FPGA. However, the terminal processing unit 43 is not limited to these examples and may be any computer that controls various parts of the mobile terminal 4.

[0075] The terminal processing unit 43 includes a reception unit 43A. The reception unit 43A receives image data and comparison results transmitted from the image analysis device 1 and displays them on the terminal-side display unit 45. The camera 44 can take images at the operation of the user 5. In this embodiment, instead of the mirror device 2, image data may be acquired by the camera 44 of the mobile terminal 4 and transmitted to the image analysis device 1. The terminal-side display unit 45 is a monitor device that displays information indicating the comparison results processed by the image analysis device 1. The terminal-side display unit 45 can display image data along with the comparison results.

[0076] Next, the processing performed by the skin condition evaluation unit 30 will be explained using Figure 10. Figure 10 is a diagram showing the processing flow (image processing method) in the image analysis device 1. As shown in Figure 10, first, image data of the user 5's face captured by the mirror device 2 is acquired (image acquisition step: S501). Next, the skin color evaluation unit 33A divides the user 5's skin into multiple stages while referring to the evaluation table (skin color evaluation step: S502).

[0077] Next, the skin abnormality identification unit 33B identifies the areas on user 5's skin where pigment abnormalities have occurred (pigmentation identification step: S503). Then, the evaluation result display unit 33C identifies blemishes and pigment spots by marking the pigment abnormalities (evaluation result display step: S504).

[0078] Next, the depth estimation unit 33D estimates the depth of the pigment abnormality from the skin surface (depth estimation step: S505). At this time, the depth estimation unit 33D refers to pre-stored data of the color of the pigment abnormality and the depth from the skin surface.

[0079] Next, the policy proposal unit 33E proposes measures to improve pigmentation to user 5 (policy proposal step: S506). Finally, the improvement data generation unit 33F generates and displays improvement data (improvement data generation step: S507). The improvement data generation unit 33F changes the hue of the pigmented area in the image data, assuming that the pigmentation in the pigmented area has been improved, and displays this to user 5. This allows user 5 to visually understand what effects can be obtained when the measures are taken, and to be motivated to continue taking the measures.

[0080] Next, another example of the processing results will be explained using Figure 11. Figure 11 shows another example of the processing results in the image processing system 100. As shown in Figure 11, even when there are more stains than in Figure 8 mentioned above, the pigment abnormalities are marked in a state where they are divided into multiple stages. Then, the number of stains can be evaluated for each corresponding stage. Furthermore, by totaling the number of stains and comparing them with the ideal value and the average value, the current status of user 5 can be objectively displayed.

[0081] Here, we will explain the method for detecting skin abnormalities other than blemishes using Figures 12 to 14. Figure 12 is a diagram showing an example of an evaluation table that serves as the evaluation criteria when the skin condition evaluation unit 30 evaluates pores. In this evaluation table, the occurrence mechanism, skin characteristics, skin type (quality of skin around the pore), palpation (feel when touched), common site (area where they often occur), cause, and countermeasures are classified according to the shape of the pore. When detecting pores, the skin abnormality identification unit 33B identifies the location of the pore from the skin color information detected by the skin color evaluation unit 33A.

[0082] In this process, information indicating the color range of the pores is pre-stored in the evaluation table, and pores are detected by referring to this value. When detecting pores, the skin abnormality identification unit 33B also evaluates the shape of the pores and classifies them into several types. For example, as shown in Figure 12, they can be classified into four types: dry pores, sagging pores, clogged pores, and shape-memory pores. Maintenance information on what treatments can be performed to make these classified pores less noticeable is described in the countermeasure column located at the far right of Figure 12. The countermeasure proposal unit 33E presents this countermeasure content along with the evaluation results, allowing the user 5 to use it for beauty purposes.

[0083] Figure 13 shows an example of an evaluation table that serves as the evaluation criteria for detecting dark circles, redness, and blemishes. When detecting dark circles, the skin abnormality identification unit 33B detects the location of the dark circles from the skin color information detected by the skin color evaluation unit 33A. At this time, information indicating the range of the dark circles' color is stored in the evaluation table in advance, and the dark circles are detected by referring to this value.

[0084] When detecting redness, the skin abnormality identification unit 33B detects areas where a red hue is prominently observed within the overall skin color detected by the skin color evaluation unit 33A as redness. The unit then identifies the area of ​​redness and determines whether it is due to telangiectasia if the redness is around the cheeks, dryness if the redness is all over the face, or acne if there is localized, severe redness. Measures to be taken for each type of redness are then prepared.

[0085] When detecting blemishes, as mentioned above, the skin tone evaluation unit 33A classifies the blemishes into stages based on their color. Since it is known that the depth from the skin surface to the area where a blemish is located varies depending on its color, the countermeasures to be taken will differ for each type of blemish. In the case of dark circles, redness, and blemishes, the countermeasure proposal unit 33E presents the user 5 with maintenance information and countermeasure information stored in the evaluation table, along with the evaluation results, as solutions.

[0086] Furthermore, as shown in Figures 13 and 14, the evaluation table displays maintenance information along with recommended maintenance products. This information is stored in the data storage unit 32. The policy proposal unit 33E may present maintenance products along with the maintenance information, or it may present only maintenance products. In other words, the display unit 20 of the mirror device 2 can display at least one of the maintenance method and maintenance products along with the evaluation results.

[0087] Figure 14 shows an example of an evaluation table that serves as the evaluation criteria for detecting skin texture, fine lines, pores, and wrinkles. When evaluating skin texture (roughness), the skin color evaluation unit 33A subdivides the skin of the cheek into minute area elements based on color differences, evaluates the density, and assigns a score. If the score is 50 points or higher, it is judged as keratin thickening; if it is between 30 and 49 points, it is judged as dry skin; and if it is 29 points or lower, it is judged as normal.

[0088] When evaluating fine wrinkles, the skin tone evaluation unit 33A detects lines formed around the eyes that have a different hue from the surrounding area. In other words, wrinkles formed around the eyes are referred to as "wrinkles" in this explanation. If one or more lines extend horizontally below the eye, these lines are judged to be sagging wrinkles. Also, if there are three or more lines on the outer left and right sides of the eye, in the area below the outer corner of the eye, these lines are judged to be fine wrinkles. Sagging wrinkles are mainly caused by muscle weakness, while fine wrinkles are mainly caused by thinning of the epidermis and dermis. The evaluation of pores is as described above, so the explanation is omitted.

[0089] When evaluating wrinkles, the skin tone evaluation unit 33A detects lines that form around the forehead and cheeks and have a different hue from the surrounding area. In this explanation, wrinkles that form around the forehead and cheeks are called fine wrinkles. If there is a line on the forehead, it is judged to be an expression wrinkle, and if there is a line on the cheek, it is judged to be a deep wrinkle. Expression wrinkles are mainly caused by daily facial expressions and weakening of the muscles of the scalp and around the eyes. Deep wrinkles are mainly caused by weakening of the muscles of the cheeks and around the mouth.

[0090] In this way, the skin condition evaluation unit 30A can detect the areas where skin abnormalities have occurred for each type of skin abnormality (fine lines, wrinkles, blemishes, enlarged pores, rough skin, redness, or dark circles).

[0091] (Facial shape evaluation section - area comparison and evaluation) Next, the configuration of the face shape evaluation unit 50 shown in Figure 3 will be explained using Figure 15. Note that the face shape evaluation unit 50 includes the form disclosed in Figure 15 and the embodiment disclosed in Figure 21, which will be shown later. Figure 15 is a block diagram showing the configuration of the face shape evaluation unit 50. The face shape evaluation unit 50 evaluates changes in the proportions of the user's face based on the position of the user's bones, muscles, and fat from the image data. The face shape evaluation unit 50 calculates the area of ​​a predetermined region defined on the user's face based on the position of the user's bones, muscles, and fat, and displays the area of ​​the predetermined region along with its past history.

[0092] The face shape evaluation unit 50 comprises a device-side communication unit 51, a data storage unit 52, a device processing unit 53, and a device-side display unit 54. The face shape evaluation unit 50 is an information processing device that analyzes the state of the user's face from imaging data obtained by capturing the user's face.

[0093] The device-side communication unit 51 is a communication interface that sends and receives various types of data via the network 3. These types of data include imaging data and data showing comparison results. In other words, the device-side communication unit 51 functions as a receiver for receiving imaging data.

[0094] The data storage unit 52 has the function of storing various control programs necessary for the operation of the device processing unit 53, as well as various data received from the outside by the device-side communication unit 51. In addition, the data storage unit 52 stores at least one reference area data. The data storage unit 52 can be implemented using various storage media such as HDD, SSD, or flash memory.

[0095] The device processing unit 53 executes the control program stored in the data storage unit 52 to realize each function that the image processing system 100 should achieve. These functions include vertex recognition, region delimitation, area calculation, area comparison, and result display functions. The device-side display unit 54 is a monitor device that displays the operation details and processing results of the face shape evaluation unit 50.

[0096] The device processing unit 53 is a computer that controls each part of the face shape evaluation unit 50, and may be, for example, a central processing unit (CPU, GPU), a microprocessor, an ASIC, an FPGA, etc. However, the device processing unit 53 is not limited to these examples and may be any computer that controls each part of the face shape evaluation unit 50.

[0097] The device processing unit 53 includes a vertex recognition unit 53A, a region delimiter 53B, an area calculation unit 53C, an area comparison unit 53D, and a display processing unit 53E. The vertex recognition unit 53A recognizes the positions of two fixed points Pf and one movable point Pm from the imaging data of the user's face 5.

[0098] Here, the fixed point Pf is a vertex determined by the facial skeletal structure. Because the fixed point Pf is determined by the facial skeletal structure, its position changes only slightly over time. Note that "fixed" here does not mean that its position does not change at all, but rather that the amount of change is extremely small compared to the movable point Pm, which will be discussed later.

[0099] On the other hand, the movable point Pm is a vertex that is determined by the facial muscles and fat. For example, as facial muscles weaken with age or as fat accumulates on the face, the movable point Pm changes position downwards. Conversely, stimulating the facial muscles can strengthen them or reduce the amount of facial fat, causing the movable point Pm to change position upwards. These changes in the position of the movable point Pm alter the proportions of the face, which greatly influences the impression the face gives to others.

[0100] Here, the vertices recognized by the vertex recognition unit 53A in this embodiment will be described with reference to Figure 16. Figure 16 is a diagram showing the vertices recognized by the vertex recognition unit 53A, and is (a) a front view and (b) a side view of the image data. Note that this is merely an example, and the vertices recognized by the vertex recognition unit 53A can be changed as needed. That is, the vertices of the face that are easy to recognize, taking into account the skeletal structure and muscle arrangement of the user 5, can be used for evaluation.

[0101] As shown in Figure 16, the vertex recognition unit 53A recognizes two fixed points Pf and one movable point Pm for one defined region. The two fixed points Pf recognize the vertices identified by the deep nasal point P1 and the vertex P2 of the temple, and the movable point Pm recognizes the vertex P3 on the cheek. The deep nasal point P1 is shared by the left and right defined regions. The specific method for identifying each vertex will be described later. In this embodiment, the vertical positions of the deep nasal point P1 and the vertex P2 of the temple are equivalent to each other. The vertex P3 on the cheek is located below the deep nasal point P1 and the vertex P2 of the temple.

[0102] Furthermore, the vertex recognition unit 53A recognizes each vertex identified by the subnasal point P4 and the sub-ear point P5 as two fixed points Pf, and recognizes the vertex P6 below the cheek as one movable point Pm. Note that the subnasal point P4 is shared by a pair of left and right defined regions. The specific method for identifying each vertex will be described later. In this embodiment, the vertical positions of the subnasal point P4 and the sub-ear point P5 are equivalent to each other. The vertex P6 below the cheek is located below the subnasal point P4 and the sub-ear point P5.

[0103] The means for recognizing each vertex in the vertex recognition unit 53A may be a method for identifying absolute coordinates relative to spatial coordinates provided for the image data, or a method for identifying relative coordinates based on one of the three vertices that define the demarcation region. In this embodiment, since the image data is 3D data, the coordinate values ​​are also represented in three dimensions.

[0104] The region definition unit 53B defines a triangular region by drawing straight lines connecting the vertices whose positions have been recognized by the vertex recognition unit 53A. The region definition unit 53B also defines a pair of left and right regions based on the midline O1 of the face. The region definition unit 53B defines either a two-dimensional or a three-dimensional region. In this embodiment, the region definition is a three-dimensional region.

[0105] In this embodiment, the region definition unit 53B defines two types of definition regions spaced apart in the vertical direction of the face. Here, the definition region located on the upper side is defined as the upper definition region A1, and the definition region located on the lower side is defined as the lower definition region A2. That is, the region definition unit 53B defines a pair of upper and lower definition regions A1 and A2, one on the left and one on the right.

[0106] The reason why the upper and lower defining regions A1 and A2 are spaced apart vertically is that the evaluation of the entire face extends across its entire vertical range, thereby allowing the evaluation by the upper and lower defining regions A1 and A2 to cover the entire face. Therefore, it is not a problem if parts of the upper and lower defining regions A1 and A2 overlap with each other.

[0107] The area calculation unit 53C calculates the area of ​​the demarcation region. In calculating the area of ​​the demarcation region, the area within the demarcation region is calculated using the coordinate data of each vertex identified by the region demarcation unit 53B. The area comparison unit 53D compares the area of ​​the demarcation region calculated by the area calculation unit 53C with a known reference area that corresponds to the demarcation region.

[0108] The area comparison unit 53D can, for example, use the area of ​​the defined region defined from user 5's image data acquired a certain period before the time of image acquisition, i.e., in the past, as a reference area. Alternatively, the area comparison unit 53D can use the area of ​​the defined region in the ideal model of the face desired by user 5 as a reference area. In this way, the reference area can be arbitrarily set as long as it can be compared with the current defined region area.

[0109] Here, we will explain one example of how to create an ideal model of the face desired by User 5. The ideal model is created using past imaging data. Approximately 100 raw data points are prepared, each with an ideal demarcation region visually specified from the past imaging data. By using this raw data and performing machine learning (deep learning) processing, the ideal model can be created. Machine learning is described in detail in Figure 25.

[0110] Next, we will explain the guidelines for comparing each area, using the area measured in the previous measurement as the reference area as an example. In the upper defined area A1 of this embodiment, the movable point Pm, the vertex P3 above the cheek, is located below the fixed points Pf, the deep nasal point P1 and the vertex P2 at the temple. Similarly, in the lower defined area A2, the movable point Pm, the vertex P6 below the cheek, is located below the subnasal point P4 and the sub-ear point P5.

[0111] Therefore, when the movable points Pm, namely the vertex P3 on the upper cheek and the vertex P6 on the lower cheek, move downward, the areas of the upper and lower defined regions A1 and A2 will each increase. On the other hand, when the movable points Pm, namely the vertex P3 on the upper cheek and the vertex P6 on the lower cheek, move upward, the areas of the upper and lower defined regions A1 and A2 will each decrease.

[0112] In other words, in a configuration like this embodiment, where the movable point Pm is positioned below the fixed point Pf, if the area of ​​the defined region is smaller than the reference area (the area measured in the previous measurement), it means that the movable point Pm has moved upward. This means that the facial proportions have improved, either because the facial muscles have become stronger or because the facial fat has decreased.

[0113] On the other hand, if the area of ​​the defined region is larger than the reference area, which was the area measured in the previous measurement, it means that the movable point Pm has moved downward. In other words, it means that the facial proportions have deteriorated due to weakening of the facial muscles or an increase in facial fat. In this way, user 5 can quantitatively understand whether the facial proportions are improving or worsening by checking the amount of change in the defined region.

[0114] In this embodiment, the configuration described is such that the position of the movable point Pm is located below the position of the fixed point Pf in both the upper and lower defining regions A1 and A2, but the embodiment is not limited to this configuration. The position of the movable point Pm may be located above the position of the fixed point Pf.

[0115] In this case, the comparison between the area of ​​the defined region and the reference area will be the opposite of the explanation given above. That is, if the area of ​​the defined region becomes larger than the reference area, it indicates that the facial proportions are improving, and if the area of ​​the defined region becomes smaller than the reference area, it indicates that the facial proportions are deteriorating.

[0116] Furthermore, if the area of ​​the defined region in the ideal face model desired by user 5 is used as the reference area, it is possible to determine whether the facial proportions have improved by checking how close the model has come to the reference area.

[0117] The display processing unit 53E compares the area of ​​the defined region with the reference area using the area comparison unit 53D and displays the comparison result on the device-side display unit 54 and the terminal-side display unit 45 of the mobile terminal 4, which will be described later. Specific examples of the display content shown by the display processing unit 53E will be described later.

[0118] Next, the control flow of the image processing system 100 and the processing content in the image processing system 100 will be explained using Figures 17 to 18. Figure 17 is a diagram showing the processing flow (image processing method) in the image processing system 100, and Figure 18 is a schematic diagram of the process in which the vertex recognition unit 53A recognizes the vertex P3 on the cheek.

[0119] As shown in Figure 17, in the beauty enhancement method according to this embodiment, first, the imaging unit 21 of the mirror device 2 receives image data of the user 5's face (S601: image reception step). In the image reception step, it is desirable to maintain the same facial expression at all times, for example by lightly clenching the back teeth, in order to suppress changes due to the user 5's facial expression.

[0120] Next, the vertex recognition unit 53A performs a vertex recognition step (S602) in which it recognizes each vertex using the imaging data transmitted from the imaging unit 21. In the vertex recognition step, the positions of two fixed points Pf and one movable point Pm are recognized as three vertices that constitute one defined region. Here, one embodiment of a specific method for determining each vertex is described. Note that this description is merely an example, and each vertex may be determined by other methods.

[0121] As shown in Figure 18, the vertex recognition unit 53A evaluates the image data in three dimensions and recognizes each vertex. First, for the deep nasal point P1, which forms one of the three vertices constituting the upper defined region A1, the deep nasal point P1 is identified as the most recessed part of the nasal root of the face and recognized as the deep nasal point P1. Next, for the temple vertex P2, which forms the other fixed point Pf, the temple vertex P2 is recognized as the most recessed part of the temple area of ​​the face. Note that the temple vertex P2 may be the part of the outer edge of the face in the left-right direction in a frontal view that passes through the straight line connecting the deep nasal point P1 and the center of the pupil or the inner corner of the eye.

[0122] Furthermore, for the vertex P3 on the cheek, which forms the movable point Pm among the three vertices, the most prominent part of the upper cheek near the vertical line outside the pupil is recognized as the vertex P3 on the cheek. In this case, as shown in Figure 18, the most prominent part may also be recognized as the vertex P3 on the cheek by projecting contour lines onto the imaging data. By performing this process on both the left and right sides, each vertex constituting the pair of upper defined regions A1 on the left and right sides is recognized.

[0123] Next, as shown in Figure 16, for the deep nasal point P1, which is one of the three vertices constituting the lower defined region A2, the most recessed part of the lower part of the face below the nose is recognized as the subnasal point P4. Next, for the infra-ear point P5, which is the other fixed point Pf, the most recessed part of the face located below the ear is recognized as the infra-ear point P5.

[0124] Furthermore, for the vertex P6 below the cheek, which forms the movable point Pm among the three vertices, the most prominent part of the lower cheek area, near the vertical line outside the pupil, and next to the corner of the mouth, is recognized as the vertex P6 below the cheek. In some cases, when recognizing the vertex P6 below the cheek, contour lines are projected onto the image data to recognize the most prominent part as the vertex P6 below the cheek. By performing this process on both the left and right sides, each vertex constituting the pair of lower defined regions A2 on the left and right sides is recognized.

[0125] Alternatively, instead of employing the aforementioned method for recognizing each vertex, image processing may be performed to identify each vertex constituting a defined region by, for example, comparing the position of each vertex in pre-registered facial data of multiple people with the captured image data. Alternatively, the position of each vertex may be determined by overlaying the latest image data with past image data. Furthermore, the position of each vertex may be determined by operator 6 selecting appropriate locations for each vertex on the image data.

[0126] Next, the region delimitation unit 53B performs a region delimitation step (S603) to define the delimitation region using the vertex data identified in the vertex recognition step. In the region delimitation step, a triangular delimitation region is defined by straight lines connecting each vertex.

[0127] Next, the area calculation unit 53C performs an area calculation step (S604) to calculate the area of ​​the defined region defined in the region demarcation step. In the area calculation step, the area of ​​the defined region is calculated using the coordinate data of each vertex.

[0128] Next, the area comparison unit 53D performs an area comparison step (S605) in which it compares the area of ​​the demarcation region calculated in the area calculation step with a reference area. In the area comparison step, the area of ​​the demarcation region is compared with a reference area known as the area of ​​the region corresponding to this demarcation region. In this explanation, the area of ​​the demarcation region obtained from past measurement results is set as the reference area.

[0129] Finally, the display processing unit 53E performs a display processing step (S606) to output information indicating the comparison results. In the display processing step, the comparison result between the area of ​​the defined region compared by the area comparison unit 53D and the reference area is displayed on the device-side display unit 54 and the terminal-side display unit 45. The comparison results may also include information suggesting insights into the results and measures that the user 5 should take in the future (such as facial massage). Note that the comparison results do not need to be displayed on the terminal-side display unit 45. By performing such a comparison, it is possible to quantitatively evaluate changes in facial proportions due to aging deterioration and improvement measures, similar to the embodiment described above, and contribute to promoting beauty.

[0130] Next, the results of the evaluation by the face shape evaluation unit 50 and its effects will be explained using Figures 19 and 20. Figure 19 shows an example of the content displayed by the display processing unit 53E, where (a) is the image data from two months ago and (b) is the image data at the time of evaluation. Figure 20 shows another example of the content displayed by the display processing unit 53E, where (a) is the image data from two months ago and (b) is the image data at the time of evaluation. In Figures 19 and 20, the same image data is placed vertically.

[0131] In one example of the comparison results shown in Figure 19, the area of ​​the upper defined region A1 decreased by approximately 23% and the area of ​​the lower defined region A2 decreased by approximately 53% compared to two months prior. This resulted in a more youthful, plump appearance and an improved visual impression.

[0132] In another example of the comparison results shown in Figure 20, the area of ​​the upper defined region A1 decreased by approximately 21.5% and the area of ​​the lower defined region A2 decreased by approximately 25% compared to two months prior. This resulted in a more youthful, plump appearance and improved overall visual impression. As a result, the face appeared more balanced and graceful, leading to an improved overall appearance.

[0133] (Facial shape evaluation section - angle comparison and evaluation) Next, the configuration of the face shape evaluation unit 50 shown in Figure 3 will be explained using Figure 21 to show a comparison of the area disclosed in Figure 15, a comparison of angles different from the evaluation, and an embodiment of the evaluation. Figure 21 is a block diagram showing an alternative configuration added to the face shape evaluation unit 50. The face shape evaluation unit 50 evaluates changes in the proportions of the user 5's face based on the position of the user 5's skeleton, muscles, and fat from the image data. Based on the position of the user 5's skeleton, muscles, and fat, the face shape evaluation unit 50 sets predetermined reference points defined on the user 5's face, calculates the angle formed by predetermined line segments set by connecting the reference points, and displays the angle between the predetermined line segments along with the past history.

[0134] The face shape evaluation unit 50 comprises a device-side communication unit 51, a data storage unit 52, a device processing unit 83, and a device-side display unit 84. The face shape evaluation unit 50 is an information processing device that analyzes the state of the user's face from imaging data captured from the user's face. The device-side communication unit 51 and the data storage unit 52 have the same configuration as in Figure 15, and their explanation is omitted.

[0135] The device processing unit 83 executes the control program stored in the data storage unit 52 to realize each function that the image processing system 100 should achieve. These functions include the reference point setting function, line segment setting function, angle calculation function, angle evaluation function, and result display function, which constitute the functional parts of the face shape evaluation unit 50. The device-side display unit 84 is a monitor device that displays the operation details and processing results of the face shape evaluation unit 50.

[0136] The device processing unit 83 is a computer that controls each part of the face shape evaluation unit 50, and may be, for example, a central processing unit (CPU, GPU), a microprocessor, an ASIC, or an FPGA. However, the device processing unit 83 is not limited to these examples and may be any computer that controls each part of the face shape evaluation unit 50.

[0137] The device processing unit 83 includes a reference point setting unit 83A, a line segment setting unit 83B, an angle calculation unit 83C, an angle evaluation unit 83D, and a display processing unit 83E.

[0138] As shown in Figure 22, the reference point setting unit 83A extracts the midpoint QM of the binocular line segment L0, which is located at the bridge of the user's nose and connects the pupils of both of the user's eyes (right eye Eyr and left eye Eyl), the first reference point Q1, which is located in the skin directly above the muscle attached to the zygomatic arch of the user's face, and the second reference point Q2, which is located in the skin directly above the muscle attached to the mandibular angle of the user's face, from the frontal image of the user's face captured by the imaging unit 21, and sets the first reference point Q1 and the second reference point Q2. Furthermore, in addition to the first reference point Q1 and the second reference point Q2, the reference point setting unit 83A can extract a third reference point Q3 located on the eyebrow of the user's face and set the third reference point Q3. The position and size of the eyebrow are determined from the contours of the face, and the third reference point Q3 is extracted as the midpoint in the length direction of the eyebrow. By increasing the number of reference points, the number of points for evaluating angles, as described later, increases, and the accuracy of the evaluation is improved.

[0139] The midpoint QM is defined as the midpoint of the line segment L0 connecting both eyes, and since it is determined by the facial skeleton and structure, its position changes only slightly over time. Therefore, it can serve as the origin for the first reference point Q1, the second reference point Q2, and the third reference point Q3.

[0140] The first reference point Q1, the second reference point Q2, and the third reference point Q3 are locations on the facial surface that are identified depending on the bone structure and shape of user 5's face. In particular, the first reference point Q1 is a point in the skin directly above the zygomatic arch and contributes to the evaluation of facial fullness. The second reference point Q2 is a point in the skin directly above the mandibular angle and contributes to the evaluation of facial sagging. The third reference point Q3 is a point located in the eyebrow and contributes to the evaluation of facial wrinkles and weakening of the muscles around the eyes.

[0141] The first reference point Q1, the second reference point Q2, and the third reference point Q3 are vertices that are identified depending on facial muscles, fat, and skin laxity. For example, as facial muscles weaken with age and fat accumulates on the face, the positions of these reference points change. Furthermore, by stimulating the facial muscles, the reference points Q1, Q2, and Q3 change to the positions of the user's face at a younger age, as the facial muscles become stronger and the amount of facial fat decreases. These changes in the position of the reference points alter the proportions of the face, and significantly alter the impression the face gives to others.

[0142] Here, the reference points recognized and set by the reference point setting unit 83A in this embodiment will be described with reference to Figure 22. Figure 22 is a diagram showing the reference points recognized by the reference point setting unit 83A, and is a front view of the image data. Note that this is merely an example, and the number, position, etc., of the reference points recognized by the reference point setting unit 83A can be arbitrarily changed. That is, considering the skeletal structure of user 5, the distribution of muscles and fat, etc., reference points on the face that are easy to recognize as described above can be used for evaluation.

[0143] As shown in Figure 22, the reference point setting unit 83A sets a binocular line segment L0 located at the bridge of the nose of user 5 and connecting both of the user's eyes, based on the frontal image of user 5's face, and also sets the midpoint QM of the binocular line segment L0. First, the midpoint QM, which will serve as the origin, is set. Next, the reference point setting unit 83A extracts and sets a first reference point Q1 located in the skin directly above the zygomatic arch of user 5's face. The zygomatic arch is directly related to the evaluation of facial fullness and apparent facial width. Then, the reference point setting unit 83A extracts and sets a second reference point Q2 located in the skin directly above the mandibular angle of user 5's face. The mandibular angle is directly related to the evaluation of jaw prominence and swelling. Furthermore, the reference point setting unit 83A extracts and sets a third reference point Q3 located in the eyebrow of user 5's face. The reference point setting unit 83A determines the position and size of the eyebrows from the contours of the face and extracts a third reference point Q3 as the midpoint position in the length direction of the eyebrows. The third reference point Q3 of the eyebrow position is directly related to the evaluation of eye sagging and the youthfulness of the facial expression.

[0144] The line segment setting unit 83B sets line segments based on the aforementioned reference points and calculates the angles between the line segments based on the resulting line segments. More specifically, as shown in Figure 22, the line segment setting unit 83B sets the first line segment L1 connecting the midpoint QM and the first reference point Q1, the second line segment L3 connecting the midpoint QM and the second reference point Q2, and the third line segment L3 connecting the midpoint QM and the third reference point Q3.

[0145] The angle calculation unit 83C calculates a first angle θ1 formed by the binocular line segment L0 and the first line segment L1 including the first reference point Q1, a second angle θ2 formed by the binocular line segment L0 and the second line segment L2 including the second reference point Q2, and a third angle θ3 formed by the binocular line segment L0 and the third line segment L3 including the third reference point Q3.

[0146] In this embodiment, the reference point and line segment are set on both the left and right sides, with respect to the midline O1 of the user's face. As shown in Figure 22, the first angle θ1, the second angle θ2, and the third angle θ3 are calculated and set individually on the left and right sides, respectively, in positions that are approximately symmetrical. This is because the evaluation of the proportions of the user's face is necessary on both the left and right sides of the face.

[0147] Furthermore, as can be seen from Figure 22, when the imaging unit 21 captures the user 5's face, the display unit 20 displays the outline 24 of the user 5's face image data that was captured in the past, in order to make it easier for the user 5, who is positioned in front of the imaging unit 21, to align their face. As a result, the user 5 can align their face with the outline 24 displayed on the display unit 20, reducing variations and measurement errors each time a photo is taken.

[0148] The means for recognizing each reference point in the reference point setting unit 83A may be a method of identifying absolute coordinates relative to the planar coordinates set for the imaging data, or a method of identifying relative coordinates based on the midpoint QM. In this embodiment, since the imaging data is 3D data and 2D data, the coordinate values ​​are represented two-dimensionally. When extracting each reference point from the planar view data of the front of the user 5's face, a known image analysis technique based on shape recognition is used.

[0149] The angle evaluation unit 83D evaluates the changes in the proportions of user 5's face based on the time-series changes of the first angle θ1 and the second angle θ2, as well as the time-series changes of the third angle θ3, each time the imaging unit 21 images user 5's face. The angle evaluation unit 83D can use, for example, individual angles obtained and calculated from imaging data of user 5 taken a certain period before the imaging data was taken, i.e., in the past, as reference angles. The angle evaluation unit 83D can also use the first angle θ1, the second angle θ2, and the third angle θ3 of the ideal model of the face desired by user 5 as the angles used as the basis for evaluation (reference angles). From the time-series increase or decrease of the numerical values ​​of each calculated angle, an evaluation of the quality of the changes in the state (proportions) of user 5's face (shape evaluation) is derived.

[0150] Here, we will explain one example of how to create an ideal model of the face desired by User 5. The ideal model is created using past imaging data. Approximately 100 (approximately 100 patterns) of original data are prepared, in which ideal reference points and line segments are visually specified from the past imaging data. The number of original data can be more or less than this. By using this original data and performing machine learning (deep learning) processing, the ideal model can be created.

[0151] The display processing unit 83E displays the first angle θ1, second angle θ2, and third angle θ3, which are calculated through the reference point setting unit 83A, line segment setting unit 83B, angle calculation unit 83C, and angle evaluation unit 83D, as well as the changes in the first angle θ1, second angle θ2, and third angle θ3 over time each time the user 5 takes a face image, and the comparison results of each corresponding angle on the device-side display unit 84 and the terminal-side display unit 45 of the mobile terminal 4. Specific examples of the display content shown by the display processing unit 83E will be described later.

[0152] Next, Figure 23 will be added to Figure 22 to explain the control flow of the image processing system 100 and the processing content in the image processing system 100. Figure 23 is a diagram showing the processing flow (image processing method) related to angle calculation and evaluation in the image processing system 100.

[0153] As shown in Figure 23, in the beauty enhancement method according to this embodiment, first, the imaging unit 21 of the mirror device 2 receives image data of the user 5's face (S801: image reception step). In the image reception step, it is desirable to maintain the same facial expression at all times, for example by lightly clenching the back teeth, in order to suppress changes due to the user 5's facial expression. Also, the user 5's face is aligned with the displayed contour 24 (see Figure 22).

[0154] Next, the reference point setting unit 83A performs a reference point setting step (S602) in which it recognizes each reference point using the imaging data transmitted from the imaging unit 21. In the reference point setting step, the first reference point Q1 and the second reference point Q2 are extracted from the 2D data of the frontal image of user 5's face. These include the midpoint QM of the binocular line segment L0, which is located at the base of user 5's nose and connects both of user 5's eyes (right eye Eyr and left eye Eyl), the first reference point Q1 located in the skin directly above the zygomatic arch of user 5's face, and the second reference point Q2 located in the skin directly above the mandibular angle of user 5's face. The first reference point Q1 and the second reference point Q2 are then set. In addition, the third reference point Q3 located in the eyebrow of user 5's face is extracted and set.

[0155] In extracting each reference point from the acquired 2D frontal view of user 5's face, a known image analysis technique based on shape recognition is used. Furthermore, as shown in Figure 22, each reference point is extracted and set from both the left and right sides, with the face's midline O1 as the reference point.

[0156] Next, the line segment setting unit 83B performs a line segment setting step (S803) to set the target line segments using the reference point data identified in the reference point setting step. In the line segment setting step, as shown in Figure 22, the line segment setting unit 83B sets the first line segment L1 connecting the midpoint QM and the first reference point Q1, the second line segment L3 connecting the midpoint QM and the second reference point Q2, and the third line segment L3 connecting the midpoint QM and the third reference point Q3.

[0157] Next, the angle calculation unit 83C performs an angle calculation step (S804) in which it calculates and sets the angle formed by each line segment using the line segment data identified in the line segment setting step. In the angle calculation step, as shown in Figure 22, the angle calculation unit 83C calculates the first angle θ1 formed by the binocular line segment L0 and the first line segment L1 including the first reference point Q1, the second angle θ2 formed by the binocular line segment L0 and the second line segment L2 including the second reference point Q2, and the third angle θ3 formed by the binocular line segment L0 and the third line segment L3 including the third reference point Q3. Since the 2D data of the front view of user 5's face is planar, it is possible to set the target line segments, set angles, and calculate them by two-dimensional geometric processing.

[0158] Next, the angle evaluation unit 83D performs an angle evaluation step (S805) in which it compares each angle calculated in the angle calculation step with the corresponding reference angle. The reference angles are the first angle θ1, second angle θ2, and third angle θ3, which are identified based on previously captured images. In the angle evaluation step, the angles θ1, θ2, and θ3 calculated from the current imaging and acquisition are compared with the corresponding angles θ1, θ2, and θ3 that have been acquired and stored in the past. This includes an evaluation of whether the changes in the proportions of user 5's face are good or bad (improvement, deterioration, etc.) based on the increase, decrease, and changes in the angle values.

[0159] Finally, the display processing unit 83E performs a display processing step (S806) to output information showing the comparison results. In the display processing step, the comparison results of each angle compared by the angle evaluation unit 83D are displayed on the device-side display unit 84 and the terminal-side display unit 45. The comparison results may also include information suggesting insights into the results and measures that the user 5 can take in the future (such as facial massage). Note that the comparison results do not need to be displayed on the terminal-side display unit 45. By performing such a comparison, and combining it with the embodiments detailed up to Figure 20, it is possible to quantitatively evaluate changes in facial proportions due to aging deterioration and improvement measures, thereby contributing to the promotion of beauty.

[0160] Next, using Figure 24, we will explain the results of the evaluation by the face shape evaluation unit 50 and its effects. Figure 24 shows an example of the display content by the display processing unit 83E, with (a) imaging data from two months ago and (b) imaging data at the time of evaluation. In the illustration, only the angles θ1, θ2, and θ3 necessary for evaluation are displayed.

[0161] In the example of comparison results shown in Figure 24, compared to two months prior, the angle values ​​of both the first angle θ1, formed by the binocular line segment L0 and the first line segment L1 including the first reference point Q1, and the second angle θ2, formed by the binocular line segment L0 and the second line segment L2 including the second reference point Q2, decreased. Furthermore, the angle value of the third angle θ3, formed by the binocular line segment L0 and the third line segment L3 including the third reference point Q3, also decreased. As a result, the widening and puffiness (swelling) of the face in a frontal view disappeared, and an improvement in youthfulness and overall appearance was observed. By including each angle θ1, θ2, and θ3 in the evaluation, it becomes possible to quantify the facial contour (face line), increasing the objectivity of the evaluation. The area comparisons and evaluations in Figures 15 to 20 represent a numerical comparison and evaluation of the internal aspects of the face, while the angle comparisons and evaluations in Figures 21 to 24 represent a comparison and evaluation of the external aspects of the face. By overlaying either or both of these, the changes and improvements in the facial condition and proportions of User 5 can be effectively judged.

[0162] The future prediction unit 60 can improve the accuracy of its predictions by using machine learning to evaluate the shape of the user 5's face (comparison of area, comparison of angles, or both) stored in the memory unit 63. In this embodiment, deep learning using a neural network or the like is performed when executing machine learning. Other machine learning methods include correlation rule learning, random forest, and support vector machine. If the prediction accuracy does not reach the target value after machine learning training, further training is performed, and additional features are extracted to compensate for the parts that could not be predicted. Prediction accuracy is improved by retraining the machine learning model. In this way, preliminary network training in machine learning is performed.

[0163] In the schematic diagram of the machine learning process shown in Figure 25, the future prediction unit 60 receives various data from one, two, or all of the following into the input layer 601: the skin health status of user 5 stored in the skin condition evaluation unit 30, the facial shape evaluation of user 5 stored in the facial shape evaluation unit 50 (comparison and evaluation of area), the facial shape evaluation of user 5 stored in the facial shape evaluation unit 50 (comparison and evaluation of angle), and the massage method (treatment area, frequency, tools used, etc.) described later. Then, in the multiple intermediate layers 602, the relationships between the input items are calculated based on the data input from the input layers, and the judgment result is output from the output layer 603.

[0164] When attempting to analyze the interrelationships and extraction of easily quantifiable aspects such as User 5's skin health and shape evaluation (area, angle), and further combine this with the relationships between effective massage methods, treatment areas, frequency, and tools used to influence changes in User 5's facial proportions, the number of conditions to consider becomes excessive, making objective evaluation difficult. Therefore, by incorporating machine learning (deep learning using neural networks), it becomes possible to identify the relationships between items and improve the accuracy of future predictions for User 5's face.

[0165] Next, a massage evaluation method using such an image processing system 100 will be described. In this massage evaluation method, a predetermined massage is applied to the face of user 5, and then the effect of the massage is evaluated using the image processing system 100. This confirms the effectiveness of the effect that the massage has on user 5.

[0166] The massage evaluation method is performed by executing a massage step and an image processing step. The massage step is performed by executing a functional agent application step, a lymph node stimulation step, and a fascial stimulation step. This explanation describes the massage step performed on the face, but the massage step may also be performed on parts other than the face.

[0167] In the functional agent application step, the functional agent is applied to the surface of user 5's face. The functional agent is a drug that has at least one of the following functions: promoting blood circulation, promoting fat dissolution, or promoting fascial relaxation. Specifically, it is preferable that the functional agent contains one of the following components: glaucine (blood circulation promoter), okra seed extract (fat dissolution promoter, fascial relaxation promoter), niacinamide (blood circulation promoter), and grecon hesperidin (blood circulation promoter). However, it is not limited to these components, and any component that produces the aforementioned effects can be used at will.

[0168] In the functional agent application step, the agent is applied to the face, particularly to areas that user 5 wants to tighten, such as the cheeks and around the eyes. Then, the lymph node stimulation step is performed. In the lymph node stimulation step, which involves lymph node massage, the area of ​​User 5's face where the lymph nodes are located is physically stimulated after the functional agent application step. The lymph nodes in the face are located below the ears.

[0169] In the lymph node stimulation step, the lymph nodes are stimulated using the beauty device 70 shown in Figure 26. Figure 26(a) is an external view of the beauty device 70 used in the massage step. Figure 26(b) is a cross-sectional view of the beauty device 70. As shown in Figure 26, the beauty device 70 is equipped with pin-shaped pressing parts 72 and 74 that are biased by biasing members 73 and 75. By pressing these pressing parts 72 and 74 against the skin of the user's face in the area where the lymph nodes are located, the lymph nodes are stimulated. The structure of the beauty device 70 will now be described in detail.

[0170] The beauty device 70 comprises a beauty device body 71, a first pressing part 72, a first biasing member 73, a second pressing part 74, and a second biasing member 75. The beauty device body 71 is cylindrical in shape, formed in a multi-stage cylindrical shape that gradually expands in diameter from one side to the other in the axial direction.

[0171] The beauty device body 71 comprises a small-diameter cylindrical portion 71A located on one side in the axial direction, an intermediate cylindrical portion 71B connected to the small-diameter cylindrical portion 71A in the axial direction, and a large-diameter cylindrical portion 71C located on the other side in the axial direction. On the inner circumferential surface of the large-diameter cylindrical portion 71C, a protruding portion 71D is formed in the portion connected to the intermediate cylindrical portion 71B in the axial direction, extending radially inward.

[0172] The first pressing portion 72 is provided at one end of the beauty device body 71 and presses against the user's face or body, and is composed of a single first pin. The first biasing member 73 is housed inside one end of the beauty device body 71 and biases the first pressing portion 72 toward the outside of the beauty device body 71. The first biasing member 73 is a coil spring.

[0173] The second pressing portion 74 is provided at the other end of the beauty device body 71 and presses against the user's face or body. It consists of three second pins and a holding portion that integrally holds the three second pins. The second biasing member 75 is housed inside the other end of the beauty device body 71 and, by contacting the end of the holding portion, biases the second pressing portion 74 toward the outside of the beauty device body 71. The second biasing member 75 is a coil spring.

[0174] The first biasing member 73 is positioned inside the intermediate cylindrical portion 71B and is in contact with the first pin while in contact with the protruding portion 71D. The second biasing member 75 is positioned inside the large-diameter cylindrical portion 71C and is in contact with the end of the holding portion while in contact with the protruding portion 71D.

[0175] When using the beauty device 70, either of the pressing parts 72 or 74 is pressed against the user's face to stimulate it. The biasing forces of the first biasing member 73 and the second biasing member 75 are different, and the user can choose which pressing part 72 or 74 to use.

[0176] Then, for example, when the first pressing part 72 is pressed against the user 5's face, the first pressing part 72 is displaced against the biasing force from the first biasing member 73. At this time, the length of the first biasing member 73, which is a coil spring, shortens and the amount of deformation increases, so that the biasing force that the first pressing part 72 receives from the first biasing member 73 gradually increases.

[0177] As a result, the force with which the first pressing part 72 presses against the user 5's face gradually increases. The first pressing part 72 may be pressed against the face until the limit of the displacement range of the first biasing member 73 is reached, or the first pressing part 72 may be stopped from pressing against the user 5's face just before pain is felt. By repeating this action multiple times, stimulation can be applied to the user 5's face. In the lymph node stimulation step, not only the lymph nodes in the face but also the lymph nodes near the clavicle are stimulated. This relieves the stagnation of lymphatic fluid in the lymph nodes.

[0178] Next, a myofascial stimulation step is performed, which involves applying myofascial massage. The myofascial stimulation step is performed using the same beauty device 70 used in the lymph node stimulation step. In the myofascial stimulation step, the top of the head, cheeks, neck, etc. are repeatedly pressed with the pressing parts 72 and 74 of the beauty device 70. This helps to release the adhesions between the fascia and the muscles. The lymph node stimulation step and the myofascial stimulation step may be repeated, or the myofascial stimulation step may be performed first, followed by the lymph node stimulation step.

[0179] Next, an image processing step is performed. In the image processing step, the face shape evaluation step performed by the face shape evaluation unit 50 described above, and a display processing step that outputs information showing the change in the proportions of the user 5's face evaluated by the face shape evaluation unit 50 are performed. In the display processing step, the display processing unit 53E displays the evaluation results on the display unit 20. This allows confirmation of the effects of lymph node massage and myofascial massage.

[0180] Here, we will explain the usefulness of stimulating lymph nodes during the massage step. Adhesion between muscles and fascia can sometimes be caused by waste products interposed between them. To remove these waste products, it is effective to improve the flow of lymphatic fluid between the muscles and fascia. And by stimulating the lymph nodes that store lymphatic fluid, the flow of lymphatic fluid can be improved.

[0181] By improving lymphatic fluid flow and stimulating the fascia in this way, waste products interposed between the muscles and fascia can be effectively removed by being carried away by the lymphatic flow. This effectively releases adhesions between the fascia and the muscles.

[0182] Next, the usefulness of stimulating lymph nodes and fascia using a beauty device 70 equipped with pressing parts 72 and 74 during the massage step will be explained. For example, in conventional massage, stimulation was provided by rolling a roller member over the skin surface where the fascia is located.

[0183] However, when stimulating the skin surface with a roller component, the pressure tends to disperse as the roller rotates, making it difficult for the stimulation to reach the deeper parts of the body where the target fascia and lymph nodes are located. As a result, there was a problem in that waste products interposed between the fascia and muscles could not be effectively removed.

[0184] On the other hand, when stimulating the area outside where the fascia is located using the pin-shaped pressing parts 72 and 74, the pressing force from the tips of the pin-shaped pressing parts 72 and 74 is concentrated at the point of contact with the tips of the pressing parts 72 and 74. This makes it easier for the pressing force to reach the deeper parts of the body where the fascia and lymph nodes are located.

[0185] Furthermore, the biasing members 73 and 75 increase the pressing force as the pressing parts 72 and 74 are displaced, allowing the pressing force to be adjusted by adjusting the degree of displacement of the pressing parts 72 and 74. In addition, since the biasing members 73 and 75 bias the pressing parts 72 and 74 with a generally constant force, there is an advantage in that the pressing force can be made generally constant by the pressing parts 72 and 74.

[0186] Next, the first example of the specific procedure for the massage method will be explained with reference to Figures 27 to 32. First, as shown in Figure 27(a), the beauty device 70 is used to stimulate the venous angle near the clavicle. At this time, the pressing parts 72 and 74 of the beauty device 70 are pressed into the venous angle. It is preferable to keep this position for about 5 seconds. Next, as shown in Figure 27(b), the pressing parts 72 and 74 of the beauty device 70 are placed on the central part of the clavicle where the subclavian lymph nodes are located, and the pressing motion of the pressing parts 72 and 74 is repeated for about 20 seconds. (Lymph node stimulation step) It is preferable to press the pressing parts 72 and 74 at a speed of about 5 times per second.

[0187] Next, as shown in Figure 27(c), place the pressing parts 72 and 74 of the beauty device 70 below the mastoid process and repeat the action of pressing the pressing parts 72 and 74 for about 10 seconds. It is preferable to press the pressing parts 72 and 74 at a speed of about 3 to 5 times per second. Next, as shown in Figure 27(d), place the pressing parts 72 and 74 of the beauty device 70 on the back of the head and repeat the action of pressing the pressing parts 72 and 74 for about 10 seconds. It is preferable to press the pressing parts 72 and 74 at a speed of about 3 to 5 times per second. It is preferable to press the part of the back of the head that is at the same height as the center of the ear and above the nape of the neck.

[0188] Next, as shown in Figure 28(a), apply the functional agent to the entire nape of one side of the neck. Apply the functional agent by spreading a few cc of it evenly. This helps to eliminate uneven application of the functional agent. Next, as shown in Figure 28(b), stretch the area in front of the ear up and down. Repeat this motion about five times.

[0189] Next, as shown in Figure 28(c), apply the functional agent around one temple. Apply several cc of the functional agent while spreading it. Next, as shown in Figure 28(d), place four fingers on the temple and pull the fingers diagonally upward (towards the temporal region) to stretch the scalp. It is preferable to hold the fingers pulled up for about 5 seconds.

[0190] Next, stimulate the temporal region. As shown in Figure 29(a), identify the part of the temporal region that is at the same height as the temple and above the ear. Then, as shown in Figure 29(b), place the pressing parts 72 and 74 of the beauty device 70 against the temple and repeat the action of pressing the pressing parts 72 and 74 for about 10 seconds. It is preferable to press the pressing parts 72 and 74 at a speed of about 3 to 5 times per second. Next, as shown in Figure 29(c), repeat the action of pressing the pressing parts 72 and 74 around the temple in the same way. In this case, it is preferable to do it at a speed of about 3 to 5 times per second for about 10 seconds.

[0191] Next, as shown in Figure 30(a), stimulate six points on the temporal region. Perform this action approximately 30 times at each point. Next, as shown in Figure 30(b), place four fingers on the temporal region and massage in an arc motion. Perform this action approximately 5 times.

[0192] Next, as shown in Figure 30(c), apply stimulation to five points on the forehead, from the center of the hairline to the temples. It is preferable to perform this action about 30 times at each point. Next, as shown in Figure 30(d), place four fingers on the forehead and massage in an arc motion. Perform this action about 5 times.

[0193] Next, as shown in Figure 31(a), apply pressure with your thumb from the top of your head and slide it down to behind your ear. Repeat this motion three times along each of the three lines. Next, as shown in Figure 31(b), pull the area behind your ear in the direction of the arrow. Perform this motion for 5 seconds in each direction.

[0194] Next, as shown in Figure 31(c), the beauty device 70 is used to stimulate eight points on the A and B sections. This action is performed 30 times for each point. Next, as shown in Figure 31(d), four fingers are placed at the hairline, and the hair is combed through three times towards the back of the head.

[0195] Next, as a finishing touch, as shown in Figure 32(a), first place your wrist below your ear. Then, as shown in Figure 32(b), move your wrist to the side of your head above your ear and place your other hand on your head. Finally, as shown in Figure 32(c), use both hands to pull upwards. This completes the entire massage.

[0196] Next, we will explain a second example of the specific procedure for the massage method. This massage is mainly for the fascia of the head. It has been confirmed that releasing adhesions in the fascia of the head affects the silhouette of the face. First, as shown in Figure 33(a), grasp the scalp above the ears with the four fingers of both hands. Next, make circular motions with the four fingers of both hands to move the temporalis muscle above the ears. When doing this, make sure to draw three circles, three times each.

[0197] Next, as shown in Figure 33(b), stimulate the scalp with the pressure points 72 and 74 of the beauty device 70, towards the Baihui acupoint located at the top of the head. Next, as shown in Figure 33(c), grasp the scalp at the hairline with the four fingers of both hands. Then, draw circles with the four fingers of both hands to move the frontalis muscle. In this case, draw three circles three times each.

[0198] Next, as shown in Figure 33(d), stimulate the hairline from the center to the temples with the pressing parts 72 and 74 of the beauty device 70. Next, apply pressure with your thumbs from the center of the hairline to the temples and lift by moving them alternately. Next, as shown in Figure 34(a), slide your thumbs from the hairline towards the crown of the head, applying pressure, all the way to behind the ears. At this time, draw each of the three lines three times.

[0199] Next, as shown in Figure 34(b), place four fingers at your hairline and run your fingers through your hair three times towards the back of your head. Then, as shown in Figure 34(c), pull the back of your ears five times in each direction of the arrows.

[0200] Next, as shown in Figure 34(d), the lower part of the mastoid process is stimulated 10 times on each side using the pressing parts 72 and 74 of the beauty device 70.

[0201] Next, as shown in Figure 35(a), pinch the ear between your index and middle fingers and rub firmly five times on each side, applying pressure up and down. Then, as shown in Figure 35(b), stimulate the entire back of the head with the pressure parts 72 and 74 of the beauty device 70. When doing so, stimulate each of the three lines five times.

[0202] Next, as shown in Figure 35(c), pinch the sternocleidomastoid muscle from the base of the neck with the thumb and index finger of one hand, and slide them down to the hollow at the back of the head, applying light pressure for about 5 seconds. Next, as shown in Figure 35(d), as a neck stretch, place one wrist above the ear and the other wrist around the acupoint called Fengchi at the back of the head. Then, while applying pressure inward, pull upward for about 5 seconds. This completes the series of massage steps for the second example.

[0203] Next, we will explain a third example of a specific massage procedure using Figures 36 to 39. This massage primarily focuses on the abdominal area. Abdominal massage aims to improve poor posture caused by hunchback and rounded shoulders. This massage can remove hardening and adhesions in the muscles and fascia around the abdomen. As a result, it can be expected to relieve stiff shoulders, correct hunchback, and tighten the abdomen.

[0204] First, as shown in Figure 36(a), press the first pressing part 72 of the beauty device 70 on three points below the clavicle, from the center of the clavicle to the coracoid process. Then, apply the functional agent and press on it about five times using the second pressing part 74. Next, as shown in Figure 36(b), press the indicated point, called the coracoid process, with the first pressing part 72 of the beauty device 70 for about 30 seconds.

[0205] Next, as shown in Figure 37(a), apply pressure to the intercostal cartilage located below the sternoclavicular joint, up to the upper chest, using the second pressing part 74 about 4 or 5 times. Then, as shown in Figure 37(b), apply pressure to the indurated area of ​​the upper arm using the first pressing part 72 of the beauty device 70. After that, place your hands on both shoulders and apply pressure to open the shoulders.

[0206] Next, as shown in Figure 38(a), with the arm raised, press the first pressing part 72 of the beauty device 70 against the inside of the armpit for about 1 minute. Next, as shown in Figure 38(b), press the first pressing part 72 of the beauty device 70 against the area located three finger-widths outside the solar plexus for about 1 minute at each spot.

[0207] Next, as shown in Figure 39(a), the first pressing part 72 of the beauty device 70 is used to press on the side of the abdomen from the side of the solar plexus to the side of the navel for about one minute at each spot. Next, as shown in Figure 39(b), the first pressing part 72 of the beauty device 70 is used to press on five spots on each side along the ribs from the solar plexus for about one minute each. After that, the functional agent is applied and then pressed with the second pressing part 74.

[0208] Next, we will explain the fourth example of a specific massage procedure using Figure 40. This massage is mainly performed on the chest area of ​​the body. The purpose of massaging the chest is to improve poor posture caused by hunching or rounded shoulders. This massage improves blood flow and lymphatic flow around the collarbone, making it easier for nutrients to reach the breasts.

[0209] First, apply pressure to the central subclavian area with the first pressing part 72 of the beauty device 70 for about 1 minute. Then, as shown in Figure 40(a), apply pressure to the hollow behind the earlobe with the first pressing part 72 of the beauty device 70 for about 1 minute. Next, as shown in Figure 40(b), apply pressure to the lower part of the mastoid process with the first pressing part 72 of the beauty device 70 for about 1 minute. After that, apply the functional agent to the area where the sternocleidomastoid muscle is located. Then, perform the massage described in Figures 35 to 37 as described above.

[0210] As described above, according to the image processing system 100 of this embodiment, the imaging unit 21 of the mirror device 2 captures the face of the user 5, and the skin condition evaluation unit 30 uses this image data to evaluate the health of the user 5's skin based on the user 5's skin color. This allows for a quantitative evaluation of skin abnormalities. As a result, the user 5 can check the condition of their own face while evaluating changes in facial shape.

[0211] The face shape evaluation unit 50 then evaluates the changes in the proportions of the user 5's face based on the image data, using the positions of the user 5's bones, muscles, and fat. Here, since the changes in face proportions are evaluated based on the positional relationship of the user 5's bones, muscles, and fat, it is possible to easily judge whether the change is good or bad before and after the change, compared to an approach that evaluates the overall change of the face. Therefore, changes in the facial morphology can be uniquely determined. In this way, skin abnormalities can also be quantitatively evaluated along with changes in facial morphology.

[0212] Furthermore, since the display unit 20 of the mirror device 2 displays the evaluation results of the skin condition evaluation unit 30 and the face shape evaluation unit 50, the user 5 can check the condition of their facial skin and changes in their proportions when they use the mirror device 2 to groom themselves, thus ensuring the convenience of the user 5.

[0213] Furthermore, the future prediction unit 60 uses at least one or both of the user 5's skin health status stored in the skin condition evaluation unit 30 and the user 5's facial shape evaluation stored in the facial shape evaluation unit 50 to make a future prediction for the user 5's face. This allows the user 5 to be motivated to promote beauty. In addition, machine learning is used when performing calculations in the future prediction unit 60, improving the prediction accuracy by taking into account the trends in various data such as the user 5's skin health status stored in the skin condition evaluation unit 30 and the user 5's facial shape evaluation stored in the facial shape evaluation unit 50, which have been collected in advance.

[0214] Furthermore, the skin color evaluation unit 33A in the skin condition evaluation unit 30 uses image data of user 5's skin to divide any area of ​​user 5's skin into several pre-set stages based on its hue value. Then, the skin abnormality identification unit 33B identifies the location of pigment abnormalities on user 5's skin based on the stages divided by the skin color evaluation unit 33A. This allows for a quantitative evaluation of the degree of pigmentation, such as blemishes and pigment spots, which can then be used for cosmetic purposes.

[0215] Furthermore, the evaluation result display unit 33C displays the location of each stage categorized by the skin color evaluation unit 33A on the image data, and marks the pigment abnormalities identified by the skin abnormality identification unit 33B. This allows the user 5 to visually understand the evaluation results, ensuring user convenience.

[0216] Furthermore, the evaluation result display unit 33C displays the position and markings for each stage of multiple image data captured for the same user 5 at different times, and displays them side by side. This allows the user 5 to visually understand the changes in their skin condition over time.

[0217] Furthermore, the depth estimation unit 33D estimates the depth of the pigmented abnormality from the skin surface on user 5's skin based on the hue value of the pigmented abnormality, thus providing information for considering measures to improve the pigmented abnormality.

[0218] Furthermore, the policy proposal unit 33E proposes measures to promote improvement of pigmentation based on the depth of the pigment abnormality, so it can propose appropriate measures according to the condition of the pigment abnormality.

[0219] Furthermore, the improvement data generation unit 33F changes the hue of the pigmented area in the image data, assuming that the pigmentation in the pigmented area has been improved, and displays this to the user 5. This visually communicates the expected effects of the proposed measures to the user 5, thereby motivating the user 5 to continue implementing the measures.

[0220] Furthermore, according to the image processing system 100 of this embodiment, the vertex recognition unit 53A recognizes the positions of two fixed points Pf, which are identified depending on the facial skeleton, and one movable point Pm, which is identified depending on the facial muscles and fat, from the image data of the user 5's face.

[0221] Next, the region definition unit 53B defines a triangular region by drawing straight lines connecting the identified vertices, and the area calculation unit 53C calculates the area of ​​the defined region. Then, the area comparison unit 53D compares the area of ​​the defined region with a reference area. This allows for a quantitative evaluation of changes in facial proportions due to aging and improvement measures, contributing to the promotion of beauty.

[0222] Furthermore, since the demarcation area is defined by two fixed points Pf and one movable point Pm, compared to a configuration in which the demarcation area is defined by, for example, two or three movable points Pm, it is possible to suppress variations in the specific position of the movable point Pm, which is difficult to recognize, and to perform accurate evaluation.

[0223] Furthermore, by evaluating the area of ​​the defined region, it is possible to use larger numerical values ​​and thus larger amounts of change compared to, for example, evaluating the position of a movable point Pm by its distance from a fixed point Pf. As a result, even if the changes in facial proportions are so subtle that they are difficult to discern from the imaging data before and after a certain period of time, the user 5 can more easily recognize the degree of facial change and gain motivation to promote beauty treatments.

[0224] Furthermore, existing facial proportion analyses are mostly dependent on skeletal structure (such as the so-called golden ratio of the face), and are based on innate factors that cannot be changed, making it difficult to increase the user's motivation for cosmetic treatments other than plastic surgery. On the other hand, the image processing system 100 does not rely solely on skeletal structure, but evaluates changes in muscle and fat due to aging that can be improved through self-care, and is therefore able to increase the user's motivation for cosmetic treatments.

[0225] Furthermore, the image processing system 100 includes a display processing unit 53E that outputs information showing the comparison result of comparing the area of ​​the defined region with the reference area. Therefore, the results of the quantitative evaluation can be easily confirmed by displaying them, for example, on the user 5's mobile terminal 4.

[0226] Furthermore, since the region definition unit 53B defines a pair of definition regions on the left and right sides based on the midline O1 of the face, it is possible to promote beauty in order to achieve a symmetrical facial proportion.

[0227] Furthermore, the region definition area 53B defines the region using two fixed points Pf, which are identified by the deep nasal point P1 and the apex P2 of the temple, and one movable point Pm, which is identified by the apex P3 on the cheek. This allows for a quantitative evaluation of the proportions around the upper part of the cheeks, making it possible to check changes in, for example, sagging of the upper cheeks that tends to become a concern with age (e.g., the marionette lines formed between the nasolabial folds and the cheekbones).

[0228] Furthermore, the vertex recognition unit 53A evaluates the image data in three dimensions to recognize the deep nose point P1, the temple vertex P2, and the cheek vertex P3. Therefore, each vertex can be easily recognized regardless of the facial features of the user 5.

[0229] Furthermore, since the region definition unit 53B defines two types of definition regions spaced apart in the vertical direction of the face, the overall proportions of the face can be quantitatively evaluated by evaluating the upper and lower parts of the face separately, thereby promoting beauty even more effectively.

[0230] Furthermore, since the region definition unit 53B defines the region using two fixed points Pf identified by the subnasal point P4 and the sub-ear point P5, and one movable point Pm identified by the apex P6 below the cheek, it is possible to quantitatively evaluate the proportions around the lower part of the cheeks and, for example, to check changes in sagging in the lower part of the cheeks, which tends to become a concern with age.

[0231] Furthermore, the vertex recognition unit 53A evaluates the image data in three dimensions and recognizes the sub-nasal point P4, the sub-ear point P5, and the vertex P6 below the cheek, so that each vertex can be easily recognized regardless of the shape of the user's face 5.

[0232] Furthermore, if the area comparison unit 53D uses the area of ​​the defined region on the user 5 at a certain period prior to the time of imaging of the image data as the reference area, it is possible to quantitatively evaluate how the facial proportions change over time. This allows for an accurate understanding of the cosmetic effect.

[0233] Furthermore, when the area comparison unit 53D uses the area of ​​the defined region in the ideal face model desired by user 5 as the reference area, it is possible to quantitatively confirm how close the user has come to the target. This helps maintain user 5's motivation for beauty and promotes effective beauty care.

[0234] In addition, the reference point setting unit 83A extracts the midpoint QM of the line segment L0 located at the base of the user's nose and connecting both of the user's eyes, the first reference point Q1 located on the skin directly above the zygomatic arch of the user's face, and the second reference point Q2 located on the skin directly above the mandibular angle of the user's face from the frontal image of the user's face captured by the imaging unit 21, and sets the first reference point (Q1) and the second reference point (Q2). Furthermore, it extracts the third reference point Q3 located on the eyebrow of the user's face and sets the third reference point Q3. This sets reference points for understanding changes in the contour (face line) of the user's face.

[0235] Furthermore, the line segment setting unit 83B and the angle calculation unit 83C calculate the first angle θ1 formed by line segment L0 and the line segment connecting the midpoint QM and the first reference point Q1, the second angle θ2 formed by line segment L0 and the line segment connecting the midpoint QM and the second reference point Q2, and the third angle θ3 formed by line segment L0 and the line segment connecting the midpoint QM and the third reference point Q3. This generates angle information that should be used as a basis for evaluating the time series.

[0236] Furthermore, the angle evaluation unit 83D evaluates the changes in the proportions of the user's face from the time-series changes of the first angle θ1, the second angle θ2, and the third angle (θ3) each time the imaging unit 21 images the user's face 5. This allows for the objective and easily understandable evaluation of changes in the user's facial contour (face line), which can lead to improvements in facial proportions.

[0237] Furthermore, since the image processing system 100 is equipped with a face shape evaluation unit 50, it can easily acquire imaging data of the user's face 5 and evaluate the imaging data using the face shape evaluation unit 50.

[0238] Furthermore, the display units 54 and 84 on the device side have the function of displaying on the display surface the image data of the user 5's face, which is facing the display surface. As a result, the user 5 can check the image data of the user 5's face captured by the imaging unit 21 of the mirror device 2, and the evaluation results of the changes in proportion performed by the device processing units 53 and 84 using the image data, as if looking in a mirror.

[0239] Furthermore, in the massage evaluation method according to the present invention, by performing the functional agent application step, the components contained in the functional agent are applied to the user's face, thereby producing at least one of the following effects: blood circulation promotion effect, fat dissolution effect, and fascia relaxation effect. This enhances the effect of the massage.

[0240] Furthermore, the lymph node stimulation step and the fascia stimulation step stimulate the lymph nodes and fascia, thereby stimulating the lymph nodes to promote lymphatic fluid flow, promoting the metabolism of waste products between muscles and fascia, and effectively resolving adhesions between fascia and muscles.

[0241] After performing the massage in this manner, an image processing step is performed using the image processing system 100. By evaluating the changes in the proportions of user 5's face from the image data of user 5's face, the effects of the massage can be reliably evaluated.

[0242] Furthermore, in the lymph node stimulation step of the massage evaluation method, the lymph nodes are stimulated using a beauty device 70 equipped with pin-shaped pressing parts 72 and 74 that are biased by biasing members 73 and 75. Therefore, even if the lymph nodes are located deep within the body, the lymph nodes can be reliably stimulated and the flow of lymphatic fluid can be promoted. This promotes the metabolism of waste products between muscles and fascia, and effectively eliminates adhesions between fascia and muscles.

[0243] Furthermore, if the functional agent contains glaucine, it can promote blood circulation. If the functional agent contains okra seed extract, it can promote blood circulation.

[0244] Furthermore, if the functional agent contains niacinamide, it can promote fat dissolution. Also, if the functional agent contains grecon hesperidin, it can produce a myofascial relaxation effect.

[0245] Furthermore, in the myofascial stimulation step of the massage evaluation method, a beauty device 70 equipped with pin-shaped pressing parts 72 and 74 biased by biasing members 73 and 75 is used to press the parts of the user's face where the fascia is located using the pressing parts 72 and 74. Therefore, compared to a configuration that stimulates the surface of the skin by rolling a roller member, for example, this method can reliably stimulate the fascia located deep within the skin.

[0246] Furthermore, if the pin-shaped pressing parts 72 and 74 are biased by a coil spring acting as a biasing member, the force exerted when the pressing parts 72 and 74 are displaced can always be kept constant, and the pressing force exerted by the pressing parts 72 and 74 can be kept constant.

[0247] Furthermore, the evaluation result provision unit 61 provides the analysis results to the personal terminal 2B, and the confirmation content reporting unit 62 reports the confirmation history information of the analysis results provided to user 5 to the store terminal 2A. In addition, the storage unit 63 stores the content that user 5 confirmed using the store terminal 2A when visiting the store, along with the confirmation history information. As a result, store staff can check what aspects of the analysis performed on user 5's own images interest them, and by providing advice on the topics that interest user 5, beauty promotion can be carried out more efficiently.

[0248] Furthermore, since the image analysis device 1 is equipped with a skin condition evaluation unit 30, a face shape evaluation unit 50, and a user identification unit 64, it is possible to evaluate the skin condition and changes in facial proportions for each user 5, and multiple users 5 can use one mirror device 2.

[0249] Furthermore, the storage unit 63 stores the facial data of each user 5 and the user 5's ID, and the user identification unit 64 identifies user 5 by referring to the image data of the user's face captured by the storage unit 63. As a result, user authentication can be performed simply by having user 5 have their face captured, eliminating the need for user 5 to, for example, enter their own ID, thus ensuring user convenience.

[0250] Furthermore, the confirmation content reporting unit 62 aggregates the number of times user 5 has checked the analysis results performed by the skin condition evaluation unit 30 and the face shape evaluation unit 50 within a predetermined period, and reports this to the store terminal 2A. Therefore, by using the confirmation history information of multiple users 5, it becomes possible to statistically understand what beauty items all users 5 are interested in. This can be used to help the store provide guidance on promoting beauty.

[0251] Furthermore, the confirmation content reporting unit 62 reports at least one of the maintenance methods and maintenance products displayed by the display unit 20 to the store terminal 2A. This allows store staff to understand which maintenance methods and products the user 5 is interested in, which can be useful for the store's sales activities.

[0252] It goes without saying that the image processing system 100 is not limited to the above embodiment and may be implemented by other methods. Various modifications will be described below. For example, the control program (image processing program) of the above embodiment may be provided stored in a storage medium readable by a computer. The storage medium is a "non-temporary tangible medium" capable of storing the image processing program. The storage medium may include any suitable storage medium such as an HDD or SSD, or two or more suitable combinations thereof. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile. It should be noted that the storage medium is not limited to these examples and may be any device or medium capable of storing the image processing program.

[0253] Furthermore, the image processing system 100 can realize each of the functions shown in the embodiment by, for example, reading a control program stored in a storage medium and executing the read control program. The image processing program may also be provided to the image processing system 100 via any transmission medium (such as a communication network or broadcast waves). The image processing system 100 realizes the functions of the multiple functional units shown in each embodiment by, for example, executing a control program downloaded via the Internet or the like.

[0254] Furthermore, the image processing program may be implemented using, for example, a scripting language such as ActionScript or JavaScript®, an object-oriented programming language such as Objective-C or Java®, or a markup language such as HTML5.

[0255] At least a portion of the processing in the image processing system 100 may be implemented by cloud computing, which consists of one or more computers. Furthermore, each functional unit of the image processing system 100 may be implemented by one or more circuits that implement the functions shown in the above embodiment, or the functions of multiple functional units may be implemented by one circuit.

[0256] Furthermore, although the above embodiment describes an example in which the image analysis device 1 and the mobile terminal 4 are separate devices, the embodiment is not limited to this. The mobile terminal 4 may have some or all of the functions of the image analysis device 1.

[0257] Furthermore, while embodiments of this disclosure have been described based on various drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions included in each means, step, etc., can be rearranged in a logically consistent manner, and multiple means or steps, etc., can be combined into one or divided. Also, the configurations shown in each embodiment may be combined as appropriate. [Explanation of Symbols]

[0258] 1. Image analysis device 2. Mirror device 30 Skin Condition Evaluation Department 50. Facial Shape Evaluation Unit 70 Beauty device 100 Image Processing Systems

Claims

1. A mirror device comprising a mirror surface that reflects the user's face, a display unit provided on the mirror surface, and an imaging unit positioned on the mirror surface to capture images of the user's face, The system includes an image analysis device that analyzes and evaluates image data of the user's face captured by the imaging unit, The aforementioned image analysis device is A reference point setting unit extracts the midpoint (QM) of the line segment (L0) located at the base of the user's nose and connecting both of the user's eyes, a first reference point (Q1) located on the skin directly above the zygomatic arch of the user's face, and a second reference point (Q2) located on the skin directly above the mandibular angle of the user's face, from the frontal image of the user's face captured by the imaging unit, and sets the first reference point (Q1) and the second reference point (Q2). An angle calculation unit that calculates a first angle (θ1) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the first reference point (Q1), and a second angle (θ2) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the second reference point (Q2), The system includes an angle evaluation unit that evaluates changes in the proportions of the user's face from the time-series changes in the first angle (θ1) and the second angle (θ2) each time the imaging unit captures the user's face. An image processing system characterized by the following:

2. The reference point setting unit extracts a third reference point (Q3) located on the user's eyebrow and sets the third reference point (Q3). The angle calculation unit calculates the third angle (θ3) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the third reference point (Q3), The image processing system according to claim 1, wherein the angle evaluation unit evaluates the change in the proportions of the user's face from the time-series changes of the first angle (θ1), the second angle (θ2), and the third angle (θ3) each time the imaging unit captures an image of the user's face.

3. The image processing system according to claim 2, wherein the first angle (θ1), the second angle (θ2), and the third angle (θ3) are set on both the left and right sides with respect to the midline of the face.

4. The image processing system according to claim 1, wherein the display unit displays the analysis results from the image analysis device.

5. The image processing system according to claim 1, characterized in that the display unit displays the outline of previously captured image data of the user's face so that the user, who is positioned in front of the imaging unit, can align their face when the imaging unit captures the user's face.

6. The image processing system according to claim 1, characterized in that the display unit is capable of adjusting the area for displaying data and the area that becomes a mirror surface.

7. The image processing system according to claim 1, wherein the angle evaluation unit analyzes and evaluates image data of the user's face.

8. The aforementioned image analysis device is A storage unit that stores the evaluation results performed by the angle evaluation unit, The image processing system according to claim 1, further comprising a future prediction unit that predicts the future shape of a user's face using the user's face shape evaluation stored in the memory unit.

9. The image processing system according to claim 8, wherein the future prediction unit performs a future prediction of the user's face by using machine learning to evaluate the shape of the user's face stored in the memory unit.

10. A mirror device comprising a mirror surface that reflects the user's face, a display unit provided on the mirror surface, and an imaging unit positioned on the mirror surface to capture images of the user's face, An image processing method in an image analysis apparatus comprising an image analysis device that analyzes and evaluates image data of a user's face captured by the imaging unit, The aforementioned image analysis device, A reference point setting step in which, from a frontal image of the user's face captured by the imaging unit, the midpoint (QM) of the line segment (L0) located at the root of the user's nose and connecting the user's eyes, a first reference point (Q1) located on the skin directly above the zygomatic arch of the user's face, and a second reference point (Q2) located on the skin directly above the mandibular angle of the user's face are extracted and the first reference point (Q1) and the second reference point (Q2) are set. An angle calculation step to calculate a first angle (θ1) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the first reference point (Q1), and a second angle (θ2) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the second reference point (Q2), The imaging unit performs an angle evaluation step, which evaluates the change in the proportions of the user's face from the time-series changes in the first angle (θ1) and the second angle (θ2) each time it images the user's face. An image processing method characterized by the following:

11. A mirror device comprising a mirror surface that reflects the user's face, a display unit provided on the mirror surface, and an imaging unit positioned on the mirror surface to capture images of the user's face, An image processing program for an image analysis device comprising an image analysis device that analyzes and evaluates image data of a user's face captured by the imaging unit, The aforementioned image analysis device, A reference point setting function that extracts the midpoint (QM) of the line segment (L0) located at the root of the user's nose and connecting both of the user's eyes, a first reference point (Q1) located on the skin directly above the zygomatic arch of the user's face, and a second reference point (Q2) located on the skin directly above the mandibular angle of the user's face, from the frontal image of the user's face captured by the imaging unit, and sets the first reference point (Q1) and the second reference point (Q2). An angle calculation function that calculates a first angle (θ1) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the first reference point (Q1), and a second angle (θ2) formed by the line segment (L0) and the line segment connecting the midpoint (QM) and the second reference point (Q2), The system provides an angle evaluation function that evaluates changes in the proportions of the user's face based on the time-series changes in the first angle (θ1) and the second angle (θ2) each time the imaging unit captures an image of the user's face. An image processing program characterized by the following:

Images