Method and device for visualizing foundation application status

Abstract

Provided is a method for visualizing the foundation application status from a face video using a general RGB camera instead of using a multispectral camera or a special filter. The present invention comprises: an acquisition step for acquiring, using an RGB camera, a time-series vector R of a color variation of a bare skin part and time-series vectors of respective pixels in a part to be visualized; and a visual representation step for visually representing the foundation application status on the basis of evaluation of similarity between the time-series vector R and the time-series vectors of the respective pixels in the part to be visualized. Specifically, an inner product of the time-series vector R and the time-series vector of the part to be visualized is calculated for each pixel in the part to be visualized, and the foundation application status is visually represented on the basis of the component size of the time-series vector R for the time-series vectors of the respective pixels in the part to be visualized. The time-series vector is acquired from a time variation of green luminance in each pixel of the camera, while utilizing the characteristics of hemoglobin to absorb a green component of light.

Description

Method and device for visualizing foundation application status

[0001] The present invention relates to a technology for visualizing the application status of foundation from facial motion images.

[0002] Foundation has two functions: it covers blemishes and uneven skin tone to make skin look beautiful, and it protects skin from external damage such as UV rays, air pollution, and dryness. Therefore, applying foundation evenly is important. However, because foundation is a color close to the skin's natural color, it is difficult to detect uneven application with the naked eye, making it difficult to master the technique of applying foundation evenly. Many companies have focused on the issues of makeup smearing and uneven foundation application that occur after makeup application, and have conducted research and development to reduce these problems. However, these have mainly focused on the ingredients and formulations of tools and foundations, and few have focused on methods to detect uneven application. Furthermore, makeup education also requires an objective indicator of whether foundation has been applied properly, but the reality is that it is difficult for beginners to apply foundation evenly by eye.

[0003] Research into detecting uneven foundation application has been known for some time. For example, Non-Patent Document 1 describes that a spectral data set of bare skin and foundation-applied skin was constructed, and a functional spectral filter with optimized spectral transmittance characteristics was created to maximize the accuracy of distinguishing between the two states, thereby distinguishing between bare skin and foundation-applied skin. However, since Non-Patent Document 1 uses a functional spectral filter with optimized spectral transmittance characteristics, it is not generally applicable.

[0004] Furthermore, uneven application can be visualized using a multispectral camera that can observe not only the visible light band but also the near-infrared wavelength band. However, multispectral cameras are expensive and not commonly available. Therefore, Non-Patent Document 2 proposes a method for classifying images of foundation-coated skin and bare skin using machine learning, as part of a study using a common RGB camera that can observe three wavelength bands: red (R), green (G), and blue (B). The proposed method demonstrated an accuracy rate of approximately 80% in experiments. However, because it was difficult to prepare foundation that closely matched the skin color of each participant, the experiment was conducted using a foundation that closely matched the author's own skin color. Detection of uneven application using machine learning requires relatively high-resolution images due to the influence of wrinkles and pores. Furthermore, Non-Patent Document 2 did not visualize uneven application of the entire face.

[0005] Furthermore, Patent Document 1 discloses a method for evaluating the application state of foundation, which generates and evaluates the standard deviation of luminance between the bare skin state and the state after application, and describes that the bare skin state and the state after application are represented by vectors, and that the dot product of these vectors indicates the thickness of the foundation application and the degree of aggregation. In an embodiment of Patent Document 1, the application state of foundation in a small area is plotted on a graph with two axes representing luminance values ​​(standard deviation of luminance) and the reflected light ratio, or the application state of foundation in a vector space with two axes representing the standard deviation of luminance and the reflected light ratio, with the coordinates of the small area on bare skin (reflected light ratio is 1) as the starting point and the coordinates of the small area after foundation application as the end point.

[0006] Furthermore, Patent Document 2 discloses a method for evaluating the covering ability of a foundation by calculating and comparing the reduced diffusion coefficient and absorption coefficient of the skin before and after application of the foundation.

[0007] JP 2019-007895 A JP 2004-163405 A

[0008] Akira Nishino et al., "Development and Application of a Foundation Quantitative and Distribution Measurement System Using Functional Spectral Filters," Journal of the Color Science Association of Japan (2013). Miho Kajita et al., "Basic Research to Eliminate Foundation Application Unevenness," 13th Forum on Data Engineering and Information Management (DEIM2021), No. C14-4, pp.1-8 (2021).

[0009] As described above, there are technologies for detecting uneven foundation application, but with a view to use on smartphones, there is a need for a method and device for capturing facial images using a general RGB camera and visualizing the application state to distinguish between areas where foundation is applied and areas where it is not. In light of this situation, an object of the present invention is to provide a method and device for visualizing the application state of foundation from moving images of a face using a general RGB camera without using a multispectral camera or special filters.

[0010] To solve the above problem, the visualization method of the present invention uses an RGB camera to visualize the application status of foundation. The method includes an acquisition step of acquiring a time series vector R of color fluctuations in the bare skin area and a time series vector of each pixel in the visualization target area, and an imaging step of imaging the application status of foundation based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area. Rather than observing the color difference between the skin and foundation, the present invention focuses on color changes due to hemoglobin synchronized with heartbeat, and uses an RGB camera to capture color fluctuations at each location in the observation area to image the application status of foundation. In other words, since skin color fluctuates subtly due to fluctuations in hemoglobin pigment caused by heartbeat, hemoglobin color fluctuations can be observed in areas where blood flow is present on the skin surface. However, when foundation is applied to the skin, the skin surface is obscured, making it difficult to observe hemoglobin color fluctuations. Therefore, in the present invention, an RGB camera is used to acquire moving image data of the face, and from that data, a time series vector R (reference vector) of the color fluctuations of the bare skin is obtained, and imaging is performed by evaluating how many components of the time series vector R (reference vector) the time series vector of each pixel in the face has.

[0011] Here, the time series vector R of color fluctuations in the bare skin portion and the time series vector of each pixel in the visualization target portion are acquired during the same time period. Note that, because the time series changes in color are synchronized across the entire skin, it is possible to evaluate subtle differences by evaluating the similarity between the time series vector of each pixel in the visualization target portion during the same time period and the time series vector R. When evaluating similarity during the same time period, there is no problem even if the heart rate fluctuates during the observation time.

[0012] The imaging step in the visualization method of the present invention preferably calculates, for each pixel of the visualization target portion, the dot product of the time series vector R and the time series vector of the visualization target portion, and images the foundation application status based on the magnitude of the component of the time series vector of each pixel of the visualization target portion that is time series vector R. Calculating the dot product of the two vectors for all pixels and imaging them has lower computational costs than performing a Fourier transform or the like.

[0013] Furthermore, in the imaging step, the magnitude of the components of the time series vector R is larger in bare skin areas, and is smaller in foundation-applied areas as the thickness of the applied film increases.

[0014] In the acquisition step of the visualization method of the present invention, the time series vector is preferably acquired from the time variation of the green luminance of each pixel of an RGB camera. It is known that the concentration of hemoglobin contained in the dermis of the skin surface changes in accordance with the heart rate. There is a method for estimating the heart rate from the change in luminance of the skin surface accompanying the change in hemoglobin concentration (see, for example, the paper "Masaaki Yasumaru et al., 'Accuracy Evaluation of a Non-Contact Heart Rate Estimation Method Using 4K Facial Video,' Research Report Audiovisual Multimedia Processing (AVM), Vol. 2018, No. 20, pp. 1-5 (2018)"). Furthermore, because hemoglobin has the property of absorbing the green component of light, measuring the green component can indirectly determine the fluctuation of hemoglobin pigment, i.e., the change in blood flow and heart rate. The pixel values ​​(brightness values) of a typical RGB camera are expressed as 256 levels ranging from 0 to 255 for red (R), green (G), and blue (B), respectively. The time variation of the green (G) brightness value in each pixel of the RGB camera matches the time variation of the hemoglobin pigment.

[0015] In the acquisition step, the time series vector is obtained from video image data or continuous shooting data for a predetermined time obtained by an RGB camera. For example, if a video of 30 frames per second is captured for 10 seconds, 300 frames of images can be acquired, and the time series vector of each pixel becomes a 300-dimensional vector. If a video is captured for 1 second, the time series vector becomes a 30-dimensional vector.

[0016] In the visualization method of the present invention, the bare skin portion is preferably a portion where the heart rate can be measured, such as the palm or neck. Alternatively, the time series vector of a signal obtained from an electroencephalograph, an electrocardiograph, or an earphone-type or watch-type heart rate monitor attached to the subject's body can also be used as the time series vector R of the color fluctuation of the bare skin portion.

[0017] In the visualization method of the present invention, data on the time series vector R of the bare skin portion and the time series vector of the entire visualization target portion without foundation may be acquired in advance, and the similarity evaluation may be corrected based on the acquired data, thereby uniformly detecting uneven foundation application throughout the entire visualization target portion. That is, a time series vector R (reference vector) is acquired in advance from the palm of the hand when no foundation is applied, and the time series vector of each pixel of the visualization target portion is also acquired. The time series vector of each pixel is assumed to have a different amplitude from the time series vector R but a similar waveform (hemoglobin visibility varies depending on the location on the skin, skin thickness, and pigmentation). If the amplitude of pixel (i, j) in the bare skin state is α(i, j) times the amplitude of the time series vector R, the time series vector of each pixel after foundation application is corrected by dividing by α(i, j). That is, in locations where hemoglobin is more easily observed α times (locations where the amplitude is α times), the correction is performed by dividing by α to match the level of the reference time series vector R. Furthermore, in addition to the case where the amplitude of pixel (i, j) is not α(i, j) times the amplitude of time series vector R, it is also possible to compare, for example, the dot product of the time series vector R (reference vector) of each pixel of the palm in a bare state with the time series vector of each pixel of the visualized target portion (e.g., the face), with the dot product of the time series vector R (reference vector) of each pixel of the palm in a bare state with the time series vector of each pixel of the visualized target portion (the face) after foundation has been applied, and calculate how many times larger it is.

[0018] The visualization method of the present invention may further include a presentation step of comparing a first image of the foundation application status of the visualization target area obtained by performing the acquisition step and the imaging step at a first timing with a second image of the foundation application status of the visualization target area obtained by performing the acquisition step and the imaging step at a second timing, and presenting the degree of change in the foundation application status superimposed on the second image. This involves performing the acquisition step and the imaging step at a timing (first timing) when the user has finished applying foundation (e.g., when makeup is complete), and then performing the acquisition step and the imaging step at a timing (second timing) when the amount of foundation applied has decreased over time (e.g., when makeup has runny), and comparing the image obtained at the first timing with the image obtained at the second timing to obtain the degree of change in the foundation application status. The degree of change in the foundation application status is then presented superimposed on the second image.

[0019] The visualization method of the present invention may further include a pre-acquisition step of acquiring in advance a time series vector of color fluctuations of each pixel in the visualization target portion before foundation application, a comparison step of comparing the time series vector of color fluctuations of each pixel in the visualization target portion after foundation application with the time series vector of color fluctuations of each pixel in the visualization target portion after foundation application, and a correction step of correcting the imaging result of the application status based on the comparison result. Since the visualization result may reflect not only the amount of foundation applied but also differences in skin thickness, etc., calibration is performed by acquiring time series data of the entire face in a bare skin state before foundation application and comparing it with data collected after foundation application.

[0020] In the visualization method of the present invention, the visualization target may be a face, and in the acquisition step, a time series vector may be acquired for a relative position with respect to the face using face tracking processing. For example, a chin rest may be used to fix the facial posture, but the chin rest may be made unnecessary by combining it with face tracking. Furthermore, by performing face tracking, even if room lighting or the like is reflected off the skin surface, it is possible to capture the image in the same way by changing the orientation of the face so that the reflected light is not captured by the camera.

[0021] The visualization program of the present invention is a program for causing a computer to execute each step of the visualization method of the present invention. By installing this visualization program as an application program on a mobile terminal or personal computer (PC), it is possible to perform a visualization process of the application status of foundation on the mobile terminal or personal computer (PC) using video captured with an RGB camera. In addition, this visualization program can be installed on a server computer on a network, and video data captured with the RGB camera can be sent from a client terminal (a computer terminal used by a user or a computer terminal used for business purposes) to the server computer, visualization process can be performed using the received data, and the visualization results can be sent back to the client terminal, which can then display them on a display.

[0022] The visualization device of the present invention is a device that visualizes the application status of foundation using an RGB camera, and is equipped with an RGB camera, a computer, and a memory in which the above-mentioned visualization program is stored.The visualization device acquires a time series vector R of color fluctuations in the bare skin area and a time series vector of each pixel in the visualization target area, and images the application status of foundation based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area.

[0023] The visualization device of the present invention may further include a smart mirror equipped with an RGB camera, and the computer may detect the user's facial area from an image acquired by the RGB camera and display an image of the foundation application status in the detected facial area superimposed on the facial area on a display in the mirror portion of the smart mirror. Use of a smart mirror makes it possible to acquire images and display application status images in accordance with the user's daily makeup application actions using a mirror, improving user convenience.

[0024] The foundation application status visualization method, visualization program, and visualization device of the present invention have the advantage of being able to visualize the foundation application status from moving images of a face using a general RGB camera, without using a multispectral camera or special filters. Furthermore, since the visualization method, visualization program, and visualization device do not look at the color difference between the skin and the foundation, but rather at color variations in each location, they have the advantage of being able to detect even foundations of colors that are indistinguishable from the naked eye, compared to conventional techniques. Furthermore, compared to methods using multispectral cameras or high-resolution cameras, they have the advantage of being able to visualize the foundation application status using equipment that is much less expensive. Furthermore, since a general RGB camera is used, it is possible to capture images with a wide angle of view, which means that the visualization method can be used in environments with a high degree of freedom for facial movement, focusing on the entire face rather than just a portion of the face.

[0025] 1. Processing flow diagram of the visualization method 2. Flow diagram of preprocessing 3. Explanatory diagram of bare skin area 4. Flow diagram of similarity evaluation 5. Explanatory diagram of foundation application pattern 6. Schematic diagram of the experimental device for visualization 7. An example of experimental results for visualization 8. An example of experimental results in which the face and palm were photographed simultaneously 9. Visualization result when foundation toned to the subject's skin color was applied (subject A) 10. Visualization result when foundation toned to the subject's skin color was applied (subject B) 11. Visualization result when foundation toned to the subject's skin color was applied (subject C) 12. Visualization result when different amounts of foundation were applied 13. Visualization result when different amounts of foundation were applied 14. Visualization result when foundation was applied to the entire face 15. Image of use of the visualization device in Example 7 16. Image of use of the visualization device in Example 8 17. Flow diagram of the visualization method in Example 8

[0026] An example of an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and variations are possible.

[0027] The visualization method of the present invention visualizes the application status of foundation by evaluating the degree to which skin color fluctuations due to hemoglobin pigments associated with heartbeat correlate with bare skin. FIG. 1 shows the process flow of the visualization method. In the visualization method of the present invention, a video including a facial region is captured using an RGB camera (step S01), and fluctuations in pixel values ​​(R, G, B) of the captured video frames are acquired (step S02). A time series vector R of color fluctuations in the bare skin region is then acquired (step S03), and a time series vector for each pixel in the visualization target region is acquired (step S04). Since time series changes in color across the entire skin are synchronized in steps S03 and S04, time series vectors for the same time period are acquired. The similarity between the time series vectors of each pixel in the visualization target region and the time series vector R is then evaluated (step S05), and the application status of foundation is visualized and displayed on a display (step S06). The evaluation of vector similarity will be described later.

[0028] Here, the preprocessing of the visualization method of the present invention will be described with reference to the flow chart in FIG. 2. The preprocessing of the visualization method consists of extracting a face region, defining the image of the face region, removing DC and low-frequency components, and extracting a bare skin region from the face image (steps S11 to S16). First, the face region can be extracted using an existing face region detection method, such as CascadeClassifier from OpenCV (an open-source library). The image of the face region can also be extracted by fitting it to a 3D mesh model using face landmarks obtained using a face landmark detection method, such as Google's MediaPipe (registered trademark), and expanding it onto the UV plane. Next, the DC and low-frequency components are removed. An infinite impulse response low-pass filter (IIR-LPF) is used to obtain low-frequency components as shown in Equation 1 below. Here, α = 0.05 is empirically determined. Then, the difference between the current frame and the previous frame is obtained and the DC and low-frequency components are removed as shown in Equation 2 below.

[0029]

[0030]

[0031] Then, if foundation is applied to a part of the face area, the time series vector R of the color fluctuation of the bare skin area where foundation is not applied, that is, the reference vector (Ref(t)), is obtained by x,y The bare skin region (Region) is extracted from (t) and the arithmetic mean (Avg) is calculated as shown in the following formula 3.

[0032]

[0033] To specify the bare skin area, a mouse was clicked on a bare skin area in the face area where no foundation was applied to designate it as a rectangular area, as shown in Figure 3. An example of a graph of the time series vector R (reference vector) is shown in Figure 4. The horizontal axis represents the frame number (index), and the vertical axis represents the value obtained by subtracting the DC component from the green component of each video frame. Fluctuations in the graph reflect changes in hemoglobin pigment concentration due to heartbeat. Each pixel in the face image is then compared to determine whether similar fluctuations exist.

[0034] (Similarity Evaluation) Next, the calculation of the similarity between the time series vector R (reference vector) of the bare skin part and the time series vector of each pixel of the visualization target part will be described. The similarity between the reference vector and the time series vector of each pixel was evaluated using the inner product and cosine similarity. The time series vector of each pixel is calculated using Image x,y , and the reference vector is represented as Ref, the inner product is calculated by the following formula 4. x,y The cosine similarity between Ref and R is calculated using the following equation 5. In these calculations, only the green component is used for visualization. FIG. 5 shows the flow of similarity evaluation using the dot product of vectors. In the similarity evaluation, the dot product of the time series vector R and the time series vector of the visualization target area is calculated for each pixel of the visualization target area (step S21), and a larger dot product value is evaluated as bare skin, while a smaller dot product value is evaluated as a thicker foundation-applied area (step S22).

[0035]

[0036]

[0037] (Visualization Demonstration Experiment) The items used in the demonstration experiment are shown in Table 1 below. The experiment was carried out in a tabletop darkroom. Surface reflections were removed by a polarizing filter (not shown) placed in front of the camera and ring light. In this experiment, video data was recorded at a resolution of 968 x 608 dots at 30 frames per second (uncompressed).

[0038]

[0039] The subjects were instructed to wash their faces and remove all makeup before the experiment. Next, two 30-second videos of their bare faces were filmed. The subjects were instructed to close their eyes and remain as still as possible during the video recording. The subjects wore a pulse monitor on their finger to measure their pulse during the video recording. Next, the subjects applied makeup. Foundation (Chacott Fit Foundation 570) was applied using a metal spatula. To facilitate application of the foundation to a 20 mm diameter circle, a sticker with a 20 mm diameter hole was attached to the application area, and the foundation was applied. After application, the sticker was removed. Five microliters of foundation was used to apply one circle. The iComes Tofutty (Tofutty is a registered trademark) electric pen dispenser, capable of dispensing small amounts of highly viscous liquid, was used.

[0040] Foundation was applied to the subjects in the pattern shown in Figure 6 (circle diameter: 20 mm). The position of the forehead circle was defined as the intersection of a line drawn vertically from the inner corner of the eye and a line drawn parallel to the eyebrows and hairline. The position of the cheek circle was defined as the intersection of a line drawn vertically from the pupil and a line drawn horizontally from the tip of the nose.

[0041] Figure 7 shows a schematic diagram of the experimental setup for visualization. Figure 8 shows an example of the experimental results. Figure 8(1) is an image captured by a camera (the rectangle in the image represents the image area selected to generate the reference vector), Figure 8(2) is an image in which the cosine similarity of all pixels has been calculated, and Figure 8(3) is an image in which the inner product of all pixels has been calculated. Here, the image in which the inner product has been calculated is the L of the time series vector R (reference vector). ２ The vector is normalized by the norm (the square root of the sum of the squares of each element of the vector).

[0042] In the image in Figure 8 (3) where the dot product was calculated, areas with and without foundation were clearly visible. However, in some areas, such as the shirt collar, the values ​​were high, making it difficult to distinguish from bare skin (not shown). We speculate that this is due to the high calculated values ​​caused by subtle body movements. On the other hand, in the image in Figure 8 (2) where the cosine similarity was calculated, the values ​​were not high in the shirt collar. However, when cosine similarity was used, areas with bare skin appeared brighter when the waveform of the time series vector R (reference vector) was similar to the waveform of the time series vector of each pixel (not shown). Therefore, for the purpose of distinguishing areas with foundation applied from areas with bare skin, which have the same fluctuations, albeit weakly, an image where the dot product was calculated taking amplitude into account is more suitable. Similar results were obtained for different subjects, confirming that the image where the dot product was calculated for all pixels effectively visualizes the application of foundation.

[0043] In the demonstration experiment of Example 1, only a portion of the face was covered with foundation. However, in actual use, foundation is applied to the entire face, so data on bare skin cannot be obtained. Therefore, the palm of the hand was also photographed at the same time as the face. A reference vector was calculated using a rectangular area designated on the palm, and the area where the foundation was applied was visualized.

[0044] Figure 9 shows an example of experimental results in which the palm of a hand was photographed simultaneously with the face. Figure 9 (1) shows an image captured by a camera, and Figure 9 (2) shows an image in which the dot product was calculated for all pixels. While foundation colors typically vary from person to person, in this experiment, we deliberately avoided using foundation that perfectly matched the subject's skin color. Therefore, it is possible to visually determine the areas covered by foundation, which is generally not easy. On the other hand, our visualization method is not affected by the color itself, but instead utilizes only color fluctuations, making it applicable even when the color of makeup-covered skin and bare skin are the same. Because hemoglobin fluctuations are not uniform on facial skin, hemoglobin can be observed strongly in some areas but weakly in others. Therefore, we obtain time-series data of the entire face when bare and compare it with data after applying foundation to visualize changes from bare skin.

[0045] The visualization device of this embodiment includes an RGB camera, a computer, and a memory storing the visualization program of the present invention. The visualization device acquires a time series vector R of color variation in the bare skin area and a time series vector of each pixel in the visualization target area, and visualizes the application status of foundation based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area. The visualization device may be a computer (such as a laptop PC, tablet device, or smartphone) equipped with an RGB camera, or a computer to which an RGB camera is connected. The computer and the RGB camera may be connected via a wired or wireless connection. The visualization program of the present invention is stored in the computer's memory. This visualization program may be installed externally or stored in the memory from the beginning. The computer acquires video data captured by the RGB camera, and the visualization program acquires from the video data the time series vector R of color variation in the bare skin area and the time series vector of each pixel in the visualization target area. The application status of foundation is then visualized based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area, and the facial image is displayed on the computer's display. The visualization device may be a server computer and its memory on a network, with the visualization program of the present invention running on the server. That is, it may also be a device in which an RGB camera is connected to a server via a network. Video data captured by the RGB camera is sent to the server, and visualization processing is performed using the data received by the server. The visualization results are stored on the server, and are displayed on a computer terminal connected to the server.

[0046] Figures 10 to 12 show the visualization results when a foundation toned to the subject's skin color was applied. Figure 10 shows the experimental results for subject A, Figure 11 shows the results for subject B, and Figure 12 shows the results for subject C. (1) or (3) in each figure displays the first frame of an image captured by a camera. The square skin area 1 in the image represents the image area selected to generate a reference vector. In (1) of each figure, a bare skin area on the face is selected, and in (3) of each figure, a bare skin area on the palm (the bare skin area near the base of the ring finger of the right hand) is selected. (2) or (4) in each figure is an image in which the dot product of all pixels is calculated. In (2), the reference vector is a bare skin area on the face, and in (4), the reference vector is a bare skin area on the palm. Images (2) or (4) in each figure are visualized images calculated based on the dot product of the reference vector and the time series vector of each pixel of the face for all pixels. The images generated using the reference vector extracted from the bare skin region of the face shown in (2) of each figure show the foundation application area 2b with higher contrast and more clearly than the images generated using the reference vector extracted from the palm region shown in (4) of each figure. This demonstrates that it is possible to visualize the application state of foundation toned to the subject's skin color.

[0047] Figures 13 and 14 show visualization results when different amounts of foundation were applied to subject A. Figure 13 shows the case where the amount of foundation applied was 3 μL, and Figure 14 shows the case where the amount of foundation applied was 7 μL. (1) or (3) in each figure displays the first frame of an image captured by a camera. The bare skin area 1 indicated by a rectangle in the image represents the image area selected to generate a reference vector. In each figure, (1) selects a bare skin area on the face, and in each figure, (3) selects a bare skin area on the palm of the hand (the bare skin area near the base of the ring finger on the right hand). Images (2) or (4) in each figure are visualized images calculated for all pixels based on the dot product of the reference vector and the time series vector of each pixel on the face. Furthermore, (2) in each figure shows the case where the reference vector is a bare skin area on the face, and (4) shows the case where the reference vector is a bare skin area on the palm of the hand. As shown in Figures 13 and 14, it was found that increasing the amount of foundation applied from 3 μL to 7 μL made the color contrast between the bare skin area and the area where the foundation was applied more pronounced, making it possible to visualize the difference in the amount of foundation applied.

[0048] FIG. 15 shows the visualization results of applying foundation to the entire face of subject B. (1) and (2) show the results when a makeup artist (a makeup artist with over 40 years of professional makeup experience and an examiner for the Japan Makeup Technician Certification Examination certified by the Japan Makeup Technician Certification Association) applied the foundation, while (3) and (4) show the results when subject B, an ordinary person who is not a makeup expert, applied the foundation himself. FIG. 15 (1) or (3) displays the first frame of an image captured by a camera. The bare skin area 1 indicated by a rectangle in the image represents the image area selected to generate a reference vector. In both FIG. 15 (1) and (3), a portion of the palm (the bare skin area near the base of the ring finger on the right hand) is selected. FIG. 15 (2) or (4) shows an image in which the dot product of all pixels has been calculated. As shown in Figure 15 (2), when the makeup artist applied the product, the facial color was uniform, whereas when subject B, who is not a makeup artist, applied the product himself, as shown in Figure 15 (4), the facial color was uneven and appeared shaded.

[0049] In this example, after applying foundation to the entire face, a reference vector was calculated from the palm of the hand to visualize the entire face. However, this visualization result does not completely reflect unevenness in the application of foundation. This is because the visualization result may reflect not only the amount of foundation applied, but also differences in skin thickness, etc. For this reason, calibration may be performed by acquiring time-series data of the entire face in a bare skin state before applying foundation, and comparing it with data collected after applying foundation, and then visualizing the changes after applying foundation to the entire face.

[0050] FIG. 16 is an image diagram of a visualization device of Example 7, where (1) shows a face region detection image and (2) shows an application mottle display image. As shown in FIG. 16, in this example, a smart mirror 3a is used as a visualization device for the foundation application status. The smart mirror 3a has a display 4a that functions as a mirror in addition to a display function, and is equipped with an RGB camera 10a, other sensors, an Internet connection function, etc. The computer and memory included in the smart mirror 3a are also used as the computer and memory constituting the visualization device. Note that a cloud server or the like may also be used for the memory. As in Example 3, the time series vector R of the color variation of the bare skin area and the time series vector of each pixel in the visualization target area are acquired, and the foundation application status is visualized based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area.

[0051] The smart mirror 3a also has a function to detect the face area 5 of the user 7 from an image acquired by the RGB camera 10a. The smart mirror 3a can image the foundation application status for the face area 5 detected using this face area detection function and save it in memory. This can be performed manually each time by the user 7's operation, or automatically. The smart mirror 3a also has a function to detect areas where foundation is not sufficiently applied in the image of the foundation application status and to display the detected areas superimposed on the user 7 on the display 4a. Specifically, as shown in FIG. 16(2), the smart mirror 3a displays the user 7 reflected in the mirror and the application mottled area 6a displayed on the display 4a superimposed on the user 7. This allows the user 7 to easily see the areas where foundation is not sufficiently applied. Note that the smart mirror 3a may also be displayed superimposed on the user 7 and the application mottled area 6a displayed on the display 4a without using the mirror function. In addition to the application mottled area, specific application conditions and advice may also be displayed as comments 8 on the display 4a.

[0052] FIG. 17 is an image diagram of the visualization device of Example 8, where (1) shows an image captured immediately after application, and (2) shows an image of the makeup removal confirmation. As shown in FIG. 17 , in this example, a smartphone 3b is used as a visualization device for the foundation application status. The smartphone 3b is a known smartphone and includes a display 4b, an RGB camera 10b, other sensors, an internet connection function, and the like. The computer and memory included in the smartphone 3b are also used as the computer and memory constituting the visualization device. Note that a cloud server or the like may be used for the memory. Similar to Example 3, the visualization device acquires a time series vector R of color variation in the bare skin portion and a time series vector of each pixel in the visualization target portion, and visualizes the foundation application status based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target portion.

[0053] The smartphone 3b also has a function for detecting a face region 5 of the user 7 from an image acquired by the RGB camera 10b. The smartphone 3b can image the foundation application status for the face region 5 detected using this face region detection function and store the image in memory. Furthermore, the smartphone 3b has a function for detecting areas in the image of the foundation application status where the foundation is not sufficiently applied, such as due to makeup smearing, or where the foundation has come off, and a function for displaying the detected areas on the display 4b. When displaying the paint mottling area on the display 4b, an image obtained by calculating the dot product or cosine similarity of all pixels may be displayed, and areas where paint mottling has occurred may be highlighted. Alternatively, for example, as shown in FIG. 17(2), the paint mottling areas (6b, 6c) may be displayed superimposed on the face image of the user 7 acquired by the RGB camera 10b.

[0054] In this embodiment, the criterion for displaying an area as a mottled area is that an amount of applied foundation is determined to be less than, for example, 70% of the area to which foundation was applied when the makeup was completed (for example, when the makeup was finished before going out), but the criterion is not limited to such a numerical criterion. For example, instead of just the amount of applied foundation, the area ratio of the mottled area to the entire facial area of ​​the user 7 may be calculated, and if the ratio of the mottled area exceeds a predetermined threshold, it may be determined that an mottled area exists and a notification may be sent on the smartphone 3 b.

[0055] FIG. 18 shows a flow diagram of the visualization method of Example 8. As shown in FIG. 18 , first, at a first timing, the face area 5 of the user 7 is detected from an image acquired by the RGB camera using the face detection function of the smartphone 3b (step S31). The first timing refers to a state in which foundation has been applied, i.e., the makeup is completed. In this state, an image of the foundation application status on the face area 5 obtained by performing the acquisition step and the imaging step is stored in memory as a first image (step S32). Next, at a second timing, the face area 5 of the user 7 is detected from an image acquired by the RGB camera using the face detection function of the smartphone 3b (step S33). The second timing refers, for example, to a state in which a predetermined time has passed since foundation was applied without removing makeup. The image of the foundation application status on the face area 5 obtained by performing the acquisition step and the imaging step is stored in memory as a second image (step S34). The smartphone 3b's application mottle detection function is used to compare the first image and the second image stored in the memory (step S35). The degree of change in the foundation application state is superimposed on the second image and displayed on the display 4b (step S36). This allows the user 7 to easily check whether or not their makeup has come off by looking at the display 4b of the smartphone 3b they are holding.

[0056] (Other Examples) By using existing tools available on the market (for example, Google's MediaPipe), face tracking and hand tracking can be performed to calculate time series vectors for the same position on the skin of the face or hand even if the face or hand moves, and visualization can be performed without using a chin rest. In addition, the use of face tracking prevents false detections caused by facial movements.

[0057] The present invention is expected to be used with a PC equipped with an RGB camera or dedicated equipment in cosmetics sales booths, beauty salons, and other makeup application settings, as well as in educational institutions. It can also be used as a smartphone application program for everyday use by individuals or as a confirmation tool for personal learning to improve skills.

[0058] 1 Bare skin area (bare skin region) 2a, 2b Foundation application area (application region) 3a Smart mirror 3b Smartphone 4a, 4b Display 5 Face area 6a to 6c Application spot area 7 User 8 Comment 10, 10a, 10b RGB camera 11 Ring light 12 Chin rest

Claims

1. A method for visualizing the application of foundation using an RGB camera, comprising: an acquisition step of acquiring a time series vector R of color fluctuations in the bare skin area and a time series vector of each pixel in the visualization target area; and an imaging step of imaging the application of foundation based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area.

2. The visualization method described in claim 1, characterized in that the imaging step calculates the dot product of the time series vector R and the time series vector of the visualization target area for each pixel of the visualization target area, and images the application status of foundation based on the magnitude of the component of the time series vector R in the time series vector of each pixel of the visualization target area.

3. The visualization method described in claim 2, characterized in that in the imaging step, the magnitude of the components of the time series vector R is larger in bare skin areas, and smaller in areas where foundation is applied, the greater the thickness of the applied film.

4. A visualization method according to any one of claims 1 to 3, characterized in that in the acquisition step, the time series vector is acquired from the time variation of green brightness at each pixel of the RGB camera.

5. The visualization method according to claim 4, wherein in the acquisition step, the time series vector is obtained from video image data or continuous shooting data for a predetermined time period obtained by the RGB camera.

6. A visualization method according to any one of claims 1 to 3, characterized in that the bare skin area is a part of the body where the heart rate can be measured, such as the palm or neck.

7. A visualization method according to any one of claims 1 to 3, characterized in that the time series vector R of the color fluctuations of the bare skin is a time series vector of a signal obtained from an electroencephalograph, an electrocardiograph, or an earphone-type or watch-type heart rate monitor attached to the subject's body.

8. A visualization method according to any one of claims 1 to 3, characterized in that data on the time series vector R of the bare skin area and the time series vector of the entire visualization target area where no foundation is applied is acquired in advance, and similarity evaluation is corrected based on this data, thereby detecting unevenness in the application of foundation evenly across the entire visualization target area.

9. The visualization method described in claim 1, further comprising a presentation step of comparing a first image of the foundation application status of the visualization target area obtained by performing the acquisition step and the imaging step at a first timing with a second image of the foundation application status of the visualization target area obtained by performing the acquisition step and the imaging step at a second timing, and presenting the degree of change in the foundation application status by superimposing it on the second image.

10. The visualization method according to claim 1, further comprising: a pre-acquisition step of acquiring in advance a time series vector of color fluctuations of each pixel in the visualization target area before foundation application; a comparison step of comparing the time series vector of color fluctuations of each pixel in the visualization target area after foundation application with the time series vector of color fluctuations of each pixel in the visualization target area after foundation application; and a correction step of correcting the imaging result of the application status based on the comparison result.

11. A visualization method according to any one of claims 1 to 3, characterized in that the visualization target part is a face, and in the acquisition step, a time series vector is acquired using face tracking processing, and data is acquired at a position where light is not reflected.

12. A visualization program for causing a computer to execute each step of the visualization method of any one of claims 1 to 3, 9 and 10.

13. A mobile terminal equipped with a computer that executes the visualization program of claim 12.

14. A device for visualizing the application status of foundation using an RGB camera, comprising: an RGB camera; a computer; and a memory storing the visualization program of claim 10; the visualization device acquires a time series vector R of color fluctuations in the bare skin area and a time series vector of each pixel in the visualization target area, and creates an image of the application status of foundation based on a similarity evaluation between the time series vector R and the time series vector of each pixel in the visualization target area.

15. The visualization device described in claim 14, further comprising a smart mirror equipped with the RGB camera, wherein the computer detects the user's facial area from the image acquired by the RGB camera, and displays an image of the foundation application status in the detected facial area superimposed on the facial area on the display of the mirror portion of the smart mirror.

Images