Beauty support system

Abstract

A system for providing beauty methods that achieve a positive impression in both self-assessment and evaluation by others, and a method for providing beauty methods using this system. [Solution] A beauty support system including an information input terminal that acquires still images of the face, as well as videos of the face, still images of the whole body, and videos of the whole body, and a computing device and an external computing device that comprehensively evaluate the customer's impression. Situations in which videos of the face are acquired include during conversations and when watching videos, etc. From videos of the face during conversations, information about the voice when speaking or nodding in agreement is acquired, for example. From this audio information, data such as voice tone, volume, voice color, and timing of responses are detected.

Description

Technical Field

[0001] The present invention relates to a beauty method providing system and a beauty method providing method using the same.

Background Art

[0002] In a general facial esthetic treatment, a counselor conducts counseling for a customer and determines the content of the facial esthetic treatment according to the result. In recent years, in addition to such a provision method, as described in Patent Document 1, instead of diagnosis by a counselor, a method has been established in which an image of a customer's face is photographed and evaluated using an evaluation device, and the treatment content is determined based on the result.

[0003] As described above, most of the conventional methods for determining the content of facial esthetic treatment provided without relying on a counselor evaluate still images. Therefore, it often happens that a person unconsciously makes an expression towards the camera, and it is difficult to say that the state is close to that in actual daily life or that it reflects the impression of the customer as seen by others. Furthermore, the observation targets of diagnosis by a counselor and a still image of the face are mostly the face, and the influence of each element of the whole body, such as posture, movement, and expression, on the face impression that others have towards the customer could not be reflected.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The method for determining the treatment that has been performed in beauty salons and the like has mainly been based on self-evaluation and still image diagnosis. Therefore, the subjective aesthetic evaluation of the customer receiving the treatment is achieved While it was possible to achieve this, it failed to attain the kind of beauty that is objectively judged by others, thus failing to satisfy the need to be seen as beautiful not only by oneself but also by others. [Means for solving the problem]

[0006] In light of the current situation, the inventors conducted thorough research and found that acquiring not only still images of the face, but also videos of the face, still images of the whole body, and videos of the whole body, and comprehensively evaluating the customer's impression, can better reflect their natural state in everyday life and the impression they give to others. This led to the completion of the present invention.

[0007] In other words, the present invention encompasses the following inventions. 1. A beauty support system that evaluates customer impressions through image-based impression evaluation analysis and video-based impression evaluation analysis. 2. The beauty support system described in item 1, which evaluates the customer's impression using at least the customer's own self-assessment and evaluations from others other than the customer. 3. The beauty support system according to claim 1, wherein the image and video used for impression evaluation are one or more parts selected from the customer's face, upper body, and whole body. 4. A beauty support method that evaluates a customer's impression using image-based impression evaluation analysis and video-based impression evaluation analysis, and provides treatment content based on at least the evaluation results of the impression evaluation device. 5. The beauty support method described in item 4, which evaluates the customer's impression using at least the customer's own self-assessment and evaluations from others other than the customer. 6. The beauty support method described in item 4, wherein the evaluation target for the impression evaluation is one or more parts selected from the customer's face, upper body, and whole body. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to objectively evaluate the impression a customer makes, and to provide more specific treatments that match the impression the customer wants to achieve, thereby further increasing customer satisfaction. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating an example of the relationship between scent and associated words. [Figure 2] A diagram illustrating examples of image words that evoke a specific impression based on facial morphology. [Figure 3] This diagram shows examples of image words that evoke certain skin tones. [Figure 4] An explanatory diagram showing the configuration of a beauty support system according to a representative embodiment of the present invention. [Figure 5] This is a block diagram showing the server configuration in the beauty support system. [Modes for carrying out the invention]

[0010] Next, embodiments of the present invention will be described in detail.

[0011] This invention is a system that diagnoses customer impressions and proposes methods for improvement using analytical information obtained from both still images and videos.

[0012] Still images can provide information about the subjective impression a customer has of themselves. They can also provide information about the impression the customer and others have of their appearance. Furthermore, still images can provide information about the localized features of the face and body.

[0013] The body parts to be captured in the still images can be selected as appropriate according to the purpose of the analysis. These include, but are not limited to, the face, upper body, and whole body. For example, these include, but are not limited to, still images of the face taken from the front, still images of the face taken from the left and right sides, still images of the face taken from the jaw direction, still images of the face taken from the forehead direction, still images of the whole body taken from the front, still images of the whole body taken from the left and right sides, still images of the whole body taken from the feet towards the head, and still images of the whole body taken from the head towards the feet.

[0014] When taking a still image of the face, the customer may be asked to intentionally make an expression. For example, a smiling face, a blank expression, an angry expression, a sad expression, etc. can be mentioned, but it is not limited to these.

[0015] By obtaining a still image of the face, for example, physical quantities such as those shown in FIG. 1 can be obtained, but it is not limited to these. The acquired information is transmitted to a support server or the like via a communication device or the like.

[0016] Using the acquired physical quantities, image words related to morphological features, skin tone, etc. can be assigned to the customer. Examples of image words received from morphological impressions are shown in FIG. 2, and examples of image words received from skin tone, etc. are shown in FIG. 3, but it is not limited to these.

[0017] Situations for taking a still image of the whole body include, for example, a standing posture, the usual way of standing, the usual way of sitting on a chair, etc.

[0018] By obtaining these whole-body still images from the front, side, or diagonal direction and analyzing the position of the head relative to the spine, the angle of the face, the position of the feet relative to the knee position, the opening of the legs, etc., for example, information regarding the posture of the spine, the position of the tongue, etc. can be obtained.

[0019] From the face and whole-body videos, mainly information about the objective impression that others have of the customer can be obtained. This is because the impression of a person seen by others is not based on a momentary thing at a certain point, but is formed in continuous movements as represented when having a conversation. Also, a person's impression is not formed by only a limited certain element, but consists of multiple elements such as shape, voice, gesture, etc., so the analysis of impressions using videos is one of the effective means.

[0020] Situations for obtaining a face video considered for obtaining the information as described in

[0019] include, for example, during a conversation, when watching a video, etc.

[0021] From the video of the face during the aforementioned conversation, information about the voice, such as when speaking or nodding in agreement, can be obtained. From this audio information, data such as voice tone, volume, voice color, and timing of responses can be detected. In addition, data such as facial, eye, and mouth movements, and timing of responses when the voice is spoken can be obtained. This obtained information is transmitted to various servers via communication devices, etc.

[0022] By acquiring both still images and videos, it is possible to obtain information about the customer's subjective and objective impressions. This allows customers to obtain information about both the impression they want to give and how they want to be perceived by others. Furthermore, by considering not only physical beauty such as appearance but also information obtained from non-physical aspects such as voice and manner of speaking, it is possible to achieve an impression that is not limited to outward appearance alone.

[0023] Next, I will explain the circumstances under which the still images and videos mentioned above are acquired.

[0024] An example of a beauty support system according to this embodiment is one that includes, as shown in Figure 4, a support server 1 for sending and receiving various information, an information input terminal 2 for obtaining images or videos, a computing device 3 for inferring customer impressions based on diagnostic results and necessary information, and for creating and selecting appropriate information to be provided to individual customers and store staff outside the support server, an external computing device 4, an external server 5 for obtaining various information other than diagnostic results necessary for inferring customer impressions, and various information not stored in the support server 1 necessary for selecting and creating useful information regarding impressions and maintenance / improvement of beauty to be provided to customers, and an information output terminal 6 for outputting diagnostic results and other information.

[0025] The information input terminal 2 and information output terminal 6 may be the same terminal equipped with the arithmetic unit 3, or they may be separate terminals connected via communication. The external arithmetic unit 4 and external server 5 may be connected via communication with the terminal equipped with the arithmetic unit 3 as needed. The method of communication may be via the internet or limited to a local network. Furthermore, the information input terminal 2 and information output terminal 6 may be separate terminals or the same terminal.

[0026] As shown in Figure 5, the terminal equipped with the beauty support system in this embodiment consists of a single electronic device comprising a processing unit 9 with an information input unit 7, a communication control unit 8, and an information output unit 11, and a storage means 10. Specifically, a conventional computer device can be used, centered around the processing unit 9, with the storage means 10, the communication control unit 8, and input operation means (not shown) such as a pointing device, keyboard, or display.

[0027] In the beauty support system according to the above embodiment, the support server 1 may be divided into two or more. In this case, the divided servers may be able to send and receive various information from each other using a connection means that does not use the internet, such as a local network, or they may be connected via a network N such as the internet to send and receive various information, or they may have both. When there are multiple support servers, each support server can be assigned to perform tasks such as recording and data processing (calculations), as well as data and system backups, in a manner that is optimal for operating the system. If the location of the information data input terminal 2 is geographically far from the support server 1, another support server 1 can be set up in the same location as the information data input terminal 2, and the two can be connected without using a network such as the internet, and the acquired and entered information can be recorded in the support server 1. This can significantly reduce the number of times personal information is sent and received via the internet, thereby reducing the risk of leakage of confidential information such as personal information. In addition, if the useful information generation unit of the processing unit of the support server 1 cannot process or complete the processing due to reasons such as information processing capacity, an external computing device can be used.

[0028] Examples of locations for installing the information input terminal 2 include, but are not limited to, beauty salons and other retail stores, department store cosmetics counters, door-to-door sales locations, and customers' homes. Furthermore, examples of the information input terminal 2 include, but are not limited to, smartphones, tablet devices, and personal computers.

[0029] Examples of locations for installing the information output terminal 6 include, but are not limited to, beauty salons and other stores, cosmetics counters in department stores, door-to-door sales locations, and customers' homes. Furthermore, examples of the information output terminal 6 include, but are not limited to, smartphones, tablet devices, and personal computers.

[0030] The processing unit 9 is mainly composed of a CPU such as a microprocessor and has a storage unit consisting of RAM and ROM (not shown) where programs and processing data that define the procedures for various processing operations are stored. The storage means 10 can be selected from memory inside or outside the support server 1, hard disk drives (HDD), solid state drives (SSD), removable disk exchange systems (RDX), redundant arrays of inexpensive disks (RAID), etc. The contents of some or all of the storage unit may be stored in the memory or hard disk of another computer connected to the support server 1.

[0031] It is desirable to acquire still images and videos of customers using a terminal equipped with a camera and / or various sensors. While it is preferable to use dedicated measuring equipment (terminals), this can be substituted by connecting the necessary accessories for information acquisition to a terminal such as a smartphone owned by the subject, or by installing a measurement application. The information acquired by these terminals will ultimately be stored in the support server 1 as either raw primary information or processed secondary information. Regarding the method of transmitting information from the terminal to the system, if the terminal has a communication unit that can connect directly to a network, such as a smartphone, the information will be transmitted to the support server 1 via a network such as the internet. If the terminal does not have such a communication unit, the information acquired from the wireless / wired communication unit of the terminal using short-range wireless communication (ANT+, Bluetooth®, ZigBee, IrDA, etc.) will be transferred to a computer or smartphone, and then transmitted to the support server 1 via a network such as the internet for storage. The following are specific examples of information acquisition, but they are not exhaustive.

[0032] Information regarding still images of the face includes, but is not limited to, information regarding the facial contour (face line), the apparent age of the face, estimated skin age, morphological evaluation results for each part of the face, and skin tone.

[0033] Methods for obtaining the facial contour (face line) include, but are not limited to, feature point detection using CNNs, Active Shape Models (ASMs), and Optical Flow.

[0034] Methods for obtaining the apparent age of a face include, but are not limited to, CNNs, multitask learning models, regression models, SVMs, k-nearest neighbors, GANs, and appearance clustering analysis using algorithms that utilize facial feature points.

[0035] Methods for obtaining the estimated skin age include, but are not limited to, CNNs, multi-task learning models, regression models, SVMs, k-nearest neighbors, GANs, and appearance clustering analysis using algorithms that utilize facial feature points.

[0036] Methods for morphological evaluation of each facial part include, but are not limited to, detecting the positional relationships of representative feature points of each morphology from feature points detected using CNN, Active Shape Model (ASM), Optical Flow, etc., and performing morphological evaluation using statistical methods.

[0037] Methods for obtaining skin tone include, but are not limited to, methods that analyze color by color space conversion and perform clustering analysis using statistical methods, SVM, Bayesian classifiers, etc.

[0038] Situations in which information about a face in a video can be obtained include, but are not limited to, acquiring video information of a customer actually having a conversation. Furthermore, information about a face in a video can include, but are not limited to, information about changes in the position of the corners of the mouth, changes in the shape of the eyes, changes in the shape of the eyebrows, the angle of nodding, the orientation and angle of the face, and the direction and movement of the gaze.

[0039] The aforementioned methods for analyzing facial video include, but are not limited to, methods that acquire and analyze feature points using the following video analysis algorithms. Recurrent Neural Network (RNN) 3D Convolutional Neural Network: 3D-CNN Time-series convolutional network: TCN

[0040] Situations in which information about a still image of the entire body can be acquired include, but are not limited to, acquiring sitting or standing posture as image information and estimating the position of the tongue from the posture by analyzing the acquired image information.

[0041] Factors used to estimate the tongue position from the aforementioned seated image information include, but are not limited to, the position of the head relative to the spine, the angle of the head relative to the spine, the position of the chin, the position of the feet relative to the knees (whether in front of or behind the knees), and the tilt from the center of the body.

[0042] Devices that can capture full-body video include, but are not limited to, smartphones, tablets, cameras built into or externally connected to PCs, cameras installed on desks used for counseling, and cameras installed in stores, etc.

[0043] In addition to the information on still images and videos mentioned above, the system can also acquire, if necessary, basic customer information such as address and name, various environmental data related to the customer's living environment (temperature, humidity, changes in UV radiation, lighting environment, etc.: automatically acquired), information on the customer's daily skincare routine, and information other than skin appearance such as skin moisture content, but is not limited to these.

[0044] According to the beauty support system of this invention, the following further effects can be obtained.

[0045] When this beauty support system is installed in a beauty salon, it can immediately propose and implement treatments based on the diagnostic results. Furthermore, by acquiring data before and after treatment, customers can immediately experience the effects of the treatment they received. Since this beauty support system can also be used from the customer's terminal, they can easily find out whether the effects are continuing and when it is time to visit the salon again without having to go to the salon, which leads to continued use of the system.

[0046] When this beauty support system is applied to door-to-door sales, salespeople can visit customers' homes directly. This allows them to acquire customer data and immediately provide customers with detailed explanations of the results.

[0047] This beauty support system can also be used by customers in their homes. Customers can obtain diagnostic results by taking images and videos of themselves using their smartphones or other devices and sending them to the server. By suggesting necessary treatments based on the results, it can encourage customers to visit beauty salons or other stores.

[0048] To encourage the use of the beauty support system, video distribution can also be conducted. The system provider will distribute videos using a video distribution platform, etc. By distributing the system screen and demonstrating actual usage, it becomes possible to clearly explain how to use the system to customers. Furthermore, it becomes possible to introduce treatment methods based on diagnostic results and cosmetics suitable for the customer through videos.

[0049] This beauty support system can also be used without an internet connection. For example, by installing the system on the customer's own device, the customer can perform all tasks, from information input to information output, using only their own device. This allows customers to use the system regardless of location or time.

[0050] This beauty support system allows customers to communicate with stores such as beauty salons and cosmetics retailers. Customers input information using their own terminals and send that information to the stores. The stores perform calculations on computers installed in the stores and collect customer information. This enables stores to suggest treatments and products that are more suitable for each customer and to provide advice to customers remotely.

[0051] With this beauty support system, companies can also collect data from their customers. By having customers enter information and send it to the company, the company can collect data from a large number of customers. Using this data, companies can improve existing beauty support systems or develop new products and services.

[0052] The embodiments described above are examples of possible forms of the beauty support system and beauty support method according to the present invention, and are not intended to limit their forms. The beauty support system and beauty support method according to the present invention may take forms different from those illustrated in the embodiments. One example is a form in which a part of the configuration of the embodiment is replaced, modified, or omitted, or a form in which a new configuration is added to the embodiment. [Explanation of Symbols]

[0053] 1. Support Server 2. Information input terminal 3 Computing device 4. External computing units 5. Servers outside the system 6. Information output terminal 7. Information Input Section 8. Communication Control Unit 9 Processing Unit 10 Memory means 11. Information Output Unit

Claims

1. A beauty support system that evaluates customer impressions through image-based and video-based impression evaluation analysis.

2. The beauty support system according to claim 1, wherein the customer's impression is evaluated using at least the customer's own self-assessment and evaluations from others other than the customer.

3. The beauty support system according to claim 1, wherein the image and video used for impression evaluation are one or more parts selected from the customer's face, upper body, and entire body.

4. A beauty support method that evaluates a customer's impression using image-based impression evaluation analysis and video-based impression evaluation analysis, and provides treatment content based on at least the evaluation results of the impression evaluation device.

5. The beauty support method according to claim 4, wherein the customer's impression is evaluated using at least the customer's own self-assessment and evaluations from others other than the customer.

6. The beauty support method according to claim 4, wherein the subject of the impression evaluation is one or more parts selected from the customer's face, upper body, and whole body.

Images