Treatment support system, treatment support program, and treatment support method

Abstract

To provide a treatment support system, method, and program for accurately identifying the facial features of a treatment recipient. [Solution] A treatment support system that supports a practitioner in treatment, the treatment support system comprises an acquisition unit, an adjustment unit, and an identification unit. The acquisition unit acquires facial information including external standards that are set as absolute standards for determining the features of the subject's facial parts. The adjustment unit generates averaged facial image data by averaging all of the face's tilt angle (roll, pitch, and yaw), size, and brightness. The identification unit determines whether facial part feature information can be identified based on the facial information, and identifies the facial part feature information.

Description

[Technical Field]

[0001] The present invention relates to a treatment support system, a treatment support program, and a treatment support method that use image processing, measurement technology, augmented reality or mixed reality technology, deep learning technology, and the like, and in particular to a treatment support system, a treatment support program, and a treatment support method that realize accurate measurement, comparison, projection, evaluation, and recording of a customer's eyebrow design. [Background technology]

[0002] The impression of a person's appearance is greatly influenced by the shape of their eyebrows. Conventional eyebrow treatments have relied heavily on the practitioner's experience and intuition, and have had issues with objective evaluation of individual face shapes and eyebrow variations, as well as reproducibility after treatment. An example of a technology to support such treatments is proposed in Patent Document 1, for example.

[0003] For example, Patent Document 1 describes receiving at least one facial image of a subject, detecting the position of natural eyebrows in the at least one facial image, receiving a representation of an eyebrow imitation, detecting the positions of one or more alignment features in the at least one facial image, determining a position of the eyebrow imitation based on the positions of the one or more alignment features, determining maximum modification parameters based on one or more product characteristics, determining a feasible eyebrow by modifying one or more of the length, arch, and thickness of the natural eyebrows within the maximum modification parameters to approximate the eyebrow imitation, and generating a presentation of the feasible eyebrow. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2024-536708 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, although it is possible to identify the natural position of the eyebrows of a person undergoing treatment using a machine learning model, there is a problem in that there remains ambiguity in identifying facial features.

[0006] In view of the above problems, an object of the present invention is to provide a novel technique that can accurately identify the facial features of a person receiving treatment. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides a treatment support system that supports a practitioner in treatment, the treatment support system comprising an acquisition unit and an identification unit, the acquisition unit acquires facial information including external criteria that are established as absolute standards for determining the characteristics of the subject's facial areas, and the identification unit identifies facial area characteristic information based on the facial information.

[0008] In addition, in order to solve the above-mentioned problems, the present invention provides a treatment support program that supports a practitioner in treatment, wherein the treatment support program causes a computer to function as an acquisition unit and an identification unit, the acquisition unit acquires facial information including external criteria that are set as absolute standards for determining the characteristics of the subject's facial areas, and the identification unit identifies facial area characteristic information based on the facial information.

[0009] In addition, in order to solve the above-mentioned problems, the present invention provides a treatment support method for supporting a practitioner in treatment, in which a computer executes a process of acquiring facial information including external criteria that are set as absolute standards for determining the characteristics of the subject's facial parts, and a process of identifying facial part characteristic information based on the facial information.

[0010] With this configuration, it is possible to accurately identify the facial parts and facial features of any subject based on absolute standards.

[0011] In a more preferred embodiment, the treatment support system further includes an adjustment unit, wherein the acquisition unit acquires a plurality of pieces of face information of a certain subject, and the adjustment unit adjusts the subject's face based on the external criterion so that some or all of the tilt angle, size, and brightness included in the face part feature information identified for each of the plurality of pieces of face information are identical among the identified face part feature information.

[0012] With this configuration, it is possible to adjust for variations in facial features of the imaged subject, enabling accurate treatment.

[0013] In a more preferred embodiment, the external reference is a ruler made of a material having a higher infrared reflectivity than the subject's face and / or a black material that provides a high contrast between the subject's face and the external reference.

[0014] With this configuration, the contour of the external reference image is clear, and the face of the subject can be adjusted accurately.

[0015] In a more preferred embodiment, the treatment support system further includes an adjustment unit, wherein the acquisition unit acquires a plurality of pieces of face information, the adjustment unit generates averaged face information based on the plurality of pieces of face information, and the identification unit identifies the facial region feature information based on the averaged face information.

[0016] In a more preferred embodiment, the treatment support system further includes a shooting instruction generation unit that identifies values of shooting instruction items based on the external criteria and generates shooting instructions based on the values of the shooting instruction items.

[0017] By adopting such a configuration, imaging errors by the practitioner can be reduced to a minimum.

[0018] In a more preferred embodiment, the treatment support system further includes a calculation unit, wherein the identification unit identifies an eyebrow region as facial part characteristic information based on the face information, and the calculation unit compares the left and right eyebrow regions.

[0019] This configuration allows the left and right eyebrow regions of the subject to be compared, allowing the subject to easily understand the degree of difference between the left and right eyebrows of the subject.

[0020] In a more preferred embodiment, the calculation unit compares the left and right eyebrow regions using pupils of both eyes based on the face information.

[0021] With this configuration, symmetry can be easily evaluated based on general facial standards by using the center of the face (left and right pupils) as a reference.

[0022] In a more preferred embodiment, the treatment support system further includes a display processing unit and a calculation unit, wherein the display processing unit renders and displays template part data on facial parts of the subject based on the facial information, and the calculation unit calculates an eyebrow position score indicating the degree of overlap between the facial parts of the subject and the template part data.

[0023] With this configuration, the target design (a design template for the final facial area) can be displayed on the actual customer's face, and the difference between the final design and the current design can be visually displayed. This allows the user to visualize the final result before the treatment, which greatly contributes to user satisfaction and alleviation of anxiety. In addition, by scoring the degree of match with the design of the targeted facial area, it becomes an easy-to-understand indicator of before and after, making it easier for the practitioner to quantitatively understand the results.

[0024] In a more preferred embodiment, the treatment support system further includes a database and a display processing unit, wherein the database stores model master information, which is information related to an eyebrow model and includes attributes of both eyes of the model, the identification unit identifies the attributes of both eyes of the subject as the facial region feature information, and the display processing unit displays template eyebrow data based on the attributes of both eyes of the model and the identified attributes of both eyes of the subject.

[0025] With this configuration, the template eyebrow data can be adjusted and displayed based on the attributes of both eyes of the model and the attributes of both eyes of the subject.

[0026] In a more preferred embodiment, the treatment support system further includes a display processing unit, which uses the external reference as an augmented reality display reference or a mixed reality display reference to render and display the template part data on the corresponding facial part of the subject.

[0027] With this configuration, the external reference can be further used as an augmented reality display reference or a mixed reality display reference to render and display the template part data on the customer's facial part, thereby determining the characteristics of the facial part and stably displaying the template part data.

[0028] In a more preferred embodiment, the treatment support system further includes a display processing unit and a calculation unit, wherein the acquisition unit acquires facial video data including the external criteria as the facial information, the display processing unit renders template part data onto an image of the subject's facial parts based on the facial video data and performs display processing, and the calculation unit calculates a facial part position score based on the facial part feature information each time the facial part feature information is identified.

[0029] With this configuration, information on the subject's facial parts and template part data can be acquired in real time, and the facial part position score can be calculated in real time based on the acquired information. This allows the practitioner to constantly check the facial part position score while treating the subject.

[0030] In a more preferred embodiment, the treatment support system further includes a calculation unit, which calculates a treatment score based on a plurality of indices related to the positional relationship between facial region feature information of the subject based on the face information, the default template region data, and the corrected template region data.

[0031] With this configuration, a treatment score can be calculated using multiple template body part data and customer body part data, making it possible to fairly evaluate the treatment of the therapist. As the number increases, it becomes clear that the therapist has improved, which has a game-like element to it, and can lead to repeat visits and motivation.

[0032] In a more preferred embodiment, the treatment support system further includes a display processing unit and a registration unit, the database stores editable template part data, the registration unit registers corrected template part data that has been edited to the template part data in association with the subject, and the display processing unit displays the corrected template part data that corresponds to the subject to be treated.

[0033] This configuration makes it easier to compare with previous data, improving reproducibility for both the practitioner and the subject.

[0034] In a more preferred embodiment, the specification unit specifies, based on the face information, ten eyebrow feature points at predetermined positions on each of the left and right eyebrows as the facial region feature information.

[0035] With this configuration, various attributes related to eyebrows can be acquired.

[0036] In a more preferred embodiment, the identification unit identifies eyebrow feature points for determining eyebrow attributes of the subject as the facial region feature information based on a straight line connecting the corresponding positions of both eyes based on the face information and an eyebrow region identified based on the face information.

[0037] With this configuration, the eyebrow attributes of the subject can be determined using both of the subject's eyes.

[0038] In a more preferred embodiment, the treatment support system further includes a position identification unit, wherein the acquisition unit acquires the facial information obtained by capturing images of the subject's face from multiple different angles, the position identification unit estimates the camera posture using corresponding points between the multiple pieces of facial information without using the external reference, and estimates facial part position information of the subject, and the identification unit identifies the facial part feature information based on the estimated facial part position information.

[0039] With this configuration, facial region feature information of the subject can be accurately identified without using an external reference.

[0040] In a more preferred embodiment, the face information includes face image data and face depth information, and the position identification unit corrects scale-indefinite elements of the subject's facial shape based on the face depth information without using the external reference, and identifies the face part position information based on the face image data and the corrected scale-indefinite elements.

[0041] With this configuration, it is possible to correct the uncertainty of the scale caused by an error in the shooting distance without using an external reference, and to identify accurate facial part position information.

[0042] In a more preferred embodiment, the treatment support system further includes an adjustment unit, which corrects the facial information without using the external standard based on an internal standard included in each acquired facial information, which is a standard for identifying the facial area feature information.

[0043] With this configuration, it is possible to normalize face information without using an external standard. [Effects of the Invention]

[0044] The present invention has the effect of providing a novel technique that can accurately identify the facial features of a person receiving treatment. [Brief explanation of the drawings]

[0045] [Figure 1]FIG. 1 illustrates a configuration of an external reference in one embodiment. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a system according to an embodiment. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of a system according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of an apparatus according to an embodiment. [Figure 5] 1 is a process flowchart of a system according to an embodiment. [Figure 6] 10 is an example of an image of a customer undergoing treatment according to an embodiment. [Figure 7] 10 is an example of an image for identifying the interocular distance according to an embodiment. [Figure 8] 10 is a diagram illustrating an example of a procedure for determining eyebrow feature points (8 points) according to an embodiment. [Figure 9] 10 is a diagram illustrating an example of a procedure for determining eyebrow feature points (10 points) according to an embodiment. [Figure 10] 10 is an example of a customer image for explaining a method for calculating a treatment score in one embodiment. [Figure 11] 10 is an example of an image of a customer undergoing treatment according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0046] (Embodiment 1: Form using external standard) The present invention will now be described more fully with reference to the accompanying drawings, in which preferred embodiments are shown, but which may be embodied in many different forms and are not limited to the embodiments set forth herein.

[0047] For example, although the configuration, operation, etc. of the treatment support system are described in this embodiment, methods, devices, computer programs, etc. with similar configurations can also achieve similar effects. Furthermore, the program may be stored on a recording medium or provided as a downloadable program from an external server. Furthermore, by using this recording medium, for example, the program can be installed on a computer, thereby configuring a treatment support device or a treatment support system. Here, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

[0048] <1. Overview of the present invention> The present invention relates to a system for supporting a practitioner in performing cosmetic surgery on a subject's facial region. In this embodiment, facial information is acquired by capturing an image of the subject's face and a reference (hereinafter referred to as an external reference) that is absolutely attached to the subject's face in a detachable manner, and information regarding the features of the subject's facial region (facial region feature information) is identified based on the facial information.

[0049] In this embodiment, facial information is adjusted using an external criterion to enable accurate identification of facial region feature information. The identified facial region feature information is then displayed together with the subject's facial information, and a template of the subject's desired facial region (hereinafter referred to as template region data) is displayed superimposed on the subject's facial region based on predetermined criteria based on the subject's facial region. The subject's facial region identified based on the facial information is then compared with the template region data to calculate a match rate indicating the degree to which the subject's facial region and the template region data match. The practitioner then repeats this process to perform treatment on the subject.

[0050] Furthermore, in the present invention, the treatment performed by the practitioner is scored. Specifically, if the position of the subject's eyebrows deviates from the position of the template eyebrows desired by the subject, the treatment is deemed to be highly difficult, and if the treatment is successful, a high treatment score is assigned.

[0051] In the present invention, the facial treatment refers to eyebrow treatment, but the content of the treatment is not limited to this. For example, facial treatment may include cosmetic medical procedures such as eyelash, eyeliner, lip, facial contouring, and blemish removal, as well as medical examinations using facial scans, corrections, and cosmetic chiropractic treatments. In these treatments, acquiring facial information including external criteria makes it possible to identify accurate characteristic information for the treatment area, and superimposing and displaying template area data on the facial area makes it possible to perform pre-operative simulations and evaluate the effects of the treatment. Below, eyebrow treatment is described, and template eyebrow data is used as the template area data.

[0052] In this embodiment, an absolutely set reference (external reference) means a reference that is artificially attached regardless of the position of the subject's eyes, nose, etc., and the external reference is absolutely set on the subject's face for the following purposes. (1) To accurately identify facial features. (2) To accurately display the template eyebrows desired by the subject and the corrected template eyebrows (details will be described later) on the subject's face. (3) To achieve (1) and (2) above, the fluctuations in the facial feature points (numeric discrepancies) and the fluctuations in the difference between the facial feature points and the template eyebrows that occur when repeatedly photographing the subject's face are corrected (normalized). (4) To generate accurate imaging instructions for the practitioner in order to achieve (1) and (2) above. Specific processing using external criteria to achieve these goals will be described later.

[0053] FIG. 1 is a diagram illustrating an example of an external reference. The external reference WS is a reference board having a feature identification index used to identify facial features, a normalization index used to normalize facial information, and a photography instruction index used to generate photography instructions for the practitioner. The surface having these indices is the front surface, and a glossy material is applied to the front surface and an adhesive is applied to the back surface. In this embodiment, the reference board is a line-symmetrical plate with curved portions at the ends relative to the center, but the shape and size are not limited to this. For example, the shape may be rod-shaped, circular, square, or the like, and the total length may be 1 cm or less. The reference board has a planar structure, but may also have a 3D structure.

[0054] As the feature identification index, in order to identify facial features (interpupillary distance, skin color, eyebrow color, eyebrow area, etc.), external reference feature points (for example, a graphic such as a logo mark, a QR code (registered trademark), an AR marker, a multi-axis scale, one or more points on the contour of the external reference, etc., hereinafter referred to as external reference feature points) and the color of the external reference are used. The multi-axis scale has a main scale MS on the horizontal axis and an auxiliary scale AS on an axis inclined relative to the horizontal axis, with vertical lines provided at predetermined intervals. The color used is a color that makes the contour of the external reference clear (high contrast with skin color) when photographing the face, such as white or black. Note that the color of the external reference is not limited to white or black, as long as it is a color complementary to or close to the complementary color of the skin color.

[0055] The normalization index is an index used to normalize (correct) the value of a normalization item in the face of a subject included in face information, and external reference feature points, etc. are used as the normalization index. The normalization item is an item to be corrected in the face information, and the normalization items include the tilt angle (roll, pitch, and yaw) of the face, size, brightness, etc. Note that the items used as normalization items may be any items used to normalize variations in face information, and are not limited to these.

[0056] The photographing instruction index is an index that is referred to when generating photographing instructions to improve the accuracy of photographing facial information by the practitioner, and external reference feature points or the like are used as the photographing instruction index.

[0057] Note that the features used as the feature identification index, normalization index, and photography instruction index are not limited to these. Specific processing using these indexes will be described later. In this embodiment, the subject is a customer visiting a salon that performs eyebrow shaping, but the subject may also be a user who uses this system. The following explanation will be given for the case where the subject is a customer.

[0058] <2. Clarifying differentiation from prior art> The advantages of the present invention will be explained below by comparing the present invention with the prior art (for example, Patent Document 1).

[0059] <2.1. Overview of Prior Art> In conventional technology (e.g., Patent Document 1), facial feature information is obtained from a facial image using internal facial landmarks (e.g., pupils, nose, mouth, etc.). This results in errors and reduced reproducibility due to variations in the inclination of the face, distance, amount of light, etc., during photography, making it difficult to objectively evaluate the effectiveness of treatment.

[0060] <2.2. Issues and problems in the prior art> The conventional technology has the following problems. (1) The reproducibility of facial feature information decreases due to variations in photographing conditions (face tilt, photographing distance, lighting conditions). (2) Internal facial landmarks alone cannot adequately address individual subject characteristics, making accurate feature extraction difficult. (3) There are few quantitative methods for evaluating the effects of treatment, making it difficult to evaluate objective improvements after treatment.

[0061] <2.3. Configuration and Differentiation of the Present Invention> The present invention is clearly differentiated from the prior art in the following respects. (1) Use of external standards: While conventional methods rely on an internal reference, the present invention provides an "external reference" using a material with high infrared reflectivity for the subject's face or a black material that provides high contrast with the face. This provides an absolute positional reference and significantly reduces errors caused by variations in shooting conditions. (2) Averaging and normalization of multiple images: By acquiring multiple pieces of facial image data and averaging or median-combining them, accidental errors that occur in a single shot can be offset, and facial region feature information can always be identified under the same conditions. (3) Error correction using machine learning: To compensate for variations in shooting conditions, a normalized learning model is used to correct the tilt angle, size, brightness, etc. of the facial image data. This method makes it possible to obtain facial feature information with higher accuracy than conventional methods. (4) Objective evaluation by calculating treatment score: By calculating the degree of overlap between the facial features and the template features, a treatment score is introduced to quantitatively evaluate the effectiveness of the treatment, which clearly shows improvements and progress in the treatment as a numerical value.

[0062] <2.4. Effects and Advantages of the Present Invention> By adopting external standards, it is possible to obtain stable facial feature information that is not dependent on fluctuations in shooting conditions, compared to conventional methods that use only internal facial landmarks. Furthermore, by combining the averaging of multiple images with error correction using machine learning, the reproducibility of facial image data is significantly improved, enabling quantitative evaluation of treatment effects. These features allow practitioners to evaluate eyebrow designs objectively and with high reproducibility, leading to improved treatment accuracy and effectiveness.

[0063] The configurations of the present invention and the prior art are compared and summarized in the table below. [Table 1]

[0064] <2.5. Summary of differentiation from prior art> The present invention solves the problems of conventional techniques, such as variations in imaging conditions, low reproducibility, and insufficient objective evaluation methods, by using multiple methods, including the adoption of external standards, averaging of multiple images, error correction using machine learning, and calculation of treatment scores, thereby achieving unprecedented technical effects and advantages. The detailed configuration of the present invention that realizes such differentiation will be described below.

[0065] <3. System Configuration> Fig. 2 is a block diagram showing the configuration of a system according to one embodiment. As shown in Fig. 2, the treatment support system 0 includes a user terminal device 1 and an information management device 2. In this embodiment, the user terminal device 1 and the information management device 2 are configured to be able to communicate with each other via a communication network NW. The communication network NW in this embodiment is an IP (Internet Protocol) network, but there are no limitations on the type of communication protocol, and there are also no limitations on the type or scale of the network.

[0066] In this embodiment, a smartphone, a tablet terminal, a personal computer, or a wearable device (such as a head-mounted display) can be used as the user terminal device 1. A general-purpose server computer or a personal computer can be used as the information management device 2. The information management device 2 may also be configured with multiple computers that are capable of sending and receiving information via a communication network NW or another network.

[0067] <3.1. Hardware configuration of the present invention> 3 is a hardware configuration diagram of the treatment support system 0. As shown in FIG. 3(a), the server 10 (information management device 2) includes a processing unit 101, a storage unit 102, and a communication unit 103.

[0068] The processing unit 101 has one or more processors such as a CPU that can execute an instruction set, and controls the overall operation and processing of the information management device 2 by executing an OS and other applications. The storage unit 102 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS. The communication unit 103 has a communication interface device with the communication network NW, and controls communication with the communication network NW to input and output information.

[0069] As shown in FIG. 3(b), the terminal device 9 (user terminal device 1) includes a processing unit 91, a storage unit 92, a communication unit 93, an input unit 94, and an output unit 95.

[0070] The processing unit 91 has one or more processors such as a CPU capable of executing an instruction set, and controls the overall operation and processing of the terminal device 9 by executing the treatment assistance program according to the present invention, an OS, and other applications. The storage unit 92 has a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS, a treatment assistance program according to the present invention, and the like. The communication unit 93 has a communication interface device for connecting to a network, and controls communication with the communication network NW to input and output information. The input unit 94 has an input device capable of input processing, such as a keyboard or a touch panel, and an imaging device capable of acquiring customer facial information, the distance between the customer and the terminal device, etc. Here, the imaging device is configured to be able to detect not only visible light but also various electromagnetic waves (infrared rays, ultraviolet rays, etc.) (for example, an infrared camera, a depth sensor, LiDAR (Light Detection and Ranging), etc.). The imaging device may be a device external to the terminal device 9, such as a camera or an unmanned aerial vehicle (drone, etc.), and the captured image data may be stored in the memory unit 92 wirelessly, wired, or via a recording medium. The output unit 95 has a display device capable of display processing such as a display. The output unit 95 may be a display device external to the user terminal device 1, or information about the treatment may be displayed on a display device of a treatment advisor or the like in a remote location.

[0071] <3.2. System Functional Configuration> FIG. 4 is a functional block diagram of the treatment support system 0 in this embodiment. As shown in FIG. 4, the user terminal device 1 includes an acquisition unit 11, an adjustment unit 12, a shooting instruction generation unit 13, a template generation unit 14, an identification unit 15, a calculation unit 16, a display processing unit 17, a registration unit 18, and a learning unit 19. These functional configurations are realized by hardware (processing unit 91) processing information by software (stored in a storage unit 92). The information management device 2 includes a data storage unit 21, a distribution unit 22, and a database 3. This is realized by hardware (processing unit 101) processing information by software (stored in a storage unit 102).

[0072] In this embodiment, the treatment support system 0 is used via a user terminal device 1 that executes a treatment support program, but some or all of the functional components (parts) provided in the user terminal device 1 may be provided in an information management device 2, which is a server.

[0073] <3.2.1. Database 3> The database 3 of the information management device 2 in FIG. 4 stores model master information and customer treatment history information.

[0074] <3.2.1.1. Model Master Information> The model master information is master information about ideal eyebrow models, and includes a model ID that uniquely identifies the model, a model category, an eyebrow contour category, eyebrow attributes, eyebrow attributes, a model image, and template eyebrow data.

[0075] The model category is data classified based on the impression of the face, and is set, for example, as a good-looking actor in his twenties, a refreshing face, etc. The eyebrow shape category is a category related to the characteristics of the eyebrow shape, and for example, straight eyebrows, soft art eyebrows, etc. are set. The eyebrow contour category is a category related to the contour of the eyebrows, and may be set to, for example, blurred, clear, etc. Eyebrow attributes are data related to the shape and arrangement of the elements of eyebrow design, and include eyebrow thickness, eyebrow angle, eyebrow length, eyebrow darkness, eyebrow color, distance between eyebrows, distance between eyes and eyebrows, and eyebrow shape category. The attributes of both eyes are data relating to the position of both eyes, and include the positions of the pupil centers of both eyes, the distance between the pupil centers of both eyes, and feature points of both eyes.

[0076] <3.2.1.2. Customer Treatment History Information> The registration unit 18 registers customer treatment history information, which is information related to the history of treatments performed on customers. The customer treatment history information includes a customer treatment history ID for uniquely identifying the customer's treatment history, a customer ID for uniquely identifying the customer, customer template eyebrow data, customer facial image data, eyebrow attributes, eyebrow position score and treatment score (described below), and registration date. The customer template eyebrow data is template eyebrow data for each customer obtained by editing the template eyebrow data of the model master information.

[0077] <3.2.2. Acquisition part 11> The acquisition unit 11 acquires facial information including a customer's face and external criteria. The acquisition unit 11 acquires facial image data or facial video data including a customer's face and external criteria as the facial information.

[0078] <3.2.3. Adjustment section 12> The adjustment unit 12 adjusts the facial information based on the facial information and normalization means. The adjustment unit 12 uses a normalization learning model as normalization means, and adjusts the values of normalization items of the facial image data by inputting facial image data based on the acquired facial information into the normalization learning model.

[0079] Furthermore, the adjustment unit 12 generates averaged face information by averaging the face information based on the acquired plurality of pieces of face information. The adjustment unit 12 generates averaged face image data based on the plurality of pieces of face image data based on the acquired face information.

[0080] <3.2.4. Shooting instruction generation unit 13> The photographing instruction generation unit 13 generates photographing instructions for photographing the facial information provided to the practitioner. The photographing instruction generation unit 13 generates photographing instructions based on a photographing instruction index included in the facial image data adjusted by the adjustment unit 12.

[0081] <3.2.5. Template Generation Unit 14> The template generating unit 14 generates template eyebrow data for each model category based on a plurality of face image data, and registers the template eyebrow data in the database 3.

[0082] Specifically, the template generation unit 14 receives, for each model category, a plurality of pieces of facial image data that match the model category and the number of votes for each piece of facial image data, and generates template eyebrow data for the model category by combining the plurality of pieces of facial image data that match the model category based on a combination of the plurality of pieces of facial image data and rankings based on the number of votes for each piece of facial image data. More specifically, the template generation unit 14 generates template eyebrow data for each model category by weighting the plurality of pieces of facial image data according to the ranking corresponding to each piece of facial image data and combining them based on a combination of the plurality of pieces of facial image data and rankings for each model category based on the number of votes for each piece of facial image data. In other words, the template generation unit 14 receives, for example, designation of facial image data that is thought to match the model category, and tallying the facial image data that match the model category and the number of votes for each piece of facial image data, averaging the facial image data for each model category, and generating template eyebrow data.

[0083] <3.2.6. Specific part 15> The identification unit 15 identifies facial part feature information indicating features of facial parts of the customer based on the customer's facial information. In this embodiment, the identification unit 15 includes an area identification unit 151 and an eyebrow attribute identification unit 152.

[0084] <3.2.6.1. Area identification part 151> The region specifying unit 151 specifies an eyebrow region as facial part characteristic information. The region specifying unit 151 uses the customer's face image data as face information to specify the shading of the customer's eyebrows, and specifies the eyebrow region based on the shading of the customer's eyebrows.

[0085] Furthermore, the region specifying unit 151 specifies eyebrow feature points, which are feature points of the customer's eyebrows, as facial part feature information. The region specifying unit 151 specifies the eyebrow feature points based on a straight line connecting the corresponding positions of both eyes based on the customer's facial information and the specified eyebrow region.

[0086] <3.2.6.2. Eyebrow attribute identification section 152> The eyebrow attribute specifying unit 152 specifies eyebrow attributes as facial part characteristic information. The eyebrow attribute specifying unit 152 specifies the eyebrow attributes of the customer based on the specified eyebrow area.

[0087] <3.2.7. Calculation Unit 16> The calculation unit 16 calculates a score related to the eyebrow treatment for the customer. In this embodiment, the calculation unit 16 has a left-right difference calculation unit 161 and a match rate calculation unit 162 that calculate an eyebrow position score (facial part position score) that is a score related to the positions of both eyebrows as the score related to the eyebrow treatment. The calculation unit 16 also has a treatment score calculation unit 163 that calculates a treatment score that is a score related to the treatment level of the practitioner as the score related to the eyebrow treatment.

[0088] <3.2.7.1. Lateral difference calculation unit 161> The left-right difference calculation unit 161 calculates the left-right difference between the eyebrows as the eyebrow position score. The left-right difference calculation unit 161 calculates the left-right difference between the eyebrows of the customer based on the identified eyebrow regions of both eyebrows.

[0089] <3.2.7.2. Match Rate Calculation Unit 162> The match rate calculation unit 162 calculates, as the eyebrow position score, a match rate indicating the degree of match between the customer's facial features and the template feature data corresponding to the facial features. The match rate calculation unit 162 accepts designation of template eyebrow data desired by the customer as template feature data, and calculates the match rate based on the area of the template eyebrow data and the customer's eyebrow area identified as a facial feature.

[0090] <3.2.7.3. Treatment score calculation unit 163> The treatment score calculation unit 163 calculates a treatment score based on multiple template eyebrow data used for a certain customer in a treatment for the customer and facial part feature information of the customer. The treatment score calculation unit 163 uses default template eyebrow data (template eyebrow data in model master information or customer template eyebrow data in customer treatment history information) and corrected template eyebrow data obtained by correcting the default template eyebrow data as the multiple template eyebrow data used for the customer, and also uses customer eyebrows as facial part feature information, and calculates a treatment score based on multiple indices related to the positional relationship of the default template eyebrow data, the corrected template eyebrow data, and the customer eyebrows.

[0091] <3.2.8. Display processing unit 17> The display processing unit 17 acquires a facial image based on the customer's facial information and displays the customer's eyebrow region. The display processing unit 17 also displays template eyebrow data. The display processing unit 17 also displays the identified eyebrow attributes and the calculated scores. The display processing unit 17 then transfers the display processing results to the output unit 95 of the user terminal device 1, and the output unit 95 outputs the display processing results.

[0092] <3.2.9. Learning Section 19> The learning unit 19 learns a normalization learning model for correcting facial image data. In this embodiment, the learning unit 19 uses, as training data, a plurality of sets of facial image data for the same customer, each set having different imaging conditions for normalization indices, and a combination of eyebrow position scores calculated based on the facial image data, to learn a normalization learning model that estimates values of normalization items for facial image data that optimize the eyebrow position score.

[0093] Specifically, the learning unit 19 acquires the values of the normalization items (face tilt angle, size, and luminance) based on the external reference feature points that are normalization indices, and further uses the match rate and eyebrow difference as eyebrow position scores to train a normalization learning model using the values of the normalization items, as well as the match rate and eyebrow difference as training data. More specifically, the learning unit 19 acquires the face tilt angle, size, and luminance, which are the values of the normalization items, based on the appearance of shapes, scales, etc. that are external reference feature points due to the degree of distortion, size, and brightness, and trains a normalization learning model that estimates the values of the normalization items that optimize the match rate and eyebrow difference using the values of the normalization items, as well as the match rate and eyebrow difference as training data. That is, by learning the correlation between the degree of distortion, size, and brightness of external reference feature points and the match rate and the difference between the left and right eyebrows, the values of normalization items that calculate an accurate eyebrow position score and the error in the eyebrow position score for each value of normalization items are learned, and a learning model is trained that estimates the values of normalization items that calculate an accurate eyebrow position score.

[0094] There is no restriction on the type of algorithm for the normalized learning model, and it may be, for example, a machine learning model (not limited to neural networks such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), but also including SVM (Support Vector Machine)), template matching, statistical processing, rule-based, etc., or a combination of some or all of these.

[0095] <3.2.10. Data storage unit 21> The data storage unit 21 acquires the template eyebrow data used for the customer from the user terminal device 1, and stores the template eyebrow data in the database 3 as customer template eyebrow data.

[0096] <3.2.11.Distribution Section 22> The distribution unit 22 distributes the template eyebrow data stored in the database 3 to the display processing unit 17 of the user terminal device 1. Note that a data storage unit (not shown) of the user terminal device 1 may store the distributed template eyebrow data in the memory unit 92. In the following description, the involvement of the data storage unit 21 and distribution unit 22 will be omitted.

[0097] <4. Processing flow> Next, a treatment support method using the treatment support system 0 will be described with reference to Figures 5 to 11. Figure 5 is a flowchart showing the process of a treatment for a customer, in which the user terminal device 1 acquires facial information of the customer and displays template eyebrow data and various scores based on the facial information.

[0098] <4.1. Acquiring facial information> In step S1 (hereinafter, "step SX" will be simply referred to as "SX"), the acquisition unit 11 acquires facial information of the client who will receive treatment. In this embodiment, the acquisition unit 11 acquires, as the client's facial information, a plurality of facial image data sets including the client's face and external references via the imaging device of the user terminal device 1. More preferably, the acquisition unit 11 acquires, as the client's facial information, facial image data in which all of the external shapes of the external references set on the client's face are captured. This improves the reproducibility of data when re-capturing. In this embodiment, the external references only need to be included in the facial image data, and may be placed on a desk. Furthermore, the facial information is not limited to image data, and may also be spectral data, point cloud data, mesh data, etc. acquired by a multi-wavelength sensor or a 3D scanning device.

[0099] <4.2. Adjusting facial image data> In S2, the adjustment unit 12 adjusts the face image data. In this embodiment, the adjustment unit 12 generates averaged face image data by averaging some or all of the face tilt, size, brightness, and contrast based on the multiple face image data acquired in S1. Specifically, the adjustment unit 12 generates averaged face image data by averaging all of the face tilt angle (roll, pitch, and yaw), size, and brightness. This allows errors in a single cut to be averaged out.

[0100] The adjustment unit 12 also inputs the averaged face image data into a normalization learning model to adjust (normalize) some or all of the tilt angle (roll, pitch, and yaw), size, and brightness of the customer's face contained in the averaged face image data.

[0101] Specifically, the adjustment unit 12 inputs the averaged face image data into a normalization learning model to identify differences between the face tilt angle, size, and luminance values that optimize the eyebrow position score and the face tilt angle, size, and luminance values that are normalization items in the averaged face image data, and adjusts the face tilt angle, size, and luminance of the averaged face image data to eliminate the differences. For example, the face tilt angle, size, and luminance that optimize the eyebrow position score may be a face tilt angle that makes a line connecting the centers of the pupils of both eyes included in the face image data horizontal, a face size that makes the size of an external reference included in the face image data a predetermined size, and a face luminance that makes the luminance of an external reference included in the face image data a predetermined luminance. This allows the normalization learning model to output the difference between the face tilt angle, size, and brightness values that optimize the eyebrow position score and the face tilt angle, size, and brightness values that are normalization items in the averaged face image data, thereby adjusting the face tilt angle, size, and brightness of the averaged face image data so that they become the face tilt angle, size, and brightness that optimize the eyebrow position score.

[0102] More specifically, the adjustment unit 12 identifies the coordinate positions of the external references included in the facial image data captured during the treatment based on the external reference feature points, and adjusts the tilt angle, size, and brightness of the face in the facial image data by inputting the facial image data, in which the coordinate positions of the external references included in the facial image data captured during each shooting, into the normalized learning model. More preferably, the adjustment unit 12 identifies the three-dimensional coordinate positions of the external references based on the acquired facial image data of the multiple facial images. This makes it possible to reduce imaging errors caused by differences in the coordinate positions of the external reference, and to obtain more accurate face image data.

[0103] In this embodiment, the adjustment unit 12 normalizes the facial image data using a normalization adjustment model as normalization means. Alternatively, the adjustment unit 12 may use an external reference image and a correspondence table including combinations of the external reference tilt angle, size, and brightness as normalization means to identify a combination of the tilt angle, size, and brightness of the facial image data from the external reference image data, and adjust the tilt angle, size, and brightness of the customer's face included in the facial image data.

[0104] Furthermore, the adjustment unit 12 identifies the parameters of the imaging device based on the face image data including the external reference, and adjusts the averaged face image data. Specifically, the adjustment unit 12 identifies internal parameters (focal length, principal point, and lens distortion) and external parameters (position and orientation of the camera) as the parameters of the imaging device based on the eyebrow feature points, eye feature points, external reference feature points, and the distance between the external reference and the user terminal device 1, which are included in the face image data, adjusts the lens distortion parameters, and adjusts the averaged face image data based on the lens distortion parameters. This makes it easier to correct errors in short-distance photography. Also, in the beauty industry, where repeat treatments are common, it is significantly effective to perform calibration using the previous photograph data of the same customer (customer treatment history information).

[0105] By executing S2 as described above, the eye position is placed at roughly the same coordinates each time using an external reference, reducing geometric variations in facial angle, distance, etc. Then, the facial feature information from S3 onward can be identified under the same conditions each time.

[0106] <4.3. Determining specific facial feature information> In S3, the identification unit 15 determines whether the facial region feature information is identifiable. If it is determined that the facial region feature information is identifiable (YES in S3), the process proceeds to S5. On the other hand, if it is not determined that the facial region feature information is identifiable (NO in S3), the process proceeds to S4.

[0107] <4.4. Generation of shooting instructions> In S4, the photographing instruction generation unit 13 generates photographing instructions for identifying facial region feature information. In this embodiment, the photographing instruction generation unit 13 identifies the value of a photographing instruction item based on the photographing instruction index included in the face image data adjusted in S2, and generates photographing instructions based on the value of the photographing instruction item.

[0108] Specifically, the shooting instruction generation unit 13 determines whether to generate a shooting instruction based on the value of the shooting instruction item and a predetermined allowable range for each shooting instruction item. The shooting instruction generation unit 13 determines to generate a shooting instruction when the value of the shooting instruction item exceeds the allowable range, and generates the shooting instruction. For example, the shooting instruction generation unit 13 identifies the size of the shooting instruction indicator based on the normalized face image data, and based on the size, identifies the distance between the imaging device of the user terminal device 1 and an external reference as the value of the shooting instruction item, and generates a shooting instruction (e.g., "move closer / farther away from the camera") that keeps the distance within the allowable range. The shooting instruction generation unit 13 also identifies the tilt angle (roll, pitch, and yaw) of the shooting instruction indicator based on the normalized face image data, and based on the tilt angle, identifies the tilt angle of the face as the value of the shooting instruction item, and generates a shooting instruction (e.g., "turn your face a little further forward") that keeps the tilt angle within the allowable range. In addition, the shooting instruction generation unit 13 identifies the brightness of the shooting instruction indicator based on the face image data, identifies the brightness of the face as the value of the shooting instruction item based on the brightness, and generates shooting instructions (for example, "There is not enough light, please move to a brighter location") that keep the brightness of the face within an acceptable range.

[0109] Furthermore, the photographing instruction generation unit 13 determines whether or not all of the external shapes of the external references are included in the face image data based on the external reference feature points, and generates a photographing instruction when all of the external shapes of the external references are not included in the face image data. For example, when all of the external shapes of the external references are not included in the face image data, the photographing instruction generation unit 13 generates a photographing instruction such as "Please take a photograph again so that all of the external references are included." This clearly reduces errors and improves the quality of the imaging results when guidance such as "move closer" or "move a little further to the left" is given. It also makes it easier for the practitioner to be aware of the situation, allowing for smoother retakes.

[0110] The above steps S1 to S4 are executed, and if it is determined that the facial region characteristic information can be specified, the subsequent steps are executed.

[0111] <4.5. Identifying facial feature information> In S5, the identification unit 15 identifies facial region feature information. In this embodiment, the identification unit 15 inputs the customer's facial image data adjusted in S2 or S4 into a well-known image recognition model to identify the customer's eyebrow attributes, skin color, eyebrow color, eyebrow area, eyebrow feature points, and eye attributes as the facial region feature information.

[0112] <4.5.1. Identifying the eyebrow area> The region identification unit 151 identifies the customer's skin color and eyebrow color based on the face image data adjusted in S2 or the face image data acquired in response to the shooting instruction generated in S4, identifies the customer's eyebrow shading based on the skin color and eyebrow color, and identifies the eyebrow region based on the customer's eyebrow shading. Specifically, the region identification unit 151 identifies the absolute values of the customer's face color and eyebrow color included in the face image data whose brightness has been adjusted in S2 or S4. Then, the region identification unit 151 identifies the region where the ratio of the absolute value of the eyebrow color to the face color exceeds a predetermined threshold (e.g., 80%) as the eyebrow region. More specifically, the region identification unit 151 uses the face color of the customer midway between both eyes as the face color included in the face image data. With this configuration, even if the color of the customer's face photographed during treatment cannot be accurately measured due to light reflection, the brightness of the face image data is adjusted using an external standard, making it possible to accurately identify the customer's face color and the eyebrow area.

[0113] Fig. 6 shows an example of a screen displaying the specified eyebrow area on the user terminal device 1. Fig. 6(a) shows facial image data obtained by capturing an external reference WS and a customer's face. Fig. 6(b) shows the specified eyebrow area BA based on the shading of the customer's eyebrows, and the specified eyebrow area BA is displayed superimposed on the customer's eyebrows CB.

[0114] 7 is an example of a diagram showing a method for specifying the pupil center distance as an eye attribute and the face center line CL as a facial feature using the scale of the external reference WS. In the illustrated example, the pupil center distance is specified by measuring the distance between the pupil perpendicular lines PL of each eye using the scale of the external reference WS, for a line PL (hereinafter referred to as the pupil perpendicular line) that passes through the pupil center and is perpendicular to a straight line connecting the pupil centers of both eyes based on the customer's facial image data.

[0115] If the shape of the external reference WS is circular or the like and does not include a scale as a feature identification indicator, the identification unit 15 uses a well-known image recognition technique to compare the size of the external reference in the face image data with the size of the interpupillary distance to identify the actual interpupillary distance of the customer. Even if the shape of the external reference WS includes a scale as a feature identification indicator, the identification unit 15 may identify the actual interpupillary distance of the customer by comparing the size of the external reference in the face image data with the size of the interpupillary distance.

[0116] <4.5.2. Identifying eyebrow feature points> 5, the region identification unit 151 identifies eyebrow feature points in the identified eyebrow region. The region identification unit 151 identifies the eyebrow feature points based on a straight line connecting the centers of the pupils of both eyes (hereinafter referred to as a reference line) based on the customer's face image data acquired in S2 or S4, and the eyebrow region.

[0117] FIG. 8 is a diagram showing an example of a procedure for determining eight eyebrow feature points. 8(a) shows an example of a procedure for determining eyebrow feature point BL1. BL1 is set as the intersection point of the inner corner of the upper left eyebrow between the perpendicular line of the reference line and the eyebrow area BA. 8(b) shows an example of a procedure for determining eyebrow feature point BL4. BL4 is set as the intersection point at the right end (end of the eyebrow) of the intersection points between the perpendicular to the reference line and the eyebrow area BA. 8(c) shows an example of a procedure for determining eyebrow feature point BL7. BL7 is set as the intersection point of the inner corner of the lower left eyebrow among the intersection points of the perpendicular line of the reference line and the eyebrow area BA. 8(d) shows an example of a procedure for determining the eyebrow feature point BL3. BL3 is set as the point of contact between a line parallel to the reference line and the curved upper end of the eyebrow area BA. 8(e) shows an example of a procedure for determining the eyebrow feature point BL2. BL2 is set as the point of contact between a straight parallel line passing through BL1 and BL3 and the curved upper end of the eyebrow area BA. 8(f) shows an example of a procedure for determining the eyebrow feature point BL5, which is set as the point of contact between a line parallel to the reference line and the curved bottom end of the eyebrow area BA. 8(g) shows an example of a procedure for determining the eyebrow feature point BL6. BL6 is set as the point of contact between a straight parallel line passing through BL5 and BL7 and the curved bottom end of the eyebrow area BA. 8(h) shows an example of a procedure for determining the eyebrow feature point BL8. BL8 is set as the lowermost intersection of the pupil perpendicular and the eyebrow area BA.

[0118] 8, eight eyebrow feature points are determined, but there is no limit to the number of eyebrow feature points, and preferably, eyebrow feature points are set according to the gender of the model. For example, eight eyebrow feature points are set for a male model, and ten eyebrow feature points are set for a female model.

[0119] FIG. 9 is a diagram showing an example of a procedure for determining 10 eyebrow feature points. 9(a) shows an example of a procedure for determining eyebrow feature point BL1. BL1 is set as the intersection point of the inner corner of the upper left eyebrow between the perpendicular line of the reference line and the eyebrow area BA. 9(b) shows an example of a procedure for determining the eyebrow feature point BL9, which is set as the point of contact between a line parallel to the reference line and the inner corner of the lower left eyebrow in the eyebrow area BA. 9(c) shows an example of a procedure for determining eyebrow feature point BL5. BL4 is set as the intersection point at the right end (end of the eyebrow) of the intersection points between the perpendicular to the reference line and the eyebrow area BA. 9(d) shows an example of a procedure for determining the eyebrow feature point BL3. BL3 is set as the point of contact between a line parallel to the reference line and the curved upper end of the eyebrow area BA. 9(e) shows an example of a procedure for determining the eyebrow feature point BL2. BL2 is set as the point of contact between a straight parallel line passing through BL1 and BL3 and the curved top end of the eyebrow area BA. 9(f) shows an example of a procedure for determining eyebrow feature point BL4. BL4 is set as the point of contact between a straight parallel line passing through BL3 and BL5 and the curved upper end of the eyebrow area BA. 9(g) shows an example of a procedure for determining the eyebrow feature point BL7. BL7 is set as the point of contact between a line parallel to the reference line and the curved bottom end of the eyebrow area BA. 9(h) shows an example of a procedure for determining eyebrow feature point BL8. BL8 is set as the point of contact between a straight parallel line passing through BL7 and BL9 and the curved bottom end of the eyebrow area BA. 9(i) shows an example of a procedure for determining the eyebrow feature point BL6. BL6 is set as the point of contact between a straight parallel line passing through BL5 and BL7 and the curved bottom end of the eyebrow area BA. 9(j) shows an example of a procedure for determining the eyebrow feature point BL10. BL10 is set as the lowermost intersection of the pupil perpendicular and the eyebrow area BA.

[0120] In a preferred embodiment of the present invention, eyebrow feature points can be identified with greater accuracy based on the scale of the external reference. Specifically, when facial image data is input to an image recognition model, the pupil center may not be accurately recognized. In this case, the region identification unit 151 identifies each eyebrow feature point by using the horizontal axis of the external reference as a reference line and a parallel line of the vertical scale as a perpendicular line to the reference line. Furthermore, the region identification unit 151 compares the eyebrow feature points identified using the perpendicular line to the reference line with eyebrow feature points identified using the horizontal axis of the external reference as a reference line and a parallel line of the vertical scale as a perpendicular line to the reference line, and if there is an error in position, the region identification unit 151 may identify the average point of each eyebrow feature point as the eyebrow feature point.

[0121] Although an example of the left eye is shown in FIGS. 8 and 9, the same setting is made for the right eye.

[0122] <4.5.3. Identifying eyebrow attributes> 5, the eyebrow attribute identification unit 152 identifies eyebrow attributes based on the face image data. In this embodiment, the eyebrow attribute identification unit 152 identifies eyebrow thickness, eyebrow angle, eyebrow length, eyebrow darkness, distance between eyebrows, distance between eyes and eyebrows, and eyebrow shape category as eyebrow attributes based on the identified multiple eyebrow feature points.

[0123] Specifically, the eyebrow attribute identification unit 152 identifies the thickness of the eyebrows as the average of the shortest distances between a line passing through two feature points in the upper part of the eyebrow region and a line passing through two feature points in the lower part of the eyebrow region. More specifically, in the case of eight points, the eyebrow attribute identification unit 152 identifies the thickness of the eyebrows using BL1 and BL3 as the two feature points in the upper part of the eyebrow region and BL5 and BL7 as the two feature points in the lower part of the eyebrow region. Furthermore, in the case of ten points, the eyebrow attribute identification unit 152 identifies the thickness of the eyebrows using BL1 and BL3 as the two feature points in the upper part of the eyebrow region and BL7 and BL9 as the two feature points in the lower part of the eyebrow region.

[0124] The eyebrow attribute identification unit 152 also identifies the eyebrow angle as the average value of the angle formed by a line passing through two feature points in the upper part of the eyebrow region and a line parallel to the reference line that passes through the lowermost of the two feature points, and a line passing through two feature points in the lower part of the eyebrow region and a line parallel to the reference line that passes through the lowermost of the two feature points. More specifically, in the case of eight points, the eyebrow attribute identification unit 152 identifies the eyebrow angle using BL1 and BL3 as the two feature points in the upper part of the eyebrow region and BL5 and BL7 as the two feature points in the lower part of the eyebrow region. In the case of ten points, the eyebrow attribute identification unit 152 identifies the eyebrow angle using BL1 and BL3 as the two feature points in the upper part of the eyebrow region and BL7 and BL9 as the two feature points in the lower part of the eyebrow region.

[0125] Furthermore, the eyebrow attribute identification unit 152 identifies the length of the eyebrows as the length of the lines connecting adjacent feature points that form the contour of the upper brow region. More specifically, when there are eight points, the eyebrow attribute identification unit 152 identifies the length of the eyebrows using BL1, BL2, BL3, and BL4 as the feature points that form the contour of the upper brow region. When there are ten points, the eyebrow attribute identification unit 152 identifies the length of the eyebrows using BL1, BL2, BL3, BL4, and BL5 as the feature points that form the contour of the upper brow region.

[0126] The eyebrow attribute specifying unit 152 also specifies the darkness of the eyebrows based on the ratio of the absolute value of the eyebrow color to the skin color used to specify the eyebrow region.

[0127] Furthermore, the eyebrow attribute specifying unit 152 specifies the distance between the eyebrows based on the distance between the tips of the two eyebrow regions. More specifically, the eyebrow attribute specifying unit 152 specifies the distance between the eyebrows based on the distance between the BL1 of the two eyebrow regions.

[0128] Furthermore, the eyebrow attribute specification unit 152 specifies the distance between the intersection of the pupil perpendicular line and the outline of the eye that is closer to the eyebrow, and the intersection of the pupil perpendicular line and the outline of the eyebrow region that is closer to the eye (BL8 if there are 8 points, BL10 if there are 10 points) as the distance between the eye and the eyebrow. Note that the distance between the eye and the eyebrow is not limited to this, and may be, for example, the distance between the center of the pupil and the intersection of the pupil perpendicular line and the outline of the eyebrow region that is closer to the eye.

[0129] The eyebrow attribute identification unit 152 also identifies the eyebrow shape category based on the angles of the middle feature point (vertex) among the interior angles of a triangle obtained by connecting three feature points in the upper part of the eyebrow region and the middle feature point (vertex) among the interior angles of a triangle obtained by connecting three feature points in the lower part of the eyebrow region. More specifically, in the case of eight points, the eyebrow attribute identification unit 152 identifies the eyebrow shape category using a combination of BL1, BL2, and BL3 as the three feature points in the upper part of the eyebrow region and a combination of BL5, BL6, and BL7 as the three feature points in the lower part of the eyebrow region. In the case of eight points, the eyebrow attribute identification unit 152 also identifies the eyebrow shape category using a combination of BL1, BL3, and BL4 as the three feature points in the upper part of the eyebrow region and a combination of BL4, BL5, and BL7 as the three feature points in the lower part of the eyebrow region. In addition, in the case of 10 points, the eyebrow attribute identification unit 152 identifies the eyebrow shape category using a combination of BL1, BL2, and BL3 as three feature points in the upper part of the eyebrow region and a combination of BL7, BL8, and BL9 as three feature points in the lower part of the eyebrow region. In addition, in the case of 10 points, the eyebrow attribute identification unit 152 identifies the eyebrow shape category using a combination of BL1, BL3, and BL5 as three feature points in the upper part of the eyebrow region and a combination of BL5, BL7, and BL9 as three feature points in the lower part of the eyebrow region.

[0130] <4.6. Display processing of template eyebrow data> In S6 of Fig. 5, the display processing unit 17 processes the display of template eyebrow data. In this embodiment, the display processing unit 17 displays multiple template eyebrow data based on model master information and customer template eyebrow history information, and accepts designation of template eyebrow data desired by the customer from among the multiple template eyebrow data. The template eyebrow data designated by the customer is configured so that its position and shape can be edited. The display processing unit 17 then displays the template eyebrow data by superimposing it on the customer's eyebrows in the facial image data acquired in S2 or S4. More preferably, the display processing unit 17 accepts input of information identifying the customer (such as the customer's name) and displays the customer template eyebrow data associated with the customer by superimposing it on the customer's eyebrows in the facial image data.

[0131] Specifically, the display processing unit 17 displays the template eyebrow data based on the eye attributes of the customer and the model identified in S5. More specifically, the identification unit 15 identifies the pupil center of the customer as an eye attribute based on the facial image data acquired in S2 or S4. Then, the display processing unit 17 superimposes the template eyebrow data on the customer's eyebrows and displays the data so that the pupil center of the customer and the pupil center of the model based on the model master information coincide with each other.

[0132] More specifically, the specifying unit 15 specifies the distance between the centers of the pupils of both eyes as an attribute of the customer's eyes based on the scale of the external reference included in the facial image data. Then, the display processing unit 17 adjusts the size of the template eyebrow data based on the ratio between the distance between the centers of the pupils of both eyes of the customer and the model based on the model master information, and displays the template eyebrow data.

[0133] When template eyebrow data based on customer template eyebrow history information is specified as the template eyebrow data, the identification unit 15 identifies the pupil center based on the customer's facial image data in the customer template eyebrow history information. The display processing unit 17 then performs display processing by superimposing the template eyebrow data on the customer's eyebrows so that the pupil center coincides with the pupil center identified based on the facial image data acquired in S1. The template eyebrow data is configured to be editable by accepting input related to editing the position and shape, but eyebrow color may also be editable. The color of the template eyebrow data may also be edited automatically. In this case, the display processing unit 17 changes the color of the template eyebrow data to match the color of the customer's eyebrows based on the customer's eyebrow color identified in S5, and then performs display processing.

[0134] In addition, in this embodiment, the pupil centers of both eyes and the distance between the pupil centers of both eyes are used as the attributes of both eyes, but feature points of both eyes may be used instead of the pupil centers of both eyes, and the distance between the inner corners of both eyes or the distance between the outer corners of both eyes may be used instead of the distance between the pupil centers of both eyes.

[0135] 4.7. Calculation of eyebrow position score In S7, the calculation unit 16 calculates the eyebrow position score. In this embodiment, the calculation unit 16 calculates the difference between the left and right eyebrows and the match rate as the eyebrow position score.

[0136] <4.7.1. Calculating the difference between the left and right eyebrows> The left-right difference calculation unit 161 calculates the left-right difference of the customer's eyebrows as an eyebrow position score based on the degree of coincidence between the left and right eyebrow areas identified in S5. Specifically, the identification unit 15 uses an external reference scale in the customer's facial image data to identify the midpoint between the tips of the eyebrows (BL1) as the center of the eyebrows. The left-right difference calculation unit 161 inverts either the left or right eyebrow area around the identified center of the eyebrows, and calculates the overlapping area of the intersection of the two eyebrow areas and the total area of the union of the two eyebrow areas. The left-right difference calculation unit 161 then calculates the ratio of the overlapping area to the total area as the left-right eyebrow difference.

[0137] The left-right difference calculation unit 161 calculates the left-right difference between the eyebrows based on the eyebrow center identified using a scale of the external reference in the customer's facial image data. On the other hand, if the shape of the external reference WS does not have a scale as a feature identification indicator, such as a circle, the identification unit 15 uses well-known image recognition technology to identify the pupil centers of both eyes as the eyebrow centers based on the pupil centers of both eyes in the facial image data, and the left-right difference calculation unit 161 calculates the left-right difference between the eyebrows based on the eyebrow centers. Even if a scale is included as a feature identification indicator, the identification unit 15 may also use well-known image recognition technology to identify the pupil centers of both eyes as the eyebrow centers based on the pupil centers of both eyes in the facial image data.

[0138] Furthermore, the left-right difference calculation unit 161 calculates the left-right difference between the eyebrows based on the attributes of the left and right eyebrows. Specifically, the left-right difference calculation unit 161 calculates the difference between the corresponding eyebrow attributes of the left and right eyebrows (eyebrow thickness, eyebrow angle, eyebrow length, eyebrow darkness, eyebrow color, distance between the eyebrows, distance between the eyes and the eyebrows, etc.) and calculates the difference as the left-right difference between the eyebrows. This allows customers to easily understand the difference between their left and right sides by combining visual results with numerical evaluations. Customers can also understand the difference in their left and right sides in concrete numerical values (for example, the difference in left and right tilt or curve by ~~%).

[0139] <4.7.2. Calculating the match rate> The match rate calculation unit 162 calculates a match rate based on the degree of agreement between the eyebrow region identified in S5 and the region of the template eyebrow data (including the corrected template eyebrow data) specified in S6. Specifically, the match rate calculation unit 162 calculates the overlapping area of the intersection of the identified customer's eyebrow region and the eyebrow region of the template eyebrow data, and the total area of the union of the customer's eyebrow region and the eyebrow region of the template eyebrow data. The match rate calculation unit 162 then calculates the ratio of the overlapping area to the total area as the match rate.

[0140] 4.8. Calculation of treatment score In S8, the treatment score calculation unit 163 calculates the treatment score. In this embodiment, the treatment score calculation unit 163 calculates the treatment score using multiple indices related to the positional relationship between the customer's eyebrow area identified in S5, the pre-edited template eyebrow data (default template eyebrow data) specified in S6, and the corrected template eyebrow data, including a corrected matching degree index based on the matching degree between the customer's eyebrow area and the area of the corrected template eyebrow data, a default matching degree index based on the matching degree between the customer's eyebrow area and the default template eyebrow data, and an edited matching degree index based on the matching degree between the corrected template eyebrow data and the default template eyebrow data.

[0141] Specifically, the treatment score calculation unit 163 calculates the treatment score by substituting the corrected matching degree index, the default matching degree index, and the edited matching degree index into a predetermined calculation formula. The predetermined calculation formula is such that the larger the corrected matching degree index, the larger the treatment score, and the smaller the edited matching degree index, the smaller the treatment score. In other words, even if the treatment that matches the corrected template eyebrow data is successful, if the corrected template eyebrow data deviates from the template eyebrow data desired by the customer, the treatment score will be small.

[0142] In addition, in a preferred embodiment of the present invention, an index for calculating a treatment score is registered for each treatment, linked to a combination of therapist and customer, and if the value of the index increases as a result of the current treatment compared to the previous treatment, the increase in the treatment score is varied according to the value of the index from the previous treatment.

[0143] Fig. 10 shows an example of a customer image to explain the calculation method of the treatment score. Fig. 10(a) shows an example of calculating a corrected matching index based on the degree of matching between the corrected template eyebrow data NTB and the customer's eyebrow area BA. Fig. 10(b) shows an example of calculating a default matching index based on the degree of matching between the default template eyebrow data OTB and the customer's eyebrow area BA. Fig. 10(c) shows an example of calculating an edited matching index based on the degree of matching between the corrected template eyebrow data NTB and the default template eyebrow data OTB.

[0144] Furthermore, the treatment score calculation unit 163 calculates the degree of improvement in the difference between the left and right eyebrows based on the difference between the left and right eyebrows before the treatment and the difference between the left and right eyebrows after the treatment. In this embodiment, the newly calculated difference between the left and right eyebrows is registered along with the time for each treatment, and the treatment score calculation unit 163 calculates the degree of improvement in the difference between the left and right eyebrows (for example, the difference value between the left and right eyebrows) based on the most recent difference between the left and right eyebrows and the oldest difference between the left and right eyebrows in one treatment.

[0145] <4.9. Screen display processing> In S9, the display processing unit 17 displays the data obtained in S1 to S8 on the screen of the user terminal device 1. In this embodiment, the display processing unit 17 superimposes the eyebrow area identified in S5 on the customer's facial image data obtained in S2 or S4 and displays the same, and further superimposes the template eyebrow data specified in S6. The display processing unit 17 also displays the eyebrow difference and match rate calculated in S7 and the treatment score calculated in S8.

[0146] FIG. 11 shows an example of a customer image captured during treatment and displayed on the user terminal device 1. FIG. 11(a) displays the customer's left and right eyebrow regions and eyebrow feature points identified based on facial image data, as well as the eyebrow lateral difference calculated in S7. FIG. 11(b) displays the customer's left and right eyebrow regions in a switchable manner, with the template eyebrow data specified in S6 superimposed on each of the left and right eyebrow regions. Also displayed are the match rate calculated in S7, the eyebrow attributes identified in S5 (in the illustrated example, the eyebrow shape category, eyebrow angle, and eyebrow thickness), the treatment score calculated in S8, and the degree of improvement in lateral difference. Although not shown in the figure, the eyebrow attributes of the template eyebrow data specified in S6 and the eyebrow attributes identified in S5 may be superimposed on a chart (e.g., a radar chart) with each axis representing an item of the eyebrow attribute.

[0147] <4.10. Registering customer treatment history information> In S10, the registration unit 18 registers customer template eyebrow data based on the corrected template eyebrow data. In this embodiment, the registration unit 18 registers the template eyebrow data edited in S6 (corrected template eyebrow data), eyebrow attributes, eyebrow position score, and treatment score as customer treatment history information.

[0148] Then, in S11, if an input regarding the end of treatment is received (YES in S10), the processing is terminated, and if an input regarding the end of treatment is not received (NO in S10), the processing returns to S1 and new facial image data is acquired.

[0149] By executing the above steps S1 to S11, the practitioner can perform the treatment while simultaneously checking the target brow area and the template brow. This allows even practitioners with limited treatment skills to perform the treatment easily. Furthermore, by calculating a score for the treatment, the practitioner's success can be checked at any time. Furthermore, using the treatment score allows the practitioner's treatment to be accurately evaluated, thereby increasing their motivation. Furthermore, objective measurement and evaluation of eyebrow designs that are not dependent on the practitioner's subjectivity can be achieved, ensuring symmetry and significantly improving post-treatment reproducibility. Furthermore, by correcting for camera errors using multiple methods, higher accuracy can be achieved compared to conventional measurement methods, enabling quantitative evaluation of treatment effects. Furthermore, by accumulating data as a customer chart, long-term treatment history management can be easily managed and used to plan future treatments.

[0150] The adjustment unit 12 performs mask processing on the acquired plurality of pieces of facial image data to generate averaged facial image data. Specifically, the adjustment unit 12 identifies the coordinate positions of external references based on the external reference feature points for the acquired plurality of pieces of facial image data, and performs mask processing to extract the external references based on the coordinate positions. The adjustment unit 12 then generates averaged facial image data using each piece of facial image data that has been subjected to mask processing.

[0151] The adjustment unit 12 may also normalize the facial image data using averaged facial image data obtained by averaging the acquired plurality of facial image data. Alternatively, the adjustment unit 12 may identify, from the acquired plurality of facial image data, facial image data having a face tilt angle, size, and brightness that optimize the eyebrow asymmetry and match rate, and use the identified facial image data to normalize the facial image data. The adjustment unit 12 may also identify, from the acquired plurality of facial image data, facial image data having the largest external reference size based on external reference feature points, and use the identified facial image data to normalize the facial image data.

[0152] The adjustment unit 12 normalizes the facial image data based on facial image data that includes all of the external reference shapes, but may also normalize the facial image data based on facial image data that does not include all of the external reference shapes. Specifically, the adjustment unit 12 may specify the inclination, size, and brightness of the face based on some of the external reference feature points (normalization indexes) of the external reference, and normalize the facial image data.

[0153] Furthermore, in the above processing flowchart, the processing from S2 onward is performed based on the customer's facial image data captured in S1. In a preferred embodiment of the present invention, the processing from S2 onward is performed by acquiring a video of the customer's face in S1. Specifically, the acquisition unit 11 acquires facial video data of the customer during treatment, and the adjustment unit 12 inputs each frame of the facial video data into a normalization learning model to adjust the values of the normalization items of the facial video data, and further adjusts the values of the normalization items through calibration using an external standard. The display processing unit 17 then renders and displays the customer's facial features, template feature data corresponding to the facial features, and calculated scores for the adjusted facial video data. In other words, various information is superimposed on the customer's eyebrows using AR (Augmented Reality) or MR (Mixed Reality). This allows the therapist to wear a head-mounted display and view the image in real time using AR, making it more persuasive to both the client and the therapist.

[0154] More specifically, the display processing unit 17 uses the external reference as the augmented reality display reference or the mixed reality display reference to render the template eyebrow data onto the customer's eyebrows for display processing. Even more specifically, the positional relationship between the pupil center of the eye, the template eyebrow data, and the augmented reality display reference or the mixed reality display reference is specified in advance based on the facial image data, and the display processing unit 17 specifies the position of the template eyebrow data based on the augmented reality display reference or the mixed reality display reference to render the template eyebrow data onto the customer's eyebrows for display processing. This makes it possible to stably render and display the template eyebrow data on the customer's eyebrows even when it is difficult to detect the center of the customer's pupil.

[0155] In conventional AR services, the accuracy of position information generally includes an error of several centimeters, which is often not a problem for general use. However, in eyebrow design treatments, even slight positional deviations can have fatal consequences. Therefore, the present invention employs a calibration and normalization process using an external standard, enabling identification of a customer's facial feature information with extremely high accuracy (e.g., an error of several millimeters or less). Based on the accurately identified facial feature location information (facial feature information), the practitioner can accurately project the target template eyebrow data onto the customer's face in real time using augmented reality (AR) or mixed reality (MR) technology. As a result, practitioners can receive extremely precise treatment support that was difficult to achieve with conventional AR services, significantly improving the reproducibility and accuracy of eyebrow design.

[0156] Furthermore, each time the eyebrow region of the subject is identified as facial region feature information, the calculation unit 16 calculates an eyebrow position score based on the eyebrow region. Specifically, the identification unit 15 identifies the eyebrow region based on the customer's eyebrows displayed based on the facial video data, and the calculation unit 16 calculates the eyebrow position score and the treatment score based on the eyebrow region and template eyebrow data displayed based on the facial video data.

[0157] Furthermore, the adjustment unit 12 adjusts the parameters of the imaging device based on the face video data including the external reference. Specifically, the adjustment unit 12 identifies the parameters (internal parameters and external parameters) of the imaging device based on the eyebrow feature points, eye feature points, external reference feature points, and the distance between the external reference and the user terminal device 1, which are included in the face video data, and adjusts the distortion parameters of the lens of the imaging device so that the distortion parameters fall within the allowable range when the distortion parameters exceed the allowable range. This allows so-called markerless calibration to be performed, and errors during re-imaging can be dynamically reduced.

[0158] In a more preferred embodiment, the adjustment unit 12 determines the distance between the external reference and the user terminal device 1 based on infrared light. Specifically, the imaging device includes an infrared sensor and an irradiation unit that irradiates infrared light, and the external reference includes a material that reflects infrared light (e.g., a retroreflective marker). The adjustment unit 12 then acquires the infrared light from the irradiation unit reflected by the external reference via the infrared sensor, and determines the distance between the external reference and the user terminal device 1 based on the infrared light.

[0159] In a more preferred embodiment, information about the customer's eyebrows after treatment is predicted. Specifically, the customer treatment history information stores facial image data of the customer before treatment, facial image data of the customer after treatment, and the treatment date for each treatment for the same customer. The learning unit 19 uses the facial image data of the customer before treatment, the facial image data of the customer after treatment, and the treatment date as training data to learn a change estimation model that estimates deterioration and changes in the customer's eyebrows and predicts the date of the next treatment. Then, a prediction unit of the treatment support system (not shown) inputs the facial image data of the customer after treatment into the change estimation model to predict the date of the customer's next treatment.

[0160] In this embodiment, the facial image data is corrected using only an external criterion, but the facial image data may also be corrected using an internal criterion, which will be described later.

[0161] (Embodiment 2: Embodiment without using an external standard) Next, a description will be given of embodiment 2. In this embodiment, the practitioner is assisted in treatment without using external references (rulers, markers, etc.) as described above.

[0162] In the first embodiment, an external standard was used to reduce errors in multiple photographs, thereby supporting the practitioner's treatment. In contrast, in this embodiment, by combining the following configurations, high reproducibility is achieved when re-photographing facial image data without using an external standard, and it is possible to display template eyebrows, calculate treatment scores, and so on. In the following explanation, the system according to this embodiment operates each functional configuration, and parts common to the first embodiment will be omitted.

[0163] <5.1. System configuration of embodiment 2> The treatment support system of this embodiment further includes a position specifying unit (not shown) in addition to the system configuration of the first embodiment.

[0164] <5.2. Location identification part> The position identification unit identifies facial feature position information, which is information related to the positions of facial features, based on the facial image data. In this embodiment, the position identification unit identifies the three-dimensional shape of the subject's face as the facial feature position information based on the facial image data captured from multiple different angles. Specifically, the position identification unit matches corresponding feature points between multiple sets of facial image data of the subject captured from different angles or facial video data (video frames) acquired by the acquisition unit 11, and applies a well-known three-dimensional shape estimation technology (e.g., SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) technology) to identify the position and orientation of the imaging device and the three-dimensional shape of the facial surface. By identifying such a three-dimensional shape, the spatial coordinate system of the subject's face can be relatively determined without using an external reference.

[0165] More preferably, the position identification unit identifies the spatial coordinate system of the subject's face using landmarks such as pupil positions and contour points as feature points between multiple facial image data of the subject photographed from different angles or facial video data (video frames). This makes it possible to track landmarks such as pupil positions and contour points, and to approximate a similar 3D coordinate system when re-taking photographs, thereby reducing photographic errors.

[0166] The position identification unit identifies facial feature position information based on an internal reference included in each piece of facial image data, which is a standard for identifying facial feature information. In this embodiment, the position identification unit uses relatively stable features, such as the subject's pupils, nose, and corners of the mouth, as internal references and measures the relative positions and angles of facial features by referring to the internal references. The position identification unit then corrects scale-invariant factors (enlargement / reduction due to shooting distance) by using, for example, the average interpupillary distance, and scales the customer's facial image data based on the measured relative positions and angles of the facial features and the corrected scale-invariant factors to identify the three-dimensional shape of the facial surface as facial feature position information.

[0167] In a more preferred embodiment, the position identifying unit corrects the scale-indefinite elements by further using the customer's facial depth information. Specifically, the acquiring unit 11 acquires face information including the customer's facial depth information and facial image data, and the position identifying unit corrects the scale-indefinite elements based on the facial depth information. Here, the facial depth information is, for example, three-dimensional face data.

[0168] In a more preferred embodiment, when there is a large change in facial expression, the position identification unit selects the data with the least change in facial expression from multiple frames of facial image data or facial video data, and identifies the relative position and angle of the facial part by referring to an internal reference of the data. This also makes it possible to further reduce positional deviations of the internal reference.

[0169] 5.3. Learning unit 19 in the second embodiment The learning unit 19 learns an error correction learning model using face image data captured under multiple different conditions that do not include an external standard as training data. Without using an external standard, the shooting angle, distance, and lighting conditions may vary each time. Therefore, the learning unit 19 uses repeatedly captured face image data as training data to construct an error correction learning model for correcting shooting errors. For example, the learning unit 19 uses shooting parameters (such as light intensity, focal length, and face angle) and actually measured eyebrow position scores as training data to learn the correlation between the shooting parameters and the eyebrow position scores, thereby constructing an error correction learning model that estimates shooting parameters for calculating an accurate eyebrow position score.

[0170] <5.4. Adjustment unit 12 in embodiment 2> The adjustment unit 12 adjusts (normalizes) the facial image data based on the facial feature position information. In this embodiment, the adjustment unit 12 adjusts the facial image data based on the three-dimensional shape of the facial surface identified as the facial feature position information. Specifically, the adjustment unit 12 adjusts the facial image data so that the feature points of the three-dimensional shape of the facial surface included in each captured facial image data, which is scaled using an internal reference, match.

[0171] The adjustment unit 12 also corrects the facial image data using the error correction learning model learned by the learning unit 19. In this embodiment, the adjustment unit 12 inputs photographed facial image data into the error correction learning model, thereby dynamically correcting the estimated shooting error according to the conditions of the facial image data. This makes it possible to improve reproducibility in stages.

[0172] The adjustment unit 12 also averages and normalizes the facial image data based on the captured facial image data. Since slight deviations occur in a single shot, if the subject allows it, the acquisition unit 11 acquires multiple sets of facial image data. The adjustment unit 12 then averages or median-combines the multiple sets of facial image data to offset errors and generate more stable facial image data (averaged facial image data).

[0173] Furthermore, the adjustment unit 12 normalizes the facial image data for each capture by projecting facial image data captured at different times onto the same coordinate system based on the three-dimensional point cloud information (three-dimensional shape of the facial surface) constructed by SLAM or SfM. This reduces variation in measurement values even without an external standard.

[0174] <5.5. Identification unit 15 in embodiment 2> The identification unit 15 identifies facial part feature information based on the facial image data adjusted by the adjustment unit 12. In this embodiment, the identification unit 15 identifies facial part feature information (eyebrow attributes, eyebrow area, and both eye attributes, etc.) using facial image data adjusted based on the three-dimensional shape of the face surface of the subject as facial part position information.

[0175] <5.6. Display Processing Unit 17 in the Second Embodiment> The display processing unit 17 performs display processing of the template eyebrow data based on the identified facial feature position information. In this embodiment, the display processing unit 17 identifies the customer's pupil center using the three-dimensional shape of the face surface as facial feature position information. Then, the display processing unit 17 superimposes the template eyebrow data on the customer's eyebrows so that the customer's pupil center and the pupil center of the model based on the model master information coincide with each other, and displays the template eyebrow data on the user terminal device 1.

[0176] The display processing unit 17 also displays template eyebrow data in AR based on facial feature position information identified by the facial video data. Specifically, the acquisition unit 11 acquires facial video data of a customer undergoing treatment, and the adjustment unit 12 adjusts the values of normalization items in the facial video data by inputting each frame of the facial video data into an error correction learning model, and further adjusts the facial video data based on the three-dimensional shape of the facial surface. The display processing unit 17 then renders and displays the customer's facial features, template feature data corresponding to the facial features, and calculated scores for the adjusted facial video data.

[0177] As described above, even when there is no need to attach external references to the subject's face or take the time to capture them in the shooting frame, the present invention ensures sufficient reproducibility by combining multiple view analysis such as SLAM / SfM, utilization of internal facial landmarks, error correction using machine learning, and averaging of multiple shots of data, etc. These methods can be combined as desired and may be combined with known technologies (such as physical fixation using a rotating table). [Explanation of symbols]

[0178] 0: Treatment support system 1: User terminal device 2: Information management device 3: Database 9: Terminal device 91: Processing section 92: Storage section 93: Communications Department 94: Input section 95: Output section 10: Server 101: Processing section 102: Storage section 103: Communications Department 11: Acquisition part 12:Adjustment section 13: Shooting instruction generation unit 14: Template generation section 15: Specific part 151: Area identification part 152: Eyebrow attribute identification part 16: Calculation section 161: Left-right difference calculation unit 162: Match rate calculation unit 163: Treatment score calculation unit 17: Display processing section 18: Registration Department 19: Learning Department 21: Data storage section 22: Distribution Department

Claims

1. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a learning unit, an adjustment unit, and an identification unit, the acquiring unit acquires a plurality of pieces of face information including external criteria established as absolute criteria for determining features of facial parts of the subject; the learning unit uses a combination of a plurality of pieces of face information and face part position scores calculated based on each piece of face information, the face part position scores indicating the degree of overlap between the face parts of the subject and template part data, as training data to learn a normalization learning model that estimates values of normalization items of the face information that calculate the accurate face part position score; the adjustment unit generates averaged face information by averaging values of the normalization items based on the plurality of pieces of face information under at least two or more different shooting conditions; Further, the averaged face information is input to the normalization learning model to adjust the value of the normalization item of the customer's face included in the averaged face information; The identification unit identifies facial region feature information based on the face information in which the normalization items have been adjusted. Treatment support system.

2. The learning unit learns the normalized learning model using a combination of a plurality of pieces of face information captured under different imaging conditions and the face part position scores calculated based on each piece of face information as training data; The identification unit identifies eyebrow features as the facial region feature information based on the face information. The treatment support system according to claim 1.

3. The treatment support system further includes a display processing unit and a calculation unit, the acquiring unit acquires successive face image data as the face information, the display processing unit executes AR display, which virtually superimposes and displays the template part data on the face parts of the subject included in the face image data, simultaneously with acquisition of the face image data; The calculation unit calculates the face part position score indicating the degree of overlap between the face part of the subject and the template part data. The treatment support system according to claim 1.

4. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a display processing unit, a calculation unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; the display processing unit renders template part data on face parts of the subject based on the face information and performs display processing; The calculation unit calculates a face part position score indicating a degree of overlap between the face part of the subject and the template part data. Treatment support system.

5. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a database, a display processing unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; The database stores model master information relating to an eyebrow model, the model master information including attributes of both eyes of the model; the specifying unit specifies attributes of both eyes of the subject as facial region feature information based on the face information; The display processing unit processes and displays template eyebrow data based on the eye attributes of the model and the eye attributes of the identified subject. Treatment support system.

6. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a display processing unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; The display processing unit uses the external reference as an augmented reality display reference or a mixed reality display reference to render template part data onto a corresponding face part of the subject and perform display processing. Treatment support system.

7. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a display processing unit, a calculation unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; the acquiring unit acquires face video data including the external criterion as the face information; the display processing unit renders template part data onto an image of a facial part of the subject based on the facial video data, and performs display processing; The calculation unit calculates a face part position score based on the face part feature information each time the face part feature information is identified. Treatment support system.

8. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a calculation unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; The calculation unit calculates a treatment score based on a plurality of indices related to the positional relationship between facial part feature information of the subject based on the face information, the default template part data, and the corrected template part data. Treatment support system.

9. A treatment support system that supports a treatment by a practitioner, The treatment support system includes an acquisition unit, a database, a display processing unit, a registration unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; the database stores editable template site data; the registration unit registers the corrected template region data, which has been edited, in association with the subject; The display processing unit processes and displays the corrected template region data corresponding to the target person to receive treatment. Treatment support system.

10. The external reference is a ruler made of a material having a higher infrared reflectivity than the face of the subject, and / or a black material that provides a high contrast between the face of the subject and the external reference. The treatment support system according to claim 1.

11. The treatment support system further includes a photographing instruction generation unit, The photographing instruction generation unit specifies a value of a photographing instruction item based on the external criterion, and generates a photographing instruction based on the value of the photographing instruction item. The treatment support system according to claim 1.

12. The treatment support system further includes a calculation unit, the specifying unit specifies an eyebrow region as facial region characteristic information based on the face information, The calculation unit compares the left and right eyebrow regions. The treatment support system according to claim 1.

13. The calculation unit compares the left and right eyebrow regions using pupils of both eyes based on the face information. The treatment support system according to claim 12.

14. The specifying unit specifies ten eyebrow feature points at predetermined positions on each of the left and right eyebrows as the facial part feature information based on the face information. The treatment support system according to claim 1.

15. The specifying unit specifies eyebrow feature points for determining eyebrow attributes of the subject as the facial part feature information based on a straight line connecting corresponding positions of both eyes based on the face information and an eyebrow area specified based on the face information. The treatment support system according to claim 1.

16. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a learning unit, an adjustment unit, and an identification unit; the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the learning unit uses a combination of a plurality of pieces of face information and face part position scores calculated based on each piece of face information, the face part position scores indicating the degree of overlap between the face parts of the subject and template part data, as training data to learn a normalization learning model that estimates values of normalization items of the face information that calculate the accurate face part position score; the adjustment unit generates averaged face information by averaging values of the normalization items based on the plurality of pieces of face information under at least two or more different shooting conditions; Further, the averaged face information is input to the normalization learning model to adjust the value of the normalization item of the customer's face included in the averaged face information; The identification unit identifies facial region feature information based on the face information in which the normalization items have been adjusted. Treatment support program.

17. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring a plurality of facial information including external criteria that are established as absolute standards for determining the characteristics of the facial parts of the subject; a process of learning a normalization learning model that estimates values of normalization items of the face information to calculate accurate face part position scores, using as training data a combination of a plurality of pieces of face information and face part position scores that are calculated based on each piece of face information and indicate the degree of overlap between the face parts of the subject and template part data; generating averaged face information by averaging the values of the normalization items based on the plurality of pieces of face information under at least two or more different photographing conditions; Further, a process of inputting the averaged face information into the normalization learning model to adjust the values of the normalization items of the customer's face included in the averaged face information; A process of identifying facial region feature information based on the face information in which the normalization items have been adjusted; A treatment support method for carrying out the above.

18. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a display processing unit, a calculation unit, and an identification unit; the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; the display processing unit renders template part data on face parts of the subject based on the face information and performs display processing; The calculation unit calculates a face part position score indicating a degree of overlap between the face part of the subject and the template part data. Treatment support program.

19. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of identifying facial region feature information based on the face information; a process of rendering and displaying template region data on the facial region of the subject based on the face information; calculating a facial feature position score indicating the degree of overlap between the subject's facial features and the template feature data; A treatment support method for carrying out the above.

20. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a database, a display processing unit, and an identification unit; the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; The database stores model master information relating to an eyebrow model, the model master information including attributes of both eyes of the model; the specifying unit specifies attributes of both eyes of the subject as facial region feature information based on the face information; The display processing unit processes and displays template eyebrow data based on the eye attributes of the model and the eye attributes of the identified subject. Treatment support program.

21. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of storing model master information, which is information about the eyebrow model and includes attributes of both eyes of the model, in a database; A process of identifying attributes of both eyes of the subject as facial region feature information based on the face information; displaying template eyebrow data based on the eye attributes of the model and the eye attributes of the identified subject; A treatment support method for carrying out the above.

22. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a display processing unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; The display processing unit uses the external reference as an augmented reality display reference or a mixed reality display reference to render template part data onto a corresponding face part of the subject and perform display processing. Treatment support program.

23. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of identifying facial region feature information based on the face information; using the external reference as an augmented reality or mixed reality display reference to render and display the template region data on the corresponding facial region of the subject; A treatment support method for carrying out the above.

24. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a display processing unit, a calculation unit, and an identification unit; the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; acquiring face video data including the external reference as the face information; the display processing unit renders template part data onto an image of a facial part of the subject based on the facial video data, and performs display processing; The calculation unit calculates a face part position score based on the face part feature information each time the face part feature information is identified. Treatment support program.

25. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of identifying facial region feature information based on the face information; A process of acquiring face video data including the external criteria as the face information; a process of rendering and displaying template region data on an image of a facial region of a subject based on the facial video data; a process of calculating a face part position score based on the face part feature information each time the face part feature information is identified; A treatment support method for carrying out the above.

26. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a calculation unit, and an identification unit, the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; The calculation unit calculates a treatment score based on a plurality of indices related to the positional relationship between facial part feature information of the subject based on the face information, the default template part data, and the corrected template part data. Treatment support program.

27. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of identifying facial region feature information based on the face information; A process of calculating a treatment score based on a plurality of indices related to the positional relationship between facial region feature information of the subject based on the face information, the default template region data, and the corrected template region data; A treatment support method for carrying out the above.

28. A treatment support program that supports a practitioner's treatment, the treatment assistance program causes a computer to function as an acquisition unit, a database, a display processing unit, a registration unit, and an identification unit; the acquiring unit acquires face information including an external standard that is set as an absolute standard for determining features of a facial part of the subject; the specifying unit specifies facial region feature information based on the face information; the database stores editable template site data; the registration unit registers the corrected template region data, which has been edited, in association with the subject; The display processing unit processes and displays the corrected template region data corresponding to the target person to receive treatment. Treatment support program.

29. A treatment support method for supporting a treatment by a practitioner, The computer A process of acquiring facial information including external criteria that are established as absolute standards for determining the characteristics of facial parts of a subject; A process of identifying facial region feature information based on the face information; storing editable template part data in a database; a process of registering the corrected template region data obtained by accepting edits to the template region data in association with the subject; a process of displaying the corrected template region data corresponding to the subject to be treated; A treatment support method for carrying out the above.

Images