Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program that can improve user convenience. [Solution] The information processing device according to the present invention comprises a specification unit, an acquisition unit, and an image processing unit. The specification unit uses a learning model, which is a model that has learned the relationship between information indicating facial features and hairstyles evaluated as suitable for the face, to identify a hairstyle that is estimated to suit the user's face. The acquisition unit acquires an image that includes a specific hair image, which is an image of the hairstyle identified by the specification unit. The image processing unit generates a composite image by combining the image of the hairstyle identified by the specification unit and the user's face image, based on the image including the specific hair image acquired by the acquisition unit and the user's face image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, a technique for synthesizing a user's face image and a hair image and presenting it to the user is known. For example, Patent Document 1 proposes a technique for performing image processing by hair simulation that changes a hair style or hair color and displaying the result of such image processing.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above conventional technology, the user has to check one by one which hair style suits them while changing the hair style. As the number of hair styles increases, the user's operation becomes more complicated, and there is room for improvement in terms of improving the user's convenience.

[0005] The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of improving user convenience.

Means for Solving the Problems

[0006] The information processing device according to the present application comprises a specification unit, an acquisition unit, and an image processing unit. The specification unit uses a learning model, which is a model that has learned the relationship between information indicating facial features and hairstyles evaluated as suitable for the face, to identify a hairstyle that is estimated to suit the user's face. The acquisition unit acquires an image that includes a specific hair image, which is an image of the hairstyle identified by the specification unit. The image processing unit generates a composite image by combining the image of the hairstyle identified by the specification unit and the user's face image, based on the image including the specific hair image acquired by the acquisition unit and the user's face image. [Effects of the Invention]

[0007] According to one embodiment, user convenience can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information providing device according to an embodiment. [Figure 4] Figure 4 shows an example of a user information table stored in the user information storage unit of the information providing device according to the embodiment. [Figure 5] Figure 5 shows an example of a post information table stored in the post information storage unit of the information provision device according to the embodiment. [Figure 6] Figure 6 shows an example of a model information table stored in the model information storage unit of the information providing device according to the embodiment. [Figure 7] Figure 7 shows an example of a content table stored in the content storage unit of the information providing device according to this embodiment. [Figure 8] Figure 8 shows an example of the configuration of a terminal device according to this embodiment. [Figure 9]Figure 9 shows an example of the configuration of the reception unit of a terminal device according to this embodiment. [Figure 10] Figure 10 shows an example of the configuration of the image processing unit of a terminal device according to this embodiment. [Figure 11] Figure 11 shows an example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 12] Figure 12 shows an example of a keyword selection screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 13] Figure 13 is a diagram illustrating the process by which the processing unit of the information providing device according to the embodiment acquires similar hair images from a specified keyword. [Figure 14] Figure 14 shows another example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 15] Figure 15 shows yet another example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 16] Figure 16 shows an example of a composite image change screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 17] Figure 17 is a diagram illustrating the process by which the estimation unit of the information providing device according to the embodiment estimates the treatment content from a user image and a composite image. [Figure 18] Figure 18 shows an example of a face shape selection screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 19] Figure 19 is a flowchart showing an example of information processing by the processing unit of the terminal device according to this embodiment. [Figure 20] Figure 20 is a flowchart showing an example of reservation processing by the processing unit of the terminal device according to this embodiment. [Figure 21] Figure 21 is a hardware configuration diagram showing an example of a computer that implements the respective functions of the information provision device and terminal device according to the embodiment.

Best Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Also, each embodiment can be appropriately combined within a range that does not conflict with the processing content. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] 〔1. An Example of Information Processing〕 First, an example of information processing according to the embodiment will be described using FIG. 1. FIG. 1 is a diagram showing an example of information processing according to the embodiment. In the following, the case where the information processing apparatus according to the embodiment is the terminal device 4 shown in FIG. 1 will be described. However, such an information processing apparatus may be the information providing apparatus 1 shown in FIG. 1, or may have a configuration combining the terminal device 4 and the information providing apparatus 1.

[0011] The information providing apparatus 1 according to the embodiment shown in FIG. 1 is, for example, an information processing apparatus that provides services such as SNS (Social Networking Service). For example, the information providing apparatus 1 receives posting information posted by a poster. The information providing apparatus 1 distributes the posting information received from the poster to the terminal device 4 of the user U who becomes the viewer.

[0012] The poster is, for example, a hair salon, a hair technician, a person receiving a hair treatment, etc., and may include the user U. A hair salon is a beauty salon (beauty parlour) or a barbershop (barber shop). A hair technician is a technician who arranges hair in a hair salon and is also called a hairstylist. A person receiving a hair treatment is a user who has received a hair treatment by a hair technician.

[0013] The information posted by the poster includes, for example, a posted image, which is an image of the poster taken by capturing the poster, and information indicating the posted text, which is the text posted by the poster. The posted image includes, for example, an image of the poster's head, including their hair and face.

[0014] Furthermore, the posted text may include, for example, the poster's comments on the posted image. The poster's comments on the posted image may include, for example, evaluation information indicating the poster's rating of the image. In the following, the poster's comments on the posted image may be referred to as "poster comments."

[0015] Furthermore, the information provider 1 receives evaluation information indicating the evaluations of other users (an example of a viewer) of the posted images included in the posted information, as well as comments from other users on the posted information. The information provider 1 then distributes the above-mentioned posted information, including the received evaluation information and comments from other users, to the viewer's device. The viewer's device is, for example, user U's terminal device 4 or another user's device. In the following, comments from other users may be referred to as viewer comments.

[0016] The evaluation of the posted information may include, for example, positive evaluations (e.g., "Good") or negative evaluations (e.g., "Bad"), but may also include intermediate evaluations (e.g., "Average"). In addition, the information provider 1 can also distribute information such as the number of times the posted information has been added to the timeline by other users as an evaluation of the posted information to the terminal device 4 of user U, who is a viewer.

[0017] The information provider 1 transmits information on multiple types of learning models to the terminal device 4 (step S1). The multiple types of learning models include a first learning model, a second learning model, a third learning model, and a fourth learning model.

[0018] First, let's explain the first learning model. The first learning model is a model that learns the relationship between keywords and hair images, and it is a learning model that takes one or more keywords as input and outputs hair images. The first learning model is generated by machine learning using training data that contains multiple combinations of keywords and hair images.

[0019] The first learning model may also be a learning model that takes one or more keywords as input and outputs hair image identification information. Hair image identification information is information that identifies a hair image.

[0020] The information provider 1 extracts one or more keywords from, for example, a poster's comment or poster's text, and generates data containing multiple combinations of such one or more keywords and the posted image that was the subject of the comment, as the training data described above.

[0021] Furthermore, if the information provider 1 cannot extract keywords from the poster's comments or text, it extracts one or more keywords from the viewer's comments and generates data containing multiple combinations of such one or more keywords and the posted image that was the subject of the comment, as the training data described above.

[0022] Furthermore, the information provider 1 can extract one or more keywords contained in both the poster's comment or poster's text and the viewer's comment, and generate data containing multiple combinations of such one or more keywords and the posted image that was the subject of the comment, as the training data described above.

[0023] The training data may include multiple combinations of keywords assigned by the operator to each posted image and the posted image. Furthermore, the training data may not only consist of posted information and other content, but also include images contained in web pages and other content, or images provided by third parties. Posted information is an example of posted content.

[0024] For example, information provider 1 can acquire content via a network that includes images containing a person's face and hair, as well as text, and extract keywords from the text contained in such content. In this case, information provider 1 generates data as training data that includes multiple combinations of keywords extracted from the text contained in the content and images contained in the content.

[0025] Next, we will explain the second learning model. The second learning model is a learning model that has learned the relationship between face images and information that indicates facial features. It takes an image containing a face as input and outputs information that indicates facial features.

[0026] The second learning model is generated by machine learning using training data that includes multiple combinations of images, including face images, and information describing facial features. The information describing facial features may, for example, indicate the type of face shape, but it may also be information describing the characteristics of one or more parts of the face, such as the size, shape, and position of the eyebrows, eyes, nose, mouth, and ears.

[0027] Furthermore, information describing facial features may include information indicating the position of each of the eyebrows, eyes, nose, mouth, and ears on the face, or it may include information indicating the position and size of each of the eyebrows, eyes, nose, mouth, and ears on the face.

[0028] Next, we will explain the third learning model. The third learning model is a model that learns the relationship between information representing facial features and hairstyles that are evaluated as suitable for that face. It is a learning model that takes information representing facial features as input and outputs information representing hairstyles that are estimated to suit that face.

[0029] The third learning model is generated by machine learning using training data that includes multiple combinations of information representing facial features, hair images, and information representing evaluations of the combination of facial feature information and hair images.

[0030] Information provider device 1 collects multiple pieces of posted information, each including posted images, viewer comments, and evaluation information. Face images include face images and hair images, and evaluation information is information indicating the evaluation of posted images by other users who are viewers, but may also include evaluation information contained in the poster's comments.

[0031] The information provider 1 extracts facial features from the facial images contained in each collected post. The information provider 1 then generates the aforementioned training data, which includes the hair images contained in the posted images, the extracted facial feature information, and evaluation information.

[0032] In the third learning model, the information indicating facial features may, for example, be information indicating the type of face shape, but it may also be information indicating the characteristics of one or more parts of the face, such as the size, shape, and position of the eyebrows, eyes, nose, mouth, and ears. Furthermore, the information indicating facial features may also be information indicating the position of each of the eyebrows, eyes, nose, mouth, and ears on the face, or information indicating the position and size of each of the eyebrows, eyes, nose, mouth, and ears on the face.

[0033] Next, we will explain the fourth learning model. The fourth learning model is a learning model that has learned the relationship between hair images before and after treatment and the treatment content. It takes hair images before and after treatment as input and outputs information indicating the estimated treatment content. The hair images before and after treatment include hair images before the hair treatment and hair images after the hair treatment.

[0034] The details of the treatment include, for example, whether or not a haircut is performed, the amount of hair cut, whether or not bleaching is done, the number of bleaching treatments, whether or not coloring is done, the type of coloring, whether or not a perm is done, the type of perm, and whether or not hair straightening is performed. The information indicating the details of the treatment includes the score information for each of the following: whether or not a haircut is performed, the amount of hair cut, whether or not bleaching is done, the number of bleaching treatments, whether or not coloring is done, the type of coloring, whether or not a perm is done, the type of perm, and whether or not hair straightening is performed.

[0035] Furthermore, the information indicating the treatment details may include scores for each combination of whether or not a haircut was performed, the amount of hair cut, whether or not bleaching was done, the number of bleaching treatments, whether or not coloring was done, the type of coloring, whether or not a perm was done, the type of perm, and whether or not hair straightening was performed.

[0036] The fourth learning model is generated by machine learning using training data that includes multiple combinations of hair images before and after treatment and information indicating the treatment content. The information providing device 1 collects treatment information, including hair images before and after treatment and information indicating the treatment content, from devices such as a hair salon or hair stylist. Then, based on the collected treatment information, the information providing device 1 generates training data that includes multiple combinations of hair images before and after treatment and information indicating the treatment content.

[0037] The learning models described above can be of any type. For example, each learning model is a model generated by machine learning, such as DNN (Deep Neural Network), GBDT (Gradient Boosting Decision Tree), and SVM (Support Vector Machine). DNNs are, for example, CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks). RNNs may also be LSTMs (Long Short-Term Memory). Furthermore, each learning model may be a model realized by combining multiple models, such as a model that combines a CNN and an RNN.

[0038] Next, terminal device 4 receives information on multiple types of learning models transmitted from information providing device 1 and stores the received information on multiple types of learning models in its internal memory (step S2).

[0039] User U of terminal device 4 may consider what hairstyle they want before making a reservation for a hair treatment at a hair salon. This consideration can be done, for example, by looking up information on the web or reading information published in magazines.

[0040] For example, user U can display hairstyle-related images obtained through keyword search or web search on terminal device 4. Hairstyle-related images are images that allow user U to confirm hairstyles, and include at least a hair image and a face image.

[0041] Here, keyword search will be explained. Terminal device 4 displays multiple keywords on the display unit (step S3). These multiple keywords are predetermined keywords, but they may also be multiple keywords randomly selected from the predetermined keywords or multiple keywords that satisfy predetermined conditions.

[0042] Multiple keywords that satisfy predetermined conditions are, for example, a predetermined number of keywords ranked in order of the frequency of selection by user U. Keywords that become candidates for extraction are, for example, keywords that will be input to the first learning model.

[0043] When user U specifies one or more keywords, terminal device 4 accepts the one or more keywords specified by user U and identifies hair images corresponding to those one or more keywords using the first learning model (step S4).

[0044] In step S4, the terminal device 4 inputs one or more keywords specified by user U into the first learning model and identifies hair images corresponding to the one or more keywords specified by user U based on the information output from the first learning model.

[0045] For example, when the first learning model outputs a hair image, terminal device 4 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by user U.

[0046] Furthermore, when the first learning model outputs hair image identification information, terminal device 4 acquires the hair image identification information output from the first learning model and retrieves the hair image identified by the acquired hair image identification information from its internal storage unit or information providing device 1. In this case as well, terminal device 4 identifies the acquired hair image as the hair image corresponding to one or more keywords specified by user U.

[0047] Next, terminal device 4 acquires multiple similar hair images, which are images containing hair images similar to the hair image identified in step S4, and displays the acquired similar hair images on the display unit (step S5). In step S5, terminal device 4 transmits a similar image search request containing the hair image identified in step S4 to a similar image search server or information providing device 1 (not shown).

[0048] The similar image search server or information provider 1 searches for multiple similar hair images based on the hair image included in the similar image search request from the terminal device 4, and transmits one or more similar hair images obtained from the search to the terminal device 4. The terminal device 4 retrieves the multiple similar hair images transmitted from the similar image search server or information provider 1 in response to the similar image search request.

[0049] In this way, the terminal device 4 displays multiple similar hair images on its display unit based on one or more keywords specified by the user U. This allows the user U to view the hair images under consideration simply by specifying keywords.

[0050] In the example described above, user U selects one or more keywords from among several keywords presented by terminal device 4, but user U's selection of one or more keywords is not limited to this example. For example, user U can also select one or more keywords by operating terminal device 4 and inputting them into terminal device 4, or by inputting one or more keywords into terminal device 4 by voice.

[0051] In addition, User U can consider hairstyles based on their facial features, either as an alternative to or in addition to keyword searches. The following describes how User U can consider hairstyles based on their facial features.

[0052] The terminal device 4 determines the facial features of user U based on the user-captured image specified by user U (step S6). The user-captured image is an image of user U's head, including their hair and face, captured by an imaging unit (not shown) of the terminal device 4 or an imaging unit (not shown) of another device, and is an example of a user image. The terminal device 4 acquires the user-captured image, for example, based on an operation by user U.

[0053] For example, user U can operate terminal device 4 to capture images of user U's head, including hair and face, using the imaging unit of terminal device 4, thereby causing terminal device 4 to acquire multiple user-captured images. User U can then operate terminal device 4 to select a desired user-captured image from among the multiple user-captured images acquired by terminal device 4.

[0054] In step S6, the terminal device 4 determines, for example, the facial features of user U shown in the user image from among a predetermined set of multiple types of facial features. For example, the terminal device 4 inputs the user image into the second learning model described above and determines the facial features of user U based on the facial feature information output from the second learning model.

[0055] Furthermore, the terminal device 4 may have information indicating each of several types of facial features. In this case, the terminal device 4 determines the facial features of user U by pattern matching technology between the information indicating each type of facial feature and the information indicating the facial features of user U shown in the user image.

[0056] Furthermore, the terminal device 4 can display facial feature line drawings, which are line drawings representing various types of facial features (e.g., face shapes), on its display unit, and allow user U to select the facial feature line drawing that most closely matches their own face shape from among these various types of facial feature line drawings.

[0057] Next, terminal device 4 uses the third learning model described above to identify a hairstyle that is estimated to suit user U's face based on the facial features of user U determined in step S6 (step S7). As described above, the third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suiting that face.

[0058] In step S7, terminal device 4 inputs information representing the facial features of user U, determined in step S6, into the third learning model, and identifies a hairstyle that is estimated to suit user U's face based on the hairstyle information output from the third learning model. The hairstyle information is a hair image, and terminal device 4 identifies the hair image output from the third learning model as a hairstyle that is estimated to suit user U's face.

[0059] Next, the terminal device 4 generates a composite image by combining the user face image, which is the face image of user U, and the hair image selected by user U, and displays the generated composite image on the display unit (step S8). The user face image is, for example, the user captured image or an image included in the user captured image described above.

[0060] The hair image selected by user U is, for example, one or more images selected by user U from among the images of the hairstyle identified in step S8 or from multiple similar hair images obtained by keyword search.

[0061] Terminal device 4 accepts the selection of one or more hair images by user U from a specific hair image, which is an image of the hairstyle identified in step S8, or from multiple similar hair images obtained by keyword search. Terminal device 4 can also accept the selection of one or more hair images by user U from multiple hair images contained in, for example, multiple web pages viewed by user U.

[0062] When user U selects two or more hair images, terminal device 4 generates a composite image by combining the composite hair image obtained by combining these two or more hair images with the user's face image.

[0063] The hair image selected by user U may be, for example, an image captured by an imaging unit (not shown) in terminal device 4. Such an image may be, for example, an image of a photograph of a person's face and hair published in a magazine. In this case, terminal device 4 generates a composite image by, for example, replacing the face image of a person included in the image selected by user U from the images captured by terminal device 4 with the face image of user U included in the user face image.

[0064] Furthermore, in images taken from photographs published in magazines, the faces of the people in the images may not be facing forward. In this case, terminal device 4 generates a composite image by projecting the captured image in a direction that makes the people's faces face forward and replacing the faces in the captured image with the user's face image. Also, if multiple captured images are selected by user U, terminal device 4 generates a composite image by combining the multiple captured images to obtain a composite image and replacing the faces in the composite image with the user's face image.

[0065] Furthermore, if the hair image selected by user U from among multiple hair images is changed based on user U's operation, terminal device 4 generates a modified composite image, which is a composite image created by combining the hair image selected by user U's operation with the modified composite image, and displays the generated modified composite image on the display unit.

[0066] For example, based on a first modification operation by user U, terminal device 4 generates a corresponding modified composite image each time the hair image selected by user U from among multiple hair images is changed, and displays the generated modified composite image on the display unit. The first modification operation is, for example, a left-right scrolling operation or a left-right swipe operation.

[0067] Furthermore, terminal device 4 can also change at least one of the hair length and color of the hair image included in the composite image based on a second modification operation by user U. Terminal device 4 changes at least one of the hair length and color of the hair image included in the composite image each time the second modification operation is performed. The first modification operation is, for example, a vertical scroll operation or a vertical swipe operation.

[0068] Furthermore, if the terminal device 4 receives a request from user U to process a hair image, it can also generate a processed hair image by applying the requested processing to the hair image. The processing of the hair image may involve changing at least one of the hair length and color of the hair image. In this case, in step S8, the terminal device 4 generates a composite image by combining the user's face image and the processed hair image, and displays the generated composite image on the display unit.

[0069] Next, terminal device 4 uses the fourth learning model described above to estimate the treatment content from the pre-treatment image of user U and the composite image generated in step S8 (step S9). The pre-treatment image of user U is, for example, the user image described above. The treatment content estimated in step S9 is the treatment content necessary to achieve the hairstyle shown in the composite image generated in step S8, and may be referred to as the estimated treatment content below.

[0070] In step S9, when the terminal device 4 inputs the pre-treatment image of user U and the composite image generated in step S8 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

[0071] Furthermore, terminal device 4 determines the treatment cost, which is the cost required for the treatment based on the determined estimated treatment content (step S10). For example, terminal device 4 can determine the treatment cost for each hair salon based on treatment cost information for each hair salon that shows the relationship between the treatment content and the treatment cost. Terminal device 4 can obtain treatment cost information for each hair salon from, for example, information providing device 1.

[0072] Furthermore, when terminal device 4 receives a specification of treatment content from user U, it compares the user-specified treatment content (the treatment content that was accepted) with the estimated treatment content and generates comparison information showing the result of the comparison (step S11). The comparison information includes, for example, information showing the difference between the user-specified treatment content and the estimated treatment content. For example, if the estimated treatment content has one more bleach than the user-specified treatment content, terminal device 4 generates comparison information that includes information indicating that the number of bleaches is one more than in the user-specified treatment content.

[0073] Next, terminal device 4 displays treatment-related information on its display unit, including information indicating the treatment content estimated in step S9, treatment cost information which is information on the treatment cost determined in step S10, and comparison information generated in step S11 (step S12). This allows user U to understand the treatment content and treatment cost estimated to be necessary to achieve the hairstyle shown in the selected hair image, and terminal device 4 can improve user U's convenience.

[0074] Next, terminal device 4 receives an acceptance / rejection response, which is a response from user U indicating acceptance or rejection of the treatment content estimated in step S9 (step S13). If user U accepts the treatment content estimated in step S9, they operate terminal device 4 to give an acceptance response. If user U does not accept the treatment content estimated in step S9, they operate terminal device 4 to give an rejection response.

[0075] Next, if the received permission response indicates permission, the terminal device 4 outputs a reservation request to the hair stylist that includes a specific image and treatment content information (step S14). The specific image includes, for example, a composite image generated in step S8 or a hair image selected by user U, and a user-captured image that includes user U's hair image before treatment. The treatment content information includes at least one of the information indicating the treatment content specified by user U and the information indicating the treatment content estimated in step S9.

[0076] The aforementioned reservation request is output to the hair stylist's store device 5 via, for example, the information providing device 1, but it may also be output to the hair stylist's store device 5 via SNS or an unillustrated mail server. The hair stylist's store device 5 receives the reservation request output from the terminal device 4 and displays the specific image and treatment details information included in the received reservation request. This allows the hair stylist to confirm the specific image and treatment details information included in the reservation request.

[0077] The hair stylist can operate the store device 5 to modify specific images included in the reservation request and modify the treatment details indicated in the treatment details information. In this case, the store device 5 outputs modification information to the user U, including the modified specific image and the modified treatment details information. The modification information is output to the terminal device 4 via, for example, the information provision device 1, but it may also be output to the terminal device 4 via SNS or an unillustrated mail server.

[0078] A hair stylist can operate the store device 5 to input a comment indicating whether they can perform the hairstyle shown in the composite image included in the reservation request. In this case, the store device 5 outputs a stylist comment, which is a comment indicating whether they can perform the hairstyle shown in the composite image, to user U. The comment is output to the terminal device 4, for example, via the information provision device 1, but may also be output to the terminal device 4 via SNS or an unillustrated mail server.

[0079] Furthermore, if the hair stylist does not need to modify a specific image included in the reservation request or modify the treatment details indicated in the treatment details information included in the reservation request, they operate the store device 5 to output reservation confirmation information to user U from the store device 5, indicating that the reservation has been accepted.

[0080] When terminal device 4 receives correction information and practitioner comments from store device 5, it displays the acquired correction information and practitioner comments on the display unit (step S15). This allows user U to check the correction suggestions and comments from the hair stylist, facilitating communication between the hair stylist and the user.

[0081] As described above, terminal device 4, an example of an information processing device, generates a composite image by combining the user's face image (user face image) and the hair image selected by user U, and outputs the generated composite image to the hair stylist. This allows user U to present the hair stylist with a composite image that combines the desired hair image and user face image, enabling them to communicate their desired hairstyle more specifically to the hair stylist than simply telling them their desired hairstyle vaguely. Therefore, terminal device 4 can improve the convenience of user U.

[0082] Furthermore, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, and a hair image selected by user U. Using a learning model that has learned the relationship between hair images before and after treatment and the treatment content, it estimates the treatment content from user U's pre-treatment image and the composite image. As a result, for example, user U can understand the treatment content required to achieve their desired hairstyle, and can also specifically communicate the treatment content to the hair stylist. Therefore, terminal device 4 can improve the convenience of user U.

[0083] Furthermore, terminal device 4, which is an example of an information processing device, accepts the specification of one or more keywords by user U and uses a learning model that takes one or more keywords as input and outputs hair images to identify hair images from the one or more keywords received by the reception unit. Then, terminal device 4 acquires similar hair images, which are images that include hair images similar to the identified hair image. As a result, user U can easily collect hair images corresponding to the desired hairstyle simply by specifying keywords. Therefore, terminal device 4 can improve the convenience of user U.

[0084] Furthermore, terminal device 4, which is an example of an information processing device, uses a learning model, which is a model that has learned the relationship between information indicating facial features and hairstyles evaluated as suitable for the face, to identify hairstyles that are estimated to suit user U's face, and acquires posted images that include the identified hairstyle image. Then, terminal device 4 generates a composite image by combining the image of the hairstyle identified by the identification unit with user U's face image, based on the posted image acquired by the acquisition unit and user U's face image. As a result, user U can easily obtain an image of what it would look like if a hairstyle estimated to suit user U's face were applied to themselves. Therefore, terminal device 4 can improve user U's convenience.

[0085] Furthermore, although the above example describes an example in which the terminal device 4 performs various processes, some or all of the processing performed by the terminal device 4 may be performed by the information providing device 1. For example, the information providing device 1 may provide information generated based on information input from the terminal device 4 via an interface such as an API (Application Programming Interface), and the terminal device 4 may display the above information based on the information provided by the information providing device 1. Also, although Figure 1 shows a case where the terminal device 4 and the information providing device 1 are separate devices, the terminal device 4 and the information providing device 1 may be an integrated information processing device.

[0086] The configuration of the information processing system, including the information providing device 1, terminal device 4, and store device 5 that perform such processing, will be described in detail below.

[0087] [2. Configuration of the Information Processing System] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. As shown in Figure 2, the information processing system 100 according to the embodiment includes an information providing device 1, a plurality of terminal devices 4, and a plurality of store devices 5.

[0088] Each of the multiple terminal devices 4 is used by a different user U. Each of the multiple store devices 5 is installed in, for example, different hair salons and used by hair stylists at different hair salons.

[0089] The information provision device 1, terminal device 4, and store device 5 are connected to each other via a network N, either by wire or wireless, enabling communication between them. Note that the information processing system 100 shown in Figure 2 may include multiple information provision devices 1. Network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0090] Information provider device 1 is an information processing device that provides various types of information. Information provider device 1 provides online services such as SNS sites, web search sites, and content distribution sites. SNS sites receive posted information transmitted from terminal devices 4 and store devices 5, and transmit the posted information to terminal devices 4 and store devices 5 based on requests from users U, hair stylists, etc. Alternatively, information provider device 1 may be an information processing device that acquires information such as posted information from the aforementioned SNS sites, etc.

[0091] Furthermore, a content distribution site is a site that provides content distribution services, and includes video and music distribution sites, map sites, route search sites, route guidance sites, route information sites, train service information sites, weather forecast sites, etc. In addition, the information providing device 1 can also provide online services through various online sites, such as electronic payment sites, online game sites, online banking sites, or accommodation and ticket reservation sites.

[0092] Each of the terminal device 4 and the store device 5 may be, for example, a desktop PC (Personal Computer), a notebook PC, a tablet device, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant). However, each of the terminal device 4 and the store device 5 is not limited to the above examples and may also be, for example, a smartwatch or a wearable device.

[0093] Furthermore, each terminal device 4 can connect to the network N via wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), or via short-range wireless communication such as Bluetooth (registered trademark) and Wi-Fi, and communicate with the information provision device 1 and each store device 5.

[0094] Furthermore, each store device 5 can connect to the network N via wireless communication networks such as LTE, 4G, and 5G, or short-range wireless communication such as Bluetooth and Wi-Fi, and communicate with the information provision device 1 and each terminal device 4.

[0095] [3. Configuration of Information Provisioning Device 1] Next, an example of the configuration of the information providing device 1 will be described with reference to Figure 3. Figure 3 is a diagram showing an example of the configuration of the information providing device 1 according to an embodiment. As shown in Figure 3, the information providing device 1 has a communication unit 10, a storage unit 11, and a processing unit 12.

[0096] [3.1. Communications Section 10] The communication unit 10 is implemented, for example, by a NIC (Network Interface Card). The communication unit 10 is connected to the network by wire or wireless and transmits and receives information with terminal devices 4 and store devices 5 via the network N.

[0097] [3.2. Storage section 11] The memory unit 11 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The memory unit 11 stores user information storage unit 20, posting information storage unit 21, model information storage unit 22, and content storage unit 23.

[0098] [3.2.1. User Information Storage Unit 20] The user information storage unit 20 stores information about user U. Figure 4 shows an example of a user information table stored in the user information storage unit 20 of the information providing device 1 according to this embodiment.

[0099] As shown in Figure 4, the user information table stored in the user information storage unit 20 includes items such as "User ID (IDentifier)", "Attribute Information", and "History Information". "User ID" is identification information that identifies user U.

[0100] "Attribute information" refers to user attributes, which are the attributes of user U corresponding to the "User ID," and includes psychographic attribute information and demographic attribute information. Demographic attributes include, for example, gender, age, place of residence, and occupation, while psychographic attributes include interests such as travel, clothing, cars, and religion, as well as lifestyle, thoughts, and ideological tendencies.

[0101] "History information" refers to history information that includes information such as the service usage history of User U corresponding to "User ID," and includes, for example, User U's posting history information, User U's search history information, and User U's browsing history information. User U's posting history information includes information posted by User U on SNS sites, User U's comments on other users' posts, and information indicating User U's evaluation of other users' posts.

[0102] User U's search history information includes, for example, search history information for posts on social networking sites and search history information on various websites. User U's browsing history information includes, for example, User U's browsing history information for posts on social networking sites and browsing history information on various websites.

[0103] [3.2.2. Submission Information Storage Unit 21] The posting information storage unit 21 stores various types of information posted by each user U. Figure 5 shows an example of a posting information table stored in the posting information storage unit 21 of the information providing device 1 according to this embodiment. As shown in Figure 5, the posting information table stored in the posting information storage unit 21 includes items such as "post ID", "post date and time", "posting user", "post text", "post image", "rating information", "number of views", and "comments".

[0104] "Post ID" is an identifier that identifies the posted information. Post information is an example of posted content. "Posting date and time" is information indicating the posting date and time of the post information corresponding to the "Post ID". "Posting user" is information indicating the user U who posted the post information corresponding to the "Post ID", and in the example shown in Figure 5, it is the user ID. "Post text" is information indicating the text contained in the post information corresponding to the "Post ID", and is shown as a string, but may also include stamps, etc.

[0105] "Posted image" refers to the posted image included in the post information corresponding to the "Post ID". "Rating information" refers to the rating of other users U for the post information corresponding to the "Post ID", but may also include information indicating the rating of the posting user for the post information corresponding to the "Post ID". "View count" refers to the number of times the post information corresponding to the "Post ID" has been viewed.

[0106] A "comment" is a comment made by another user U on the post information corresponding to the "post ID," and is represented as a string, for example, but may also include stamps, etc. Note that a "comment" may also include a comment from the original poster on the post information corresponding to the "post ID."

[0107] The evaluation information shown in Figure 5 includes information indicating positive evaluations (e.g., "good") and negative evaluations (e.g., "bad"), but may also include information indicating intermediate evaluations (e.g., "average").

[0108] [3.2.3. Model Information Storage Unit 22] The model information storage unit 22 stores information on various learning models. Figure 6 shows an example of a model information table stored in the model information storage unit 22 of the information providing device 1 according to this embodiment.

[0109] As shown in Figure 6, the model information table stored in the model information storage unit 22 includes items such as "Model ID" and "Model Information". "Model ID" is identification information that identifies the learning model. "Model Information" is information about the learning model corresponding to "Model ID". The learning model information is information such as the parameters of the learning model, but is not limited to this example; any information that allows the terminal device 4 to use the learning model is acceptable.

[0110] The learning models whose information is included in the model information table include, for example, the first learning model, the second learning model, the third learning model, and the fourth learning model described above. The first learning model is a model that has learned the relationship between keywords and hair images, and is a learning model that takes one or more keywords as input and outputs a hair image or hair image identification information.

[0111] The second learning model is a model that has learned the relationship between face images and facial features. It takes an image containing a face as input and outputs information indicating facial features. The third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suitable for that face. It takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit that face.

[0112] The fourth learning model is one that has learned the relationship between hair images before and after treatment and the treatment content. It takes hair images before and after treatment as input and outputs information indicating the estimated treatment content. Note that the learning models in which information is included in the model information table are not limited to the examples described above.

[0113] The learning models described above can be of any type. For example, each learning model is a model generated by machine learning, such as DNN, GBDT, or SVM. A DNN is, for example, a CNN or RNN. An RNN may also be an LSTM. Furthermore, each learning model may be a model realized by combining multiple models, such as a model that combines a CNN and an RNN.

[0114] [3.2.4. Content Storage Unit 23] The content storage unit 23 contains information about the content. Figure 7 shows an example of a content table stored in the content storage unit 23 of the information providing device 1 according to this embodiment. The content table shown in Figure 7 includes items such as "Content ID" and "Content".

[0115] A "Content ID" is an identifier that identifies content. "Content" is information about the content associated with the "Content ID". Specifically, information about the content includes, for example, information about the content's content or information about the content's address.

[0116] In the example shown in Figure 7, the content with content ID "C1" is content CNT1, the content with content ID "C2" is content CNT2, and the content with content ID "C3" is content CNT3. Note that the content storage unit 23 is not limited to the above and may store various types of information depending on the purpose.

[0117] [3.3. Processing Unit 12] The processing unit 12 is a controller, and is realized by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information providing device 1, using RAM or the like as a working area.

[0118] Furthermore, the processing unit 12 is a controller, and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or GPGPU (General Purpose Graphic Processing Unit).

[0119] As shown in Figure 3, the processing unit 12 comprises an acquisition unit 30, a learning unit 31, and a providing unit 32, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the processing unit 12 is not limited to the configuration shown in Figure 3, and other configurations are also acceptable as long as they perform the information processing described later.

[0120] [3.3.1. Acquisition part 30] The acquisition unit 30 acquires various types of information. The acquisition unit 30 acquires various types of information from the storage unit 11. The acquisition unit 30 acquires various types of information from the user information storage unit 20, the posting information storage unit 21, the model information storage unit 22, and the content storage unit 23, etc.

[0121] Furthermore, the acquisition unit 30 acquires information about user U (for example, user U's attribute information and behavioral history information) from, for example, terminal device 4, store device 5, or external device, and stores the acquired user U information in the user information storage unit 20. In addition, the acquisition unit 30 acquires various types of content from, for example, external device, and stores the acquired content in the content storage unit 23.

[0122] Furthermore, the acquisition unit 30 acquires posting information from devices such as terminal devices 4 or store devices 5 via the communication unit 10. The acquisition unit 30 adds the acquired posting information to the posting information table stored in the posting information storage unit 21.

[0123] Furthermore, the acquisition unit 30 acquires acquisition requests regarding posted information from the terminal device 4 via the communication unit 10. For example, the acquisition unit 30 notifies the provision unit 32 of the information regarding the acquired acquisition requests.

[0124] Furthermore, the acquisition unit 30 acquires various information such as reservation requests from the terminal device 4 via the communication unit 10, stores the acquired information such as reservation requests in the storage unit 11, and notifies the provision unit 32 of the acquired information such as reservation requests. The acquisition unit 30 also acquires various information such as correction information from the store device 5 via the communication unit 10, and notifies the provision unit 32 of the acquired information such as correction information.

[0125] Furthermore, the acquisition unit 30 acquires training data for generating the learning model described above and notifies the learning unit 31 of the acquired training data. For example, the acquisition unit 30 acquires information stored in the storage unit 11 from the storage unit 11 and notifies the learning unit 31 of the acquired information as training data.

[0126] The training data for the first learning model includes, for example, multiple combinations of keywords and hair images. The training data for the second learning model includes, for example, multiple combinations of images including face images and information indicating facial features.

[0127] The training data for the third learning model is, for example, training data that includes multiple combinations of information indicating facial features, hair images, and information indicating evaluations of the combination of facial feature information and hair images. The evaluations of the combinations of facial feature information and hair images are, for example, information indicating positive evaluations or negative evaluations, and are used as labels. The training data for the fourth learning model is, for example, training data that includes multiple combinations of hair images before and after treatment and information indicating the treatment content.

[0128] Furthermore, the acquisition unit 30 can also generate the aforementioned training data based on the information stored in the storage unit 11. In addition, the acquisition unit 30 can acquire and generate training data other than those described above.

[0129] [3.3.2. Learning Section 31] The learning unit 31 generates various learning models through machine learning using various training data notified by the acquisition unit 30, and stores information about the generated learning models in the posted information storage unit 21.

[0130] For example, the learning unit 31 generates a first learning model using training data for the first learning model, and generates a second learning model using training data for the second learning model. Furthermore, the learning unit 31 generates a third learning model using training data for the third learning model, and generates a fourth learning model using training data for the fourth learning model.

[0131] Furthermore, the learning unit 31 can also generate learning models other than the first learning model, the second learning model, the third learning model, and the fourth learning model.

[0132] [3.3.3.Providing Department 32] The provisioning unit 32 activates the communication unit 10 and provides various content to the user U's terminal device 4 and the hair stylist's store device 5. For example, based on requests from the terminal device 4 and the store device 5, the provisioning unit 32 generates browsing content including SNS posting information and provides such browsing content to the terminal device 4 and the store device 5 via the communication unit 10 and the network N, etc.

[0133] Furthermore, the providing unit 32 provides various types of information notified by the acquisition unit 30 to terminal devices 4 or store devices 5, etc., via the communication unit 10 and network N, etc. For example, the providing unit 32 provides information from terminal devices 4 notified by the acquisition unit 30 to store devices 5, etc., via the communication unit 10 and network N, etc., or provides information from store devices 5 notified by the acquisition unit 30 to terminal devices 4, etc., via the communication unit 10 and network N, etc.

[0134] Furthermore, the providing unit 32 provides information on various learning models generated by the learning unit 31 to the terminal device 4 via the communication unit 10 and the network N, etc. For example, the providing unit 32 provides the first learning model, second learning model, third learning model, and fourth learning model generated by the learning unit 31 to the terminal device 4 via the communication unit 10 and the network N, etc.

[0135] Furthermore, the supply unit 32 provides the terminal device 4 with an application program (hereinafter referred to as "app") to be used in the terminal device 4 via the communication unit 10 and the network N, etc. Such an app is installed in the terminal device 4 and executes the processes described above in the terminal device 4. For example, depending on the functions of such an app, the terminal device 4 can generate the composite image described above or acquire various types of information using the learning model described above.

[0136] [4. Terminal device 4] Figure 8 shows an example of the configuration of a terminal device 4 according to the embodiment. As shown in Figure 8, the terminal device 4 according to the embodiment includes a communication unit 40, a display unit 41, an operation unit 42, an imaging unit 43, a sensor unit 44, a storage unit 45, and a processing unit 46.

[0137] [4.1. Communications Section 40] The communication unit 40 is implemented by, for example, a NIC. The communication unit 40 is connected to the network N by wire or wireless connection and transmits and receives information with the information provision device 1 and the store equipment 5 via the network N. Communication with the store equipment 5 is performed via, for example, the information provision device 1, SNS, or an unillustrated mail server, but direct communication with the store equipment 5 is also possible without going through these.

[0138] [4.2.Display section 41] The display unit 41 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0139] [4.3. Operation unit 42] The control unit 42 includes, for example, a keyboard with keys for entering letters, numbers, and spaces, an enter key and arrow keys, a mouse, and a power button. If the display unit 41 is a touch panel compatible display, the control unit 42 includes a touch panel.

[0140] [4.4. Imaging Unit 43] The imaging unit 43 is an image sensor (camera) that captures images of the subject. For example, the imaging unit 43 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor. Note that the imaging unit 43 is not limited to an internal camera; it may also be an external camera such as a wireless camera capable of communicating with the terminal device 4 or a webcam.

[0141] [4.5. Sensor section 44] The sensor unit 44 includes a position detection unit and a gyro sensor, etc. The position detection unit, for example, detects the position of the terminal device 4, which is the current position of user U, and outputs information indicating the detected current position of user U to the processing unit 46.

[0142] The position detection unit receives multiple positioning signals transmitted from multiple positioning satellites in the GNSS (Global Navigation Satellite System) and detects the current position of user U based on the received multiple positioning signals. The gyro sensor is a sensor that detects the attitude of the terminal device 4, such as its tilt and rotation.

[0143] [4.6. Storage section 45] The memory unit 45 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs.

[0144] The memory unit 45 stores information such as information transmitted from the information providing device 1 and the store device 5 and acquired by the processing unit 46 via the network N and the communication unit 40, images captured by the imaging unit 43, and detection information detected by the sensor unit 44. For example, the memory unit 45 stores information on applications and various learning models transmitted from the information providing device 1.

[0145] [4.7. Processing Unit 46] The processing unit 46 is a controller and is implemented, for example, by a CPU or MPU executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the terminal device 4 using RAM as a working area. The processing unit 46 may be partially or entirely implemented by an integrated circuit such as an ASIC or FPGA.

[0146] The processing unit 46 comprises a reception unit 50, an acquisition unit 51, a identification unit 52, an image processing unit 53, an estimation unit 54, a determination unit 55, a comparison unit 56, and an output unit 57. The processing unit 46 functions as a functional unit including the reception unit 50, acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, determination unit 55, comparison unit 56, and output unit 57, for example, by executing the above-mentioned application on the OS (Operating System).

[0147] Multiple functions, including the reception unit 50, acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, determination unit 55, comparison unit 56, and output unit 57, may be pre-programmed in the processing unit 46.

[0148] [4.7.1. Reception Desk 50] The reception unit 50 receives various requests and information. The reception unit 50 receives requests and information transmitted from terminal devices 4 and store devices 5.

[0149] For example, the reception unit 50 receives correction information transmitted from the store device 5. The correction information is information indicating the correction proposed by the hair stylist. The reception unit 50 also receives reservation confirmation information transmitted from the store device 5. The reservation confirmation information includes information indicating that a reservation has been accepted based on the reservation request, such as the details of the treatment.

[0150] Furthermore, the reception unit 50 accepts various operations from user U using the operation unit 42, etc. For example, the reception unit 50 accepts various operations such as user U selecting a hair image, user U processing a hair image, user U specifying the treatment content, user U approving or rejecting the estimated treatment content, and user U specifying the treatment content.

[0151] Furthermore, the reception unit 50 receives imaging operations from the user U. When the reception unit 50 receives an imaging operation from the user U, it operates the imaging unit 43 to cause the imaging unit 43 to perform the imaging operation.

[0152] Furthermore, the reception unit 50 receives requests from user U for a first modification operation of the hair image and a second modification operation of the hair image by user U. The first modification operation is an operation to change the composite image by changing the hair image used to generate the composite image. The second modification operation is an operation to change at least one of the hair length and color of the hair image.

[0153] Figure 9 shows an example of the configuration of the reception unit 50 of the terminal device 4 according to the embodiment. As shown in Figure 9, the reception unit 50 includes a selection reception unit 60, a processing reception unit 61, a designation reception unit 62, a revised proposal reception unit 63, and an approval / rejection reception unit 64.

[0154] The selection reception unit 60 accepts various selections from the user U. For example, the selection reception unit 60 accepts the user U's selection of a hair image from among several types of hair images. The user U can perform a hair image selection operation by operating the operation unit 42, and the selection reception unit 60 accepts the hair image selection operation.

[0155] The hair image selected by user U is, for example, one or more hair images from among images of hairstyles identified by the identification unit 52, multiple similar hair images acquired by the acquisition unit 51, and hair images included in captured images captured by the imaging unit 43.

[0156] The hair images selected by user U are not limited to the examples described above. For example, the hair images selected by user U may include hair images selected by the processing unit 46 when the automatic hair image selection mode by the processing unit 46 is selected by user U's operation.

[0157] The processing reception unit 61 accepts the processing of the hair image. For example, the processing reception unit 61 accepts the selection of at least one of the hair length and color of the hair image. The hair in the hair image is the hair shown in the hair image, and the user U can perform processing operations on the hair image by operating the operation unit 42, and the processing reception unit 61 accepts the processing of the hair image based on the processing operations on the hair image.

[0158] The designation reception unit 62 accepts various designations from user U. For example, the designation reception unit 62 accepts designations of treatment content from user U. User U can perform a designation operation on the operation unit 42, and the designation reception unit 62 accepts the designation of treatment content based on the designation operation.

[0159] Furthermore, the designation reception unit 62 accepts the designation of a user-captured image, which is an image captured by the imaging unit 43 and includes a hair image, based on the user U's operation of the operation unit 42.

[0160] Furthermore, the designated reception unit 62 receives the user U's approval or rejection of the estimated treatment content. User U can approve or reject the estimated treatment content by operating the operation unit 42, and the designated reception unit 62 accepts the approval or rejection of the treatment content based on the approval or rejection operation.

[0161] Furthermore, the specification reception unit 62 accepts the specification of one or more keywords by user U. For example, the specification reception unit 62 accepts the specification of one or more keywords by user U from among multiple keywords. User U can perform the operation of specifying one or more keywords by operating the operation unit 42, and the specification reception unit 62 accepts the specification of one or more keywords based on the operation of specifying one or more keywords.

[0162] The revision proposal reception unit 63 receives revision proposals from hair stylists regarding the estimated treatment content. For example, the revision proposal reception unit 63 receives revision proposal information from the store device 5 as revision proposals for the estimated treatment content from hair stylists, which is information indicating revision proposals from hair stylists regarding the estimated treatment content.

[0163] Furthermore, the revision suggestion receiving unit 63 receives revision suggestions from hair stylists for hairstyles shown in composite images. For example, the revision suggestion receiving unit 63 receives revision suggestion information, which is information indicating revision suggestions from hair stylists, from the store device 5 as revision suggestions for hairstyles from hair stylists.

[0164] The approval / denial reception unit 64 receives the approval or denial of the estimated treatment content, which is the treatment content estimated by the estimation unit 54, from the user U. The user U can perform an approval operation to indicate approval of the estimated treatment content or a denial operation to indicate denial of approval of the estimated treatment content by operating the operation unit 42, and the approval / denial reception unit 64 receives the approval or denial of the estimated treatment content from the user U based on the approval or denial operation.

[0165] [4.7.2. Acquisition part 51] The acquisition unit 51 acquires various types of information. For example, the acquisition unit 51 acquires various types of information from the storage unit 45. The acquisition unit 51 acquires information about the learning model stored in the storage unit 45.

[0166] Furthermore, the acquisition unit 51 acquires the captured image, which is the image captured by the imaging unit 43, and stores the acquired captured image in the storage unit 45. In addition, the acquisition unit 51 acquires the detection information, which is the information detected by the sensor unit 44, and stores the acquired detection information in the storage unit 45.

[0167] The acquisition unit 51 acquires information such as applications and learning models transmitted from the information providing device 1 via the network N and the communication unit 40, and stores the acquired information such as applications and learning models in the storage unit 45. The acquisition unit 51 acquires multiple posts containing specific hair images from the information providing device 1, and stores the acquired multiple posts in the storage unit 45.

[0168] Furthermore, the acquisition unit 51 acquires the practitioner's comments and stores the acquired comments in the storage unit 45. The practitioner's comments are comments from the hair stylist indicating whether or not it is possible to perform the treatment on the hairstyle shown in the composite image.

[0169] Furthermore, the acquisition unit 51 acquires information corresponding to the user U's operation of the operation unit 42. For example, based on the user U's operation, the acquisition unit 51 acquires an image captured by the imaging unit 43, including a hair image, from the storage unit 45.

[0170] Furthermore, the acquisition unit 51 acquires multiple similar hair images, which are images containing hair images similar to the specific hair image, and stores the acquired multiple similar hair images in the storage unit 45. The specific hair image is an image of a hairstyle identified by the identification unit 52. For example, the acquisition unit 51 sends a similar image search request containing the specific hair image to a search server or information providing device 1 (not shown), and acquires multiple similar hair images sent from the search server or information providing device 1 in response to the similar image search request.

[0171] Furthermore, the acquisition unit 51 transmits a search request to a search server or information providing device 1 (not shown) that includes information indicating the hairstyle identified by the identification unit 52, and acquires a plurality of specific hair images transmitted from the search server or information providing device 1 in response to the search request.

[0172] Furthermore, the acquisition unit 51 acquires treatment cost information for each hair salon, which shows the relationship between the treatment content and the treatment cost, from, for example, the information provision device 1 or the store device 5.

[0173] [4.7.3. Specification part 52] The identification unit 52 uses the first learning model to identify a hair image from one or more keywords specified by the reception unit 50. As described above, the first learning model is a model that has learned the relationship between keywords and hair images, and takes one or more keywords as input and outputs a hair image or hair image identification information.

[0174] When the first learning model outputs a hair image, the identification unit 52 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by user U.

[0175] Furthermore, if the first learning model outputs hair image identification information instead of hair images, the identification unit 52 inputs one or more keywords specified by user U into the first learning model and obtains the hair image identification information output from the first learning model. The hair image identification information is information that identifies a hair image, for example, a score for each hair image. For example, the identification unit 52 identifies the hair image with the highest score among the scores for each hair image as the hair image corresponding to one or more keywords specified by user U.

[0176] The identification unit 52 acquires the hair image identified by the image identification information from the storage unit 45 or the information providing device 1, etc., and identifies the acquired hair image as the hair image corresponding to one or more keywords specified by the user U.

[0177] Furthermore, the identification unit 52 uses a third learning model to identify hairstyles that are estimated to suit user U's face. For example, the identification unit 52 uses a third learning model to identify hairstyles that are estimated to suit user U's face based on the facial features of user U specified by user U or the facial features of user U determined by the determination unit 55.

[0178] The third learning model, as described above, is a model that learns the relationship between information representing facial features and hairstyles that are evaluated as suitable for that face. It takes information representing facial features as input and outputs information representing hairstyles that are estimated to suit that face.

[0179] The identification unit 52 inputs information representing the facial features of user U into the third learning model and identifies a hairstyle estimated to suit user U's face based on the hairstyle information output from the third learning model. The hairstyle information output from the third learning model is, for example, a score for each hair image, and the identification unit 52 identifies the hairstyle represented by the hair image with the highest score among the scores for each hair image as the hairstyle estimated to suit user U's face. Alternatively, the hairstyle information output from the third learning model may also be a hair image. The information representing the facial features of user U is, for example, information representing the facial features of user U specified by user U or information representing the facial features of user U determined by the determination unit 55.

[0180] Facial features may include, for example, information indicating the type of face shape, but may also include information indicating the characteristics of one or more of the facial features, such as the size, shape, and position of the eyebrows, eyes, nose, mouth, and ears. For example, information indicating facial features may include information indicating the position of each of the eyebrows, eyes, nose, mouth, and ears on the face, or it may include information indicating the position and size of each of the eyebrows, eyes, nose, mouth, and ears on the face.

[0181] [4.7.4. Image Processing Unit 53] The image processing unit 53 performs various image processing operations. For example, the image processing unit 53 generates a composite image by combining the user face image, which is the face image of user U, and the hair image selected by user U. In the following, the hair image selected by user U may be referred to as the selected hair image.

[0182] The selected hair image is a hair image whose selection has been received by the selection receiving unit 60, but is not limited to this example. For example, the selected hair image may include a hair image selected by the processing unit 46 when the automatic hair image selection mode by the processing unit 46 is selected by the operation of the user U. The user face image is, for example, a user-captured image or a user-captured image of the user U included in the user-captured image whose specification has been received by the specification receiving unit 62.

[0183] For example, the image processing unit 53 generates a composite image by combining the selected hair image with the user's face image. The image processing unit 53 generates a composite image by replacing the hair image of user U included in the user's face image with the selected hair image.

[0184] The image processing unit 53 extracts, for example, the features of the user's face image and the features of the selected hair image. The image processing unit 53 generates a composite image by replacing the features of the hair image included in the user's face image with the features of the selected hair image.

[0185] Furthermore, the image processing unit 53 can also generate a composite image by deleting the hair image of user U included in the user face image and superimposing the selected hair image onto the user face image from which user U's hair image has been deleted.

[0186] Furthermore, when the image processing unit 53 generates a composite image by combining a selected image, which is an image containing a selected hair image, and a user image, which is an image containing a user face image, it can also replace the user face image included in the user image with the selected hair image.

[0187] Furthermore, if two or more hair images are selected by the selection receiving unit 60, the image processing unit 53 generates a composite hair image by combining these two or more selected hair images. Then, the image processing unit 53 generates a composite image by combining the generated composite hair image with the user's face image.

[0188] For example, the image processing unit 53 extracts feature quantities from each of two or more selected hair images and generates a composite hair image by combining the extracted feature quantities from the two or more selected hair images. For example, the image processing unit 53 generates a composite hair image by combining the feature quantities from two or more selected hair images by weighted addition, but the synthesis method is not limited to this example. For example, the image processing unit 53 may also have a learning model that synthesizes two or more images and is a synthesis model generated by machine learning, and by inputting two or more selected hair images into such a synthesis model, a composite hair image output from the synthesis model can also be obtained.

[0189] When the image processing unit 53 generates a composite image by combining a selected image, which is an image containing a selected hair image, with a user image (for example, a user-captured image), which is an image containing a user's face image, it can also replace the face images of people included in the selected image with the user's face images included in the user image.

[0190] For example, suppose the selected image is a selected captured image, which is an image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the face images of people included in the selected captured image with the user face images included in the user image.

[0191] When the image processing unit 53 generates a composite image by combining a selected image, which is an image containing a selected hair image, and a user image, which is an image containing a user face image, it can also replace the user face image included in the user image with the face image of the person included in the selected image.

[0192] For example, suppose the selected image is a selected captured image, which is an image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the user's face image in the user image with the face image of a person included in the selected captured image.

[0193] Furthermore, if there are multiple selected images, the image processing unit 53 combines the multiple selected images to generate a composite selected image. For example, the image processing unit 53 uses a specific selected image as a reference to combine multiple selected images and generate a composite selected image. The specific selected image is, for example, the first selected image or the selected image specified by user U.

[0194] For example, the image processing unit 53 extracts feature quantities from each hair image of multiple selected images, generates a composite hair image by combining the extracted feature quantities of multiple hair images, and generates a composite selected image by replacing the hair image of a specific selected image with the composite hair image. Alternatively, the image processing unit 53 may have a learning model for combining multiple images, which is a composite model generated by machine learning, and can obtain a composite hair image output from such a composite model by inputting multiple selected images into the composite model.

[0195] Furthermore, when the image processing unit 61 receives a request for processing of a hair image included in the composite image, the image processing unit 53 generates a user face image that includes the processed hair image obtained by processing the hair image received by the processing unit 61.

[0196] For example, when the image processing unit 61 receives a request for hair image processing, the image processing unit 53 generates a processed hair image obtained by processing the hair image received by the processing unit 61. Then, the image processing unit 53 generates a composite image by combining the processed hair image with the user's face image.

[0197] The image processing unit 53 generates a processed hair image by, for example, extracting feature quantities from the hair image and adjusting the extracted feature quantities, but is not limited to this example. Furthermore, if the image processing unit 53 receives a request for processing of the hair image in the composite image from the processing reception unit 61, it can also process the hair image in the composite image by extracting feature quantities from the hair image in the composite image and adjusting the extracted feature quantities.

[0198] Furthermore, the image processing unit 53 can also perform a projection transformation of the selected image. Figure 10 is a diagram showing an example of the configuration of the image processing unit 53 of the terminal device 4 according to the embodiment. As shown in Figure 10, the image processing unit 53 comprises a transformation unit 65 and a synthesis unit 66. The transformation unit 65 performs a projection transformation of the selected image. The synthesis unit 66 generates a composite image by replacing the face image of a person included in the selected image or the face image of a person included in the selected image that has been projected by the transformation unit 65 with a user face image.

[0199] Furthermore, the conversion unit 65 can also project an image of a user or a user's face in addition to the selected image. In this case, the synthesis unit 66 generates a composite image by replacing the face image of a person included in the selected image or the face image of a person included in the selected image that has been projected by the conversion unit 65 with the face image of user U included in the user image that has been projected by the conversion unit 65 or the user's face image that has been projected by the conversion unit 65.

[0200] Furthermore, the synthesis unit 66 can also generate a composite image by replacing the face image of user U included in the user image projected by the transformation unit 65 with the face image of a person included in the selected image or the face image of a person included in the selected image projected by the transformation unit 65.

[0201] [4.7.5. Estimation section 54] The estimation unit 54 estimates the treatment content from the pre-treatment image and the composite image of user U. For example, the estimation unit 54 uses a fourth learning model to estimate the treatment content from the pre-treatment image and the composite image of user U.

[0202] The fourth learning model is a learning model that has learned the relationship between hair images before and after treatment and the treatment content. It takes hair images before and after treatment as input and outputs information indicating the estimated treatment content.

[0203] The image of user U before treatment is, for example, the user image captured as described above. The treatment content estimated by the estimation unit 54 is the estimated treatment content described above, and is the treatment content necessary to achieve the hairstyle shown in the composite image generated by the image processing unit 53.

[0204] When the estimation unit 54 receives the pre-treatment image of user U and the composite image generated by the image processing unit 53 as input to the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

[0205] The information indicating the treatment content output from the fourth learning model includes, for example, scores for whether or not a haircut was performed, the amount of haircut, whether or not bleaching was done, the number of bleaching treatments, whether or not coloring was done, the type of coloring, whether or not a perm was performed, the type of perm, and whether or not hair straightening was performed. The estimation unit 54 determines the estimated treatment content based on each score output from the fourth learning model.

[0206] For example, the estimation unit 54 includes a haircut in the estimated treatment content if the score for whether or not a haircut is present is above a threshold, and excludes a haircut in the estimated treatment content if the score for whether or not a haircut is present is below the threshold. Also, the estimation unit 54 includes, for example, an amount of haircut proportional to the score for the amount of haircut in the estimated treatment content. Also, the estimation unit 54 includes, for example, colors with scores above a threshold among the scores of each type of color in the estimated treatment content.

[0207] Furthermore, the information indicating the treatment content from the fourth learning model may include information indicating the treatment content, including the treatment items and the order of treatment. In this case, the fourth learning model is configured to output scores for multiple treatment methods, each including the treatment items and the order of treatment, and the treatment content indicated by the treatment method with the highest score is determined to be the estimated treatment content.

[0208] [4.7.6. Judgment unit 55] The determination unit 55 determines the treatment cost, which is the cost required for the treatment based on the estimated treatment content determined by the estimation unit 54.

[0209] For example, the determination unit 55 has treatment cost information for each hair salon, which shows the relationship between the treatment content and the treatment cost, and determines the treatment cost for each hair salon based on this treatment cost information. The treatment cost information for each hair salon is information acquired by the acquisition unit 51 as described above, but is not limited to this example.

[0210] Furthermore, the determination unit 55 determines, for example, the facial features of user U shown in the user-captured image. For example, the determination unit 55 inputs the user-captured image to the second learning model described above and determines the facial features of user U based on the facial feature information output from the second learning model.

[0211] The information indicating facial features output from the second learning model is, for example, a score for each facial feature. In this case, the determination unit 55 determines that the facial feature with the highest score among the scores for each facial feature belongs to user U.

[0212] Furthermore, the information indicating facial features output from the second learning model may also be scores for each feature of each part (for example, eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines that the facial feature containing the feature with the highest score among the scores for each feature of the parts (combination of position, size, and shape) is the facial feature of user U.

[0213] Furthermore, the information indicating facial features output from the second learning model may also be scores for each feature, such as the size and shape of each part (for example, eyebrows, eyes, nose, mouth, and ears). In this case, the determination unit 55 determines that the facial feature containing the feature with the highest score among the scores for each feature of size and shape of the parts is the facial feature of user U.

[0214] Furthermore, the determination unit 55 may have information indicating each of several types of facial features. In this case, the determination unit 55 determines the facial features of user U by pattern matching technology between each type of facial feature and the facial features of user U shown in the user image.

[0215] Furthermore, the determination unit 55 can display face feature line drawings, which are line drawings representing each of several types of facial features (for example, face shape), on the display unit, and allow the user U to select the face feature line drawing that is closest to their own face shape from among these multiple types of face feature line drawings.

[0216] [4.7.7. Comparison Section 56] The comparison unit 56 compares the user-specified treatment content, which is the treatment content specified by the designation reception unit 62, with the estimated treatment content, which is the treatment content estimated by the estimation unit 54.

[0217] The comparison unit 56, for example, compares the user-specified treatment content with the estimated treatment content and generates comparison information showing the results of the comparison. The comparison information includes, for example, information showing the difference between the user-specified treatment content and the estimated treatment content.

[0218] For example, if the estimated treatment content has one more bleaching step than the user-specified treatment content, the comparison unit 56 generates comparison information that includes information indicating that the number of bleaching steps is one greater than that of the user-specified treatment content.

[0219] Furthermore, if the estimated treatment content or user-specified treatment content includes the order of treatments, and some of the treatment order differs, the comparison unit 56 generates comparison information that includes information indicating the different order of treatments.

[0220] [4.7.8. Output section 57] The output unit 57 outputs information such as the identification result by the identification unit 52, the image processing result by the image processing unit 53, the estimation result by the estimation unit 54, the determination result by the determination unit 55, and the comparison result by the comparison unit 56 as output information.

[0221] Furthermore, the output unit 57 can also output information indicating the reception result by the reception unit 50 and information indicating the acquisition result by the acquisition unit 51 as output information. The output unit 57 can, for example, display the output information on the display unit 41 by outputting the output information to the display unit 41, or it can output the output information to the hair stylist via the communication unit 40 and the network N. Output to the hair stylist is done, for example, via the information provision device 1, SNS, or an unillustrated mail server, but it may also be output directly to the store device 5.

[0222] The output unit 57 outputs information from the reservation setting screen for booking hair treatments at a hair salon as output information. The reservation setting screen is a GUI (Graphical User Interface) screen for setting user images, hair images, hair salons, and reservation dates and times, and will be described in detail later.

[0223] The output unit 57 outputs a reservation request containing treatment-related information to the hair stylist, for example, by sending the reservation request containing treatment-related information to the hair stylist.

[0224] Treatment-related information includes, for example, a user image including a hair image of user U before treatment, a composite image generated by the image processing unit 53, information indicating the treatment content for which the designation was accepted by the designation acceptance unit 62, and information indicating the treatment content estimated by the estimation unit 54.

[0225] The treatment-related information may include, in place of or in addition to, the composite image generated by the image processing unit 53, a selected hair image which is a hair image whose selection has been received by the selection receiving unit 60.

[0226] Furthermore, the output unit 57 outputs multiple keywords that can be specified by the user U. For example, the output unit 57 outputs output information containing multiple keywords to the display unit 41, thereby displaying multiple keywords on the display unit 41 that can be specified by the user U.

[0227] Furthermore, the output unit 57 outputs the practitioner comments acquired by the acquisition unit 51. For example, the output unit 57 displays the practitioner comments on the display unit 41 by outputting the practitioner comments acquired by the acquisition unit 51 to the display unit 41.

[0228] Furthermore, the output unit 57 outputs revision proposal information, which is information indicating revision proposals from the hair stylist that have been received by the revision proposal reception unit 63. The revision proposal information may include, for example, information indicating revision proposals for the estimated treatment content or information indicating revision proposals for the hairstyle shown in the composite image. For example, the output unit 57 outputs the revision proposal information received by the revision proposal reception unit 63 to the display unit 41, thereby displaying the revision proposals from the hair stylist on the display unit 41.

[0229] Furthermore, the output unit 57 can output the above-mentioned reservation request to the hair stylist if permission is accepted by the permission acceptance unit 64, and can refrain from outputting the above-mentioned reservation request to the hair stylist if permission is not accepted by the permission acceptance unit 64.

[0230] Furthermore, when the output unit 57 wants to display the output information on the display unit 41, it converts the output information into displayable information by, for example, rendering, and then outputs the converted output information to the display unit 41. Also, if the display unit 41 has a rendering function, the output unit 57 outputs the output information directly to the display unit 41 and displays the output information on the display unit 41.

[0231] Figure 11 shows an example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in Figure 11, the reservation setting screen 70 includes a user image selection area 71, a hair image selection area 72, a hair salon selection area 73, and a reservation target date and time setting area 74.

[0232] The reservation setting screen 70 is displayed on the display unit 41 by the output unit 57 when, for example, the operation performed by user U on the operation unit 42 is a specific operation, namely a reservation setting operation. Although not shown in Figure 11, the reservation setting screen 70 also includes, for example, a treatment content input area, a treatment content estimation button, and a reservation button.

[0233] The user image selection area 71 includes a shooting start button 711 and a folder button 712. The shooting start button 711 is a button for driving the imaging application program (hereinafter sometimes referred to as the imaging app) that performs imaging with the imaging unit 43. User U can operate the imaging app by operating the shooting start button 711, displaying the imaging app screen on the display unit 41, and operating the imaging app by operating the operation unit 42. By operating the imaging app, User U can capture an area including their face. As a result, the acquisition unit 51 acquires the user image.

[0234] The folder button 712 is a button for opening a specific folder (for example, a photo folder where images captured by the imaging unit 43 are placed). User U can operate the folder button 712 to display multiple images contained in a specific folder on the display unit 41. User U can select a user image from the multiple images contained in a specific folder by operating the operation unit 42. As a result, the acquisition unit 51 acquires the user image.

[0235] The hair image selection area 72 includes a keyword setting button 721, a shooting start button 722, and a folder button 723. The keyword setting button 721 is a button that displays a keyword selection screen on the display unit 41 for the user U to specify one or more desired keywords from a set of multiple keywords.

[0236] The shooting start button 722 is a button for driving the imaging application. User U can operate the imaging application by operating the shooting start button 722, displaying the imaging application screen on the display unit 41, and then operating the imaging application by operating the operation unit 42. By operating the imaging application, User U can capture an area containing a desired hair image (for example, an area containing an image of a hair model published in a hair catalog or fashion magazine). As a result, the acquisition unit 51 acquires the selected image.

[0237] The folder button 723 is a button for opening a specific folder (for example, a photo folder where images captured by the imaging unit 43 are placed). User U can operate the folder button 723 to display multiple images contained in a specific folder on the display unit 41. User U can also operate the operation unit 42 to select a desired image from the multiple images contained in a specific folder. The selected image is then acquired by the acquisition unit 51.

[0238] The hair salon selection area 73 includes a current location search button 731 and a keyword search button 732. The current location search button 731 is a button that displays hair salons located in the vicinity of the current location (for example, within a predetermined distance from the current location) on the display unit 41, making them selectable. By operating the operation unit 42, the user U can select a desired hair salon from those displayed on the display unit 41.

[0239] The keyword search button 732 is a button that displays the keyword search screen on the display unit 41 for performing a keyword search. By operating the operation unit 42, user U can select a desired hair salon from the hair salons that have been found and displayed on the display unit 41 on the keyword search screen.

[0240] The reservation target date and time setting area 74 is an area for setting the reservation target date and time. User U can set the date and time on which they wish to receive hair treatment in the reservation target date and time setting area 74 by operating the operation unit 42.

[0241] Figure 12 shows an example of a keyword selection screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to this embodiment. As shown in Figure 12, the keyword selection screen 80 includes a group of keyword selection buttons 81, 82, 83, and 84, and a search button 85.

[0242] The keyword selection button group 81 includes multiple buttons for specifying keywords corresponding to hair length. The keyword selection button group 81 shown in Figure 12 includes buttons for specifying the keyword "short," buttons for specifying the keyword "medium," buttons for specifying the keyword "long," and so on.

[0243] The keyword selection button group 82 includes multiple buttons for specifying keywords corresponding to hair color. The keyword selection button group 82 shown in Figure 12 includes buttons for specifying the keyword "brown," "beige," "pink," and "black," among others.

[0244] The keyword selection button group 83 includes multiple buttons for specifying keywords corresponding to the desired atmosphere. The keyword selection button group 83 shown in Figure 12 includes buttons for specifying the keyword "cute," "cool," "pop," and "natural," among others.

[0245] The keyword selection button group 84 includes multiple buttons for specifying keywords corresponding to TPO (Time Place Occasion). The keyword selection button group 84 shown in Figure 12 includes buttons for specifying the keyword "Harajuku," "Ginza," "Shinjuku," and "Roppongi," among others.

[0246] The multiple keywords indicated by the keyword selection button group 81, 82, 83, 84 are a predetermined set of keywords, but they may also be multiple keywords randomly selected from the set of keywords or multiple keywords that satisfy predetermined conditions. The multiple keywords that satisfy predetermined conditions are, for example, a predetermined number of keywords that are selected most frequently by user U. The keywords that become candidates for extraction are, for example, keywords that will be input to the first learning model.

[0247] On the keyword selection screen 80, user U can specify hair length, hair color, atmosphere, TPO, etc., by operating the buttons corresponding to the desired keywords on the operation unit 42. Then, by operating the operation unit 42 and selecting the search button 85, user U will have the processing unit 46 execute the process of obtaining similar hair images from the specified keywords. Note that the multiple keywords included in the keyword selection screen 80 are not limited to the examples described above.

[0248] Figure 13 is a diagram illustrating the process by which the processing unit 46 of the information providing device 1 according to this embodiment acquires similar hair images from specified keywords. In the example shown in Figure 13, the keywords specified by user U are "long," "pink," "cute," and "Harajuku."

[0249] In this case, the reception unit 50 receives the keywords specified by user U as "long," "pink," "cute," and "Harajuku." The identification unit 52 uses the first learning model to identify hair images from the keywords "long," "pink," "cute," and "Harajuku" that were received by the reception unit 50.

[0250] The acquisition unit 51 then sends a similar image search request including a specific hair image to a search server or information providing device 1 (not shown), and acquires multiple similar hair images transmitted from the search server or information providing device 1 in response to the similar image search request.

[0251] Figure 14 shows another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. In the reservation setting screen 70 shown in Figure 14, a user image selected by user U is set in the user image selection area 71, and multiple similar hair images corresponding to one or more keywords specified by user U are set in the hair image selection area 72.

[0252] User U can select one or more similar hair images from among the multiple similar hair images set in the hair image selection area 72 as the selected image by operating the operation unit 42.

[0253] Figure 15 shows yet another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. The reservation setting screen 70 shown in Figure 15 is displayed on the display unit 41 when, in the reservation setting screen 70 shown in Figure 14, user U selects one similar hair image from the multiple similar hair images shown in Figure 14 by operating the operation unit 42.

[0254] The reception unit 50 accepts the selection of a similar hair image by the user U as the selected image. The image processing unit 53 generates a composite image by combining the similar hair image selected by the reception unit 50 with the user image selected by the user U, and places the generated composite image on the reservation setting screen 70 as shown in Figure 15. This allows the user U to check the composite image displayed on the display unit 41 and to more realistically confirm whether the hairstyle shown in the selected hair image suits them.

[0255] Furthermore, by operating the operation unit 42, user U can select a composite image on the reservation setting screen 70, thereby changing the selected image used to generate the composite image and displaying the composite image on the display unit 41.

[0256] Figure 16 shows an example of a composite image change screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to this embodiment. The composite image change screen 90 shown in Figure 16 includes a length selection button 91, a color selection button 92, and a composite image display area 93.

[0257] The length selection button 91 is used to select the hair length, which is changed by swiping (or scrolling) in the up and down direction. The color selection button 92 is used to select the hair color, which is changed by swiping (or scrolling) in the up and down direction. Swiping (or scrolling) in the up and down direction is an example of a second modification operation on the hair image.

[0258] In the example shown in Figure 16, the length selection button 91 is selected, and user U can change the length of the hair in the composite image by operating the operation unit 42 to swipe (or scroll) up and down. When the image processing unit 53 receives a swipe (or scroll) in the up and down direction from the reception unit 50, it changes the length of the hair in the composite image and displays the composite image with the changed hair length on the display unit 41.

[0259] Furthermore, user U can change the hair color of the composite image by operating the operation unit 42 to select the color selection button 92, and then operating the operation unit 42 to swipe (or scroll) up or down. When the image processing unit 53 receives an up or down swipe from the reception unit 50, it changes the hair color of the composite image and displays the composite image with the changed hair color on the display unit 41.

[0260] Furthermore, user U can display a composite image on the display unit 41 by swiping (or scrolling) in the left or right direction to change the selected image. Swiping (or scrolling) in the left or right direction is an example of the first modification operation of the hair image. When the image processing unit 53 receives a swipe (or scroll) in the left or right direction from the reception unit 50, it changes the selected image to generate a composite image and displays the composite image with the changed selected image on the display unit 41.

[0261] As described above, the reservation setting screen 70 includes a treatment details input area (not shown). User U can input the treatment details they specify or desire into the treatment details input area by operating the operation unit 42.

[0262] Furthermore, as mentioned above, the reservation setting screen 70 includes buttons for estimating treatment content and making a reservation (not shown). User U selects a composite image, enters the treatment content that User U specifies or desires in the treatment content input area, then selects a hair salon and sets the reservation date and time. Then, User U operates the operation unit 42 to select the reservation button, and a reservation request is sent from the output unit 57 to the hair stylist. As a result, the reservation request is output to the hair stylist.

[0263] The reception unit 50 accepts the selection of the treatment content estimation button by user U. When the reception unit 50 accepts the selection of the treatment content estimation button, the estimation unit 54 estimates the treatment content from the user image and composite image shown on the reservation setting screen 70. The user image is an example of user U's image before treatment. When the estimation unit 54 inputs the user image and composite image into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

[0264] Figure 17 is a diagram illustrating the process by which the estimation unit 54 of the information providing device 1 according to this embodiment estimates the treatment content from a user image and a composite image. In the example shown in Figure 17, the estimation unit 54 determines from the user image and the composite image that the estimated treatment content is "cut, bleach once, color bleach twice, perm once".

[0265] When the reception unit 50 accepts the selection of the reservation button, the output unit 57 outputs a reservation request to the hair stylist that includes a user image, a composite image, information indicating the treatment content entered in the treatment content input area, information indicating the estimated treatment content, and information indicating the reservation date and time. Also, when the reception unit 50 accepts the selection of a specific button (not shown), the output unit 57 displays, for example, a screen indicating the end of the operation on the display unit 41. The selection of the reservation button by user U is an example of a response indicating user U's consent to the estimated treatment content, and the selection of a specific button by user U is an example of a response indicating user U's refusal to consent to the estimated treatment content.

[0266] Furthermore, the output unit 57 can output a face shape selection screen to the display unit 41 in response to user U's operation on the operation unit 42, allowing user U to select a face shape, and display the face shape selection screen on the display unit 41.

[0267] FIG. 18 is a diagram showing an example of a face type selection screen displayed on the display unit 41 by the output unit 57 of the information providing apparatus 1 according to the embodiment. As shown in FIG. 18, the face type selection screen 95 includes a plurality of face types. The user U can operate the operation unit 42 to select the face type that is closest to the face type of the user U among the face types.

[0268] The reception unit 50 receives the selection of the face type by the user U among the plurality of face types included in the face type selection screen 95. The specifying unit 52 designates the face type selected by the user U as the feature of the face of the user U specified by the user U, and specifies a hairstyle estimated to suit the face of the user U using the third learning model from such features of the face of the user U.

[0269] The specifying unit 52 inputs information indicating the face type selected by the user U into the third learning model, and specifies a hairstyle estimated to suit the face of the user U based on the hairstyle information output from the third learning model.

[0270] 〔5. Processing Procedure〕 Next, a procedure of information processing by the processing unit 46 of the terminal device 4 according to the embodiment will be described. FIG. 19 is a flowchart showing an example of information processing by the processing unit 46 of the terminal device 4 according to the embodiment.

[0271] As shown in FIG. 19, the processing unit 46 of the terminal device 4 determines whether or not the information of the learning model has been acquired from the information providing apparatus 1 (step S20). When the processing unit 46 determines that the information of the learning model has been acquired (step S20: Yes), the acquired information of the learning model is stored in the storage unit 45 (step S21).

[0272] If the processing in step S21 is completed, or if it determines that it has not obtained information about the learning model (step S20: No), the processing unit 46 determines whether or not there is a reservation setting operation by user U (step S22). If the processing unit 46 determines that there is a reservation setting operation (step S22: Yes), it executes the reservation process (step S23). The reservation process in step S23 is the process in steps S40 to S49 shown in Figure 20, which will be described in detail later.

[0273] If the processing in step S23 is completed, or if it is determined that there is no reservation setting operation (step S22: No), the processing unit 46 determines whether or not there are any revision suggestions from the hair stylist (step S24). If the processing unit 46 determines that there are revision suggestions (step S24: Yes), it displays the revision suggestions from the hair stylist on the display unit 41 (step S25).

[0274] If the processing in step S25 is completed, or if it is determined that there are no proposed corrections (step S24: No), the processing unit 46 determines whether or not there are practitioner comments (step S26). If the processing unit 46 determines that there are practitioner comments (step S26: Yes), it displays the practitioner comments on the display unit 41 (step S27).

[0275] If the processing in step S27 is completed, or if it is determined that there are no operator comments (step S26: No), the processing unit 46 determines whether or not there is an image synthesis operation (step S28). If the processing unit 46 determines that there is an image synthesis operation (step S28: Yes), it performs the synthesis process (step S29). The synthesis process in step S29 is the same as steps S40 to S47 shown in Figure 20.

[0276] When the processing in step S29 is completed, or when it is determined that there is no image synthesis operation (step S28: No), the processing unit 46 determines whether or not it is time to terminate the operation (step S30). For example, the processing unit 46 determines that it is time to terminate the operation when the power to the information providing device 1 is turned off, or when it is determined that a termination operation has been performed by operating on an unillustrated control unit of the information providing device 1.

[0277] If the processing unit 46 determines that it is not yet time to terminate the operation (step S30: No), it proceeds to step S20. If it determines that it is time to terminate the operation (step S30: Yes), it terminates the process shown in Figure 19.

[0278] Figure 20 is a flowchart showing an example of reservation processing by the processing unit 46 of the terminal device 4 according to the embodiment. As shown in Figure 20, the processing unit 46 receives the selection of a user image and acquires the selected user image (step S40).

[0279] Next, the processing unit 46 accepts the specification of one or more keywords (step S41). The processing unit 46 identifies a hair image from the one or more keywords specified in step S41 (step S42). Then, the processing unit 46 obtains a similar hair image, which is an image similar to the hair image identified in step S42 (step S43).

[0280] Next, the processing unit 46 combines the user's face image with the selected similar hair image acquired in step S42 (step S44). Then, the processing unit 46 displays the combined image created in step S44 on the display unit 41 (step S45).

[0281] Next, the processing unit 46 determines whether or not there is a modification operation (step S46). Modification operations include, for example, the first modification operation of the hair image and the second modification operation of the hair image described above. If the processing unit 46 determines that there is a modification operation (step S46: Yes), it performs image processing according to the modification operation to modify the composite image (step S47), and then proceeds to step S46.

[0282] If the processing unit 46 determines that there is no change operation (step S46: No), it estimates the treatment content based on the pre-treatment image and the composite image (step S48). The processing unit 46 determines that there is no change operation, for example, when the reservation button is selected by user U.

[0283] Then, the processing unit 46 outputs a reservation request to the hair stylist, including the pre-treatment image, the composite image, the estimated treatment content, and the specified treatment content (step S49). The specified treatment content is the treatment content entered by user U. When the processing in step S49 is completed, the processing unit 46 terminates the process shown in Figure 20.

[0284] [6. Variant Example] In the example described above, the processing unit 46 displays a composite image on the display unit 41 in which the selected image has been changed by user operation in the left-right direction, and displays a composite image on the display unit 41 in which the length or color of the hair has been changed by user operation in the up-down direction, but the example is not limited to this.

[0285] For example, the processing unit 46 can display a composite image on the display unit 41 in which the selected image has been changed by user operation in the vertical direction, and can also display a composite image on the display unit 41 in which the length or color of the hair has been changed by user operation in the horizontal direction.

[0286] Furthermore, the processing unit 46 can also display a composite image on the display unit 41 in which the selected image and the length or color of the hair have been changed by a user operation in a diagonal direction. The user operation is a swipe operation or a scroll operation, but is not limited to these examples; for example, it may also be a tap operation or a click operation, or a pinch-in operation or a pinch-out operation.

[0287] Furthermore, the hairstyle information output from the third learning model is not limited to hair images. For example, the hairstyle information output from the third learning model may be hair image identification information, which is information used to identify a hair image. This information could be, for example, hair image identification information or keywords used to identify a hair image.

[0288] In this case, the third learning model is generated using training data that includes multiple combinations of information representing facial features, information identifying hair images, and information representing evaluations of combinations of facial feature information and hair image identification information.

[0289] Furthermore, each learning model, including the first learning model, the second learning model, and the third learning model, may be a learning model specific to user U. For example, by using hair images included in posted images viewed by user U as the hair images used to generate the first and third learning models, each of the first and third learning models can be generated for user U.

[0290] Furthermore, by using hair images included in the posted images viewed by user U as the face images used to generate the second learning model, a second learning model can be generated for each user U.

[0291] The information providing device 1 can execute some or all of the functions of the acquisition unit 51, the specifying unit 52, the image processing unit 53, the estimating unit 54, the determination unit 55, the comparison unit 56, and the output unit 57 in cooperation with the terminal device 4, and can function as a part of the information processing device. In the following, in some cases, a configuration including some or all of the above-described terminal device 4 and the information providing device 1 may be described as an information processing device.

[0292] [7. Hardware Configuration] Each of the information providing device 1 and the terminal device 4 according to the above-described embodiment is realized by a computer 200 having a configuration as shown in, for example, FIG. 21. FIG. 21 is a hardware configuration diagram showing an example of a computer 200 that realizes the functions of each of the information providing device 1 and the terminal device 4 according to the embodiment. The computer 200 includes a CPU 201, a RAM 202, a ROM (Read Only Memory) 203, an HDD (Hard Disk Drive) 204, a communication interface (I / F) 205, an input / output interface (I / F) 206, and a media interface (I / F) 207.

[0293] The CPU 201 operates based on a program stored in the ROM 203 or the HDD 204 and controls each unit. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 is started up, a program depending on the hardware of the computer 200, and the like.

[0294] The HDD 204 stores a program executed by the CPU 201, data used by such a program, and the like. The communication interface 205 receives data from other devices via the network N (see FIG. 2) and sends it to the CPU 201, and sends data generated by the CPU 201 to other devices via the network N.

[0295] The CPU 201 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 206. The CPU 201 acquires data from input devices via the input / output interface 206. The CPU 201 also outputs data it has generated to output devices via the input / output interface 206.

[0296] The media interface 207 reads a program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program from the recording medium 208 onto the RAM 202 via the media interface 207 and executes the loaded program. The recording medium 208 can be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording medium, or semiconductor memory.

[0297] For example, when the computer 200 functions as an information providing device 1 or terminal device 4 according to the embodiment, the CPU 201 of the computer 200 realizes the functions of the processing unit 12 and processing unit 46 by executing programs loaded on the RAM 202. In addition, data in the storage unit 11 or storage unit 45 is stored in the HDD 204. The CPU 201 of the computer 200 reads and executes these programs from the recording medium 208, but as another example, these programs may be obtained from other devices via the network N.

[0298] [8. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0299] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0300] For example, the information provision device 1 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms via APIs or network computing, allowing for flexible configuration changes.

[0301] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0302] [9. Effects] As described above, the information processing device according to the embodiment comprises a identification unit 52, an acquisition unit 51, and an image processing unit 53. The identification unit 52 uses a learning model, which is a model that has learned the relationship between information indicating facial features and hairstyles evaluated as suitable for the face, to identify a hairstyle that is estimated to suit user U's face. The acquisition unit 51 acquires a posted image that includes a specific hairstyle image, which is an image of the hairstyle identified by the identification unit 52. The image processing unit 53 generates a composite image by combining the image of the hairstyle identified by the identification unit 52 and the image of user U's face, based on the posted image acquired by the acquisition unit 51 and the image of user U's face. As a result, user U can easily obtain an image of what it would look like if a hairstyle estimated to suit user U's face were applied to themselves. Therefore, the information processing device can improve the convenience of user U.

[0303] Furthermore, the information describing facial features is information indicating the type of face shape. This allows the information processing device to accurately identify hairstyles that are estimated to suit the face.

[0304] Furthermore, the information processing device includes an output unit 57 that outputs a composite image generated by the image processing unit 53. This allows the information processing device to improve the convenience of the user U.

[0305] Furthermore, the learning model is generated through machine learning using training data that includes multiple combinations of information representing facial features, hair images, and information representing evaluations of the combination of facial feature information and hair images. As a result, the information processing device can accurately identify hairstyles that are estimated to suit the face.

[0306] Furthermore, the training data is generated based on user-submitted images, including facial and hair images, and user-submitted information, which includes evaluation information indicating the evaluation of the user-submitted images. This allows the information processing device to accurately identify hairstyles that are estimated to suit a person's face.

[0307] Furthermore, the information processing device includes a reception unit 50 that receives a first modification operation for a hair image from user U. The acquisition unit 51 acquires multiple posted images containing a specific hair image, the image processing unit 53 generates a composite image corresponding to each of the multiple posted images, and the output unit 57 changes the composite image to be output each time the first modification operation is received by the reception unit 50. This allows the information processing device to improve the convenience of user U.

[0308] Furthermore, the reception unit 50 receives a second modification operation for the hair image from the user U, and the output unit 57 changes at least one of the hair length and color of the hair image included in the composite image each time the reception unit 50 receives a second modification operation. This improves the convenience of the information processing device for the user U.

[0309] Although embodiments of the present application have been described in detail based on the drawings, these are illustrative examples, and the present invention can be implemented in various other forms, including those described in the disclosure section of the invention, based on the knowledge of those skilled in the art.

[0310] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of symbols]

[0311] 1 Information provision device 4 Terminal devices 5 Store Equipment 10,40 Communications Department 11,45 Storage section 12,46 Processing Unit 20 User information storage unit 21 Post Information Storage Unit 22 Model Information Storage Unit 23 Content storage unit 30,51 Acquisition Department 31. Learning Department 32 Providing Department 41 Display section 42 Operation section 43 Imaging Unit 44. Detector Unit 50 Reception Department 52 Specific part 53 Image Processing Unit 54 Estimation part 55 Judgment section 56 Comparison Section 57 Output section 60 Selection Reception Section 61 Processing Reception Department 62 Designated Reception Department 63 Revision Proposal Reception Department 64 Approval / Rejection Department 65 Conversion section 66 Synthesis section 100 Information Processing Systems

Claims

1. A selection unit identifies hairstyles that are estimated to suit the user's face, using a learning model that has learned the relationship between information representing facial features and hairstyles evaluated as suitable for the face. An acquisition unit that acquires an image including a specific hair image, which is an image of the hairstyle identified by the identification unit, The system includes an image processing unit that generates a composite image by combining an image of the hairstyle identified by the identification unit with the user's face image, based on an image including the specific hair image acquired by the acquisition unit and the user's face image. An information processing device characterized by the following:

2. The information describing the facial features is, This is information indicating the type of face shape. The information processing apparatus according to feature 1.

3. The system includes an output unit that outputs the composite image generated by the image processing unit. The information processing apparatus according to claim 1 or 2.

4. The aforementioned learning model, This is generated by machine learning using training data that includes multiple combinations of information indicating facial features, hair images, and information indicating an evaluation of the combination of the facial features information and the hair images. The information processing apparatus according to claim 1 or 2.

5. The aforementioned training data is This is generated based on post information that includes a post image containing a face image and a hair image, and evaluation information indicating an evaluation of the post image. The information processing apparatus according to feature 4.

6. The system includes a reception unit that receives a first modification operation of the hair image from the user, The acquisition unit is, Multiple images, including the aforementioned specific hair image, are acquired. The aforementioned image processing unit, A composite image is generated that corresponds to each of the images containing multiple specific hair images. The output unit is, The reception unit changes the composite image to be output each time the first modification operation is received. The information processing apparatus according to claim 3.

7. The aforementioned reception unit is The user requests a second modification operation for the hair image. The output unit is, Each time the reception unit receives the second modification operation, at least one of the hair length and color of the hair image included in the composite image is changed. The information processing apparatus according to feature 6.

8. A method of information processing performed by a computer, The process involves identifying a hairstyle that is estimated to suit the user's face, using a learning model that has learned the relationship between information representing facial features and hairstyles that have been evaluated as suitable for the face, and An acquisition step to acquire an image including a specific hair image, which is an image of the hairstyle identified by the aforementioned specific step, The image processing step includes generating a composite image by combining the image of the hairstyle identified in the identification step with the image of the user's face, based on the image including the specific hair image acquired in the acquisition step and the user's face image. An information processing method characterized by the following:

9. A procedure for identifying hairstyles that are estimated to suit a user's face, using a learning model that has learned the relationship between information representing facial features and hairstyles evaluated as suitable for the said face, An acquisition procedure for acquiring an image that includes a specific hair image, which is an image of the hairstyle identified by the identification procedure, The computer is instructed to perform an image processing procedure that generates a composite image by combining the image of the hairstyle identified in the identification procedure with the image of the user's face, based on the image including the specific hair image acquired in the acquisition procedure and the user's face image. An information processing program characterized by the following features.

Images