Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program that are capable of increasing the convenience of users.SOLUTION: An information processing device includes a specification unit, an acquisition unit, and an image processing unit. The specification unit specifies a hair style that is estimated to be suitable for a face of a user, by using a learning model that is a model having learned a relationship between information indicating the characteristics of a face and a hair style evaluated to suit the face. The acquisition unit obtains a posted image including a specified hair image that is an image of the hair style specified by the specification unit. The image processing unit generates a composite image that is obtained by combining the image of the hairstyle specified by the specification unit and the face image of the user, based on the posted image obtained by the acquisition unit and the face image of the user.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there are known techniques for synthesizing a facial image and a hair image of a user and presenting the synthesized image to the user. For example, Patent Document 1 proposes a technique for performing image processing using a hair simulation to change the hairstyle and hair color, and displaying the results of the image processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-178789 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned conventional technology, the user has to check which hairstyle suits them one by one as they change hairstyles, and the more hairstyles there are, the more complicated the user's operations become, so there is room for improvement in terms of improving user convenience.

[0005] The present application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can improve user convenience. [Means for solving the problem]

[0006] The information processing device according to the present application includes an identification unit, an acquisition unit, and an image processing unit. The identification unit identifies a hairstyle that is estimated to suit a user's face using a learning model that is a model that has learned the relationship between information indicating facial features and hairstyles evaluated as suiting the face. The acquisition unit acquires posted images including a specific hair image that is an image of the hairstyle identified by the identification unit. The image processing unit generates a composite image by combining the image of the hairstyle identified by the identification unit with the user's facial image, based on the posted images acquired by the acquisition unit and the user's facial image. [Effects of the Invention]

[0007] According to one aspect of the embodiment, it is possible to improve user convenience. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to the embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an information providing device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a user information table stored in the user information storage unit of the information providing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a posted information table stored in the posted information storage unit of the information providing device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a model information table stored in the model information storage unit of the information providing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a content table stored in the content storage unit of the information providing device according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 9]FIG. 9 is a diagram illustrating an example of the configuration of the reception unit of the terminal device according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an image processing unit of the terminal device according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a keyword specification screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 13] FIG. 13 is a diagram for explaining a process of acquiring a similar hair image from a specified keyword by the processing unit of the information providing device according to the embodiment. [Figure 14] FIG. 14 is a diagram showing another example of the reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 15] FIG. 15 is a diagram showing yet another example of the reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a composite image change screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 17] FIG. 17 is a diagram illustrating a process of estimating a treatment content from a user image and a composite image by an estimation unit of an information providing device according to an embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a face shape selection screen displayed on the display unit by the output unit of the information providing device according to the embodiment. [Figure 19] FIG. 19 is a flowchart illustrating an example of information processing by the processing unit of the terminal device according to the embodiment. [Figure 20] FIG. 20 is a flowchart illustrating an example of a reservation process by the processing unit of the terminal device according to the embodiment. [Figure 21] FIG. 21 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information providing device and the terminal device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, modes for implementing an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the respective embodiments can be appropriately combined within the scope of not causing any contradiction in the processing content. Furthermore, the same components in the following embodiments will be assigned the same reference numerals, and redundant explanations will be omitted.

[0010] [1. An example of information processing] First, an example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the embodiment. Note that, in the following, a case will be described in which the information processing device according to the embodiment is the terminal device 4 shown in Fig. 1, but the information processing device may be the information providing device 1 shown in Fig. 1, or may be a configuration in which the terminal device 4 and the information providing device 1 are combined.

[0011] 1 is an information processing device that provides services such as a social networking service (SNS). For example, the information providing device 1 receives information posted by a poster. The information providing device 1 distributes the information received from the poster to a terminal device 4 of a user U who will be a viewer.

[0012] A poster may be, for example, a hair salon, a hair practitioner, a treatment recipient, etc., and may include user U. A hair salon is a beauty salon (beauty parlor) or a barber shop (hairdressing parlor). A hair practitioner is a practitioner who styles hair at a hair salon, and is also called a hair stylist. A treatment recipient is a user who has received hair treatment by a hair practitioner.

[0013] The posted information posted by the poster includes, for example, a posted image, which is a poster-captured image obtained by capturing an image of the poster, information indicating posted text, which is text posted by the poster, etc. The posted image includes, for example, a captured image of the poster's head, including the poster's hair and face.

[0014] The posted text may include, for example, a poster's comment on the posted image. The poster's comment on the posted image may include, for example, evaluation information indicating the poster's evaluation of the posted image. Hereinafter, the poster's comment on the posted image may be referred to as a poster's comment.

[0015] The information providing device 1 also accepts evaluation information indicating the evaluations of other users (an example of viewers) on posted images included in the posted information, and comments from other users on the posted information. The information providing device 1 then distributes the posted information, including the received evaluation information and comments from other users, to a viewer's device. The viewer's device is, for example, the terminal device 4 of user U or a device of another user. Hereinafter, comments from other users may be referred to as viewer comments.

[0016] The evaluation of the posted information may be, for example, a positive evaluation (e.g., "good") or a negative evaluation (e.g., "bad"), but may also include a neutral evaluation (e.g., "average"), etc. The information providing device 1 can also distribute information such as the number of times the posted information has been added to a timeline by other users as an evaluation of the posted information to the terminal device 4 of the user U who will be a viewer.

[0017] The information providing device 1 transmits information on a plurality of types of learning models to the terminal device 4 (step S1). The plurality of types of learning models include a first learning model, a second learning model, a third learning model, and a fourth learning model.

[0018] First, we will explain the first learning model. The first learning model is a model that learns the relationship between keywords and hair images, and is a learning model that inputs one or more keywords and outputs hair images. The first learning model is generated by machine learning using learning data that includes multiple combinations of keywords and hair images.

[0019] The first learning model may be a learning model that receives one or more keywords as input and outputs hair image specifying information. The hair image specifying information is information that specifies a hair image.

[0020] The information providing device 1 extracts, for example, one or more keywords contained in the poster's comments or poster's text, and generates data containing multiple combinations of the one or more keywords and the posted image that was the subject of the comment as the above-mentioned learning data.

[0021] In addition, if the information providing device 1 cannot extract keywords from the poster's comments or the poster's text, it extracts one or more keywords contained in the viewer's comments and generates data containing multiple combinations of the one or more keywords and the posted image that was the subject of the comment as the above-mentioned learning data.

[0022] In addition, the information providing device 1 can extract one or more keywords included in both the poster's comments or poster's text and the viewer's comments, and generate data containing multiple combinations of such one or more keywords and the posted image that was the subject of the comment as the above-mentioned learning data.

[0023] The learning data may include multiple combinations of keywords assigned by an operator to each posted image and the posted image. Instead of content such as posted information, the learning data may be images included in content such as a web page, or images provided by a third party. Posted information is an example of posted content.

[0024] For example, the information providing device 1 can acquire content including text and images of people's faces and hair via a network, and extract keywords from the text included in the content. In this case, the information providing device 1 generates data including multiple combinations of keywords extracted from the text included in the content and images included in the content as learning data.

[0025] Next, we will explain the second learning model. The second learning model is a learning model that learns the relationship between face images and information indicating facial features, and takes images including face images as input and outputs information indicating facial features.

[0026] The second learning model is generated by machine learning using learning data including a plurality of combinations of images including face images and information indicating facial features. The information indicating facial features is, for example, information indicating the type of face shape, but may also be information indicating the characteristics of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears.

[0027] In addition, the information indicating facial features may be information indicating the position of each of the eyebrows, eyes, nose, mouth, and ears on the face, or information indicating the position and size of each of the eyebrows, eyes, nose, mouth, and ears on the face.

[0028] Next, we will explain the third learning model. The third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face, and is a learning model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit the face.

[0029] The third learning model is generated by machine learning using learning data including multiple combinations of information indicating facial features, hair images, and information indicating an evaluation of the combination of the information indicating facial features and the hair images.

[0030] The information providing device 1 collects a plurality of pieces of posted information, each of which includes a posted image, a viewer's comment, and evaluation information. The face image includes a face image and a hair image, and the evaluation information is information indicating the evaluation of the posted image by other users who are viewers, but may also include information indicating the evaluation included in the poster's comment.

[0031] The information providing device 1 extracts facial features indicated by face images included in each collected piece of posted information. Then, the information providing device 1 generates data including a hair image included in the posted image of the posted information, information indicating the extracted facial features, and evaluation information as the learning data described above.

[0032] In the third learning model, the information indicating facial features is, for example, information indicating the type of face shape, but may also be information indicating the features of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears.Furthermore, the information indicating facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears.

[0033] Next, the fourth learning model will be described. The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment details, and is a learning model that inputs hair images before and after a treatment and outputs information indicating the estimated treatment details. The hair images before and after a treatment include hair images before the hair treatment and hair images after the hair treatment.

[0034] The treatment details include, for example, whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, whether or not to straighten hair, etc. The information indicating the treatment details includes information on the scores for whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, and whether or not to straighten hair.

[0035] In addition, the information indicating the treatment content may include score information for each combination of whether or not a cut was performed, the amount of cut, whether or not bleaching was performed, the number of bleachings, whether or not coloring was performed, the type of coloring, whether or not perming was performed, the type of perming, and whether or not hair straightening was performed.

[0036] The fourth learning model is generated by machine learning using learning data including multiple combinations of hair images before and after a treatment and information indicating the treatment details. The information providing device 1 collects treatment information including hair images before and after a treatment and information indicating the treatment details from devices such as a hair salon or a hair stylist. Then, based on the collected treatment information, the information providing device 1 generates learning data including multiple combinations of hair images before and after a treatment and information indicating the treatment details.

[0037] Any type of model can be adopted as each of the above-mentioned learning models. For example, each learning model is a model generated by machine learning, such as a deep neural network (DNN), a gradient boosting decision tree (GBDT), or a support vector machine (SVM). The DNN is, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). The RNN may also be a long short-term memory (LSTM). Each learning model may also be a model realized by combining multiple models, such as a model combining a CNN and an RNN.

[0038] Next, the terminal device 4 receives information on the multiple types of learning models transmitted from the information providing device 1, and stores the received information on the multiple types of learning models in an internal storage unit (step S2).

[0039] A user U of the terminal device 4 may consider what hairstyle to have before making a reservation for a hair treatment at a hair salon. The user U may consider a hairstyle from, for example, information found through a web search or information published in a magazine.

[0040] For example, the user U can display hairstyle-related images obtained by a keyword search or a web search on the terminal device 4. The hairstyle-related images are images that allow the user U to check the hairstyle, and include at least a hair image and a face image.

[0041] Here, the keyword search will be explained. The terminal device 4 displays a plurality of keywords on the display unit (step S3). These plurality of keywords are predetermined keywords, but may also be a plurality of keywords randomly extracted from the plurality of keywords or a plurality of keywords that satisfy a predetermined condition.

[0042] The plurality of keywords that satisfy the predetermined conditions are, for example, a predetermined number of keywords that are ranked in order of frequency of selection by the user U. The keywords that are candidates for extraction are, for example, keywords that are to be input to the first learning model.

[0043] When one or more keywords are specified by the user U, the terminal device 4 accepts the one or more keywords specified by the user U and identifies a hair image corresponding to the one or more keywords using the first learning model (step S4).

[0044] In step S4, the terminal device 4 inputs one or more keywords specified by the user U into a first learning model and, based on the information output from the first learning model, identifies hair images corresponding to the one or more keywords specified by the user U.

[0045] For example, when the first learning model outputs a hair image, the terminal device 4 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords designated by the user U.

[0046] Furthermore, when the first learning model outputs hair image specifying information, the terminal device 4 acquires the hair image specifying information output from the first learning model, and acquires the hair image specified by the acquired hair image specifying information from the internal storage unit or the information providing device 1. In this case as well, the terminal device 4 identifies the acquired hair image as a hair image corresponding to one or more keywords designated by the user U.

[0047] Next, the terminal device 4 acquires a plurality of similar hair images, which are images including hair images similar to the hair image identified in step S4, and displays the acquired similar hair images on the display unit (step S5). In step S5, the terminal device 4 transmits a similar image search request including the hair image identified in step S4 to a similar image search server or the information providing device 1 (not shown).

[0048] The similar image search server or information providing device 1 searches for multiple similar hair images based on the hair image included in the similar image search request from the terminal device 4, and transmits one or more similar hair images obtained by the search to the terminal device 4. The terminal device 4 acquires the multiple similar hair images transmitted from the similar image search server or information providing device 1 in response to the similar image search request.

[0049] In this way, the terminal device 4 displays a plurality of similar hair images on the display unit based on one or more keywords specified by the user U. This allows the user U to check the hair images to be considered simply by specifying a keyword.

[0050] In the above example, the user U designates one or more keywords from among the multiple keywords presented by the terminal device 4, but the designation of one or more keywords by the user U is not limited to this example. For example, the user U can designate one or more keywords by operating the terminal device 4 to input one or more keywords into the terminal device 4, or can designate one or more keywords by inputting one or more keywords into the terminal device 4 by voice.

[0051] Furthermore, instead of or in addition to a keyword search, the user U can also consider a hairstyle based on the user U's facial features. Consideration of a hairstyle based on the user U's facial features will be described below.

[0052] The terminal device 4 determines the facial features of the user U based on the user-captured image specified by the user U (step S6). The user-captured image is an image of the head of the user U, including the hair and face, captured by an imaging unit (not shown) of the terminal device 4 or an imaging unit of a device (not shown), and is an example of a user image. The terminal device 4 acquires the user-captured image based on, for example, an operation by the user U.

[0053] For example, the user U can operate the terminal device 4 to capture an image of the user U's head, including the user's hair and face, with an imaging unit of the terminal device 4, and cause the terminal device 4 to acquire a plurality of user-captured images. The user U can operate the terminal device 4 to specify a desired user-captured image from the plurality of user-captured images acquired by the terminal device 4.

[0054] In step S6, the terminal device 4 determines the facial features of the user U shown in the user-captured image from, for example, a plurality of predetermined types of facial features. For example, the terminal device 4 inputs the user-captured image into the second learning model described above, and determines the facial features of the user U based on the information indicating the facial features output from the second learning model.

[0055] In addition, the terminal device 4 may have information indicating each of multiple types of facial features, in which case the terminal device 4 determines the facial features of the user U using a pattern matching technique between the information indicating each type of facial feature and the information indicating the facial features of the user U shown in the user-captured image.

[0056] The terminal device 4 can also display facial feature line drawings, which are line drawings showing each of multiple types of facial features (e.g., face shapes), on the display unit, and allow the user U to select the facial feature line drawing that is closest to their own face shape from among these multiple types of facial feature line drawings.

[0057] Next, the terminal device 4 uses the above-mentioned third learning model to identify a hairstyle that is estimated to suit the face of the user U, based on the facial features of the user U determined in step S6 (step S7). As described above, the third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated to suit the face.

[0058] In step S7, the terminal device 4 inputs information indicating the facial features of the user U determined in step S6 into a third learning model, and identifies a hairstyle that is estimated to suit the face of the user U based on the hairstyle information output from the third learning model. The hairstyle information is a hair image, and the terminal device 4 identifies the hair image output from the third learning model as the hairstyle that is estimated to suit the face of the user U.

[0059] Next, the terminal device 4 generates a composite image by combining the user face image, which is a facial image of the user U, with the hair image selected by the user U, and displays the generated composite image on the display unit (step S8). The user face image is, for example, the above-mentioned user captured image or an image included in the user captured image.

[0060] The hair image selected by the user U is, for example, one or more images selected by the user U from among the images of the hairstyle identified in step S8 or a plurality of similar hair images acquired by keyword search.

[0061] The terminal device 4 accepts the user U's selection of one or more hair images from the specific hair image, which is an image of the hairstyle identified in step S8, or from multiple similar hair images acquired by keyword search. The terminal device 4 can also accept the user U's selection of one or more hair images from multiple hair images included in multiple web pages viewed by the user U, for example.

[0062] When two or more hair images are selected by the user U, the terminal device 4 generates a composite image by combining the two or more hair images to obtain a composite hair image with the user's face image.

[0063] The hair image selected by the user U may be, for example, a captured image captured by an imaging unit (not shown) in the terminal device 4. Such a captured image is, for example, an image captured from a photograph of a person's face and hair published in a magazine. In this case, the terminal device 4 generates a composite image by, for example, replacing the facial image of the person included in the captured image selected by the user U among the captured images captured by the terminal device 4 with the facial image of the user U included in the user facial image.

[0064] Furthermore, in a captured image obtained by capturing a photo published in a magazine, the face of a person included in the captured image may not be facing forward. In this case, the terminal device 4 generates a composite image by replacing the facial image included in the captured image obtained by projectively transforming the captured image so that the person's face faces forward with the user's facial image. Furthermore, when multiple captured images are selected by the user U, the terminal device 4 generates a composite image by replacing the facial image included in the composite captured image obtained by combining the multiple captured images with the user's facial image.

[0065] In addition, when the hair image selected by the user U from among multiple hair images is changed based on the user U's operation, the terminal device 4 generates a modified composite image that is a composite image synthesized with the hair image whose selection was changed by the user U's operation, and displays the generated modified composite image on the display unit.

[0066] For example, each time a hair image selected by the user U from among the plurality of hair images is changed based on a first change operation by the user U, the terminal device 4 generates a corresponding changed composite image and displays the generated changed composite image on the display unit. The first change operation is, for example, a scroll operation in the left / right direction or a swipe operation in the left / right direction.

[0067] Furthermore, the terminal device 4 can also change at least one of the hair length and color of the hair image included in the composite image based on a second change operation by the user U. For example, the terminal device 4 changes at least one of the hair length and color of the hair image included in the composite image each time the second change operation is performed. The first change operation is, for example, a vertical scroll operation or a vertical swipe operation.

[0068] Furthermore, when terminal device 4 receives a hair image edit selected by user U, it can also generate an edited hair image by applying the received edit to the hair image. The hair image edit is, for example, editing to change at least one of the hair length and color of the hair in the hair image. In this case, in step S8, terminal device 4 generates a composite image by combining the user face image and the edited hair image, and displays the generated composite image on the display unit.

[0069] Next, the terminal device 4 uses the above-described fourth learning model to estimate the treatment details from the image of the user U before the treatment and the composite image generated in step S8 (step S9). The image of the user U before the treatment is, for example, the above-described user-captured image. The treatment details estimated in step S9 are treatment details necessary to achieve the hairstyle shown in the composite image generated in step S8, and may hereinafter be referred to as estimated treatment details.

[0070] In step S9, when the terminal device 4 inputs the image of the user U before the treatment and the composite image generated in step S8 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

[0071] Furthermore, the terminal device 4 determines the treatment cost, which is the cost required for the treatment based on the estimated treatment content, based on the determined estimated treatment content (step S10). For example, the terminal device 4 can determine the treatment cost for each hair salon based on treatment cost information for each hair salon that indicates the relationship between the treatment content and the treatment cost. The terminal device 4 can acquire the treatment cost information for each hair salon from, for example, the information providing device 1.

[0072] Furthermore, when the terminal device 4 receives a treatment specification from the user U, it compares the user-specified treatment specification, which is the treatment specification received, with the estimated treatment specification and generates comparison information indicating the comparison result (step S11). The comparison information includes, for example, information indicating the difference between the user-specified treatment specification and the estimated treatment specification. For example, if the estimated treatment specification includes, in addition to the user-specified treatment specification, one more bleaching, the terminal device 4 generates comparison information including information indicating that the number of bleachings is one more than the user-specified treatment specification.

[0073] Next, the terminal device 4 displays on the display unit treatment-related information including information indicating the treatment details estimated in step S9, treatment cost information which is information on the treatment costs determined in step S10, and comparison information generated in step S11 (step S12). This allows the user U to understand the treatment details and treatment costs estimated to be necessary to achieve the hairstyle shown in the selected hair image, and the terminal device 4 can improve convenience for the user U.

[0074] Next, the terminal device 4 receives an acceptance / rejection response, which is a response indicating whether the user U accepts or rejects the treatment content estimated in step S9 (step S13). If the treatment content estimated in step S9 is acceptable, the user U operates the terminal device 4 to issue a response indicating acceptance, and if the treatment content estimated in step S9 is not acceptable, the user U operates the terminal device 4 to issue a response indicating rejection.

[0075] Next, if the received permission response indicates permission, the terminal device 4 outputs a reservation request including a specific image and treatment content information to the hair salon professional (step S14). The specific image includes, for example, the composite image generated in step S8 or a hair image selected by the user U, and a user-captured image including a hair image of the user U before the treatment. The treatment content information includes at least one of information indicating the treatment content specified and accepted by the user U and information indicating the treatment content estimated in step S9.

[0076] The reservation request described above is output to the hair practitioner's store device 5 via the information providing device 1, for example, but may also be output to the hair practitioner's store device 5 via an SNS or a mail server (not shown). The hair practitioner's store device 5 receives the reservation request output from the terminal device 4 and displays the specific image and treatment content information included in the received reservation request. This allows the hair practitioner to check the specific image and treatment content information included in the reservation request.

[0077] The hairdresser can operate the in-store device 5 to modify the specific image included in the reservation request and the treatment details indicated in the treatment detail information. In this case, the in-store device 5 outputs modification information to the user U, including a modified specific image, which is the specific image after modification, and modified treatment detail information, which is information indicating the treatment details after modification. The modification information is output to the terminal device 4 via the information providing device 1, for example, but may also be output to the terminal device 4 via an SNS or a mail server (not shown).

[0078] The hairdresser can operate the in-store device 5 to input a comment to the in-store device 5 indicating whether or not the hairdresser can perform the hairstyle shown in the composite image included in the reservation request. In this case, the in-store device 5 outputs a hairdresser comment indicating whether or not the hairdresser can perform the hairstyle shown in the composite image to the user U. The comment is output to the terminal device 4 via the information providing device 1, for example, but may also be output to the terminal device 4 via an SNS or a mail server (not shown).

[0079] In addition, if there is no need to modify the specific image included in the reservation request or the treatment content indicated by the treatment content information included in the reservation request, the hair practitioner operates the store device 5 to output reservation acceptance information to the user U from the store device 5 indicating that the reservation has been accepted.

[0080] When the terminal device 4 acquires the correction information and the practitioner's comments from the in-store device 5, it displays the acquired correction information and the practitioner's comments on the display unit (step S15). This allows the user U to check the correction suggestions and comments from the hair practitioner, facilitating communication with the hair practitioner.

[0081] As described above, the terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U, and outputs the generated composite image to the hairdresser. This allows the user U to present the composite image, which combines the desired hair image and the user face image, to the hairdresser, and allows the hairdresser to specifically communicate the desired hairstyle compared to vaguely communicating the desired hairstyle to the hairdresser. Therefore, the terminal device 4 can improve the convenience for the user U.

[0082] Furthermore, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, with a hair image selected by user U, and estimates the treatment details from the pre-treatment image of user U and the composite image using a learning model that has learned the relationship between the hair images before and after the treatment and the treatment details. This allows user U to, for example, understand the treatment details required to achieve a desired hairstyle and can specifically communicate the treatment details required to achieve the desired hairstyle to the hairdresser. Therefore, terminal device 4 can improve convenience for user U.

[0083] Furthermore, terminal device 4, which is an example of an information processing device, accepts one or more keywords specified by user U, and identifies a hair image from the one or more keywords specified by the accepting unit using a learning model that inputs the one or more keywords and outputs a hair image. Then, terminal device 4 acquires similar hair images, which are images that include hair images similar to the identified hair image. This allows user U to easily collect hair images corresponding to a desired hairstyle simply by specifying a keyword. Therefore, terminal device 4 can improve convenience for user U.

[0084] Furthermore, terminal device 4, which is an example of an information processing device, identifies a hairstyle that is estimated to suit user U's face using a learning model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face, and acquires posted images including a specific hair image that is an image of the identified hairstyle. Then, based on the posted image acquired by the acquisition unit and the facial image of user U, terminal device 4 generates a composite image by combining the image of the hairstyle identified by the identification unit with the facial image of user U. This allows user U to easily acquire an image of himself or herself wearing the hairstyle that is estimated to suit user U's face. Therefore, terminal device 4 can improve the convenience of user U.

[0085] Furthermore, in the above example, various processes are performed by the terminal device 4, but some or all of the processes of the terminal device 4 described above may be performed mainly by the information providing device 1. For example, the information providing device 1 may provide the terminal device 4 with information generated based on information input from the terminal device 4 via an interface such as an API (Application Programming Interface), and the terminal device 4 may display the information based on the information provided by the information providing device 1. Furthermore, while FIG. 1 shows a case where the terminal device 4 and the information providing device 1 are separate devices, the terminal device 4 and the information providing device 1 may be integrated as an information processing device.

[0086] The configuration of an information processing system including the information providing device 1, the terminal device 4, and the store device 5 that perform such processing will be described in detail below.

[0087] [2. Information Processing System Configuration] 2 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. As illustrated in FIG. 2, the information processing system 100 according to the embodiment includes an information providing device 1, a plurality of terminal devices 4, and a plurality of in-store devices 5.

[0088] Each of the plurality of terminal devices 4 is used by a different user U. Each of the plurality of in-store devices 5 is installed, for example, in a different hair salon and used by a hair practitioner at a different hair salon.

[0089] The information providing device 1, the terminal device 4, and the store device 5 are connected to each other via a network N so as to be able to communicate with each other via a wired or wireless connection. The information processing system 100 shown in Fig. 2 may include a plurality of information providing devices 1. The network N is, for example, a local area network (LAN) or a wide area network (WAN) such as the Internet.

[0090] The information providing device 1 is an information processing device that provides various types of information. The information providing device 1 provides online services such as SNS sites, web search sites, and content distribution sites. The SNS sites accept posted information sent from terminal devices 4, store devices 5, and the like, and transmit the posted information to the terminal devices 4 and store devices 5 based on requests from users U, hair practitioners, and the like. The information providing device 1 may also be an information processing device that acquires information such as posted information from the above-mentioned SNS sites and the like.

[0091] Furthermore, the content distribution site is a site that provides content distribution services, and includes video / music distribution sites, map sites, route search sites, route guidance sites, line information sites, traffic information sites, weather forecast sites, etc. Furthermore, the information providing device 1 can also provide online services through various online sites, such as electronic payment sites, online game sites, online banking sites, or accommodation / ticket reservation sites.

[0092] Each of the terminal device 4 and the in-store device 5 is, for example, a desktop personal computer (PC), a notebook PC, a tablet terminal, a smartphone, a mobile phone, or a personal digital assistant (PDA). Note that each of the terminal device 4 and the in-store device 5 is not limited to the above examples and may be, for example, a smart watch or a wearable device.

[0093] In addition, each terminal device 4 can connect to the network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN, and communicate with the information providing device 1 and each store device 5.

[0094] In addition, each store device 5 can connect to the network N via a wireless communication network such as LTE, 4G, or 5G, or short-range wireless communication such as Bluetooth or wireless LAN, and communicate with the information providing device 1 and each terminal device 4.

[0095] 3. Configuration of Information Providing Device 1 Next, a configuration example of the information providing device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information providing device 1 according to an embodiment. As shown in Fig. 3, the information providing device 1 has a communication unit 10, a storage unit 11, and a processing unit 12.

[0096] [3.1. Communication Unit 10] The communication unit 10 is realized by, for example, a network interface card (NIC). The communication unit 10 is connected to a network by wire or wirelessly, and transmits and receives information to and from the terminal device 4, the store device 5, and the like via the network N.

[0097] [3.2. Storage section 11] The storage unit 11 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 11 stores a user information storage unit 20, a posted information storage unit 21, a model information storage unit 22, and a content storage unit 23.

[0098] 3.2.1. User Information Storage Unit 20 The user information storage unit 20 stores information about the user U. Fig. 4 is a diagram showing an example of a user information table stored in the user information storage unit 20 of the information providing device 1 according to the embodiment.

[0099] 4, the user information table stored in the user information storage unit 20 includes items such as “user ID (IDentifier),” “attribute information,” and “history information.” The “user ID” is identification information that identifies the user U.

[0100] "Attribute information" is information about user attributes, which are attributes of user U corresponding to "user ID," and includes information about psychographic attributes, information about demographic attributes, etc. Demographic attributes include, for example, gender, age, place of residence, and occupation, while psychographic attributes include interests such as travel, clothing, cars, and religion, lifestyle, thoughts, and ideological tendencies.

[0101] "History information" is history information including information such as the service usage history of user U corresponding to "user ID," and includes, for example, posting history information of user U, search history information of user U, and browsing history information of user U. Posting history information of user U includes posted information posted by user U to the SNS site, comments by user U on posted information of other users U, and information indicating user U's evaluation of posted information of other users U.

[0102] The search history information of the user U is, for example, information on the search history of posted information on the SNS site, information on the search history on various websites, etc. The browsing history information of the user U is, for example, information on the browsing history of posted information on the SNS site by the user U, information on the browsing history on various websites, etc.

[0103] 3.2.2. Posted Information Storage Unit 21 The posted information storage unit 21 stores various types of information posted by each user U. Fig. 5 is a diagram showing an example of a posted information table stored in the posted information storage unit 21 of the information providing device 1 according to the embodiment. As shown in Fig. 5, the posted information table stored in the posted information storage unit 21 includes items such as "post ID," "posting date and time," "posting user," "posted text," "posted image," "rating information," "number of views," and "comments."

[0104] "Post ID" is identification information that identifies posted information. Post information is an example of posted content. "Posting date and time" is information that indicates the posting date and time of the posted information corresponding to the "Posting ID." "Posting user" is information that indicates the user U who posted the posted information corresponding to the "Posting ID," and in the example shown in Figure 5, it is the user ID. "Posting text" is information that indicates the text included in the posted information corresponding to the "Posting ID," and is expressed as a character string, but may also include stamps, etc.

[0105] "Posted image" is a posted image included in the posted information corresponding to the "post ID." "Rating information" is rating information indicating the ratings of other users U regarding the posted information corresponding to the "post ID," but may also include information indicating the rating of the posting user regarding the posted information corresponding to the "post ID." "Number of views" is information indicating the number of times the posted information corresponding to the "post ID" has been viewed.

[0106] "Comment" is a comment from another user U on the posted information corresponding to the "Posting ID", and is indicated by, for example, a character string, but may also include stamps, etc. Note that "Comment" may also include a comment from the posting user on the posted information corresponding to the "Posting ID".

[0107] The evaluation information shown in FIG. 5 is information indicating a positive evaluation (e.g., "good") or a negative evaluation (e.g., "bad"), but may also include information indicating an intermediate evaluation (e.g., "average").

[0108] 3.2.3. Model Information Storage Unit 22 The model information storage unit 22 stores information on various learning models. Fig. 6 is a diagram showing an example of a model information table stored in the model information storage unit 22 of the information providing device 1 according to the embodiment.

[0109] As shown in FIG. 6, the model information table stored in the model information storage unit 22 includes items such as "model ID" and "model information." The "model ID" is identification information that identifies a learning model. The "model information" is information about the learning model corresponding to the "model ID." The information about the learning model is information such as parameters of the learning model, but is not limited to this example and may be information for using the learning model in the terminal device 4.

[0110] The learning models whose information is included in the model information table include, for example, the first learning model, the second learning model, the third learning model, and the fourth learning model described above. The first learning model is a model that learns the relationship between keywords and hair images, and is a learning model that inputs one or more keywords and outputs a hair image or hair image-specific information.

[0111] The second learning model is a model that has learned the relationship between face images and facial features, and is a learning model that takes an image containing a face image as input and outputs information indicating the facial features. The third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face, and is a learning model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit the face.

[0112] The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment content, and is a learning model that inputs hair images before and after a treatment and outputs information indicating the estimated treatment content. Note that the learning model whose information is included in the model information table is not limited to the above-mentioned examples.

[0113] Any type of model can be adopted as each of the above-mentioned learning models. For example, each learning model is a model generated by machine learning, such as DNN, GBDT, or SVM. The DNN is, for example, CNN or RNN. The RNN may be, for example, LSTM. Each learning model may also be a model realized by combining multiple models, such as a model combining CNN and RNN.

[0114] 3.2.4. Content Storage Unit 23 The content storage unit 23 stores information about content. Fig. 7 is a diagram showing an example of a content table stored in the content storage unit 23 of the information providing device 1 according to the embodiment. The content table shown in Fig. 7 includes items such as "content ID" and "content."

[0115] The "content ID" is an identifier that identifies the content. The "content" is information related to the content associated with the "content ID." Specifically, the information related to the content is, for example, information related to the content content or information related to the address of the content.

[0116] 7, the content with content ID "C1" is content CNT1, the content with content ID "C2" is content CNT2, and the content with content ID "C3" is content CNT3. Note that the content storage unit 23 is not limited to the above and may store various types of information depending on the purpose.

[0117] [3.3. Processing Unit 12] The processing unit 12 is a controller, and is realized by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the information providing device 1 using RAM or the like as a working area.

[0118] The processing unit 12 is a controller, and may be realized by an integrated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a general purpose graphic processing unit (GPGPU).

[0119] 3, the processing unit 12 has an acquisition unit 30, a learning unit 31, and a provision unit 32, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the processing unit 12 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs the information processing described below.

[0120] [3.3.1. Acquisition part 30] The acquisition unit 30 acquires various types of information. The acquisition unit 30 acquires various types of information from the storage unit 11. The acquisition unit 30 acquires various types of information from the user information storage unit 20, the posted information storage unit 21, the model information storage unit 22, the content storage unit 23, etc.

[0121] The acquisition unit 30 also acquires information about the user U (for example, attribute information and behavior history information about the user U) from, for example, the terminal device 4, the store device 5, or an external device, and stores the acquired information about the user U in the user information storage unit 20. The acquisition unit 30 also acquires various types of content from, for example, an external device, and stores the acquired content in the content storage unit 23.

[0122] The acquisition unit 30 also acquires posted information from devices such as the terminal device 4 or the store device 5 via the communication unit 10. The acquisition unit 30 adds the acquired posted information to a posted information table stored in the posted information storage unit 21.

[0123] Furthermore, the acquiring unit 30 acquires an acquisition request for posted information from the terminal device 4 via the communication unit 10. For example, the acquiring unit 30 notifies the providing unit 32 of information relating to the acquired acquisition request.

[0124] The acquisition unit 30 also acquires various information such as reservation requests from the terminal devices 4 via the communication unit 10, stores the acquired various information such as reservation requests in the storage unit 11, and notifies the provision unit 32 of the acquired various information such as reservation requests. The acquisition unit 30 acquires various information such as correction information from the in-store devices 5 via the communication unit 10, and notifies the provision unit 32 of the acquired various information such as correction information.

[0125] Furthermore, the acquisition unit 30 acquires learning data for generating the above-described learning model, and notifies the acquired learning data to the learning unit 31. For example, the acquisition unit 30 acquires information stored in the storage unit 11 from the storage unit 11, and notifies the acquired information to the learning unit 31 as learning data.

[0126] The training data for the first training model is, for example, training data including a plurality of combinations of keywords and hair images, and the training data for the second training model is, for example, training data including a plurality of combinations of images including face images and information indicating facial features.

[0127] The learning data for the third learning model is, for example, learning data including a plurality of combinations of information indicating facial features, a hair image, and information indicating an evaluation of the combination of the information indicating facial features and the hair image. The evaluation of the combination of the information indicating facial features and the hair image is, for example, information indicating a positive evaluation or information indicating a negative evaluation, and is used as a label. The learning data for the fourth learning model is, for example, learning data including a plurality of combinations of hair images before and after a treatment and information indicating the treatment content.

[0128] The acquiring unit 30 can also generate the above-mentioned learning data based on the information stored in the storage unit 11. The acquiring unit 30 can also acquire or generate learning data other than those described above.

[0129] [3.3.2. Learning Section 31] The learning unit 31 generates various learning models by machine learning using the various learning data notified by the acquisition unit 30, and stores information on the various learning models generated in the posted information storage unit 21.

[0130] For example, the learning unit 31 generates a first learning model using the learning data for the first learning model, generates a second learning model using the learning data for the second learning model, generates a third learning model using the learning data for the third learning model, and generates a fourth learning model using the learning data for the fourth learning model.

[0131] The learning unit 31 can also generate learning models other than the first learning model, the second learning model, the third learning model, and the fourth learning model.

[0132] [3.3.3.Providing Department 32] The providing unit 32 starts the communication unit 10 and provides various contents to the terminal device 4 of the user U and the store device 5 of the hair practitioner. For example, the providing unit 32 generates viewing content including information posted on an SNS based on a request from the terminal device 4 or the store device 5, and provides the viewing content to the terminal device 4 or the store device 5 via the communication unit 10 and the network N.

[0133] Furthermore, the providing unit 32 provides the various types of information notified by the acquiring unit 30 to the terminal device 4 or the store device 5, etc. via the communication unit 10 and the network N, etc. For example, the providing unit 32 provides the information from the terminal device 4 notified by the acquiring unit 30 to the store device 5, etc. via the communication unit 10 and the network N, etc., and provides the information from the store device 5 notified by the acquiring unit 30 to the terminal device 4, etc. via the communication unit 10 and the network N, etc.

[0134] In addition, the providing unit 32 provides information on the various learning models generated by the learning unit 31 to the terminal device 4 via the communication unit 10, the network N, etc. For example, the providing unit 32 provides the first learning model, the second learning model, the third learning model, and the fourth learning model generated by the learning unit 31 to the terminal device 4 via the communication unit 10, the network N, etc.

[0135] Furthermore, the providing unit 32 provides an application program (hereinafter referred to as an app) used in the terminal device 4 to the terminal device 4 via the communication unit 10, the network N, etc. Such an app is installed in the terminal device 4, for example, and executes the above-described processes in the terminal device 4. For example, the functions of such an app enable the terminal device 4 to generate the above-described composite image and acquire various pieces of information using the above-described learning model.

[0136] [4. Terminal Device 4] Fig. 8 is a diagram showing an example of the configuration of the terminal device 4 according to the embodiment. As shown in Fig. 8, the terminal device 4 according to the embodiment includes a communication unit 40, a display unit 41, an operation unit 42, an imaging unit 43, a sensor unit 44, a storage unit 45, and a processing unit 46.

[0137] 4.1. Communication Unit 40 The communication unit 40 is realized by, for example, a NIC. The communication unit 40 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the information providing device 1, the in-store device 5, and the like via the network N. Communication with the in-store device 5 is performed, for example, via the information providing device 1, an SNS, or a mail server (not shown), but communication with the in-store device 5 may also be performed directly without going through these.

[0138] [4.2.Display section 41] The display unit 41 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0139] [4.3. Operation unit 42] The operation unit 42 includes, for example, a keyboard including keys for inputting letters, numbers, and spaces, an enter key, arrow keys, etc., a mouse, a power button, etc. If the display unit 41 is a touch panel compatible display, the operation unit 42 includes a touch panel.

[0140] [4.4. Imaging unit 43] The imaging unit 43 is an image sensor (camera) that captures an image of a subject. For example, the imaging unit 43 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor. Note that the imaging unit 43 is not limited to a built-in camera, and may be a wireless camera that can communicate with the terminal device 4, an external camera such as a web camera, or the like.

[0141] [4.5. Sensor unit 44] The sensor unit 44 includes a position detection unit, a gyro sensor, etc. The position detection unit detects, for example, the position of the terminal device 4, which is the current position of the user U, and outputs information indicating the detected current position of the user U to the processing unit 46.

[0142] The position detection unit receives a plurality of positioning signals transmitted from a plurality of positioning satellites in the Global Navigation Satellite System (GNSS), and detects the current position of the user U based on the received plurality of positioning signals. The gyro sensor is a sensor that detects the attitude of the terminal device 4, such as tilt and rotation.

[0143] [4.6. Storage section 45] The storage unit 45 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0144] The storage unit 45 stores, for example, information transmitted from the information providing device 1, the store device 5, etc. and acquired by the processing unit 46 via the network N and the communication unit 40, captured images that are images captured by the imaging unit 43, and detected information that is information detected by the sensor unit 44. For example, the storage unit 45 stores information on apps and various learning models transmitted from the information providing device 1.

[0145] [4.7. Processing section 46] The processing unit 46 is a controller, and is realized by, for example, a CPU or an MPU executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the terminal device 4 using a RAM as a work area. The processing unit 46 may be realized in part or in whole by an integrated circuit such as an ASIC or an FPGA.

[0146] The processing unit 46 includes a reception unit 50, an acquisition unit 51, an identification unit 52, an image processing unit 53, an estimation unit 54, a determination unit 55, a comparison unit 56, and an output unit 57. The processing unit 46 functions as a functional unit including the reception unit 50, the acquisition unit 51, the identification unit 52, the image processing unit 53, the estimation unit 54, the determination unit 55, the comparison unit 56, and the output unit 57, for example, by executing the above-mentioned application on an OS (Operating System).

[0147] In addition, multiple functions including the reception unit 50, acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, judgment unit 55, comparison unit 56, and output unit 57 may be pre-integrated in the processing unit 46.

[0148] 4.7.1. Reception unit 50 The reception unit 50 receives various requests and information. The reception unit 50 receives requests and information transmitted from the terminal device 4 and the in-store device 5.

[0149] For example, the reception unit 50 receives correction information transmitted from the in-store device 5. The correction information is information indicating correction suggestions made by the hair treatment professional. The reception unit 50 also receives reservation acceptance information transmitted from the in-store device 5. The reservation acceptance information includes information indicating that a reservation has been accepted, such as the treatment content, in the reservation request.

[0150] The reception unit 50 also receives various operations by the user U using the operation unit 42, etc. For example, the reception unit 50 receives various operations such as the user U selecting a hair image, the user U editing a hair image, the user U specifying the treatment details, the user U approving or not approving the estimated treatment details, and the user U specifying the treatment details.

[0151] Furthermore, the reception unit 50 receives an imaging operation from the user U. When the reception unit 50 receives an imaging operation from the user U, the reception unit 50 operates the imaging unit 43 to cause the imaging unit 43 to perform an imaging operation.

[0152] The receiving unit 50 also receives a first change operation of the hair image by the user U, a second change operation of the hair image by the user U, etc. The first change operation is an operation for changing the composite image by changing the hair image used to generate the composite image. The second change operation is an operation for changing at least one of the hair length and color of the hair image.

[0153] 9 is a diagram illustrating an example of the configuration of the reception unit 50 of the terminal device 4 according to the embodiment. As illustrated in FIG. 9, the reception unit 50 includes a selection reception unit 60, an edit reception unit 61, a designation reception unit 62, a revision proposal reception unit 63, and an approval / non-approval reception unit 64.

[0154] The selection receiving unit 60 receives various selections by the user U. For example, the selection receiving unit 60 receives a selection of a hair image by the user U from a plurality of types of hair images. The user U can perform a hair image selection operation by operating the operation unit 42, and the selection receiving unit 60 receives the selection of the hair image based on the hair image selection operation.

[0155] The hair image selected by the user U is, for example, one or more of an image of a hairstyle identified by the identification unit 52, a plurality of similar hair images acquired by the acquisition unit 51, and a hair image included in a captured image captured by the imaging unit 43.

[0156] It should be noted that the hair image selected by the user U is not limited to the above-described example. For example, the hair image selected by the user U may include a hair image selected by the processing unit 46 when an automatic hair image selection mode by the processing unit 46 is selected by an operation of the user U.

[0157] The processing receiving unit 61 receives processing of the hair image. For example, the processing receiving unit 61 receives a selection of at least one of the hair length and color of the hair image. The hair in the hair image is the hair shown in the hair image, and the user U can perform processing operations on the hair image by operating the operation unit 42, and the processing receiving unit 61 receives processing of the hair image based on the processing operations on the hair image.

[0158] The specification receiving unit 62 receives various specifications from the user U. For example, the specification receiving unit 62 receives a specification of treatment content from the user U. The user U can perform a specification operation of the treatment content by operating the operation unit 42, and the specification receiving unit 62 receives the specification of the treatment content based on the specification operation of the treatment content.

[0159] Furthermore, the designation receiving unit 62 receives, based on an operation by the user U on the operation unit 42, a designation of a user-captured image that is an image captured by the imaging unit 43 and that includes a hair image.

[0160] Furthermore, the designation receiving unit 62 receives the user U's permission or denial of the estimated treatment content. The user U can perform an operation to permit or deny the estimated treatment content by operating the operation unit 42, and the designation receiving unit 62 receives the permission or denial of the treatment content based on the permission operation or denial operation.

[0161] Furthermore, the designation receiving unit 62 receives designation of one or more keywords by the user U. For example, the designation receiving unit 62 receives designation of one or more keywords from among a plurality of keywords by the user U. The user U can perform a designation operation of one or more keywords by operating the operation unit 42, and the designation receiving unit 62 receives designation of one or more keywords based on the designation operation of one or more keywords.

[0162] The revision proposal receiving unit 63 receives a revision proposal from the hair practitioner to the estimated treatment content. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating a revision proposal from the hair practitioner to the estimated treatment content, from the in-store device 5 as a revision proposal to the estimated treatment content by the hair practitioner.

[0163] The revision proposal receiving unit 63 also receives a revision proposal from the hairdresser for the hairstyle displayed in the composite image. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating a revision proposal from the hairdresser, from the in-store device 5 as a revision proposal for the hairstyle from the hairdresser.

[0164] The permission acceptance unit 64 accepts the user U's permission or denial of the estimated treatment content, which is the treatment content estimated by the estimation unit 54. The user U can perform a permission operation indicating permission for the estimated treatment content or a denial operation indicating denial of the estimated treatment content by operating the operation unit 42, and the permission acceptance unit 64 accepts the user U's permission or denial of the estimated treatment content based on the permission operation or denial operation.

[0165] [4.7.2. Acquisition part 51] The acquisition unit 51 acquires various types of information. For example, the acquisition unit 51 acquires various types of information from the storage unit 45. The acquisition unit 51 acquires information on the learning model stored in the storage unit 45.

[0166] The acquisition unit 51 also acquires captured images, which are images captured by the imaging unit 43, and stores the acquired captured images in the storage unit 45. The acquisition unit 51 also acquires detection information, which is information detected by the sensor unit 44, and stores the acquired detection information in the storage unit 45.

[0167] The acquisition unit 51 acquires information on apps and learning models transmitted from the information providing device 1 via the network N and the communication unit 40, and stores the acquired information on apps and learning models in the storage unit 45. The acquisition unit 51 acquires multiple pieces of posted information each including a specific hair image from the information providing device 1, and stores the acquired multiple pieces of posted information in the storage unit 45.

[0168] The acquisition unit 51 also acquires a practitioner's comment and stores the acquired comment in the storage unit 45. The practitioner's comment is a comment from the hair practitioner as to whether or not the hair style shown in the composite image can be performed.

[0169] Furthermore, the acquisition unit 51 acquires information according to an operation on the operation unit 42 by the user U. For example, the acquisition unit 51 acquires, from the storage unit 45, an image captured by the imaging unit 43 and including a hair image, based on the operation by the user U.

[0170] The acquisition unit 51 also acquires a plurality of similar hair images, which are images including hair images similar to the specific hair image, and stores the acquired plurality of similar hair images in the storage unit 45. The specific hair image is an image of a hairstyle identified by the identification unit 52. For example, the acquisition unit 51 transmits a similar image search request including the specific hair image to a search server or information providing device 1 (not shown), and acquires a plurality of similar hair images transmitted from the search server or information providing device 1 in response to the similar image search request.

[0171] In addition, the acquisition unit 51, for example, transmits a search request including information indicating the hairstyle identified by the identification unit 52 to a search server or information providing device 1 (not shown), and acquires multiple identified hair images transmitted from the search server or information providing device 1 in response to the search request.

[0172] The acquisition unit 51 also acquires treatment cost information for each hair salon, which indicates the relationship between the treatment content and the treatment cost, from the information providing device 1 or the store device 5, for example.

[0173] [4.7.3. Specification part 52] The identification unit 52 uses the first learning model to identify a hair image from one or more keywords whose designation has been accepted by the acceptance unit 50. As described above, the first learning model is a model that has learned the relationship between keywords and hair images, and takes one or more keywords as input and outputs a hair image or hair image identification information.

[0174] When the first learning model outputs a hair image, the specifying unit 52 specifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by the user U.

[0175] Furthermore, when the first learning model outputs hair image identification information instead of a hair image, the identification unit 52 inputs one or more keywords specified by the user U into the first learning model and acquires the hair image identification information output from the first learning model. The hair image identification information is information that identifies a hair image, such as a score for each hair image. For example, the identification unit 52 identifies the hair image with the highest score among the scores for each hair image as the hair image corresponding to the one or more keywords specified by the user U.

[0176] The identification unit 52 acquires a hair image identified by the image identification information from the memory unit 45 or the information providing device 1, etc., and identifies the acquired hair image as a hair image corresponding to one or more keywords specified by the user U.

[0177] Furthermore, the identification unit 52 uses the third learning model to identify a hairstyle that is estimated to suit the face of the user U. For example, the identification unit 52 uses the third learning model to identify a hairstyle that is estimated to suit the face of the user U from the facial features of the user U specified by the user U or the facial features of the user U determined by the determination unit 55.

[0178] As described above, the third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting that face, and is a model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit that face.

[0179] The identification unit 52 inputs information indicating the facial features of the user U into a third learning model, and identifies a hairstyle that is estimated to suit the face of the user U based on the hairstyle information output from the third learning model. The hairstyle information output from the third learning model is, for example, a score for each hair image, and the identification unit 52 identifies the hairstyle indicated by the hair image with the highest score among the scores for each hair image as the hairstyle that is estimated to suit the face of the user U. The hairstyle information output from the third learning model may also be a hair image. The information indicating the facial features of the user U is, for example, information indicating the facial features of the user U specified by the user U or information indicating the facial features of the user U determined by the determination unit 55.

[0180] Facial features are, for example, information indicating the type of face shape, but may also be information indicating the features of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears. For example, the information indicating facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears.

[0181] [4.7.4. Image Processing Unit 53] The image processing unit 53 performs various types of image processing. For example, the image processing unit 53 generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U. Hereinafter, the hair image selected by the user U may be referred to as a selected hair image.

[0182] The selected hair image is a hair image whose selection is accepted by the selection accepting unit 60, but is not limited to this example. For example, the selected hair image may include a hair image selected by the processing unit 46 when an automatic hair image selection mode by the processing unit 46 is selected by an operation of the user U. The user face image is, for example, a user-captured image whose specification is accepted by the specification accepting unit 62 or a face image of the user U included in the user-captured image.

[0183] For example, the image processing unit 53 generates a composite image by combining the selected hair image with the user face image. The image processing unit 53 generates a composite image by replacing the hair image of the user U included in the user face image with the selected hair image.

[0184] The image processing unit 53 extracts, for example, feature amounts of the user's face image and feature amounts of the selected hair image, and generates a composite image by replacing the feature amounts of the hair image included in the user's face image with the feature amounts of the selected hair image.

[0185] In addition, the image processing unit 53 can also generate a composite image by deleting the hair image of the user U included in the user face image and superimposing the selected hair image on the user face image from which the hair image of the user U has been deleted.

[0186] In addition, when the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image, which is an image including a user face image, it can also replace the user face image included in the user image with the selected hair image.

[0187] Furthermore, when two or more hair images are selected by the selection receiving unit 60, the image processing unit 53 generates a composite hair image by combining the two or more selected hair images. Then, the image processing unit 53 generates a composite image by combining the generated composite hair image with the user's face image.

[0188] For example, the image processing unit 53 extracts feature amounts from each of two or more selected hair images and generates a composite hair image by combining the extracted feature amounts of the two or more selected hair images. For example, the image processing unit 53 generates a composite hair image by combining the feature amounts of the two or more selected hair images by weighted addition, but the combining method is not limited to this example. For example, the image processing unit 53 has a synthesis model that is a learning model for synthesizing two or more images and is generated by machine learning, and by inputting two or more selected hair images into the synthesis model, a composite hair image output from the synthesis model can be obtained.

[0189] When the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image (e.g., a user-captured image), which is an image including a user's face image, it can also replace the face image of a person included in the selected image with the user's face image included in the user image.

[0190] For example, assume that the selected image is a selected captured image that is captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the facial image of a person included in the selected captured image with the user facial image included in the user image.

[0191] When the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image, which is an image including a user face image, it can also replace the user face image included in the user image with the face image of a person included in the selected image.

[0192] For example, assume that the selected image is a selected captured image that is captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the user's face image in the user image with the face image of a person included in the selected captured image.

[0193] Furthermore, when there are multiple selected images, the image processing unit 53 synthesizes the multiple selected images to generate a composite selected image. For example, the image processing unit 53 synthesizes the multiple selected images based on a specific selected image to generate a composite selected image. The specific selected image is, for example, the initially selected selected image or the selected image designated by the user U.

[0194] For example, the image processing unit 53 extracts a feature amount of each hair image of a plurality of selected images, synthesizes the extracted feature amounts of the plurality of hair images to generate a composite hair image, and replaces the hair image of a specific selected image with the composite hair image to generate a composite selected image. The image processing unit 53 also has a synthesis model that is a learning model for synthesizing a plurality of images and is generated by machine learning, and can obtain a composite hair image output from the synthesis model by inputting a plurality of selected images to the synthesis model.

[0195] In addition, when the processing reception unit 61 accepts processing of a hair image included in the composite image, the image processing unit 53 generates a user face image including a processed hair image obtained by processing the hair image accepted by the processing reception unit 61.

[0196] For example, when the processing receiving unit 61 receives a request to process a hair image, the image processing unit 53 generates a processed hair image obtained by processing the hair image received by the processing receiving unit 61. Then, the image processing unit 53 generates a composite image by combining the processed hair image with the user's face image.

[0197] For example, image processing unit 53 extracts feature amounts of the hair image and adjusts the extracted feature amounts to generate an edited hair image, but is not limited to this example. Furthermore, when edit receiving unit 61 receives an edit request for the hair image in the composite image, image processing unit 53 can also extract feature amounts of the hair image in the composite image and edit the hair image in the composite image by adjusting the extracted feature amounts.

[0198] The image processing unit 53 can also perform projective transformation of the selected image. Fig. 10 is a diagram showing an example of the configuration of the image processing unit 53 of the terminal device 4 according to the embodiment. As shown in Fig. 10, the image processing unit 53 includes a transformation unit 65 and a synthesis unit 66. The transformation unit 65 performs projective transformation of the selected image. The synthesis unit 66 generates a synthesized image by replacing the facial image of a person included in the selected image or the facial image of a person included in the selected image projectively transformed by the transformation unit 65 with the user's facial image.

[0199] The transformation unit 65 can also perform projection transformation on a user image or a user face image in place of the selected image. In this case, the synthesis unit 66 generates a composite image by replacing the face image of a person included in the selected image or the face image of a person included in the selected image projectively transformed by the transformation unit 65 with the face image of user U included in the user image projectively transformed by the transformation unit 65 or the user face image projectively transformed by the transformation unit 65.

[0200] In addition, the synthesis unit 66 can also generate a composite image by replacing the facial image of user U included in the user image projectively transformed by the transformation unit 65 with the facial image of a person included in the selected image or the facial image of a person included in the selected image projectively transformed by the transformation unit 65.

[0201] [4.7.5. Estimation section 54] The estimation unit 54 estimates the treatment details from the pre-treatment image and the composite image of the user U. For example, the estimation unit 54 estimates the treatment details from the pre-treatment image and the composite image of the user U using a fourth learning model.

[0202] The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment details, and takes hair images before and after a treatment as input and outputs information indicating the estimated treatment details.

[0203] The image of the user U before the treatment is, for example, the user-captured image described above. The treatment details estimated by the estimation unit 54 are the estimated treatment details described above, and are treatment details necessary to achieve the hairstyle shown in the composite image generated by the image processing unit 53.

[0204] When the estimation unit 54 inputs the image of the user U before the treatment and the composite image generated by the image processing unit 53 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

[0205] The information indicating the treatment content output from the fourth learning model includes, for example, information on the scores of whether or not a haircut was performed, the amount of haircut, whether or not bleaching was performed, the number of times bleaching was performed, whether or not coloring was performed, the type of coloring, whether or not perming was performed, the type of perming, and whether or not straightening was performed. The estimation unit 54 determines the estimated treatment content based on the scores output from the fourth learning model.

[0206] For example, the estimation unit 54 includes a cut in the estimated treatment details when the score for whether or not a cut was performed is equal to or greater than a threshold, and does not include a cut in the estimated treatment details when the score for whether or not a cut was performed is less than the threshold. Furthermore, the estimation unit 54 may, for example, include in the estimated treatment details the amount of cut proportional to the score for the amount of cut. Furthermore, the estimation unit 54 may, for example, include in the estimated treatment details colors with scores equal to or greater than a threshold among the scores for each type of color.

[0207] Furthermore, the information indicating the treatment content from the fourth learning model may be information indicating the treatment content including the treatment items and the treatment order. In this case, the fourth learning model is formed to output scores of multiple treatment methods each including the treatment items and the treatment order, and the treatment content indicated by the treatment method with the highest score is determined as the estimated treatment content.

[0208] [4.7.6. Judgment unit 55] The determining unit 55 determines the treatment cost, which is the cost required for the treatment according to the estimated treatment content, based on the estimated treatment content determined by the estimating unit 54.

[0209] For example, the determination unit 55 has treatment cost information for each hair salon that indicates the relationship between the treatment content and the treatment cost, and determines the treatment cost for each hair salon based on the treatment cost information. The treatment cost information for each hair salon is information acquired by the acquisition unit 51 as described above, but is not limited to this example.

[0210] Furthermore, the determination unit 55 determines, for example, facial features of the user U shown in the user-captured image. For example, the determination unit 55 inputs the user-captured image into the second learning model described above, and determines the facial features of the user U based on information indicating the facial features output from the second learning model.

[0211] The information indicating the facial features output from the second learning model is, for example, a score for each facial feature, and in this case, the determination unit 55 determines the facial feature with the highest score among the scores for each facial feature as the facial feature of user U.

[0212] Furthermore, the information indicating the facial features output from the second learning model may be a score for each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature including the feature with the highest score among the scores for each feature (combination of position, size, and shape) as the facial feature of the user U.

[0213] Furthermore, the information indicating the facial features output from the second learning model may be a score for each feature of the size and shape of each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature including the feature with the highest score among the scores for each feature of the size and shape as the facial feature of the user U.

[0214] In addition, the determination unit 55 may have information indicating each of multiple types of facial features, in which case the determination unit 55 determines the facial features of the user U using a pattern matching technique between each type of facial feature and the facial features of the user U shown in the user-captured image.

[0215] In addition, the determination unit 55 can display facial feature line drawings, which are line drawings showing each of multiple types of facial features (for example, face shapes), on the display unit, and allow the user U to select the facial feature line drawing that is closest to his or her own face shape from among these multiple types of facial feature line drawings.

[0216] [4.7.7. Comparison Unit 56] The comparison unit 56 compares the user-specified treatment content, which is the treatment content specified by the specification receiving unit 62, with the estimated treatment content, which is the treatment content estimated by the estimation unit .

[0217] The comparison unit 56, for example, compares the user-specified treatment content with the estimated treatment content and generates comparison information indicating the comparison result. The comparison information includes, for example, information indicating the difference between the user-specified treatment content and the estimated treatment content.

[0218] For example, if the estimated treatment content includes one more bleach than the user-specified treatment content, the comparison unit 56 generates comparison information including information indicating that the number of bleaches is one more than the user-specified treatment content.

[0219] In addition, when the estimated treatment content or the user-specified treatment content includes a treatment order, if part of the treatment order is different, the comparison unit 56 generates comparison information including information indicating the different order of the treatment orders.

[0220] [4.7.8. Output section 57] The output unit 57 outputs, as output information, information indicating the identification result by the identification unit 52, information indicating the result of image processing by the image processing unit 53, information indicating the estimation result by the estimation unit 54, information indicating the judgment result by the judgment unit 55, and information indicating the comparison result by the comparison unit 56.

[0221] The output unit 57 can also output, as output information, information indicating the reception result by the reception unit 50, information indicating the acquisition result by the acquisition unit 51, and the like. For example, the output unit 57 can output the output information to the display unit 41 to display the output information on the display unit 41, or output the output information to the hair practitioner via the communication unit 40 and the network N, for example. The output to the hair practitioner is performed, for example, via the information providing device 1, an SNS, or a mail server (not shown), but may also be output directly to the in-store device 5.

[0222] The output unit 57 outputs information on a reservation setting screen for reserving a hair treatment at a hair salon as output information. The reservation setting screen is a GUI (Graphical User Interface) screen for setting a user image, a hair image, a hair salon, a reservation date and time, etc., and will be described in detail later.

[0223] The output unit 57 outputs the reservation request including the treatment-related information to the hair treatment provider, for example, by transmitting the reservation request including the treatment-related information to the hair treatment provider.

[0224] The treatment-related information includes, for example, a user image including a hair image of the user U before the treatment, a composite image generated by the image processing unit 53, information indicating the treatment content specified and accepted by the specification acceptance unit 62, and information indicating the treatment content estimated by the estimation unit 54.

[0225] The treatment-related information may include a selected hair image, which is a hair image whose selection has been accepted by the selection accepting unit 60, instead of or in addition to the composite image generated by the image processing unit 53.

[0226] Furthermore, the output unit 57 outputs a plurality of keywords so that the user U can specify them. For example, the output unit 57 outputs output information including the plurality of keywords to the display unit 41, thereby causing the display unit 41 to display the plurality of keywords so that the user U can specify them.

[0227] Furthermore, the output unit 57 outputs the practitioner comments acquired by the acquisition unit 51. For example, the output unit 57 outputs the practitioner comments acquired by the acquisition unit 51 to the display unit 41, thereby causing the practitioner comments to be displayed on the display unit 41.

[0228] Furthermore, the output unit 57 outputs revision proposal information, which is information indicating revision proposals made by the hair practitioner and received by the revision proposal receiving unit 63. The revision proposal information is, for example, information indicating revision proposals for the estimated treatment content or information indicating revision proposals for the hairstyle shown in the composite image. For example, the output unit 57 outputs the revision proposal information received by the revision proposal receiving unit 63 to the display unit 41, thereby causing the display unit 41 to display the revision proposals made by the hair practitioner.

[0229] In addition, if permission is accepted by the permission acceptance unit 64, the output unit 57 outputs the above-mentioned reservation request to the hair practitioner, and if permission is not accepted by the permission acceptance unit 64, the output unit 57 does not output the above-mentioned reservation request to the hair practitioner.

[0230] When output unit 57 displays the output information on display unit 41, it converts the output information into information for display by, for example, rendering the output information, and outputs the converted output information to display unit 41. When display unit 41 has a rendering function, output unit 57 outputs the output information to display unit 41 as is, and causes display unit 41 to display the output information.

[0231] 11 is a diagram showing an example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 11, the reservation setting screen 70 includes a user image selection area 71, a hair image selection area 72, a hair salon selection area 73, and a reservation date and time setting area 74.

[0232] The appointment setting screen 70 is displayed on the display unit 41 by the output unit 57, for example, when the operation by the user U on the operation unit 42 is a specific appointment setting operation. Although not shown in Fig. 11, the appointment setting screen 70 also includes, for example, a treatment content input area, a treatment content estimation button, and an appointment button.

[0233] The user image selection area 71 includes a start shooting button 711 and a folder button 712. The start shooting button 711 is a button for driving an imaging application program (hereinafter, may be referred to as an imaging app) that captures an image with the imaging unit 43. The user U can operate the start shooting button 711 to display a screen of the imaging app on the display unit 41 and operate the imaging app by operating the operation unit 42. The user U can capture an image of an area including his or her own face by operating the imaging app. As a result, the acquisition unit 51 acquires a user image.

[0234] The folder button 712 is a button for opening a specific folder (for example, a photo folder in which images captured by the imaging unit 43 are placed). The user U can operate the folder button 712 to display multiple captured images contained in the specific folder on the display unit 41. The user U can select a user image from the multiple captured images contained in the specific folder by operating the operation unit 42. As a result, the acquisition unit 51 acquires the user image.

[0235] The hair image selection area 72 includes a keyword setting button 721, a start shooting button 722, and a folder button 723. The keyword setting button 721 is a button for displaying on the display unit 41 a keyword specification screen that enables the user U to specify one or more desired keywords from among a plurality of keywords.

[0236] The shooting start button 722 is a button for driving the imaging app. The user U can operate the shooting start button 722 to display the imaging app screen on the display unit 41 and operate the imaging app by operating the operation unit 42. By operating the imaging app, the user U can capture an image of an area including a desired hair image (for example, an area including an image of a hair model published in a hair catalog or fashion magazine). As a result, the acquisition unit 51 acquires the selected image.

[0237] The folder button 723 is a button for opening a specific folder (for example, a photo folder in which images captured by the imaging unit 43 are placed). The user U can operate the folder button 723 to display multiple captured images contained in the specific folder on the display unit 41. The user U can select a desired selected image from the multiple captured images contained in the specific folder by operating the operation unit 42. As a result, the selected image is acquired by the acquisition unit 51.

[0238] The hair salon selection area 73 includes a current location vicinity search button 731 and a keyword search button 732. The current location vicinity search button 731 is a button for displaying selectable hair salons located near the current location (for example, an area within a predetermined distance from the current location) on the display unit 41. The user U can select a desired hair salon from the hair salons displayed on the display unit 41 by operating the operation unit 42.

[0239] The keyword search button 732 is a button for displaying a keyword search screen for performing a keyword search on the display unit 41. By operating the operation unit 42, the user U can select a desired hair salon from the hair salons searched for on the keyword search screen and displayed on the display unit 41.

[0240] The reservation target date and time setting area 74 is an area for setting a reservation target date and time. The user U can set the date and time at which he or she wishes to receive a hair treatment in the reservation target date and time setting area 74 by operating the operation unit 42.

[0241] 12 is a diagram showing an example of a keyword specification screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 12, a keyword specification screen 80 includes a group of keyword specification buttons 81, 82, 83, and 84, and a search button 85.

[0242] The keyword specification button group 81 includes a plurality of buttons for specifying keywords corresponding to hair length. The keyword specification button group 81 shown in Fig. 12 includes a button for specifying the keyword "short", a button for specifying the keyword "medium", a button for specifying the keyword "long", etc.

[0243] The keyword specification button group 82 includes a plurality of buttons for specifying keywords corresponding to hair colors. The keyword specification button group 82 shown in Fig. 12 includes a button for specifying the keyword "brown," a button for specifying the keyword "beige," a button for specifying the keyword "pink," a button for specifying the keyword "black," etc.

[0244] The keyword specification button group 83 includes a plurality of buttons for specifying keywords corresponding to atmospheres. The keyword specification button group 83 shown in Fig. 12 includes a button for specifying the keyword "cute", a button for specifying the keyword "cool", a button for specifying the keyword "pop", a button for specifying the keyword "natural", etc.

[0245] The keyword specification button group 84 includes a plurality of buttons for specifying keywords corresponding to TPO (Time Place Occasion). The keyword specification button group 84 shown in Fig. 12 includes a button for specifying the keyword "Harajuku", a button for specifying the keyword "Ginza", a button for specifying the keyword "Shinjuku", a button for specifying the keyword "Roppongi", etc.

[0246] The multiple keywords indicated by the keyword designation button groups 81, 82, 83, and 84 are multiple predetermined keywords, but may also be multiple keywords randomly extracted from the multiple keywords or multiple keywords that satisfy predetermined conditions. The multiple keywords that satisfy the predetermined conditions are, for example, a predetermined number of keywords that are ranked in order of frequency of selection by user U. The keywords that are candidates for extraction are, for example, keywords to be input to the first learning model.

[0247] The user U can specify hair length, hair color, mood, TPO, etc. by operating the operation unit 42 and pressing a button corresponding to a desired keyword on the keyword specification screen 80. Then, the user U operates the operation unit 42 to select the search button 85, which causes the processing unit 46 to execute a process of acquiring similar hair images from the specified keyword. Note that the multiple keywords included in the keyword specification screen 80 are not limited to the examples described above.

[0248] 13 is a diagram for explaining the process of acquiring similar hair images from designated keywords by the processing unit 46 of the information providing device 1 according to the embodiment. In the example shown in FIG. 13, the keywords designated by the user U are "long," "pink," "cute," and "Harajuku."

[0249] In this case, the receiving unit 50 receives the keywords "long hair," "pink," "cute," and "Harajuku" specified by the user U. The identifying unit 52 uses the first learning model to identify a hair image from the keywords "long hair," "pink," "cute," and "Harajuku" specified and accepted by the receiving unit 50.

[0250] Then, the acquisition unit 51 sends a similar image search request including the specific hair image to a search server or information providing device 1 (not shown), and acquires multiple similar hair images sent from the search server or information providing device 1 in response to the similar image search request.

[0251] 14 is a diagram showing another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. In the reservation setting screen 70 shown in Fig. 14, a user image selected by the user U is set in a user image selection area 71, and a plurality of similar hair images corresponding to one or more keywords designated by the user U are set in a hair image selection area 72.

[0252] By operating the operation unit 42, the user U can select one or more similar hair images from the plurality of similar hair images set in the hair image selection area 72 as selected images.

[0253] Fig. 15 is a diagram showing yet another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. The reservation setting screen 70 shown in Fig. 15 is displayed on the display unit 41 when the user U selects one of the similar hair images shown in Fig. 14 by operating the operation unit 42 on the reservation setting screen 70 shown in Fig. 14.

[0254] The reception unit 50 receives the selection of a similar hair image by the user U as a selected image. The image processing unit 53 generates a composite image by combining the similar hair image whose selection is received by the reception unit 50 with the user image selected by the user U, and places the generated composite image on the reservation setting screen 70 as shown in Fig. 15. This allows the user U to check the composite image displayed on the display unit 41 and more realistically confirm whether the hairstyle shown in the selected hair image suits them.

[0255] Furthermore, the user U can select a composite image on the reservation setting screen 70 by operating the operation unit 42, thereby displaying the composite image on the display unit 41 while changing the selected image from which the composite image is generated.

[0256] 16 is a diagram showing an example of a composite image change screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. A composite image change screen 90 shown in FIG. 16 includes a length selection button 91, a color selection button 92, and a composite image display area 93.

[0257] The length selection button 91 is a button for changing the target to be changed by swiping (or scrolling) up and down to the hair length. The color selection button 92 is a button for changing the target to be changed by swiping (or scrolling) up and down to the hair color. Swiping (or scrolling) up and down is an example of a second change operation for the hair image.

[0258] 16, the length selection button 91 is selected, and the user U can change the hair length of the composite image by swiping (or scrolling) up or down by operating the operation unit 42. When the reception unit 50 receives the up or down swipe (or scroll), the image processing unit 53 changes the hair length of the composite image and causes the display unit 41 to display the composite image with the changed hair length.

[0259] Furthermore, the user U can change the hair color of the composite image by operating the operation unit 42 to select the color selection button 92, and then operating the operation unit 42 to swipe (or scroll) up or down. When the reception unit 50 receives a swipe up or down, the image processing unit 53 changes the hair color of the composite image and causes the display unit 41 to display the composite image with the changed hair color.

[0260] Furthermore, the user U can display a composite image in which the selected image has been changed by swiping (or scrolling) in the left or right direction on the display unit 41. Swiping (or scrolling) in the left or right direction is an example of a first change operation for the hair image. When the reception unit 50 receives a swipe (or scroll) in the left or right direction, the image processing unit 53 changes the selected image to generate a composite image, and causes the display unit 41 to display the composite image in which the selected image has been changed.

[0261] As described above, the appointment setting screen 70 includes a treatment content input area (not shown). The user U can input the treatment content that the user U has specified or desired into the treatment content input area by operating the operation unit 42.

[0262] As described above, the reservation setting screen 70 also includes a treatment content estimation button and a reservation button (not shown). After the user U determines the composite image and inputs the treatment content specified or desired by the user U in the treatment content input area, the user U selects a hair salon and sets the reservation date and time. The user U then operates the operation unit 42 to select the reservation button, which causes a reservation request to be sent from the output unit 57 to the hair practitioner. This causes the reservation request to be output to the hair practitioner.

[0263] The reception unit 50 receives selection of the treatment content estimation button by the user U. When the reception unit 50 receives selection of the treatment content estimation button, the estimation unit 54 estimates the treatment content from the user image and composite image displayed on the appointment setting screen 70. The user image is an example of an image of the user U before the treatment. When the user image and composite image are input into a fourth learning model, the estimation unit 54 determines the estimated treatment content based on information indicating the treatment content output from the fourth learning model.

[0264] 17 is a diagram for explaining the process of estimating treatment details from a user image and a composite image by the estimation unit 54 of the information providing device 1 according to the embodiment. In the example shown in FIG. 17, the estimation unit 54 determines "cut, bleach once, color bleach twice, and perm once" as the estimated treatment details from the user image and the composite image.

[0265] When the reception unit 50 receives the selection of the reservation button, the output unit 57 outputs to the hair practitioner a reservation request including the user image, the composite image, information indicating the treatment content entered in the treatment content input area, information indicating the estimated treatment content, and information indicating the reservation target date and time. When the reception unit 50 receives the selection of a specific button (not shown), the output unit 57 causes the display unit 41 to display, for example, a screen indicating completion of the operation. The selection of the reservation button by the user U is an example of a response indicating the user U's consent to the estimated treatment content, and the selection of the specific button by the user U is an example of a response indicating the user U's refusal to accept the estimated treatment content.

[0266] In addition, the output unit 57 can output a face shape selection screen to the display unit 41 to allow the user U to select the face shape of the user U in accordance with the user U's operation on the operation unit 42, and display the face shape selection screen on the display unit 41.

[0267] 18 is a diagram showing an example of a face shape selection screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 18, the face shape selection screen 95 includes a plurality of face shapes. The user U can operate the operation unit 42 to select the face shape that is closest to the user U's face shape from among the face shapes.

[0268] The receiving unit 50 receives a face shape selection by the user U from among a plurality of face shapes included in the face shape selection screen 95. The identifying unit 52 regards the face shape selected by the user U as the facial features of the user U designated by the user U, and identifies a hairstyle that is estimated to suit the face of the user U using a third learning model from the facial features of the user U.

[0269] The identification unit 52 inputs information indicating the face shape selected by the user U into a third learning model, and identifies a hairstyle that is estimated to suit the face of the user U based on the hairstyle information output from the third learning model.

[0270] [5. Processing Procedure] Next, a procedure of information processing by the processing unit 46 of the terminal device 4 according to the embodiment will be described. Fig. 19 is a flowchart showing an example of information processing by the processing unit 46 of the terminal device 4 according to the embodiment.

[0271] 19, the processing unit 46 of the terminal device 4 determines whether or not information about the learning model has been acquired from the information providing device 1 (step S20). If the processing unit 46 determines that information about the learning model has been acquired (step S20: Yes), the processing unit 46 stores the acquired information about the learning model in the storage unit 45 (step S21).

[0272] When the processing of step S21 is completed or when it is determined that the information of the learning model has not been acquired (step S20: No), the processing unit 46 determines whether or not a reservation setting operation has been performed by the user U (step S22). When it is determined that a reservation setting operation has been performed (step S22: Yes), the processing unit 46 executes reservation processing (step S23). The reservation processing of step S23 is the processing of steps S40 to S49 shown in FIG. 20, and will be described in detail later.

[0273] When the processing of step S23 is completed or when it is determined that no reservation setting operation has been performed (step S22: No), the processing unit 46 determines whether or not there is a revision suggestion from the hair practitioner (step S24). When it is determined that there is a revision suggestion (step S24: Yes), the processing unit 46 causes the display unit 41 to display the revision suggestion from the hair practitioner (step S25).

[0274] When the processing of step S25 is completed or when it is determined that there is no correction suggestion (step S24: No), the processing unit 46 determines whether there is a practitioner comment (step S26). When it is determined that there is a practitioner comment (step S26: Yes), the processing unit 46 causes the practitioner comment to be displayed on the display unit 41 (step S27).

[0275] When the processing of step S27 is completed or when it is determined that there is no comment from the practitioner (step S26: No), the processing unit 46 determines whether or not an image synthesis operation has been performed (step S28). When it is determined that an image synthesis operation has been performed (step S28: Yes), the processing unit 46 performs synthesis processing (step S29). The synthesis processing of step S29 is the same as steps S40 to S47 shown in FIG. 20.

[0276] When the processing of step S29 is completed or when it is determined that there is no image synthesis operation (step S28: No), the processing unit 46 determines whether or not it is time to end the operation (step S30). For example, when the power supply of the information providing device 1 is turned off or when it is determined that an end operation has been performed by operating an operation unit (not shown) of the information providing device 1, the processing unit 46 determines that it is time to end the operation.

[0277] If the processing unit 46 determines that the operation end time has not arrived (step S30: No), it proceeds to step S20, and if it determines that the operation end time has arrived (step S30: Yes), it terminates the processing shown in Figure 19.

[0278] 20 is a flowchart showing an example of a reservation process by the processing unit 46 of the terminal device 4 according to the embodiment. As shown in FIG. 20, the processing unit 46 accepts a selection of a user image and acquires the user image whose selection has been accepted (step S40).

[0279] Next, the processing unit 46 accepts the designation of one or more keywords (step S41). The processing unit 46 identifies a hair image from the one or more keywords designated in step S41 (step S42). Then, the processing unit 46 acquires a similar hair image that is an image similar to the hair image identified in step S42 (step S43).

[0280] Next, the processing unit 46 combines the similar hair image selected from the similar hair images acquired in step S42 with the user's face image (step S44), and then the processing unit 46 causes the display unit 41 to display the combined image combined in step S44 (step S45).

[0281] Next, the processing unit 46 determines whether or not a change operation has been performed (step S46). The change operation is, for example, the first change operation of the hair image or the second change operation of the hair image described above. If the processing unit 46 determines that a change operation has been performed (step S46: Yes), it performs image processing according to the change operation to change the composite image (step S47), and then proceeds to step S46.

[0282] If the processing unit 46 determines that no change operation has been performed (step S46: No), the processing unit 46 estimates the treatment content based on the pre-treatment image and the composite image (step S48). The processing unit 46 determines that no change operation has been performed, for example, when the user U selects the reservation button.

[0283] Then, the processing unit 46 outputs a reservation request including the before-treatment image, the composite image, the estimated treatment content, and the specified treatment content to the hair treatment professional (step S49). The specified treatment content is the treatment content input by the user U. When the processing of step S49 is completed, the processing unit 46 ends the processing shown in FIG. 20.

[0284] [6. Modifications] In the above-described example, the processing unit 46 displays on the display unit 41 a composite image in which the selected image is changed by a user operation in the left-right direction, and a composite image in which the hair length or color, etc. is changed by a user operation in the up-down direction, but is not limited to such an example.

[0285] For example, the processing unit 46 can display on the display unit 41 a composite image in which the selected image has been changed by user operation in the up and down directions, and can display on the display unit 41 a composite image in which the hair length or color, etc. has been changed by user operation in the left and right directions.

[0286] Furthermore, the processing unit 46 can also display a composite image in which the length or color of the hair of the selected image has been changed by a user operation in a diagonal direction on the display unit 41. Note that the user operation is a swipe operation or a scroll operation, but is not limited to such examples and may be, for example, a tap operation or a click operation, or a pinch-in operation or a pinch-out operation.

[0287] Furthermore, the hairstyle information output from the third learning model is not limited to a hair image. For example, the hairstyle information output from the third learning model may be hair image identification information, which is information for identifying a hair image. The information for identifying a hair image may be, for example, identification information of the hair image or a keyword for identifying the hair image.

[0288] In this case, the third learning model is generated using learning data including multiple combinations of information indicating facial features, hair image identification information, and information indicating an evaluation of the combination of the information indicating facial features and the hair image identification information.

[0289] Furthermore, each learning model, including the first learning model, the second learning model, and the third learning model, may be a learning model for each user U. For example, by using hair images included in posted images viewed by user U as hair images used to generate each of the first learning model and the third learning model, each of the first learning model and the third learning model can be generated for each user U.

[0290] Furthermore, by using a hair image included in the posted image viewed by the user U as the face image used to generate the second learning model, the second learning model can be generated for each user U.

[0291] The information providing device 1 can execute some or all of the functions of the acquiring unit 51, identifying unit 52, image processing unit 53, estimating unit 54, determining unit 55, comparing unit 56, and output unit 57 in cooperation with the terminal device 4, and can function as a part of an information processing device. Note that hereinafter, a configuration including some or all of the above-described terminal device 4 and the information providing device 1 may be referred to as an information processing device.

[0292] [7. Hardware Configuration] Each of the information providing device 1 and the terminal device 4 according to the embodiment described above is realized by a computer 200 having a configuration as shown in Fig. 21. Fig. 21 is a hardware configuration diagram showing an example of the computer 200 that realizes each of the functions of the information providing device 1 and the terminal device 4 according to the embodiment. The computer 200 has a CPU 201, a RAM 202, a ROM (Read Only Memory) 203, an HDD (Hard Disk Drive) 204, a communication interface (I / F) 205, an input / output interface (I / F) 206, and a media interface (I / F) 207.

[0293] The CPU 201 operates and controls each unit based on programs stored in the ROM 203 or the HDD 204. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 starts up, programs dependent on the hardware of the computer 200, and the like.

[0294] The HDD 204 stores programs executed by the CPU 201, data used by such programs, etc. The communication interface 205 receives data from other devices via the network N (see FIG. 2) and sends it to the CPU 201, and transmits data generated by the CPU 201 to other devices via the network N.

[0295] The CPU 201 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 206. The CPU 201 acquires data from the input devices via the input / output interface 206. The CPU 201 also outputs generated data to the output devices via the input / output interface 206.

[0296] The media interface 207 reads a program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program or data from the recording medium 208 onto the RAM 202 via the media interface 207 and executes the loaded program. The recording medium 208 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0297] For example, when the computer 200 functions as the information providing device 1 or the terminal device 4 according to the embodiment, the CPU 201 of the computer 200 executes a program loaded onto the RAM 202 to realize the functions of the processing unit 12 or the processing unit 46. The HDD 204 also stores data stored in the storage unit 11 or the storage unit 45. The CPU 201 of the computer 200 reads and executes these programs from the recording medium 208, but as another example, the CPU 201 may also acquire these programs from another device via the network N.

[0298] [8. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0299] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0300] For example, the information providing device 1 described above may be realized by a plurality of server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using API or network computing.

[0301] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0302] [9. Effects] As described above, the information processing device according to the embodiment includes the identification unit 52, the acquisition unit 51, and the image processing unit 53. The identification unit 52 identifies a hairstyle that is estimated to suit the face of the user U using a learning model that is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated to suit the face. The acquisition unit 51 acquires posted images including a specific hair image that is an image of the hairstyle identified by the identification unit 52. The image processing unit 53 generates a composite image by combining the image of the hairstyle identified by the identification unit 52 and the facial image of the user U, based on the posted images acquired by the acquisition unit 51 and the facial image of the user U. This allows the user U to easily acquire an image of himself or herself wearing the hairstyle that is estimated to suit the face of the user U. Therefore, the information processing device can improve convenience for the user U.

[0303] Furthermore, the information indicating facial features is information indicating the type of face shape, which allows the information processing device to accurately identify a hairstyle that is estimated to suit the face.

[0304] The information processing device also includes an output unit 57 that outputs the composite image generated by the image processing unit 53. This allows the information processing device to improve convenience for the user U.

[0305] The learning model is generated by machine learning using learning data that includes multiple combinations of facial feature information, hair images, and information indicating evaluations of the combinations of the facial feature information and hair images, allowing the information processing device to accurately identify hairstyles that are estimated to suit the face.

[0306] The learning data is generated based on posted images including face images and hair images, and posted information including evaluation information indicating evaluations of the posted images, thereby enabling the information processing device to accurately identify hairstyles that are estimated to suit the face.

[0307] The information processing device also includes a receiving unit 50 that receives a first change operation for the hair image from the user U. An acquiring unit 51 acquires a plurality of posted images including a specific hair image, an image processing unit 53 generates a composite image corresponding to each of the plurality of posted images, and an output unit 57 changes the composite image to be output each time the first change operation is received by the receiving unit 50. This allows the information processing device to improve convenience for the user U.

[0308] Furthermore, the receiving unit 50 receives a second change operation for the hair image from the user U, and the output unit 57 changes at least one of the hair length and color of the hair image included in the composite image every time the second change operation is received by the receiving unit 50. This allows the information processing device to improve convenience for the user U.

[0309] The above describes the embodiments of the present application in detail based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have been modified and improved in various ways based on the knowledge of those skilled in the art.

[0310] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0311] 1 Information provision device 4 Terminal Devices 5 Store equipment 10,40 Communications Department 11,45 Storage part 12,46 Processing section 20 User information storage unit 21 Posted information storage unit 22 Model information storage unit 23 Content storage unit 30,51 Acquisition Department 31 Learning Department 32 Providing Department 41 Display section 42 Operation section 43 Imaging unit 44 Sensor section 50 Reception 52 Specific part 53 Image processing section 54 Estimation part 55 Judgment section 56 Comparison section 57 Output section 60 Selection Reception Section 61 Processing Reception Department 62 Designated Reception Department 63 Revision Reception Department 64 Permission Reception Department 65 Conversion unit 66 Synthesis section 100 Information Processing Systems

Claims

1. an identification unit that identifies a hairstyle that is estimated to suit the user's face using a learning model that is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated to suit the face; an acquisition unit that acquires posted images including a specific hair image that is an image of the hairstyle identified by the identification unit; an image processing unit that generates a composite image by combining the image of the hairstyle identified by the identification unit with the facial image of the user, based on the posted image acquired by the acquisition unit and the facial image of the user.

1. An information processing device comprising:

2. The facial feature information includes: This is information that indicates the type of face shape.

2. The information processing apparatus according to claim 1, wherein:

3. an output unit that outputs the composite image generated by the image processing unit; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The learning model is Generated by machine learning using learning data including a plurality of combinations of information indicating facial features, a hair image, and information indicating an evaluation of the combination of the information indicating the facial features and the hair image.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The learning data is The image data is generated based on posted information including a posted image including a face image and a hair image and evaluation information indicating an evaluation of the posted image.

5. The information processing apparatus according to claim 4,

6. a reception unit that receives a first change operation of the hair image from the user, The acquisition unit Acquire a plurality of the posted images including the specific hair image; The image processing unit generating the composite image corresponding to each of the plurality of posted images; The output unit The composite image to be output is changed every time the first change operation is accepted by the accepting unit.

4. The information processing apparatus according to claim 3,

7. The reception unit receiving a second change operation of the hair image from the user; The output unit At least one of the length and color of the hair of the hair image included in the composite image is changed each time the second change operation is accepted by the accepting unit.

7. The information processing apparatus according to claim 6,

8. 1. A computer-implemented information processing method, comprising: a step of identifying a hairstyle that is estimated to suit the user's face using a learning model that has learned the relationship between information indicating facial features and hairstyles that are evaluated to suit the face; an acquiring step of acquiring a posted image including a specific hair image, which is an image of the hairstyle identified by the identifying step; an image processing step of generating a composite image by combining the image of the hairstyle identified in the identifying step with the facial image of the user, based on the posted image acquired in the acquiring step and the facial image of the user.

1. An information processing method comprising:

9. a step of identifying a hairstyle that is estimated to suit the user's face using a learning model that has learned the relationship between information indicating facial features and hairstyles that are evaluated to suit the face; an acquisition step of acquiring a posted image including a specific hair image, which is an image of the hairstyle identified by the identification step; an image processing procedure for generating a composite image by combining the image of the hairstyle specified by the specifying procedure with the facial image of the user, based on the posted image acquired by the acquiring procedure and the facial image of the user. An information processing program characterized by:

Images