Information processing device, information processing method, and information processing program

Abstract

To provide an appropriate content regarding a synthesis image.SOLUTION: The information processor according to the present application includes: a determination unit for determining whether a synthesization target image is an appropriate image appropriate for synthesization; and a provision unit for providing a content for synthesizing the synthesization target image and a taken image taken by a user when the determination unit determines that the synthesization target image is an appropriate image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, Augmented Reality (AR) has become popular as content provided via the Internet. In such AR, for example, a composite image is provided in which three-dimensional data viewed from a virtual viewpoint is superimposed on a captured image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7072706 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art leaves room for improvement in providing appropriate content related to composite images.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing device, an information processing method, and an information processing program that are capable of providing appropriate content related to a composite image. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention is characterized by comprising a judgment unit that judges whether an image to be synthesized is an appropriate image suitable for synthesis, and a provision unit that, when the judgment unit judges that the image to be synthesized is the appropriate image, provides content for synthesizing the image to be synthesized with an image taken by a user. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide appropriate content related to a composite image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. [Figure 2] FIG. 2 is a diagram (part 1) showing a specific example of information processing according to the embodiment. [Figure 3] FIG. 3 is a diagram (part 2) illustrating a specific example of information processing according to the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of an information providing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a posted information database according to the embodiment. [Figure 6] FIG. 6 is a block diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an outline of the estimation process according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a composite image according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a display related to the appropriate image according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of processing executed by the information providing device according to the embodiment. [Figure 11] FIG. 11 is a flowchart (part 1) illustrating an example of processing executed by a terminal device according to the embodiment. [Figure 12] FIG. 12 is a flowchart (part 2) illustrating an example of the process executed by the terminal device according to the embodiment. [Figure 13] FIG. 13 is a flowchart (part 3) illustrating an example of processing executed by the terminal device according to the embodiment. [Figure 14]FIG. 14 is a flowchart (part 4) illustrating an example of processing executed by the terminal device according to the embodiment. [Figure 15] FIG. 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments.

[0010] [Embodiment] [1. Information Processing] First, an example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the embodiment. Note that, although the following describes a case where the information processing device according to the embodiment is a terminal device 10, it may also be an information providing device 100.

[0011] 1 is an information processing device that operates, for example, a social networking service (SNS). For example, the information providing device 100 distributes posted information posted by contributors such as hair salons to a terminal device 10 of a user U who is a viewer.

[0012] When providing such an SNS, the information providing device 100 provides a service related to hairstyle simulation, which allows the user U to try out various hairstyles by combining a user image of the user U with a hairstyle posted on the SNS.

[0013] 1 is a terminal operated by a user U. For example, the terminal device 10 displays a browsing page of an SNS provided by the information providing device 100 in response to an operation by the user U. The terminal device 10 also generates a composite image by combining a hairstyle posted on the SNS with a user image of the user U, and displays the generated composite image.

[0014] Specifically, as shown in Fig. 1, the information providing device 100 acquires posted information from posters such as beauty salons, hair salons, and stylists (step S1).

[0015] Next, the information providing device 100 generates a browsing page for the user U based on the access by the terminal device 10, and provides the generated browsing page to the terminal device 10 (step S2). The terminal device 10, which has received the browsing page, displays the browsing page on its screen (step S3). Specific examples of the browsing page will be described later with reference to FIG. 2 etc.

[0016] Next, the terminal device 10 determines a composite target image from among the posted images displayed on the browsing page based on an operation by the user U (step S4). Note that the composite target image here is a posted image posted by each poster, and is an image of a hairstyle.

[0017] That is, the user U can select any compositing target image from the browsing page. As will be described later, the terminal device 10 can, for example, determine whether the compositing target image is an appropriate image suitable for compositing, and based on the determination result, can accept selection of only the appropriate image as the compositing target image.

[0018] Next, the terminal device 10 acquires a user image to be composited with the composite target image (step S5). For example, the terminal device 10 acquires the user image by taking a new user image with a camera or by reading a user image taken in the past from a memory.

[0019] Next, the terminal device 10 generates a composite image by combining the composite target image and the user image. For example, the terminal device 10 generates the composite image by inputting the composite target image and the user image into a synthesis model. The synthesis model here is a model that has been trained to output an image of the user U with the hairstyle that appears in the synthesis target image when the user image and the synthesis target image are input. Note that any model can be used for the synthesis model, such as a model that, when a face region of the user image is selected, outputs an image in which only the hairstyle is changed so that the face region is not changed.

[0020] Then, the terminal device 10 provides the synthesized composite image to the user U (step S7). This allows the user U to try out the hairstyle in the image posted on the SNS, thereby providing a highly convenient service for the user U.

[0021] In providing such hairstyle simulation services, each user posts photos of their hairstyles to the SNS. As a result, the posted images include both appropriate images suitable for compositing and unsuitable images not suitable for compositing.

[0022] Specifically, each poster can post an image of a hair model as the subject of their choice, so the posted images will include images of the hair model taken from various angles.

[0023] On the other hand, for example, when performing a hairstyle simulation, the hairstyle is created based on the face of user U and the posted image, so it is desirable that there is little error between the direction of user U's face in the user image and the direction of the hair model's face in the image to be synthesized, and that the face of user U is easy to recognize in the synthesized image after synthesis.

[0024] Therefore, in the information processing according to the embodiment, it is determined whether or not the appropriate image for synthesis is an appropriate image, and content for synthesizing the appropriate image into a composite image is provided only for the appropriate image.

[0025] Furthermore, for example, when a user U photographs a user image after selecting an image to be composited, it is desirable that the user image be easily photographed as a suitable image for compositing. Therefore, in the information processing according to the embodiment, it is determined whether or not the posted image posted by each poster is an appropriate image, and photographing content for appropriately photographing the user image is provided based on the appropriate image selected by the user U.

[0026] 2 and 3 are diagrams showing a specific example of information processing according to the embodiment. As shown in Fig. 2, a browsing page P1 is displayed on the terminal device 10. For example, the browsing page P1 displays a list image area GL in which posted images are displayed as a list, a search box, and the like.

[0027] For example, when the user U selects a posted image displayed in the list image area GL, the screen transitions from the viewing page P1 to a details page P2. The details page P2 displays a detailed image GR of the posted image selected by the user U, a description TR of the detailed image GR, information about the poster, the number of "likes," and the like.

[0028] Furthermore, when the detailed image GR is an appropriate image suitable for compositing, a compositing icon C1 is displayed on the detailed image GR, which transitions to compositing the compositing target image and the photographed image photographed by the user.

[0029] For example, the terminal device 10 estimates the facial direction of the subject in the detailed image GR using a model that estimates the facial direction of the subject in the detailed image GR in three axes, and displays a composite icon C1 for the detailed image GR when a predetermined condition is met, for example, when the subject's face is facing forward.

[0030] That is, the user U can easily recognize whether the detailed image GR is an appropriate image or not by whether or not the composite icon C1 is displayed on the detail page P2. Next, when the user U selects the composite icon C1, the screen transitions from the detail page P2 to a user image selection page P3. Note that when the composite icon C1 is selected, it is also possible to transition to a shooting screen for taking a new user image instead of the selection page P3.

[0031] On the selection page P3, it is possible to select an image that has been registered in advance in the terminal device 10, and for example, a guide C2 is displayed. For example, the guide C2 suggests the position, size, and body orientation of the user U's face that will appear in the user image when synthesizing a composite image. For example, the user U can select an image that matches the guide C2, and the terminal device 10 can synthesize an appropriate composite image.

[0032] In this way, the terminal device 10 displays the composition icon C1 for transitioning to composition for the detailed image GR that is the appropriate image, and provides composition content for combining the detailed image GR with the user image.

[0033] 3, when a composite icon C1 displayed on the details page P2 is selected, the screen transitions from the details page P2 to a photography page P4. In this case, the terminal device 10 provides photography content for taking a user image.

[0034] For example, the terminal device 10 sets the shooting conditions for the user image based on the detailed image GR and then provides the shooting content. The terminal device 10 estimates the facial orientation of the subject shown in the detailed image GR and sets the shooting conditions so that the facial orientation of the user U shown in the user image is approximately the same. Then, for example, the terminal device 10 successively estimates the facial orientation of the user U shown in the camera during shooting, and activates the shooting button C3 when, for example, the facial orientation of the user U satisfies the shooting conditions.

[0035] In other words, if the photographing conditions are not met, the terminal device 10 inactivates the photographing button C3. In other words, if the photographing conditions are met, the terminal device 10 allows the user to photograph an image.

[0036] This allows the user U to easily capture an appropriate user image corresponding to the detailed image GR. Note that the photography content can be arbitrarily changed, for example, by notifying the user by audio output when the photography conditions are met, automatically photographing the user image, or displaying the degree of deviation between the photography conditions and the current user image using a graphic such as a bar.

[0037] [2. Example of information provision device configuration] Next, a configuration example of the information providing device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing a configuration example of the information providing device 100 according to the embodiment. As shown in Fig. 4, the information providing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0038] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network via a wired or wireless connection.

[0039] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 2 , the storage unit 120 has a posted information database 121, a score estimation model database 122, and a combined model database 123.

[0040] (Posted Information Database 121) The posted information database 121 is a database that stores the posted information posted by each contributor. Fig. 5 is a diagram illustrating an example of the posted information database 121 according to the embodiment.

[0041] As shown in FIG. 5, the posted information database 121 has items such as "posted ID," "poster ID," "image information," "text information," "score," and "photography conditions."

[0042] "Post ID" is an identifier for identifying posted information, and "Poster ID" is an identifier for identifying the poster. "Image information" is information about the image included in the posted information (posted image). "Text information" is text information included in the posted information (for example, description of the posted image, hashtags, comments by viewers).

[0043] The "score" is a score related to the posted image included in the posted information. The score here indicates whether the posted image is an appropriate image. The "photography conditions" are the photography conditions required for the user image when the corresponding posted image is used as the composite target image.

[0044] (Score estimation model database 122) Returning to the explanation of Fig. 4, the score estimation model database 122 will be described. The score estimation model database 122 stores a model that estimates a score indicating whether or not a posted image is an appropriate image. For example, as will be described later, the score estimation model is a model that outputs a score depending on the orientation of the face of the subject appearing in the posted image.

[0045] (Synthetic Model Database 123) The synthesis model database 123 is a database that stores synthesis models. The synthesis model is a model that has been trained to output a synthetic image in which the hairstyle of the synthesis target image is synthesized onto the face of the user U when a posted image that is a synthesis target image and a user image are input.

[0046] (control unit 130) The control unit 130 is a controller, and is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the information providing device 100 using a storage area such as a RAM as a working area. In the example shown in FIG. 4, the control unit 130 has an acquisition unit 131, an assignment unit 132, and a content providing unit 133.

[0047] (Acquisition part 131) The acquisition unit 131 acquires posted information from the poster's terminal via the communication unit 110. The acquisition unit 131 registers the acquired posted information in the posted information database 121. The acquisition unit 131 also acquires an acquisition request for a browse page from the terminal device 10 of the user U via the communication unit 110. For example, the acquisition unit 131 passes information related to the acquired acquisition request to the content providing unit 133.

[0048] (Giving unit 132) The assigning unit 132 assigns information related to a "score" or "photography conditions" to the posted information registered in the posted information database 121. For example, the assigning unit 132 obtains a score related to the posted image by inputting the posted image of the posted information into a score estimation model registered in the score estimation model database 122, which estimates the score, and registers the score in the posted information database 121.

[0049] Furthermore, the assigning unit 132 estimates the face orientation of the subject appearing in the posted information using a model, sets the shooting conditions based on the face orientation, and registers the shooting conditions in the posted information database 121. Note that the series of processes performed by the assigning unit 132 may be performed on the side of each terminal device 10.

[0050] (Content provider 133) The content providing unit 133 starts the communication unit 110 and provides various contents to the terminal device 10 of the user U. For example, the content providing unit 133 generates a browsing page of an SNS based on a request from the terminal device 10 and provides it to the terminal device 10.

[0051] The content providing unit 133 also provides various contents (for example, applications) for combining the composite target image with the user image. The content providing unit 133 provides contents such as the composite target image or a model for estimating the facial orientation of a subject appearing in the user image, and a composite model for combining the composite target image and the user image.

[0052] Note that any type of model can be adopted as the above model. For example, a support vector machine (SVM) or a deep neural network (DNN) may be adopted as the model. Here, the DNN may be a convolutional neural network (CNN) or a recurrent neural network (RNN). The RNN may be a long short-term memory (LSTM) or the like. That is, any type of model can be adopted. The model may also be realized by combining multiple models, such as a model that combines a CNN and an RNN.

[0053] The learning may be, for example, deep learning using a deep neural network (DNN), or may be data mining or other machine learning algorithms.

[0054] [3. Example of terminal device configuration] Next, a configuration example of the terminal device 10 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing a configuration example of the terminal device 10 according to the embodiment. As shown in Fig. 6, the terminal device 10 has a communication unit 11, a display unit 12, an input unit 13, a camera 20, a control unit 30, and a storage unit 40.

[0055] (Communications Department 11) The communication unit 11 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network by wire or wirelessly.

[0056] (Display section 12) The display unit 12 is, for example, any of various displays that displays images input from the control unit 30.

[0057] (Input section 13) The input unit 13 is an input device for various operations by the user U. The input unit 13 is a keyboard, a mouse, or a microphone, but if the display unit 12 is a touch panel display, the display unit 12 and the input unit 13 are integrally formed.

[0058] (Camera 20) The camera 20 is an imaging device having various imaging elements. Although Fig. 6 shows a case where the camera 20 is included in the terminal device 10, the camera 20 may be externally attached to the terminal device 10.

[0059] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 40 stores, for example, a user image captured by the camera 20 and various information transmitted from the information providing device 100.

[0060] (control unit 30) The control unit 30 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, by executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the information providing device 100 using a storage area such as a RAM as a working area. In the example shown in FIG. 6, the control unit 130 has an acquisition unit 31, a reception unit 32, an estimation unit 33, a determination unit 34, a generation unit 35, and a provision unit 36.

[0061] (Acquisition part 31) The acquisition unit 31 acquires various pieces of information from the information providing device 100 via the communication unit 11. For example, the acquisition unit 31 acquires various pieces of content related to SNS and various pieces of content for generating a composite image.

[0062] (Reception Department 32) The receiving unit 32 receives various operations by the user U via the input unit 13. For example, the receiving unit 32 receives an operation to select an image to be synthesized for the browse page P1 shown in FIG.

[0063] (Estimation part 33) The estimation unit 33 estimates the facial orientation of the subject appearing in the composite target image selected by the user U, or the facial orientation of the user U appearing in the captured image being captured by the camera 20. For example, the estimation unit 33 estimates the facial orientation of the subject using an estimation model that estimates the facial orientation of the subject appearing in each image. For example, the estimation unit 33 estimates the facial orientation of the subject using on-device AI that has learned the facial orientation of the subject on the terminal device 10.

[0064] 7 is a diagram illustrating an outline of the estimation process according to the embodiment. For example, as shown in FIG. 7, the estimation unit 33 estimates the angle of the subject's facial orientation along three axes (roll, pitch, and yaw) using on-device AI. That is, the estimation unit 33 estimates the subject's facial orientation or the rotation angle of the face (deviation from the frontal direction). Then, the estimation unit 33 passes information related to the estimation result to the determination unit 34.

[0065] (Judgment unit 34) The determination unit 34 determines whether or not the compositing target image selected by the user U is an appropriate image suitable for compositing. For example, the determination unit 34 determines whether or not the image is appropriate based on the face direction of the subject appearing in the compositing target image estimated by the estimation unit 33.

[0066] For example, when the face orientation of the subject estimated by the estimation unit 33 falls within a predetermined range, the determination unit 34 determines that the image is appropriate. In other words, when the face orientation of the subject estimated by the estimation unit 33 deviates from the predetermined range, the determination unit 34 does not determine that the image is appropriate. For example, the predetermined range here means that the face orientation of the subject is facing forward, but any value may be set.

[0067] The determination unit 34 may also determine whether or not a compositing target image is an appropriate image based on, for example, the out-of-focus state of a subject appearing in the compositing target image and the size of the subject. For example, the determination unit 34 may determine that a compositing target image is an appropriate image if it has little out-of-focus state and the subject is of an appropriate size. The determination unit 34 may also determine whether or not a compositing target image is an appropriate image based on the background of the subject in the compositing target image. In this case, for example, the determination unit 34 may determine that a compositing target image in which the background of the subject is plain is an appropriate image.

[0068] The above-described processing by the estimation unit 33 and the determination unit 34 regarding the composite target image may be performed on the side of the information providing device 100. In this case, when a browsing page is requested, the terminal device 10 determines whether the composite target image is an appropriate image based on the score assigned by the assignment unit 132 of the information providing device 100.

[0069] (Generation unit 35) The generation unit 35 generates a composite image by combining the composite target image with a captured image (i.e., a user image) captured by the user U. For example, the generation unit 35 determines a user image to be composited with the composite target image selected by the user U.

[0070] For example, based on an operation by the user U, the generation unit 35 determines a user image stored in the storage unit 40 of the terminal device 10 or a newly captured user image as the user image to be composited with the composite target image.

[0071] For example, the generation unit 35 generates a composite image using a synthesis model. The synthesis model is a model that is trained to output an image in which the user U has a hairstyle that appears in the synthesis target image when a user image and a synthesis target image are input.

[0072] 8 is a diagram illustrating an example of a composite image according to the embodiment. As illustrated in FIG. 8, the generation unit 35 generates a composite image Gn by inputting a composite target image Gm selected by a user U from an SNS and a user image Gy into a composite model.

[0073] For example, the composite image Gn is an image obtained by combining the face of the user U reflected in the user image Gy with the hairstyle of the composite target image Gm. For example, by changing the composite image Gn and the user image Gy, the user U can try out various hairstyles for the user images Gy, each of which has different clothing, accessories (necklaces, earrings, etc.), facial expressions, etc. As one example of use, the user U can take a user image after wearing the clothing, accessories, and makeup required to attend a wedding, and try out various hairstyles, thereby more clearly imagining what the actual wedding will look like. In such a case, the user image may be an image of the user U's entire body.

[0074] (Providing Department 36) The providing unit 36 ​​provides various contents provided from the information providing device 100 to the user U via the display unit 12. For example, the providing unit 36 ​​provides contents related to a browsing page P1, a details page P2, and a user image selection page P3 of the SNS as shown in FIG. 2, and photographing contents related to a photographing page P4 as shown in FIG.

[0075] Furthermore, when the detail page P2 is selected before the user image, the providing unit 36 ​​displays the composite icon C1 (see FIG. 2) depending on whether the detailed image displayed on the detail page P2 is an appropriate image.

[0076] In addition, when a user image is selected before the image to be synthesized, the providing unit 36 ​​displays on the viewing page P1 where the list image area GL is displayed that suitable images that can be properly synthesized with the user image are capable of being synthesized.

[0077] 9 is a diagram showing an example of a display related to an appropriate image according to the embodiment. As shown in FIG. 9, when a user U selects a user image on a selection page P11 for selecting a user image to be composited, the screen transitions to a browsing page P12 on which a list image area is displayed.

[0078] At this time, a composition possibility icon C10 is displayed for each posted image, indicating whether the image is suitable for the user image selected by the user, with "Composite OK" and "Composite NG" being displayed for images that are difficult to combine properly.

[0079] More specifically, when the user U selects a user image, the estimation unit 33 estimates the orientation of the user's face in the user image. Then, the determination unit 34 compares the orientation of the face with the orientation of each image displayed on the browsing page P12, and determines an appropriate image for the user image.

[0080] In response to these processes, the providing unit 36 ​​displays a composite possibility icon C10 for the posted image displayed on the browsing page P12, thereby allowing the user U to easily recognize the posted images that can be composited with the user image selected by the user U.

[0081] Although the case where a user image is selected from the selection page P11 has been described here, the same processing may be performed on a user image newly taken by the user U with the camera 20.

[0082] Furthermore, for example, the providing unit 36 ​​may display only appropriate images that can be combined with the user image on the browsing page P12. In this case, for example, the providing unit 36 ​​transmits information about the orientation of the face of the user U in the user image to the information providing device 100 along with an acquisition request for the browsing page. The information providing device 100 extracts posted information that is an appropriate image from the posted information database 121 based on the information about the orientation of the face of the user U in the user image, and then generates and provides the browsing page to the terminal device 10.

[0083] [4. Processing flow] Next, a processing procedure executed by the information providing device 100 according to the embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of processing executed by the information providing device 100 according to the embodiment.

[0084] 10, first, the information providing device 100 acquires posted information from each poster (step S101). Next, the information providing device 100 assigns shooting conditions and composition conditions to each piece of posted information based on the posted images included in the posted information (step S102). Note that the shooting conditions here are shooting conditions required for the user image when the posted image is used as a composite target image, and the composition conditions are conditions required for the composite target image when the user image is selected before the composite target image.

[0085] Next, the information providing device 100 distributes various contents, for example, a browsing page, to the terminal device 10 of the user U (step S103). Then, the information providing device 100 ends the processing. Note that the processing related to step S102 may be performed on the terminal device 10 side.

[0086] Next, a processing procedure executed by the terminal device 10 according to the embodiment will be described with reference to Fig. 11 to Fig. 14. Fig. 11 to Fig. 14 are flowcharts showing an example of a processing procedure executed by the terminal device 10 according to the embodiment.

[0087] First, a series of processing steps until the terminal device 10 provides a composite image will be described with reference to Fig. 11. As shown in Fig. 11, first, the terminal device 10 acquires a browsing page related to the composite target image from the information providing device 100 (step S201).

[0088] Next, the terminal device 10 accepts the selection of a synthesis target image by the user U (step S202). Next, the terminal device 10 accepts the selection of a user image to be synthesized with the synthesis target image (step S203).

[0089] Next, the terminal device 10 generates a composite image based on the composite target image and the user image (step S204), and then displays (provides) the composite image (step S205), and ends the process.

[0090] Next, a series of processing steps relating to providing a page to be viewed will be described with reference to Fig. 12. As shown in Fig. 12, first, the terminal device 10 acquires a page to be viewed relating to an image to be composited from the information providing device 100 (step S211).

[0091] Next, the terminal device 10 determines whether each posted image displayed on the browsing page is an appropriate image (step S212). Next, the terminal device 10 assigns, to the appropriate image, composition content (for example, content related to the composition icon C1 shown in FIG. 2) that enables composition with the user image.

[0092] Then, the terminal device 10 provides the browsing page to the user U (step S214), and ends the process.

[0093] Next, a series of processes relating to provision of photographed content will be described with reference to Fig. 13. As shown in Fig. 13, first, the terminal device 10 acquires a browsing page relating to a composite target image from the information providing device 100 (step S221).

[0094] Next, the terminal device 10 provides a browsing page to the user U (step S222). Next, the terminal device 10 accepts a selection operation of a composite target image (step S223).

[0095] Next, the terminal device 10 sets the photographing conditions of the user image based on, for example, the face direction of the subject of the selected composite target image (step S224).Then, the terminal device 10 provides photographed content based on the photographing conditions that have been set (step S225), and ends the process.

[0096] Next, a series of processing steps when a user image is determined before a composite target image will be described with reference to Fig. 14. As shown in Fig. 14, the terminal device 10 determines a user image to be composited based on an operation by a user (step S231). Note that the user image may be an image registered in advance in the terminal device 10, or may be a newly captured image.

[0097] Next, the terminal device 10 determines an appropriate image from among the posted images displayed on the browsing page based on the user image (step S232). Next, the terminal device 10 assigns compositing content (for example, the compositing appropriateness icon C10 shown in FIG. 9) to the appropriate image from among the posted images displayed on the browsing page (step S233).

[0098] Then, the terminal device 10 provides the browsing page to the user U (step S234), and ends the process.

[0099] [5. Modifications] In the above-described embodiment, a case has been described in which the appropriateness of an image is determined based on the facial orientation of the subject appearing in the user image and the composite target image, respectively. However, this is not limiting. For example, the appropriateness of an image may be determined based on an evaluation of the hairstyle of the subject appearing in the composite target image. More specifically, the number of "likes" on the posting page of each posted image or evaluations may be collected using CrowdWorks or the like, and when a posted image is input, a model may be trained that calculates a higher score for higher evaluations, and the appropriateness of an image may be determined based on the output result of the model.

[0100] In the above embodiment, a composite image is generated by combining a hairstyle of a composite target image with a user image of the user's face, but the present invention is not limited to this. For example, a composite image may be generated by combining glasses, a hat, accessories, etc. that appear in the composite target image with the user's face.

[0101] In the above-described embodiment, the case where an image posted on an SNS is acquired as the composite target image has been described, but the present invention is not limited to this. For example, an image taken by a user may be acquired as the composite target image.

[0102] Furthermore, the user image and the composite target image are not limited to these, and the user image and the composite target image may be changed as appropriate, for example, by compositing furniture that appears in the composite target image with an image of a house taken by the user.

[0103] [6. Effects] As described above, the terminal device 10 (an example of an information processing device) according to the present application includes a determination unit 34 that determines whether or not the image to be synthesized is an appropriate image suitable for synthesis, and a provision unit 36 ​​that provides content for synthesizing the image to be synthesized with an image captured by a user when the determination unit 34 determines that the image to be synthesized is the appropriate image.

[0104] The terminal device 10 according to the present application also includes a generation unit 35 that generates a composite image by combining a composite target image and a captured image. The generation unit 35 generates a composite image by combining the hairstyle of a subject appearing in the composite target image with the face of a user appearing in the captured image.

[0105] The determination unit 34 also determines whether or not an image is appropriate based on the orientation of the face of the subject appearing in the composite target image. The determination unit 34 also determines whether or not an image is appropriate depending on whether or not the orientation of the face of the subject is facing forward.

[0106] The determination unit 34 determines whether the image is appropriate based on the focus of the subject and the size of the subject in the composite target image. If the provision unit 36 ​​determines that the composite target image is appropriate, it displays a composite icon C1, which initiates generation of a composite image. The terminal device 10 also includes an acquisition unit 31 that acquires an image posted to a social networking service as the composite target image.

[0107] By performing any one or a combination of the above-described processes, the information processing device according to the present application can provide appropriate content related to a composite image.

[0108] [7. Hardware Configuration] The information processing device according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 15. Fig. 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0109] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0110] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a network (communication network) N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

[0111] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse (in FIG. 15, the output devices and input devices are collectively referred to as "input / output devices") via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0112] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0113] For example, when the computer 1000 functions as the information processing device according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via the network N.

[0114] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art.

[0115] [8. Other] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0116] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0117] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0118] For example, the information providing device 100 described above may be realized using multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0119] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0120] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0121] 10 Terminal Equipment 11 Communications Department 12 Display section 13 Input section 20 Camera 30 Control Unit 31 Acquisition Department 32 Reception Department 33 Estimation part 34 Judgment section 35 Generation part 36 Providing Department 40 Storage section 100 Information provision device 110 Communications Department 120 Storage section 121 Posting Information Database 122 Score Estimation Model Database 123 Synthetic Model Database 130 control section 131 Acquisition Department 132 Granting Department

Claims

1. a determination unit that determines whether or not the composite target image is an appropriate image suitable for composite; a providing unit that provides content for combining the composite target image with a captured image taken by a user when the determining unit determines that the composite target image is the appropriate image; Equipped with The determination unit Determining whether the image to be synthesized is an appropriate image based on a background image of the subject appearing in the image to be synthesized. An information processing device characterized by:

2. a generating unit for generating a composite image by combining the composite target image and the photographed image; To be prepared 2. The information processing device according to claim 1,

3. The generation unit generating a composite image by combining the hairstyle of the subject appearing in the composite target image with the face of the user appearing in the photographed image; 3. The information processing device according to claim 2, wherein:

4. The determination unit Determining whether the image is appropriate or not based on the orientation of the face of the subject appearing in the composite target image.

4. The information processing device according to claim 3,

5. The determination unit Determining whether the image is appropriate or not depending on whether the face of the subject reflected in the image to be synthesized is facing forward.

5. The information processing device according to claim 4,

6. The determination unit Determining whether the image is appropriate or not based on at least one of the focus deviation of the subject in the composite target image and the size of the subject.

2. The information processing device according to claim 1,

7. The providing unit To provide content that displays a composition icon that transitions to generation of the composite image when the composition target image is determined to be the appropriate image.

3. The information processing device according to claim 2, wherein:

8. an acquisition unit that acquires an image posted on a social networking service as the composite target image; 2. The information processing apparatus according to claim 1, further comprising:

9. 1. A computer-implemented information processing method, comprising: a determination step of determining whether the composite target image is an appropriate image suitable for composite; a providing step of providing content for combining the composite target image with a photographed image taken by a user when the composite target image is determined to be the appropriate image by the determining step; Including, The determination step includes: Determining whether the image to be synthesized is an appropriate image based on a background image of the subject appearing in the image to be synthesized. An information processing method comprising:

10. a determination step for determining whether the composite target image is an appropriate image suitable for composite; a providing step of providing content for combining the composite target image with a photographed image taken by a user when the composite target image is determined to be the appropriate image by the determining step; on the computer, Determining whether the image to be synthesized is an appropriate image based on a background image of the subject appearing in the image to be synthesized. An information processing program characterized by:

Images