Mobile device, information processing method, and non-transitory computer-readable medium

Abstract

Disclosed are a mobile device, an information processing method, and a non-transitory computer-readable medium. The mobile device includes: an imaging unit configured to capture a through-image; a display unit configured to display the through-image; and a processor configured to: acquire mesh data for at least one first facial image based on feature points of the first facial image included in the through-image; acquire texture data for at least one second facial image different from the first facial image; determine whether the through-image includes a plurality of the first facial images; and, based on the determination that the through-image includes a plurality of the first facial images, control the display unit to display the texture data for the second facial image on each of the plurality of the first facial images so as to correspond to the mesh data for the plurality of the first facial images.

Description

[0001] This invention application is a divisional application of invention patent application No. 201680041354.8, whose application date is May 9, 2016, international application number is PCT / JP2016 / 063746, and which entered the Chinese national phase on January 12, 2018, and whose invention name is “Information processing device, information processing method and program”. Technical Field

[0002] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0003] In recent years, with the advancement of image processing technology, technologies for superimposing and displaying images on virtual or real images have been studied and developed. For example, there is also augmented reality (AR) technology as one of such technologies, and this technology has been applied to various fields.

[0004] For example, Patent Document 1 discloses an information processing device that superimposes and displays a makeup image on an image based on the position of a facial portion identified in the image. In addition to makeup images, various other images can be superimposed and displayed. Specifically, this includes a composite image obtained by combining two or more images.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: JP 2012-181688A Summary of the Invention

[0008] Technical issues

[0009] However, the synthesis of images in the related art technology is fixed and does not always suit the user's intention. Therefore, the present disclosure will propose a novel and improved information processing device, information processing method, and program that can perform synthesis desired by the user.

[0010] Solution to the problem

[0011] According to the present disclosure, an information processing device is provided, including a control unit configured to generate a composite object model by synthesizing a first object model and a second object model, and to display the generated composite object model in association with an image corresponding to the first object model or the second object model. The control unit controls a mode of the synthesis based on input information.

[0012] In addition, according to the content of the present disclosure, an information processing method is provided, which includes: generating a synthetic object model by synthesizing a first object model and a second object model through a control unit; displaying the generated synthetic object model in association with an image corresponding to the first object model or the second object model; and controlling the synthesis mode based on input information.

[0013] In addition, according to the content of the present disclosure, a program is provided that enables a computer to implement the following functions: a function of generating a synthetic object model by synthesizing a first object model and a second object model; a function of displaying the generated synthetic object model in association with an image corresponding to the first object model or the second object model; and a function of controlling a synthesis mode based on input information.

[0014] According to one aspect of the present application, a mobile device is provided, including: an imaging unit configured to capture a through-image; a display unit configured to display the through-image; and a processor configured to: acquire grid data of at least one first facial image based on feature points of the first facial image included in the through-image; acquire texture data of at least one second facial image different from the first facial image; determine whether the through-image includes a plurality of the first facial images; and, based on the determination that the through-image includes the plurality of the first facial images, control the display unit to display the texture data of the second facial image on each of the plurality of the first facial images to correspond to the grid data of the plurality of the first facial images.

[0015] According to another aspect of the present application, a mobile device is provided, including: a storage unit; an imaging unit configured to capture a through-image; a display unit configured to display the through-image; and a processor configured to: acquire grid data of at least one first facial image based on feature points of the first facial image included in the through-image; acquire texture data of at least one second facial image different from the first facial image; control the display unit to display the texture data of the second facial image on the through-image to correspond to the grid data of the first facial image; record the grid data in the storage unit; and control the display unit to display the texture data of the second facial image based on the recorded grid data even when the number of feature points of the first facial image decreases.

[0016] According to another aspect of the present application, a mobile device is provided, including: an imaging unit configured to capture a through-image; a display unit configured to display the through-image; and a processor configured to: acquire grid data of at least one first facial image included in the through-image based on feature points of the first facial image; acquire texture data of at least one second facial image different from the first facial image; control the display unit to display the texture data of the second facial image on the through-image to correspond to the grid data of the first facial image; and control the display unit to display a plurality of second facial images different from each other together with the first facial image.

[0017] According to another aspect of the present application, an information processing method is provided, including: acquiring a through-image captured by an imaging unit of a mobile device; acquiring grid data of at least one first facial image based on feature points of the first facial image included in the through-image; acquiring texture data of at least one second facial image different from the first facial image; determining whether the through-image includes a plurality of the first facial images; and, based on the determination that the through-image includes the plurality of the first facial images, controlling a display unit of the mobile device to display the texture data of the second facial image on each of the plurality of the first facial images to correspond to the grid data of the plurality of the first facial images.

[0018] According to another aspect of the present application, a non-transitory computer-readable medium having a program embodied thereon is provided, which, when executed by a mobile computing device, causes the mobile computing device to perform a method, the method comprising: acquiring a through-image captured by an imaging unit of the mobile computing device; acquiring mesh data of at least one first facial image based on feature points of a first facial image included in the through-image; acquiring texture data of at least one second facial image different from the first facial image; determining whether the through-image includes a plurality of the first facial images; and, based on the determination that the through-image includes a plurality of the first facial images, controlling a display unit of the mobile computing device to display the texture data of the second facial image on each of the plurality of the first facial images to correspond to the mesh data of the plurality of the first facial images.

[0019] Beneficial effects of the present invention

[0020] According to the present disclosure, an information processing device, information processing method, and program capable of performing the synthesis desired by the user as described above can be provided. Note that the above effects are not necessarily restrictive. Together with or in place of the above effects, any of the effects described in this specification or other effects that can be understood from this specification can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] [ Figure 1 ] Figure 1 is a block diagram illustrating a schematic functional configuration example of the information processing device according to the first embodiment of the present disclosure.

[0022] [ Figure 2 ] Figure 2 2 is a diagram illustrating image synthesis processing according to an embodiment.

[0023] [ Figure 3 ] Figure 3 is a diagram showing a combination example of src and dst according to an embodiment.

[0024] [ Figure 4 ] Figure 4 is a diagram illustrating operations and processes related to synthesizing a face mask according to an embodiment.

[0025] [ Figure 5 ] Figure 5 is a diagram illustrating an example of a face mask displayed by an information processing device according to an embodiment.

[0026] [ Figure 6 ] Figure 6 is a flowchart conceptually illustrating processing executed by an information processing apparatus according to an embodiment.

[0027] [ Figure 7 ] Figure 7 : is a diagram illustrating a process of generating a plurality of synthesized face masks based on a face mask selected in a face list or the like according to a second modified example of the embodiment.

[0028] [ Figure 8 ] Figure 8 : is a diagram illustrating a process of generating a plurality of synthesized face masks based on a face mask selected from a face image or the like displayed on a screen according to a second modified example of the embodiment.

[0029] [ Figure 9 ] Figure 9 : is a diagram illustrating a registration process in a face list according to a third modified example of the embodiment.

[0030] [ Figure 10 ] Figure 10 : are diagrams illustrating a synthesis process according to a fourth modified example of the embodiment.

[0031] [ Figure 11 ] Figure 11 : is a diagram showing a preview in a face list according to a fifth modified example of the embodiment.

[0032] [ Figure 12 ] Figure 12: are diagrams illustrating synthesis processing without a fixed face mask according to a sixth modified example of the embodiment.

[0033] [ Figure 13 ] Figure 13 : are diagrams illustrating synthesis processing in a case where a face mask is fixed according to a sixth modified example of the embodiment.

[0034] [ Figure 14 ] Figure 14 : is a diagram showing an example of an interface for adjusting facial feature information in a synthesized face mask according to a seventh modified example of the embodiment.

[0035] [ Figure 15A ] Figure 15A : is a diagram illustrating a process of changing mesh information in a synthesized face mask according to a seventh modified example of the embodiment.

[0036] [ Figure 15B ] Figure 15B : is a diagram illustrating a process of changing mesh information in a synthesized face mask according to a seventh modified example of the embodiment.

[0037] [ Figure 15C ] Figure 15C : is a diagram illustrating a process of changing mesh information in a synthesized face mask according to a seventh modified example of the embodiment.

[0038] [ Figure 16 ] Figure 16 : is a diagram illustrating another example of a process of changing mesh information in a synthesized face mask according to a seventh modified example of the embodiment.

[0039] [ Figure 17 ] Figure 17 : is a diagram illustrating a process of changing texture information in a synthesized face mask according to a seventh modified example of the embodiment.

[0040] [ Figure 18 ] Figure 18 : is a diagram illustrating a process of changing texture information in a synthesized face mask according to a seventh modified example of the embodiment.

[0041] [ Figure 19 ] Figure 19 : is a diagram illustrating an example of a process of synthesizing an object other than a face according to a ninth modification example of the embodiment.

[0042] [ Figure 20 ] Figure 20 is a diagram illustrating another example of a process of synthesizing an object other than a face according to a ninth modification example of the embodiment.

[0043] [ Figure 21 ] Figure 21 : is a diagram illustrating an operation of moving a face mask as a target of synthesis, performed by the information processing apparatus according to the second embodiment of the present disclosure.

[0044] [ Figure 22 ] Figure 22 : is a diagram illustrating a process of specifying a target for synthesis based on a user's movement, which is performed by the information processing apparatus according to the embodiment.

[0045] [ Figure 23 ] Figure 23 : is a diagram showing an example of moving a face mask or the like as a target of synthesis by a person according to a first modified example of the embodiment.

[0046] [ Figure 24 ] Figure 24 : is a diagram illustrating generation and sharing of link information according to a second modified example of the embodiment.

[0047] [ Figure 25 ] Figure 25 is an explanatory diagram illustrating a hardware configuration of an information processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, structural elements having substantially the same function and structure are denoted by the same reference numerals, and repeated explanation of these structural elements is omitted.

[0049] In the specification and drawings, in some cases, multiple components having substantially the same functional configuration will be distinguished by different numbers added to the same reference numeral. For example, multiple configurations having substantially the same function will be distinguished as, for example, face mask 20A and face mask 20B, as needed. However, in cases where it is not necessary to distinguish between substantially the same functional configurations, the same reference numeral will be used. In cases where it is not particularly necessary to distinguish between face mask 20A and face mask 20B, the face mask will simply be referred to as face mask 20.

[0050] The description will be given in the following order.

[0051] 1. First embodiment (application example for mobile terminals)

[0052] 1-1. Device Configuration

[0053] 1-2. Technical features of the device

[0054] 1-3. Processing by the device

[0055] 1-4. Summary of the First Embodiment

[0056] 1-5. Modification Example

[0057] 2. Second Embodiment (Application Example for Digital Signage Devices)

[0058] 2-1. Device Configuration

[0059] 2-2. Technical features of the device

[0060] 2-3. Processing by the device

[0061] 2-4. Summary of the Second Embodiment

[0062] 2-5. Modification Example

[0063] 3. Application Examples

[0064] 4. Hardware Configuration of an Information Processing Device According to One Embodiment of the Present Disclosure

[0065] 5. Conclusion

[0066] <1. First Embodiment (Application Example for Mobile Terminals)>

[0067] First, the information processing device 100-1 according to the first embodiment of the present disclosure will be described. The information processing device 100-1 according to this embodiment is a portable mobile terminal such as a smartphone or tablet terminal. For ease of description, the information processing devices 100 according to the first embodiment and the second embodiment will be distinguished by adding numbers corresponding to the embodiment at the end, such as information processing device 100-1 and information processing device 100-2.

[0068] <1-1. Device Configuration>

[0069] Will refer to Figure 1 The functional configuration of the information processing device 100 - 1 according to the first embodiment of the present disclosure is described. Figure 1 is a block diagram illustrating a schematic functional configuration example of the information processing device 100 - 1 according to the first embodiment of the present disclosure.

[0070] like Figure 1 As shown, the information processing apparatus 100 - 1 includes an imaging unit 102 , a facial recognition unit 104 , an operation detection unit 106 , a control unit 108 , a storage unit 110 , and a display unit 112 .

[0071] The imaging unit 102 images the surroundings of the information processing device 100-1. Specifically, the imaging unit 102 captures still images or moving images based on instructions from the control unit 108. For example, the imaging unit 102 may include an imaging optical system such as a zoom lens and an imaging lens for collecting light, and a signal conversion element such as a charge-coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). An input unit for acquiring images may be included instead of the imaging unit 102. Images may be acquired through communication.

[0072] The facial recognition unit 104 recognizes a face included in an image as an object (including a part of an object). Specifically, the facial recognition unit 104 recognizes a face appearing in an image obtained by image capture by the imaging unit 102 or a face included in an image stored in the storage unit 110. The facial recognition result is then provided to the control unit 108. For example, the facial recognition unit 104 performs facial recognition processing by using a typical facial recognition technology.

[0073] Operation detection unit 106 detects user operations performed on information processing device 100-1. Specifically, if the user performs an operation, operation detection unit 106 obtains information related to the operation and detects the content of the operation from the obtained information related to the operation. The operation detection result is then provided to control unit 108. For example, the detected operation may be a button operation, a click operation, a tap operation, a drag-and-drop operation, a slide operation, a flick operation, a gesture operation, or the like.

[0074] The control unit 108 controls the image display. Specifically, the control unit 108 performs image synthesis processing based on the face recognition result of the face recognition unit 104 and the operation detection result detected by the operation detection unit 106. Figure 2 The image synthesis process is described in detail. Figure 2 2 is a diagram illustrating a process of synthesizing an image according to an embodiment.

[0075] First, the control unit 108 detects feature points in the recognized face. For example, the control unit 108 detects Figure 2 The images of the face recognized by the face recognition unit 104 (hereinafter, also referred to as face images) 10A and 10B shown in the left figure are as follows. Figure 2 The feature points shown in the middle left figure.

[0076] Next, the control unit 108 generates a face model (hereinafter, also referred to as a face mask) based on the detected feature points. Specifically, the control unit 108 generates information including mesh information and texture information (hereinafter, also referred to as facial feature information) based on the feature points. Then, the control unit 108 generates a face mask based on the facial feature information. For example, the control unit 108 generates a face mask based on the feature points. Figure 2 The control unit 108 then generates the following information for each of the facial images 10A and 10B based on the mesh information and the texture information: Figure 2 The facial masks 20A and 20B are shown in the middle right figure of FIG.

[0077] Next, the control unit 108 generates a composite face mask by synthesizing the face masks. Specifically, the control unit 108 combines the facial feature information of the respective face masks and generates a composite face mask based on the facial feature information obtained by the combination. For example, the control unit 108 generates a composite face mask including Figure 2 The facial feature information of the mesh information of the facial mask 20A and the texture information of the facial mask 20B shown in the middle right figure of the image is combined, and a synthesized facial mask is generated based on the facial feature information.

[0078] Next, the control unit 108 causes the generated synthesized face mask to be displayed in association with the image corresponding to the face mask serving as a synthesis destination or synthesis source (hereinafter, also referred to as a synthesis target). Figure 2 As shown in the right figure of , the control unit 108 causes the generated synthetic facial mask 30 to be superimposed and displayed on the facial image 10A corresponding to the facial mask 20A.

[0079] Return to reference Figure 1 As described in the functional configuration of the information processing apparatus 100-1, the storage unit 110 stores information related to image processing. Specifically, the storage unit 110 stores an image as an input for face recognition processing and dictionary information (feature point information) for face recognition.

[0080] The display unit 112 displays an image. Specifically, the display unit 112 displays an image supplied from the control unit 108. For example, the display unit 112 may be a liquid crystal panel, an organic electroluminescence (EL) display, or the like.

[0081] <1-2. Technical Features of the Device>

[0082] Next, technical features of the information processing device 100 - 1 according to the embodiment will be described.

[0083] (A. Designation of synthesis target)

[0084] The information processing device 100-1 specifies the target of synthesis in response to the user's input. Specifically, the control unit 108 specifies the face mask specified based on the user's input operation or the face mask generated from the specified face image as the target of synthesis. The designation of the target of synthesis is the designation of the synthesis source (src) and the synthesis destination (dst). In addition, the reference Figure 3 Describes the combination of src and dst. Figure 3 is a diagram showing an example of a combination of src and dst according to an embodiment.

[0085] First, there are mainly three types of facial images corresponding to the facial masks used as the target of synthesis. For example, the facial image may be a static image, a moving image, or an image temporarily recorded by image capture (hereinafter, also referred to as a through image). The control unit 108 synthesizes the facial mask generated from the facial image that is a static image, a moving image, or a through image and the facial mask generated from the facial image that is a static image, a moving image, or a through image. Therefore, for example, Figure 3 As shown, there are nine synthesis methods. In addition, src or dst can be an already generated face mask, and if this case is included, there are sixteen synthesis methods.

[0086] There are several times when a face mask is generated. Specifically, a face mask is generated before or at the start of the synthesis process. Examples of the time before the synthesis process include when the face recognition unit 104 recognizes a face. Examples of the time when the synthesis process starts include when the movement of the face mask, which will be described later, begins, or when the designation of a synthesis target is completed without moving the face mask.

[0087] The control unit 108 moves the facial mask that is the target of synthesis in response to input based on the user's movement. Specifically, input based on the user's movement includes an operation for specifying both a starting point and an end point, or an operation for specifying only the starting point. A drag-and-drop operation is an operation for specifying both a starting point and an end point. A slide operation or a flick operation is an operation for specifying only the starting point. For example, the control unit 108 designates the facial mask or facial image (hereinafter, also referred to as a facial mask, etc.) that is the target of the drag operation as src, and the facial mask, etc. that is the target of the drop operation as dst. Alternatively, the control unit 108 designates the facial mask, etc. that has been subjected to a slide operation or a flick operation as src, and the facial mask, etc. that is positioned in the direction of the slide operation or the flick operation as dst. The facial mask, etc. that has been subjected to the slide operation or the flick operation is moved as if it were flying in the direction of the operation.

[0088] (A-1. Designation of Synthesis Source)

[0089] The control unit 108 designates the face mask selected by the user's input operation or the face mask generated from the selected face image as src. Specifically, the control unit 108 displays a face list in which face masks or face images are arranged on the screen. Then, the control unit 108 designates the face mask selected from the face list by the user's input operation or the face mask generated from the selected face image as src. For example, the control unit 108 designates the face mask selected as the target of movement as src. In addition, the control unit 108 designates the face mask selected as the target of movement as src. Figure 4 The process of specifying a synthesis source is described in detail. Figure 4 is a diagram illustrating operations and processes related to synthesized facial masks according to an embodiment.

[0090] The control unit 108 displays a face list in which face masks or face images are arranged on the screen. For example, the control unit 108 displays a face list in which face masks or face images are arranged. Figure 4 A face list of the static facial images 12A1 to 12A3 shown in the lower portion of is displayed on the screen.

[0091] Then, the control unit 108 designates a face mask designated from the face list by the user's input operation or a face mask generated from a designated face image as src. Figure 4 As shown in the left figure of , the control unit 108 specifies the static facial image 12A1 as the target of the drag operation as src.

[0092] Although the example of selecting src from a face list has been described so far, src may be selected from a still image, a moving image, or a through image displayed on the screen. For example, the control unit 108 specifies as src a face mask or the like selected by the user from a face image included in a face recognition result obtained from a still image, a moving image, or a through image displayed on the screen, or a face mask or the like selected from a face mask generated from the face image.

[0093] Control unit 108 selects facial masks and the like arranged in the face list as candidates for synthesis based on the attributes of the face represented by the facial mask. Specifically, facial attributes include age and gender. For example, control unit 108 selects a facial mask or facial image corresponding to an age close to the age estimated from the facial image provided as the facial recognition result by facial recognition unit 104, or selects a facial mask or facial image representing a face of the same gender as the estimated gender. Control unit 108 then displays the face list in which the selected facial masks and the like are arranged.

[0094] The face masks and the like arranged in the face list may be face masks and the like of faces recognized in a predetermined period of time.

[0095] (A-2. Designation of synthesis purpose)

[0096] The control unit 108 designates the face mask selected by the user's input operation or the face mask generated from the selected face image as dst. Specifically, the control unit 108 designates the face mask at the movement destination, such as the face mask to be moved, as dst. Figure 4 Describe in detail the process of specifying the synthesis purpose.

[0097] The control unit 108 moves the face mask or the like designated as src in response to an input based on the user's motion. For example, the control unit 108 moves the face mask or the like designated as src. Figure 4 The static facial image 12A1 shown in the left figure of FIG. 1 , which is designated as the target of the drag operation, moves according to the user's finger movement.

[0098] Next, the control unit 108 designates the face mask or the like that exists at the position where the input ends when the input based on the user's motion ends as dst. Figure 4 As shown in the left figure of FIG, the static facial image 12A1 is moved toward the through facial image 16A and a release operation is performed on the facial image 16A, the control unit 108 designates the facial image 16A as dst.

[0099] (B. Generation of synthetic facial masks based on synthetic patterns)

[0100] The information processing device 100-1 generates a synthesized facial mask according to a mode for synthesis (hereinafter also referred to as a synthesis mode) based on the facial mask designated as the target of synthesis, etc. Specifically, the control unit 108 generates a synthesized facial mask by synthesizing the facial mask designated as src or generated from the designated facial image and the facial mask designated as dst or generated from the designated facial image according to the synthesis mode.

[0101] Here, the synthesis mode includes a first mode and a second mode, wherein in the first mode, the synthesized facial mask is generated based on the mesh information of the facial mask as src, and in the second mode, the synthesized facial mask is generated based on the mesh information of the facial mask as dst. In the first mode, the synthesized facial mask is generated based on the texture information of the facial mask as dst, and in the second mode, the synthesized facial mask is generated based on the texture information of the facial mask as src.

[0102] For example, the control unit 108 receives Figure 4 The facial image 12A1 shown in the left figure is generated by the facial mask that has been designated as src and the facial mask generated by Figure 4 The facial image 16A shown on the left is synthesized with the facial mask that has been designated as dst. Figure 4 In the example of FIG. 1 , the control unit 108 generates a face mask as shown in FIG. 1 according to the second mode by using the mesh information of the face mask generated from the face image 16A as dst and the texture information of the face mask generated from the face image 12A1 as src. Figure 4 The middle figure shows the synthesized facial mask 26A.

[0103] If the purpose of synthesis is not specified, control unit 108 may not perform synthesis. Specifically, if the input based on the user's movement is interrupted, control unit 108 does not synthesize the face mask. For example, if a release operation is performed at a location where a face mask or facial image is not displayed, control unit 108 does not perform synthesis.

[0104] (C. Control of synthesis mode)

[0105] The information processing device 100-1 controls the aforementioned synthesis mode. Specifically, the control unit 108 controls the synthesis mode based on the input information. More specifically, corresponding to the input of the processing performed by the control unit 108, the input information includes information related to the image corresponding to the face mask as src or the face mask as dst. Examples of information related to the image include information related to the attributes of the image or the content of the image. As attributes of the image, the type of medium (electronic medium) such as a still image, a moving image or a through image and the type of content such as a photograph or a computer image (CG) are exemplified. For example, in the case where the face mask of the still image is dst and the face mask of the moving image is src, the control unit 108 selects the first mode. As the content of the image, the presence or content of speech and actions of people appearing in the image such as music playing or sports are exemplified.

[0106] The input information may be information related to the user's input operation. Specifically, the user's input operation includes an operation of selecting a face mask used as src or a face mask used as dst, and the control unit 108 specifies the synthesis mode according to the position of the face mask selected as src or the face mask selected as dst. Figure 5 The process of specifying the synthesis mode according to the selected position is described in detail. Figure 5 is a diagram illustrating an example of a face mask displayed by the information processing device 100 - 1 according to an embodiment.

[0107] The control unit 108 causes the position of the face mask selected as src or the face mask selected as dst corresponding to each synthesis mode to be displayed in a visually distinguishable form. Specifically, the control unit 108 causes the selected position to be displayed in a form different from the form specified by the texture information of the displayed face mask. For example, the control unit 108 causes the position to be displayed in a form different from the form specified by the texture information of the displayed face mask. Figure 5 The face mask 20 is displayed with different colors applied to the shaded area A1 and the grid line area A2. For example, if the user starts a drag operation from the starting point in area A1, the first mode is designated as the composite mode. If the user starts a drag operation from the starting point in area A2, the second mode is designated as the composite mode.

[0108] Although the example in which the visually distinguishable form uses color has been described so far, the form may use brightness, luminance, pattern, or flicker. When a position other than the selected position corresponding to each synthesis mode is selected, the control unit 108 does not specify a synthesis mode based on the selected position.

[0109] While the user input operation described so far is an example of selecting a face mask, in addition to the aforementioned example, the user input operation may be a click or tap operation on the face mask to be synthesized. The user input operation may be an operation performed on a dedicated graphical user interface (GUI). For example, the mode may be selected based on the result of operating the synthesis mode switch button or synthesis mode selection menu displayed when the release operation is performed. Furthermore, a display object indicating the selected synthesis mode may be displayed on the screen.

[0110] (D. Display of Synthesized Facial Mask)

[0111] The information processing device 100-1 associates and displays the generated synthetic facial mask with the facial image. Specifically, the control unit 108 associates and displays the synthetic facial mask with the facial image specified based on the user's input from the facial images corresponding to the facial mask as src and the facial mask as dst. More specifically, after the movement from the facial mask src to the facial mask dst is completed, the control unit 108 associates and displays the synthetic facial mask with the image corresponding to the facial mask as dst. For example, the control unit 108 Figure 4 The synthesized facial mask 26A obtained by synthesis, shown in the middle figure of FIG, is superimposed and displayed on the facial image 16A designated as dst by the release operation. The control unit 108 may represent the texture of the synthesized facial mask as transparent, and may cause the transparency to decrease over time when the synthesized facial mask is displayed.

[0112] (E. Operations on Synthesized Facial Mask)

[0113] The information processing device 100-1 moves the synthetic face mask in the same manner as the face mask in response to the user's operation. Specifically, the control unit 108 moves the synthetic face mask according to the input based on the user's movement. For example, the control unit 108 moves the synthetic face mask in response to the user's drag operation. Figure 4 The synthesized face mask 36A shown in the right figure of FIG is moved. Even when the synthesized face mask is not superimposed and displayed on the face image, the synthesized face mask is changed according to the change of src or dst.

[0114] (F. Image Storage)

[0115] The information processing device 100-1 saves the image on which the synthetic face mask is superimposed and displayed (hereinafter, also referred to as the processed image). Specifically, if the operation detection unit 106 detects an operation to save the image while the synthetic face mask is displayed on the screen, the control unit 108 causes the storage unit 110 to store the processed image.

[0116] Information processing apparatus 100-1 allows the user to select a processed image or an unprocessed image as a target for saving. For example, if operation detection unit 106 detects an operation to save an image, control unit 108 displays a GUI on the screen, which allows the user to select which of the processed and unprocessed images to save. Control unit 108 then saves either the processed or unprocessed image based on the selection result.

[0117] Information processing device 100-1 may display a display object on the screen indicating that the image is a processed image. For example, control unit 108 may display a marker object at a corner of the processed image. Furthermore, control unit 108 may display a display object, such as a finger grip, adjacent to the synthesized face mask, reduce the transparency of the outline of the synthesized face mask, or display the synthesized face mask as if it were flipped.

[0118] <1-3. Processing by Device>

[0119] Next, we will refer to Figure 6 Processing performed by the information processing apparatus 100 - 1 according to the embodiment is described. Figure 6 1 is a flowchart conceptually illustrating processing performed by the information processing apparatus 100 - 1 according to the embodiment.

[0120] The information processing apparatus 100 - 1 determines whether an operation of dragging the face mask or the like has been performed (step S302 ) Specifically, the control unit 108 determines whether the operation detection unit 106 has detected a drag operation.

[0121] If it is determined that a drag operation has been performed, the information processing apparatus 100-1 specifies src based on the drag operation (step S304). Specifically, if the operation detection unit 106 detects a drag operation, the control unit 108 specifies the face mask at the start position of the detected drag operation as src.

[0122] Next, the information processing apparatus 100-1 determines whether an operation of releasing the face mask or the like has been performed (step S306). Specifically, the control unit 108 determines whether the operation detection unit 106 has detected a releasing operation.

[0123] If it is determined that a release operation has been performed, the information processing device 100-1 specifies dst based on the release operation (step S308). Specifically, if the operation detection unit 106 detects a release operation, the control unit 108 specifies the face mask or the like at the position where the release operation was detected as dst.

[0124] If the release operation is not performed normally, the process ends. Specifically, if the release operation is performed at a position where there is no face mask or the like, the control unit 108 ends the process.

[0125] Next, the information processing device 100-1 specifies the synthesis mode based on the input information (step S310). Specifically, the control unit 108 specifies the synthesis mode based on information about the facial image specified as src or dst or the facial image corresponding to the face mask.

[0126] Next, the information processing apparatus 100 - 1 determines the designated synthesis mode (step S312 ). Specifically, the control unit 108 determines which of the first mode and the second mode the designated synthesis mode is.

[0127] If the designated synthesis mode is the first mode, the information processing device 100-1 generates a synthesized facial mask based on the mesh information of the facial mask src and the texture information of the facial mask dst (step S314). Specifically, if the synthesis mode is the first mode, the control unit 108 generates a synthesized facial mask including the mesh information of the facial mask src as mesh information and the texture information of the facial mask dst as texture information.

[0128] If the designated synthesis mode is the second mode, the information processing device 100-1 generates a synthesized facial mask based on the mesh information of the facial mask dst and the texture information of the facial mask src (step S316). Specifically, if the synthesis mode is the second mode, the control unit 108 generates a synthesized facial mask including the mesh information of the facial mask dst as mesh information and the texture information of the facial mask src as texture information.

[0129] Next, the information processing device 100-1 associates the synthesized face mask with the facial image of dst and displays it (step S318). Specifically, the control unit 108 superimposes and displays the generated synthesized face mask on the facial image corresponding to the facial mask dst.

[0130] <1-4. Summary of First Embodiment>

[0131] According to the first embodiment of the present disclosure, information processing device 100-1 generates a composite object mask by synthesizing a first object mask and a second object mask, and displays the generated composite object mask in association with an image corresponding to the first object mask or the second object mask. Furthermore, information processing device 100-1 controls the synthesis mode based on input information. Therefore, by performing synthesis based on the input information, the user's desired synthesis can be performed.

[0132] The first object mask includes a first face mask, the second object mask includes a second face mask, and the synthesized object mask includes a synthesized face mask. Thus, the user can appreciate a face as the target of the synthesis, where the face is a characteristic part of the human body. The target of the object mask can be an object other than a person, for example, a natural object such as a vegetable or a rock, or an artificial object such as a clock. The object includes a portion of the aforementioned objects. Therefore, the target of the object mask can be a part of the human body other than the face, or can be a portion of an object other than a person.

[0133] The modes include a first mode, in which a composite facial mask is generated based on the mesh information of the first facial mask, and a second mode, in which a composite facial mask is generated based on the mesh information of the second facial mask. Therefore, by switching the mesh information of the composite facial mask according to the composite mode, the representation of the face of the composite facial mask can be selected from the representation of the face of the source or destination facial mask. As a result, user convenience can be improved.

[0134] In the first mode, the composite facial mask is generated based on the texture information of the second facial mask. In the second mode, the composite facial mask is generated based on the texture information of the first facial mask. Thus, the composite facial mask is automatically switched, and user convenience can be improved by changing the combination of grid information and texture information according to the composite mode.

[0135] Information processing device 100-1 displays the composite face mask in association with an image specified based on user input from images corresponding to the first and second face masks. Thus, user convenience can be improved by superimposing and displaying the composite face mask on the facial image specified by the user.

[0136] The input information includes information related to an image corresponding to the first facial mask or the second facial mask as input to the control process. Therefore, by synthesizing a facial image suitable for the facial image, particularly a facial image with a synthesized facial mask superimposed and displayed, it is possible to suppress the user's unfamiliar feeling or, conversely, intentionally give the user an unfamiliar feeling.

[0137] The aforementioned image-related information includes information related to the aforementioned image attributes and the aforementioned image content. Therefore, compared to synthesis based on the image content, the processing load can be reduced when synthesizing based on the image attributes. Furthermore, compared to synthesis based on the image attributes, synthesis based on the image content can perform synthesis that is suitable for the display scene.

[0138] In addition, the input information includes information related to the user's input operation. Therefore, the user's convenience can be improved by performing synthesis that meets the user's intention.

[0139] The user's input operation includes an operation of selecting the first face mask or the second face mask, and the information processing device 100-1 specifies the mode according to the selection position of the first face mask or the second face mask. Therefore, it is possible to improve operability by enabling the selection of the synthesis mode when selecting the synthesis target.

[0140] The information processing device 100-1 displays the selected position of the first face mask or the second face mask corresponding to each mode in a visually distinguishable form. Therefore, the user can easily know the selected position and the convenience of the user can be improved.

[0141] At least one of the images corresponding to the first and second facial masks includes an image temporarily recorded during image capture. This allows the user to observe how the synthesis progresses during image capture and to experiment with trial and error. This improves user convenience.

[0142] According to the first embodiment of the present disclosure, the information processing device 100-1 causes the first or second object mask to be moved for compositing in response to input based on a user's movement relative to a screen on which an object mask or an image corresponding to the object mask is displayed. Thus, the operational feel of compositing can be improved by moving the facial mask according to the user's intention.

[0143] After completing the movement from one of the first and second facial masks to the other, information processing device 100-1 displays the synthesized facial mask in association with the image corresponding to the other facial mask. Therefore, the user's sense of operation can be improved by specifying the timing for displaying the synthesized facial mask based on the user's movement.

[0144] If the input based on the user's motion is interrupted, the information processing device 100-1 does not perform synthesis. Therefore, the convenience of the user can be improved by interrupting the synthesis according to the user's intention.

[0145] The input based on the user's movement includes an operation of specifying both the start point and the end point or an operation of specifying only the start point. Therefore, the user can operate the face mask sensuously, and the operability can be further improved.

[0146] The first or second face mask to be moved includes face masks listed in the face list displayed on the screen and face masks generated from images listed in the face list. Therefore, it is possible to set a face not shown on the screen as a target for compositing, and to broaden the user's selection by selecting a face mask or face image from the face list to use as a source.

[0147] Information processing device 100-1 selects facial masks or images arranged in the face list based on the attributes of the face represented by the first facial mask or the second facial mask. Therefore, by arranging facial masks or facial images that are suitable as faces for synthesis candidates in the face list, the user's satisfaction with the listed facial masks or facial images can be improved.

[0148] At least one of the first face mask and the second face mask includes a face mask of the user. Therefore, the user's face can be synthesized with another face, or another face can be synthesized with the user's face, thereby simulating a hairstyle, fashion, etc. that suits the user.

[0149] <1-5. Modification Example>

[0150] The first embodiment of the present disclosure has been described so far. The embodiment is not limited to the foregoing example. Hereinafter, first to tenth modified examples of the embodiment will be described.

[0151] (First Modification Example)

[0152] According to a first modified embodiment, information processing apparatus 100-1 may generate a synthesized facial mask based on two mesh information pieces or two texture information pieces of the synthesized target. Specifically, the synthesizing mode includes a third mode in which a synthesized facial mask is generated based on the mesh information of the facial mask serving as source and the mesh information of the facial mask serving as destination, as well as the texture information of the facial mask serving as source and the texture information of the facial mask serving as destination.

[0153] For example, the control unit 108 specifies a mesh for synthesizing the facial mask by synthesizing the mesh used as the source face mask and the mesh used as the destination face mask in the third mode. For example, the mesh for synthesizing the facial mask in the third mode can be obtained by averaging the meshes used for the source face mask and the destination face mask. Texture processing is performed in the same manner as for the aforementioned meshes. Of course, the third mode is controlled based on input information in the same manner as in the first and second modes.

[0154] As described above, according to the first modified example of the embodiment, the synthesis mode includes a third mode in which the synthesized facial mask is generated based on the mesh information of the facial mask used as source and the mesh information of the facial mask used as destination, as well as the texture information of the facial mask used as source and the texture information of the facial mask used as destination. Therefore, the degree of synthesis can be set more finely, thereby meeting the needs of various users.

[0155] (Second Modification Example)

[0156] According to the second modified example of the embodiment, when there are multiple face masks to be referred to as dst, the information processing device 100-1 can generate a composite face mask for all of the multiple face masks. Specifically, when multiple face masks or face images are displayed on the screen, the control unit 108 generates a composite face mask based on the face mask being moved and each of the multiple face masks or face images. In addition, referring to Figure 7 and Figure 8 The processing according to the present modification example is described in detail. Figure 7 is a diagram illustrating a process of generating a plurality of synthesized face masks based on a face mask selected from a face list or the like according to a second modified example of the embodiment, Figure 8 1 is a diagram illustrating a process of generating a plurality of synthesized face masks based on a face mask or the like selected from a face image or the like displayed on a screen according to a second modified example of the embodiment.

[0157] If the face mask or the like is moved to one of a plurality of face masks or a plurality of face images corresponding to the face mask, the control unit 108 performs synthesis. For example, it will be considered that a plurality of synthesized face masks are generated based on the face mask or the like selected from the face list. Figure 7 If the facial image 12B3 arranged in the face list shown in the left figure of FIG is moved to any one of the static facial images 14B1 to 14B4, the control unit 108 performs synthesis of each of the facial masks of the facial images 14B1 to 14B4 and the facial mask of the facial image 12B3 in the second mode. As a result, Figure 7 As shown in the right diagram of , each of the generated synthetic face masks 34B1 to 34B4 is superimposed on each of the facial images 14B1 to 14B4 and displayed.

[0158] For example, it will be considered that a plurality of synthetic face masks are generated based on a face mask or the like selected from a face image or the like displayed on a screen. Figure 8 If the static facial image 14C1 shown in the left figure of FIG is moved to any one of the other facial images 14C2 to 14C4, the control unit 108 performs synthesis of each of the facial masks of the facial images 14C2 to 14C4 with the facial mask of the facial image 14C1 in the second mode. As a result, as shown in FIG. Figure 8 As shown in the right figure of , each of the generated synthetic facial masks 34C2 to 34C4 is superimposed on each of the facial images 14C2 to 14C4 and displayed.

[0159] The control unit 108 can perform synthesis without moving the face mask or the like used as src. For example, if a face mask or the like used as src is selected, the control unit 108 performs synthesis using a plurality of face masks or the like used as dst,

[0160] Of course, a synthetic face mask may be generated for one selected from a plurality of face masks or the like.

[0161] In the case where a plurality of face masks are displayed on the screen, the control unit 108 can switch the target for generating a synthetic face mask based on the user's operation. Specifically, the control unit 108 switches the target for generating a synthetic face mask based on the operation performed on the display object. For example, if Figure 8 If the button object 40 shown is set to be turned on, the control unit 108 sets all the multiple face masks as the target for generating the composite face mask. Figure 8 The button object 40 shown is set to off, and the control unit 108 sets one of a plurality of face masks or the like as a target for generating a synthesized face mask.

[0162] According to the second modified embodiment, when multiple facial masks or facial images are displayed on the screen, information processing device 100-1 generates a composite facial mask based on the moved facial mask and each of the multiple facial masks or facial images. Therefore, the user can save the time and effort required to use the composite facial mask.

[0163] If the face mask or the like is moved to one of a plurality of face masks or a plurality of face images corresponding to the face mask, the control unit 108 performs synthesis. Therefore, regardless of the number of dst, the ease of operation can be improved by integrating an interface for synthesis.

[0164] (Third Modification Example)

[0165] According to the third modified example of the embodiment, the information processing device 100-1 can register a face mask or the like in the face list based on the user's operation. Specifically, the control unit 108 adds the face mask moved toward the face list to the face list. In addition, the face mask is added to the face list with reference to Figure 9 The registration process in the face list is described in detail. Figure 9 : is a diagram illustrating a registration process in a face list according to a third modified example of the embodiment.

[0166] The control unit 108 prepares a registration frame in the face list. Although before registration in the face list, as shown in FIG. Figure 9 In the face list shown in the left diagram of , face masks and the like are not arranged, but objects indicating frames in which registration is possible are arranged.

[0167] Next, if the face mask or the like is moved to the face list, the control unit 108 adds the face mask or the like to the face list. For example, if the user moves the face mask or the like from the face list Figure 9The face mask generated by the through face image 16D shown on the upper left side of the left figure is moved to the face list, as shown in FIG. Figure 9 As shown in the right figure of , the control unit 108 adds the face mask 22D to the face list.

[0168] Next, the control unit 108 performs synthesis using a face mask or the like registered in the face list as the source. For example, the control unit 108 moves the face mask 22D registered in the face list to the facial images 14D1 to 14D3. This generates a synthesized facial mask in the second mode, using the face mask 22D as the source and each facial image at the destination as the destination. Each of the generated synthesized facial masks 34D1 to 34D3 is then superimposed on each of the facial images 14D1 to 14D3 and displayed.

[0169] If the facial image is moved to the face list, control unit 108 may register the face mask generated from the facial image in the face list. For example, if the user moves their face or information processing device 100-1 moves their face appearing in the through image to the face list, control unit 108 generates a face mask from the user's face in the through image and registers the generated face mask in the face list.

[0170] Control unit 108 may register a face mask in the face list based on an operation that is not accompanied by movement of the face mask, etc. For example, if a selection operation, such as a touch operation, on a face mask, etc. continues for a predetermined time, control unit 108 may add the face mask associated with the selection operation to the face list. In such a case, control unit 108 may display a GUI for checking the availability of registration after detecting the continuation of the selection operation. Control unit 108 may add an image selected by the user from images obtained through image capture or images stored in storage unit 110.

[0171] The control unit 108 can automatically generate a face list. For example, the control unit 108 generates a face mask for a face that appears in an image obtained through image capture (including a still image, a moving image, and a through image), and adds the generated face mask to the face list. In addition, the control unit 108 can generate a face mask for the face that was the subject during image capture, and add the generated face mask to the face list. In addition, a previously registered face mask can be updated. For example, if a face image or face mask pointing in a direction closer to the front than an already registered face mask is acquired, the control unit 108 registers the newly acquired face mask in place of the already registered face mask.

[0172] A face mask or the like can be deleted from the face list. Specifically, the control unit 108 deletes a face mask or the like from the face list based on a user operation. For example, if a face mask or the like arranged in the face list is moved outside the face list by dragging and dropping, the control unit 108 deletes the moved face mask or the like from the face list.

[0173] The control unit 108 can automatically delete facial masks and the like from the face list. For example, the control unit 108 can delete facial masks and the like that have been registered in the face list for a predetermined period of time. Furthermore, the control unit 108 can delete facial masks and the like that are outside the imaging range from the face list. If image capture ends or the imaging screen is printed, the control unit 108 can delete facial masks and the like from the face list. Furthermore, the control unit 108 can delete facial masks and the like based on the user's location information. When the user leaves a predetermined area, the control unit 108 deletes the facial masks and the like that correspond to the predetermined area.

[0174] According to the third modification example of the embodiment, as described above, the information processing device 100-1 adds the face mask moved toward the face list to the face list. Therefore, the user can create the face list sensually and the operability can be improved.

[0175] (Fourth Modified Example)

[0176] According to a fourth modified example of the embodiment, the information processing device 100-1 can cause a synthetic face mask to be displayed in association with a face other than a real person's face. Specifically, the facial mask used for synthesis includes a facial mask of an artificially generated face (hereinafter, also referred to as an artificial face). For example, the control unit 108 synthesizes a facial mask generated from a facial image of an artificial face with other facial masks. Then, the control unit 108 causes the synthetic face mask obtained by synthesis to be superimposed and displayed on the facial image of the artificial face. In addition, referring to Figure 10 The processing according to the present modification example is described. Figure 10 : are diagrams illustrating a synthesis process according to a fourth modified example of the embodiment.

[0177] Artificial faces include faces that are illustrated as pictures. For example, if a face mask generated from a through-face image 16E faces the face illustrated as Figure 10 If the facial image 14E of the female face of the picture shown in the left figure of FIG is moved, the control unit 108 synthesizes the facial mask generated from the facial image 14E with the moved facial mask in the second mode. Figure 10As shown in the right figure of FIG, the control unit 108 causes the synthesized face mask 34E obtained by synthesis to be superimposed and displayed on the facial image 14E. Of course, a face mask registered in the face list, such as one of the face masks 22E1 to 22E4, can be used for synthesis instead of the face mask generated from the through-face image 16E.

[0178] The artificial face may be a face created by CG. Although not an artificial object, a natural object such as a vegetable or a rock that is similar in form to a face may be used instead of the artificial object.

[0179] According to the fourth modified example of the embodiment, as described above, the face mask used for synthesis includes a face mask of an artificially generated face. Therefore, it is possible to address various user needs by expanding the target of synthesis.

[0180] (Fifth Modified Example)

[0181] According to a fifth modified example of the embodiment, information processing device 100-1 can present a preview of a second synthesized facial mask related to the synthesized facial mask (hereinafter also referred to as a preview of the synthesized facial mask). Specifically, the preview of the synthesized facial mask includes changing the facial mask being moved to the second synthesized facial mask, and the second synthesized facial mask includes the synthesized facial mask. For example, control unit 108 changes the facial mask selected as the target of movement to a synthesized facial mask obtained by synthesizing the facial mask with the facial mask estimated as the target of movement. For example, if the movement operation is a drag operation, control unit 108 performs a preview by using the facial mask existing in the drag direction as the facial mask used as the destination. Slide operations or flick operations are also performed in the same manner as in the case of a drag operation.

[0182] The second synthesized facial mask includes a synthesized facial mask obtained by changing facial feature information in the synthesized facial mask based on information related to the movement of the facial mask being moved. Specifically, the facial feature information of the preview-related synthesized facial mask is changed based on the distance between the facial mask serving as source and the facial mask serving as destination, where the distance is changed by the movement. For example, when the distance decreases, the control unit 108 changes the facial mask being moved to a synthesized facial mask that is closer to the synthesized facial mask after the movement. More specifically, as the distance decreases, the control unit 108 makes the mesh and texture of the facial mask being moved closer to the mesh and texture of the synthesized facial mask. The movement operation can be a drag operation or a slide operation.

[0183] The facial feature information of the preview-related synthesized facial mask may be changed according to the direction of movement of the facial mask. For example, if the facial mask is moved in a predetermined direction, the control unit 108 changes the moved facial mask into a synthesized facial mask that emphasizes the mesh and texture of the facial mask for source. Furthermore, if the facial mask is moved in a direction opposite to the predetermined direction, the control unit 108 changes the moved facial mask into a synthesized facial mask that emphasizes the mesh and texture of the facial mask for destination.

[0184] Furthermore, the facial feature information of the synthesized facial mask associated with the preview may be changed according to the movement speed of the facial mask. For example, as the movement speed of the facial mask increases, the control unit 108 may adjust the mesh and texture of the moved facial mask to be closer to the mesh and texture of the synthesized facial mask. If the movement speed is equal to or greater than a threshold, the control unit 108 may change the moved facial mask to the synthesized facial mask.

[0185] The preview of the synthesized face mask can change the face mask arranged in the face list or the face mask generated from the arranged face image to the second synthesized face mask. Specifically, the control unit 108 changes the face mask arranged in the face list, the face mask selected from the face image, or the face mask generated from the selected face image to the second synthesized face mask. In addition, the reference Figure 11 Describes the preview in the face list in detail. Figure 11 : is a diagram showing a preview in a face list according to a fifth modified example of the embodiment.

[0186] The control unit 108 combines one or each of a plurality of face masks selected from the face list by the user's operation with the face mask of the face on the screen that has been recognized by the face recognition unit 104. For example, the control unit 108 combines the face masks arranged in the face list with the face masks generated from the facial images 14F1 to 14F3 corresponding to the face masks, respectively, in the first mode.

[0187] Then, the control unit 108 causes each of the generated synthesized face masks to be displayed in place of the selected corresponding face mask in the face list. Figure 11 As shown, the control unit 108 causes the synthesized face masks 32F1 to 32F3 to be displayed instead of the face masks in the face list. When a face image in the face list is selected, the synthesized face mask can be superimposed on the face image and displayed.

[0188] The control unit 108 can use all face masks arranged in the face list or the like as targets for preview. The target for preview can be switched based on a user's operation.

[0189] The synthesized face mask in the preview can be updated based on the mesh information of src or dst. For example, the facial representation of the synthesized face mask in the preview can be changed according to the movement of the face of src or dst.

[0190] Although an example has been described so far in which a face mask or the like used as dst is changed into a synthesized face mask, a face mask or the like used as src may be changed into a synthesized face mask.

[0191] Although an example of performing preview by changing a face mask or the like that is a target of synthesis to a synthesized face mask has been described so far, an additional screen for preview may be provided, and a preview of the synthesized face mask may be displayed on the screen.

[0192] According to the fifth modified example of the embodiment, the information processing device 100-1 presents a preview of the second synthesized face mask related to the synthesized face mask. Therefore, since the situation after synthesis is presented in advance, the user can determine whether to perform synthesis before synthesis, and the convenience of the user can be improved.

[0193] The preview of the second synthesized face mask includes changing the face mask being moved to the second synthesized face mask. Therefore, since the preview of the synthesized face mask is performed during the movement operation for synthesis, the user can determine whether to perform synthesis during the movement.

[0194] The second synthetic face mask includes a synthetic face mask whose facial feature information changes according to the distance between the face mask used as source and the face mask used as destination. Therefore, the user can check the synthetic process by moving the mask, which can provide fun for the user.

[0195] Previewing the second synthesized facial mask includes changing the facial mask arranged in the face list or the facial mask generated from the facial image arranged in the face list to the second synthesized facial mask. Therefore, since the facial mask arranged in the face list becomes the target of the preview, the user can observe the preview without performing a movement operation.

[0196] Information processing device 100-1 changes one or all of the facial masks selected from the facial masks or facial images arranged in the face list to one or more second synthesized facial masks. Therefore, since only the facial mask intended by the user is considered as a preview target, the user can focus on and observe the synthesized facial mask that the user wants to check.

[0197] (Sixth Modification Example)

[0198] According to the sixth modified example of the embodiment, the information processing device 100-1 can fix the face mask of the face image on which the synthetic face mask is superimposed and displayed when the synthetic face mask is superimposed and displayed. Specifically, when the synthetic face mask is displayed, the control unit 108 causes the face mask corresponding to the image associated with the synthetic face mask to be displayed adjacent to the synthetic face mask. In addition, referring to Figure 12 and Figure 13 The processing according to the present modification example is described in detail. Figure 12 is a diagram illustrating a synthesis process in a case where a face mask is not fixed according to a sixth modified example of the embodiment, Figure 13 : are diagrams illustrating synthesis processing in a case where a face mask is fixed according to a sixth modified example of the embodiment.

[0199] First, a case where the face mask is not fixed will be described. If the face mask used as src is moved to the face mask used as dst, etc., the control unit 108 generates a composite face mask. Figure 12 As shown in the left figure of , the facial mask 26G generated from the through facial image 16G is moved to the moving facial image 18G by a drag operation, and the control unit 108 synthesizes the facial mask 26G with the facial mask generated from the facial image 18G in the second mode.

[0200] Next, control unit 108 superimposes and displays the generated synthetic facial mask on the facial image serving as the destination. For example, control unit 108 superimposes and displays the generated synthetic facial mask 38G on facial image 18G serving as the destination. Since synthetic facial mask 38G is superimposed and displayed on facial image 18G, it is difficult for the user to perform an operation to generate a facial mask from facial image 18G, even if the user wishes to generate a facial mask for facial image 18G. Therefore, as will be described later, the facial mask is fixed in information processing device 100-1.

[0201] If the face mask used as src is moved to the face mask used as dst, etc., the control unit 108 generates a synthesized face mask. Figure 13 As shown in the left figure of , the facial mask generated from the static facial image 14H is moved to the through facial image 16H by a drag operation, and the control unit 108 synthesizes the moved facial mask with the facial mask generated from the facial image 16H in the second mode.

[0202] Next, the control unit 108 causes the generated synthetic face mask to be superimposed and displayed on the facial image serving as dst, and causes the facial mask generated from the facial image serving as dst to be displayed at a position adjacent to the synthetic face mask. Figure 13As shown in the central figure of , for example, the control unit 108 causes the generated synthetic facial mask 36H to be superimposed and displayed on the facial image 16H used as dst, and causes the facial mask 26H generated from the facial image 16H to be displayed on the synthetic facial mask 36H, i.e., the facial image 16H.

[0203] The fixed face mask can be moved and synthesized in the same way as the face mask described above. Figure 13 As shown in the middle figure of , the generated face mask 26H to be fixed is moved to the face image 14H, and the control unit 108 synthesizes the face mask 26H with the face mask generated from the face image 14H in the second mode.

[0204] Next, control unit 108 superimposes and displays the generated synthesized facial mask on the facial image used as dst, and displays the facial mask generated from the facial image used as dst adjacent to the synthesized facial mask. For example, control unit 108 superimposes and displays generated synthesized facial mask 34H on facial image 14H, and displays facial mask 24H generated from facial image 14H on synthesized facial mask 34H, i.e., facial image 14H. As a result, the display is performed so that the face associated with the facial image has been switched.

[0205] It is possible to switch whether to fix the aforementioned face mask. Specifically, the control unit 108 switches whether to fix the face mask based on the user's operation. For example, the control unit 108 switches based on the following Figure 13 Whether or not to fix the face mask is switched by switching the button object 50 shown in the left figure between on and off.

[0206] According to the sixth modified embodiment, as described above, when a composite facial mask is displayed, information processing apparatus 100-1 displays a facial mask corresponding to an image associated with the composite facial mask adjacent to the composite facial mask. Therefore, operations can be performed on the facial mask on which the facial image of the composite facial mask is superimposed and displayed, thereby improving user convenience.

[0207] (Seventh Modification Example)

[0208] According to the seventh modified example of the embodiment, the information processing device 100-1 can change the facial feature information of the synthesized face mask. Specifically, the control unit 108 changes the facial feature information of the synthesized face mask based on the input information. More specifically, the control unit 108 changes the facial feature information of the synthesized face mask between the facial feature information of the face mask used as src and the facial feature information of the face model used as dst based on the user's input operation. In addition, referring to Figure 14 The processing according to the present modification example is described in detail. Figure 14: is a diagram illustrating an example of an interface for adjusting facial feature information of a synthetic face mask according to a seventh modified example of the embodiment.

[0209] The control unit 108 provides the user with an interface for adjusting the facial feature information of the synthesized face mask. For example, the control unit 108 enables the user to display the following information: Figure 14 GUI 60 is shown. GUI 60 includes an interface, such as a slider, for adjusting mesh information of the synthesized face mask and an interface, such as a slider, for adjusting texture information of the synthesized face mask. For example, the left end of the slider corresponds to src and the right end of the slider corresponds to dst.

[0210] When the finger grip of the slider for the grid information is set to the left end and the finger grip of the slider for the texture information is set to the right end, the facial feature information of the composite facial mask corresponds to the facial feature information in the first mode described above. When the finger grips of the two sliders are set in opposite directions, the facial feature information of the composite facial mask corresponds to the facial feature information in the second mode described above. When the finger grips of the two sliders are set between the left and right ends, the facial feature information of the composite facial mask corresponds to the facial feature information in the third mode described above.

[0211] Then, the control unit 108 causes the display of the synthetic face mask to be changed based on the facial feature information set through the interface. For example, the control unit 108 changes the display of the synthetic face mask to Figure 14 The mesh information and texture information of the synthesized facial mask shown are changed to the mesh information and texture information set by the sliders in the GUI 60. The control unit 108 may cause the synthesized facial mask to change in real time in response to an operation on the GUI 60, or may cause the synthesized facial mask to change after the operation on the GUI 60 is completed, for example, after the operation is ended.

[0212] Regardless of the facial feature information of the facial masks used as src and dst, the facial feature information of the synthesized facial mask can be changed. Specifically, the control unit 108 changes the mesh information of the synthesized facial mask based on the attribute information of the facial mask used as src or the facial mask used as dst. Figures 15A to 15C Describe the configuration in detail. Figures 15A to 15C : is a diagram illustrating a process of changing mesh information of a synthesized face mask according to a seventh modified example of the embodiment.

[0213] The control unit 108 selects a mesh type based on the attribute information of the mask and changes the mesh information based on the selected mesh type. For example, the type of facial wear such as a facial mask or a medical mask, a cartoon or CG character, or the presence of bangs is exemplified as the attribute information of the facial mask. For example, in the case where the type of the facial mask is a facial wear, the control unit 108 selects a mesh type on a part of the face such as Figure 15A or Figure 15B In the case where the type of the face mask is a person, the control unit 108 selects a mesh type that is set to be larger than other mesh types so that the face mask is not as large as the other mesh types. Figure 15C A mesh type showing the deformation of a part of a face, such as an eye.

[0214] Instead of or in addition to selecting a grid type, the control unit 108 may dynamically change the grid information. Figure 16 Describe the configuration in detail. Figure 16 : is a diagram illustrating another example of a process of changing mesh information of a synthesized face mask according to a seventh modified example of the embodiment.

[0215] The control unit 108 dynamically changes the grid information in response to the user's operation. Specifically, the control unit 108 changes a portion of the grid information to have a size selected by the user's operation. For example, Figure 16 In the synthetic face masks 36I1 to 36I3 shown, the control unit 108 changes the size of a portion corresponding to the eyes in the grid to 1.0 to 1.5 times the size set by the user's operation.

[0216] The control unit 108 may change the texture information of the synthesized facial mask based on the facial image associated with the synthesized facial mask or the facial mask of the synthesized target. Specifically, the control unit 108 changes the texture information of the synthesized facial mask based on a comparison between the texture information of the facial mask used as src and the texture information of the facial mask used as dst, etc. Figure 17 and Figure 18 Describe the configuration in detail. Figure 17 and Figure 18 Each is a diagram illustrating a process of changing texture information of a synthesized face mask according to a seventh modified example of the embodiment.

[0217] First, the case where the texture information is not changed will be described. If a synthetic face mask is generated, the control unit 108 causes the generated synthetic face mask to be superimposed and displayed on the face image used as dst. For example, the control unit 108 causes the generated synthetic face mask to be superimposed and displayed on the face image used as dst. Figure 17The synthesized facial mask 36J1 shown in the left figure is displayed. Here, because the hue of the texture of the synthesized facial mask 36J1 differs from the hue of the superimposed and displayed facial image, there is a possibility that the user will perceive the synthesis as unnatural. Therefore, the control unit 108 changes the transparency of the texture of the synthesized facial mask according to the degree of visual difference between the texture of the facial mask used as source and the texture of the facial mask used as destination.

[0218] Specifically, the control unit 108 changes the transparency of the texture of the synthesized facial mask according to the degree of difference in brightness or color attributes (color phase, brightness or saturation) between the texture of the facial mask used as src and the texture of the facial mask used as dst. Figure 17 In the synthetic face mask 36J2 shown in the right figure of FIG, the control unit 108 increases the transparency of the texture around the outline of the synthetic face mask. In this way, the feeling of strangeness can be suppressed.

[0219] Instead of or in addition to the change in transparency of the texture, the control unit 108 may change the color attribute of the texture. Specifically, the control unit 108 changes the color attribute of the texture of the synthesized facial mask according to the degree of difference in brightness or color attribute between the texture of the facial mask used as src and the texture of the facial mask used as dst. For example, the control unit 108 changes the color attribute of the texture of the synthesized facial mask from src to dst by reducing the brightness of the texture of the synthesized facial mask. Figure 18 The synthetic face mask 36K1 shown in the right figure is changed to Figure 18 The synthetic face mask 36K2 shown in the right figure of FIG. In this way, the feeling of strangeness can be suppressed.

[0220] The aforementioned process of changing the facial feature information of the synthesized face mask may be performed after synthesis or during synthesis.

[0221] According to the seventh modification example of the embodiment, as described above, the information processing device 100-1 changes the facial feature information of the composite face mask based on the input information. Therefore, since the composite face mask suitable for the input information is displayed, the user's visibility of the composite face mask can be improved.

[0222] Information processing device 100-1 changes the facial feature information of the synthesized facial mask between the facial feature information of the facial mask serving as source and the facial feature information of the facial model serving as destination. This makes it easier to fine-tune the facial feature information of the synthesized facial mask, thereby improving user convenience. Specifically, if the facial feature information of the synthesized facial mask changes in real time in response to user input after synthesis, the user can make adjustments while checking the changes to the synthesized facial mask, further improving convenience.

[0223] Information processing device 100-1 changes the mesh information of the synthesized facial mask based on the attribute information of the facial mask used as source or destination. Therefore, the synthesized facial mask is changed regardless of whether the facial mask corresponds to source or destination, thus expanding user options and addressing various user needs. Furthermore, since the mesh information of the synthesized facial mask is changed, the facial representation represented by the synthesized facial mask can be changed and adapted to the intended synthesis purpose.

[0224] The information processing device 100-1 changes the texture information of the synthesized facial mask based on the facial image associated with the synthesized facial mask or the facial mask that is the target of synthesis. Therefore, since the texture information of the synthesized facial mask is changed, the color of the face represented by the synthesized facial mask can be changed, and the synthesized facial mask can be made to conform to the purpose of synthesis.

[0225] (Eighth Modification Example)

[0226] According to the eighth modified embodiment, information processing device 100-1 can use facial feature information from another facial mask when facial feature information for generating a composite facial mask is insufficient. This can occur when a person appearing in a moving or direct image shakes their head or changes camera motion, reducing the number of feature points detected in the facial image corresponding to the facial mask. In such cases, the amount of mesh and texture information in the source or destination facial mask decreases, potentially creating a composite facial mask that appears unfamiliar to the user. Therefore, control unit 108 uses facial feature information from a previous facial mask to supplement the insufficient facial feature information.

[0227] Specifically, if a face mask is generated or updated, the control unit 108 causes the facial feature information to be stored. For example, the control unit 108 temporarily causes the storage unit 110 to store the facial feature information of the face mask until a predetermined time has passed after the face mask is generated or updated.

[0228] Then, the control unit 108 acquires the facial feature information stored in the past and supplements the insufficient facial feature information if the facial feature information is insufficient. For example, if the number of detected feature points is equal to or less than a threshold, the control unit 108 supplements the insufficient feature points with the feature points stored in the past.

[0229] The other facial mask having facial feature information for supplementation may be the facial mask of src or dst as the synthesis object. For example, when the number of feature points detected in facial mask src is equal to or less than a threshold value, the control unit 108 supplements the feature points of src by using feature points of facial mask dst corresponding to feature points not detected in facial mask src.

[0230] The transparency of the synthesized facial mask can be changed based on the number of detected feature points. For example, when the number of detected feature points in the facial mask used as source or destination is small, control unit 108 sets a higher transparency for the texture of the synthesized facial mask. Here, as the number of detected feature points decreases, the mesh information and texture information decrease, and the appearance of the synthesized facial mask becomes rougher. However, since this configuration increases the transparency of the synthesized facial mask as the number of detected feature points decreases, the user may not be able to clearly visually recognize the synthesized facial mask, and the roughness of the synthesized facial mask may be blurred.

[0231] If no feature points are detected, the synthesized face mask may be deleted. For example, if no feature points of the face mask used as src or dst are detected, the control unit 108 fades out the synthesized face mask by gradually increasing the transparency of the texture of the synthesized face mask.

[0232] The face mask src or dst in which the feature points have been detected may be superimposed and displayed instead of the synthesized face mask, rather than deleting the synthesized face mask. In this case, the synthesizing process is not performed.

[0233] According to the eighth modified example of the embodiment, when the facial feature information for generating the composite facial mask is insufficient, information processing device 100-1 uses the facial feature information of another facial mask. Therefore, since the composite facial mask continues to be displayed even when some of the feature points are not detected, frequent switching between display and non-display of the composite facial mask can be suppressed, and visibility to the user can be maintained.

[0234] The other facial masks include past facial masks, or facial masks used as source or destination for synthesis. Therefore, since the synthetic facial mask is supplemented with feature points compatible with the synthetic facial mask, the likelihood of the synthetic facial mask appearing unfamiliar to the user is reduced.

[0235] (Ninth Modification Example)

[0236] According to the ninth modified example of the embodiment, the information processing device 100-1 can perform synthesis of objects other than faces. Specifically, the control unit 108 performs synthesis in which at least one of the models of src and dst is a model of an object other than a face. For example, a facial model is pre-connected to an image of an object other than a face. Then, the control unit 108 synthesizes the facial model of the image connected to the object with the facial model of the facial image, and causes the synthesized facial model obtained by the synthesis to be superimposed and displayed on the image of the object or the facial image. In addition, reference will be made to Figure 19 and Figure 20 The processing according to the present modification example is described in detail. Figure 19 is a diagram illustrating a process of synthesizing an object other than a face according to a ninth modified example of the embodiment, Figure 20 is a diagram illustrating another example of a process of synthesizing an object other than a face according to a ninth modification example of the embodiment.

[0237] Objects other than faces include illustrations. For example, a face model is pre-linked to Figure 19 The Kabuki illustration image 12L is arranged in the face list shown. In the case where the subject is an object close to the face, a face model is generated by face recognition processing. If the illustration image 12L or the face model connected to the image 12L is moved to another face model or face image, the control unit 108 performs synthesis of the face model connected to the image 12L with the face model or the face model generated from the face image at the movement destination. Then, the control unit 108 makes the face model 12L and the face model connected to the image 12L be combined. Figure 19 The synthesized face model 36L obtained by synthesis is shown superimposed and displayed on the face image at the movement destination.

[0238] Objects other than faces include images for superimposition. For example, Figure 20 The makeup image 12M arranged in the face list shown is previously connected to the face model. If the makeup image 12M or the face model connected to the makeup image 12M is moved to another face model or face image, the control unit 108 performs synthesis of the face model connected to the makeup image 12M with the face model or the face model generated from the face image at the move destination. Then, the control unit 108 makes the following Figure 20 The synthesized face model 36M obtained by synthesis is shown superimposed and displayed on the face image at the movement destination.

[0239] Although an example has been described so far in which an object other than a face is an illustration image or an image for superimposition, the object may be a natural object such as vegetables or an artificial object such as a clock.

[0240] Although the example in which the face model is pre-connected to the object image has been described so far, the information processing device 100-1 may synthesize the object model generated from the object image. For example, the control unit 108 may synthesize the tomato model generated from the tomato image with other tomato models.

[0241] Furthermore, control unit 108 can combine models of different objects. For example, control unit 108 can combine a face model generated from a face image with a tomato model generated from a tomato image. In this case, since the method of picking feature points and the number of feature points differ between the face model and the tomato model, the mesh information and texture information are different. Therefore, the feature points, mesh information, and texture information between the models are pre-associated.

[0242] According to the ninth modified example of the embodiment, as described above, information processing device 100-1 performs synthesis in which at least one of the models of src and dst is a model of an object other than a face. Therefore, since the application purpose of the synthesis processing is expanded to objects other than faces, it is possible to meet the needs of various users.

[0243] (Tenth Modification Example)

[0244] According to the tenth modified example of the embodiment, a composite facial mask can be superimposed and displayed regardless of the user's operation. Specifically, if a predetermined time has passed after facial recognition, the control unit 108 performs synthesis of a facial mask generated from a facial image related to facial recognition with a predetermined facial mask, and causes the synthesized facial mask obtained by synthesis to be superimposed on the facial image and displayed.

[0245] The facial mask to be synthesized can be moved toward the facial image associated with the recognized face. For example, if the user's face is displayed on the screen and a predetermined time has passed since the user's face was recognized, the facial mask to be used for synthesis can be moved toward the recognized user's face, i.e., the facial image on the screen, as if it were flying toward the user's face. If the moved facial mask then reaches the user's face, the synthesized facial mask is superimposed and displayed on the user's face. If the moved facial mask does not reach the user's face, synthesis is not performed.

[0246] The face mask to be synthesized can be moved as if it were carried by the display object displayed on the screen. For example, if a person holds a face mask and moves, and the person moves to the user's face, the synthesized face mask is superimposed and displayed on the user's face.

[0247] When the number of facial masks used as DST is a predetermined number, the facial masks corresponding to the predetermined number and the facial masks used as DST can be synthesized. For example, when there are seven faces on the screen, the facial masks of the facial mask group corresponding to seven, such as the facial masks of the faces of seven comic heroes, and the facial masks of the seven faces on the screen are associated with each other and synthesized, and the seven synthesized facial masks obtained by the synthesis are superimposed and displayed on the seven faces on the screen.

[0248] According to the tenth modified embodiment, as described above, the composite face mask is superimposed and displayed regardless of the user's operation. Therefore, the user does not need to perform any operation, saving the user's time and effort. Furthermore, user convenience is improved in situations where the user has difficulty performing the operation.

[0249] <2. Second Embodiment (Application Example to Digital Signage Device)>

[0250] The information processing device 100-1 according to the first embodiment of the present disclosure has been described so far. Next, the information processing device 100-2 according to the second embodiment of the present disclosure will be described. The information processing device 100-2 according to the present embodiment is a static device such as a digital signage.

[0251] <2-1. Device Configuration>

[0252] Since the functional configuration of the information processing device 100 - 2 according to the second embodiment of the present disclosure is substantially the same as that of the information processing device 100 - 1 according to the first embodiment, description will be omitted.

[0253] <2-2. Technical Features of the Device>

[0254] Next, the technical features of the information processing apparatus 100 - 2 according to the embodiment will be described. Description of substantially the same features as those in the first embodiment will be omitted.

[0255] (G. Designation of Synthesis Target)

[0256] The information processing apparatus 100 - 1 specifies a target for synthesis according to a gesture operation of the user as an input based on the motion of the user.

[0257] (G-1. Designation of Target for Synthesis Based on Movement of Facial Mask)

[0258] The control unit 108 moves the face mask of the synthesized target in response to the user's gesture operation. Figure 21 A process for moving a facial mask in response to a gesture operation is described. Figure 21: is a diagram illustrating an operation of moving a face mask of a synthesized target in the information processing device 100 - 2 according to the embodiment.

[0259] First, the control unit 108 causes a face mask or the like that is a target of synthesis to be displayed. For example, Figure 21 As shown in the left diagram, the control unit 108 causes a face list in which face masks serving as src and the like are arranged to be displayed.

[0260] Then, the control unit 108 causes the face mask etc. to be moved in response to the user's gesture of selecting the face mask etc. from the face list. For example, if Figure 21 As shown in the middle figure of , the user takes an action to pick up the facial image 12N from the face list, and the control unit 108 causes the facial image 12N to be moved in response to the user's action.

[0261] Next, if a face mask or the like is moved to the user's face, the control unit 108 causes the composite face mask to be superimposed and displayed on the user's face. For example, if the face image 12N is moved to the user's face, Figure 21 As shown in the right diagram of , the control unit 108 causes a synthesized face mask 36N obtained by synthesizing the face mask of the face image 12N and the face mask of the user's face to be superimposed and displayed on the user's face.

[0262] (G-2. Designation of Target Based on Synthesis of User's Movement)

[0263] In addition, the control unit 108 can specify the target of the synthesis based on the movement of the user. Figure 22 The processing of this configuration is described in detail. Figure 22 1 is a diagram illustrating a process of designating a target for synthesis based on movement of a user in the information processing apparatus 100 - 2 according to the embodiment. Description of substantially the same functions as those described above will be omitted.

[0264] First, as described above, the control unit 108 causes the face mask that is the target of synthesis to be displayed.

[0265] If the user's face and the face mask overlap due to the user's movement, the control unit 108 causes the synthesized face mask to be superimposed and displayed on the user's face. Figure 22 As shown in the middle figure of FIG, the user's face overlaps with the facial image 12O arranged in the face list, then Figure 22 As shown in the right figure of , the control unit 108 causes a synthesized face mask 36O obtained by synthesizing the face mask of the face image 12O and the face mask of the user's face to be superimposed and displayed on the user's face.

[0266] <2-3. Processing by Device>

[0267] Since the processing performed by the information processing apparatus 100 - 2 according to the second embodiment of the present disclosure is substantially the same as the processing performed by the information processing apparatus 100 - 1 according to the first embodiment, a description thereof will be omitted.

[0268] <2-4. Summary of Second Embodiment>

[0269] According to the second embodiment of the present disclosure, the input based on the user's movement includes a gesture operation, and as described above, information processing device 100-2 moves the face mask used for synthesis based on the gesture operation. Therefore, the user can perform the synthesis operation sensorily, and the operability of the synthesis-related operation can be improved.

[0270] <2-5. Modification Example>

[0271] The second embodiment of the present disclosure has been described so far. The embodiment is not limited to the aforementioned example. Hereinafter, a first modified example and a second modified example of the embodiment will be described.

[0272] (First Modification Example)

[0273] According to the first modified example of the embodiment, the synthesized face mask can be superimposed and displayed regardless of the user's operation. Specifically, the control unit 108 causes the face mask of the synthesized target to be moved as if the face mask is carried by the display object displayed on the screen. In addition, referring to Figure 23 The processing according to the present modification example is described in detail. Figure 23 : is a diagram illustrating an example of moving a face mask or the like as a target of synthesis by a person according to a first modified example of the embodiment.

[0274] The control unit 108 moves the display object accompanied by the face mask toward the face of the user appearing on the screen. Figure 23 As shown in the left figure of , a person 70 holding a face mask 22P moves toward the face of the user on the screen.

[0275] Next, if the display object accompanied by the face mask arrives near the face of the user appearing on the screen, the control unit 108 moves the face mask from the display object to the face of the user. Figure 23 As shown in the middle figure of , if the character 70 moves to the vicinity of the user's face, the character 70 throws the face mask 22P held by the character 70 toward the user's face.

[0276] Next, if the face mask reaches the face of the user appearing on the screen, the control unit 108 causes the synthetic face mask to be superimposed and displayed on the face of the user. Figure 23 As shown in the right figure of , the facial mask 22P reaches the user's face, and a synthesized facial mask 36P obtained by synthesizing the facial mask 22P and the facial mask of the user's face is superimposed on the user's face and displayed.

[0277] According to the first modified embodiment, information processing device 100-2 causes the facial mask of the target of synthesis to be moved as if the facial mask were carried by the displayed object on the screen. This allows the user to appreciate the process before performing synthesis, making the process of synthesizing facial masks more attractive to the user.

[0278] (Second Modification Example)

[0279] According to a second modified embodiment, information processing device 100-2 can perform synthesis based on a link between synthesis targets. Specifically, control unit 108 links synthesis targets based on a user's selection operation and performs synthesis based on the linked synthesis targets. For example, if a user selects a synthesis target such as a facial mask for a destination, such as an image of their own face, control unit 108 generates a facial mask for the user's face and links the selected facial mask to the user's facial mask. Control unit 108 then performs synthesis based on the facial mask linked to the user's facial mask, and superimposes and displays the generated synthesized facial mask on the image in which the user's face appears.

[0280] One or two targets for synthesis can be selected on the face list. Furthermore, multiple links may be provided, and in such a case, the user can select which link to use. For example, the user selects a link using a mobile terminal such as a smartphone or tablet terminal, and information about the selected link (hereinafter also referred to as link information) is provided to information processing device 100-2, such as a digital signage, and information processing device 100-2 displays the synthesized face mask based on the provided link information.

[0281] The link information of the synthesized target can be shared with another device. Specifically, the control unit 108 provides the link information to another information processing device through communication. Figure 24 Describes in detail the sharing of link information. Figure 24 : is a diagram illustrating generation and sharing of link information according to a second modified example of the embodiment.

[0282] The control unit 108 generates link information of the synthesized target based on the user's selection and shares the generated link information. Figure 24As shown in the left figure, the facial appearance position of the comic foreground 200 and the facial mask of the person at the facial appearance position are previously associated with each other. The user selects the face to which the user wishes to link, fits his or her face into the facial appearance hole of the face to which the user wishes to link, and then images the comic foreground 200 with the face fitting into the comic foreground 200. If an image obtained through image capture is provided to information processing device 100-2 via communication, control unit 108 generates a facial mask that fits the user's face in the image. Control unit 108 then generates link information between the generated facial mask and the facial mask associated with the facial appearance position of the comic foreground 200. Control unit 108 then provides the link information to another information processing device 100-2 via communication.

[0283] If the link information is provided, the control unit 108 stores the link information. For example, if the link information is received, the control unit 108 causes the storage unit 108 to store the link information.

[0284] Next, if a face corresponding to the facial mask matching the linked information is recognized in the image captured, control unit 108 generates a facial mask for the recognized face based on the facial mask and the linked facial mask, and superimposes and displays the synthesized facial mask. For example, if a user associated with the stored linked information enters the imaging range and the user's face is recognized, control unit 108 generates a facial mask for the user's face. Control unit 108 then superimposes and displays synthesized facial masks 34Q1 and 34Q2, respectively, obtained by synthesizing the generated facial mask with the linked facial mask, on the facial image of the person corresponding to the linked facial mask.

[0285] Although an example using the physical comic foreground 200 has been described so far, a virtual foreground to be displayed on a digital signage may be used.

[0286] The information processing apparatus that images the comic foreground may be the information processing apparatus 100-1 according to the first embodiment. In this case, the information processing apparatus 100-1 that performs image capturing generates link information, and the information processing apparatus 100-1 provides the link information.

[0287] According to the second modified example of the embodiment, information processing apparatus 100-2 links the synthesized targets based on the user's selection operation and performs synthesis based on the linked synthesized targets. Therefore, since the synthesized targets are managed by linking, it becomes easy to store combinations of synthesized targets and re-synthesize the synthesized targets.

[0288] The link information is shared with other devices. Therefore, since the synthesized face mask is also displayed on other devices according to the link, the time and effort required to set the synthesis target for each device can be saved, and the convenience of the user can be improved.

[0289] <3. Application Examples>

[0290] The configurations described in the aforementioned embodiments can be applied to various cases. Hereinafter, some cases other than the aforementioned embodiments will be described.

[0291] (Privacy Protection)

[0292] Privacy can be protected by superimposing and displaying a synthetic facial mask on the face of a person appearing in an image. For example, the information processing device 100 uses the facial mask of the person appearing in the image as the destination and a separately selected facial mask as the source. The synthetic facial mask is generated based on the mesh information of the destination and the texture information of the source. The information processing device 100 then superimposes and displays the generated facial mask on the face of the person appearing in the destination image.

[0293] The face mask used as src is superimposed and displayed on the face image corresponding to the face mask used as dst. In this case, the face appearing in the image looks like a face covered with the face mask.

[0294] (Karaoke movie)

[0295] By superimposing and displaying a synthesized facial mask on the face of a person appearing in a movie being played during karaoke, the perception of the person singing or listening to karaoke can be enhanced. For example, the information processing device 100 uses a facial mask of a person appearing in a karaoke movie as the destination and a facial mask of the person singing as the source. The synthesized facial mask is generated based on the mesh information of the destination and the texture information of the source. The information processing device 100 then superimposes and displays the generated synthesized facial mask on the face of the person appearing in the karaoke movie. The synthesized facial mask can also be generated in a reverse manner based on the texture information of the source and the texture information of the destination.

[0296] It is possible to perform synthesis between the faces of people appearing in a karaoke movie. For example, the information processing device 100 superimposes a synthesized face mask obtained by synthesizing the face mask of person X's face with the face mask of another person Y appearing in the karaoke movie, and displays the resultant face mask on the face of person X appearing in the karaoke movie.

[0297] (Video Chat Movie)

[0298] By superimposing and displaying a synthetic facial mask on the face of a person appearing in a video chat movie, communication via video chat can be smoothed. For example, information processing device 100 superimposes and displays a synthetic facial mask, obtained by synthesizing the eye portion of a facial mask of a person appearing in a video chat movie with the eyes of the person viewing the camera used for the video chat, on that person. In this way, the person appearing in the video chat movie is displayed as if their gaze is directed toward the camera.

[0299] (Embedding facial feature information into the medium)

[0300] By pre-associating facial feature information with a medium, a synthesized facial mask can be superimposed and displayed on a face appearing in the image, even when it is difficult to directly generate a facial mask from an image associated with the medium. For example, the medium may be a moving image, and facial feature information or feature point information associated with each frame of the moving image is stored as meta-information in the moving image information. When reproducing the moving image, the information processing device 100 performs synthesis using the facial feature information, etc., stored as the moving image information of the moving image. The facial feature information, etc., may be associated with a portion of the moving image frame. The information stored as the moving image information may include feature point information and color information associated with the feature points.

[0301] The medium may be paper, such as a magazine, book, or newspaper. For example, readable facial feature information may be pre-written near a person posted on the magazine, such as a two-dimensional AR marker. If the user images the two-dimensional AR marker, the information processing device 100 generates a facial mask based on the facial feature information read from the two-dimensional AR marker and performs synthesis using the generated facial mask.

[0302] <4. Hardware Configuration of Information Processing Device According to One Embodiment of the Present Disclosure>

[0303] Figure 25 1 is an explanatory diagram showing the hardware configuration of the information processing apparatus 100 according to an embodiment of the present disclosure. Figure 25 As shown, the information processing device 100 includes: a central processing unit (CPU) 142, a read-only memory (ROM) 144, a random access memory (RAM) 146, a bridge 148, a bus 150, an interface 152, an input device 154, an output device 156, a storage device 158, a drive 160, a connection port 162 and a communication device 164.

[0304] The CPU 142 functions as a computational processing device and cooperates with various types of programs to implement the operations of the facial recognition unit 104, the operation detection unit 106, and the control unit 108 in the information processing device 100. Furthermore, the CPU 142 may be a microprocessor. The ROM 144 stores programs, computational parameters, and the like used by the CPU 142. For example, the RAM 146 temporarily stores programs used during the execution of the CPU 142 or parameters that change appropriately during execution. The ROM 144 and the RAM 146 constitute part of the storage unit 110 in the information processing device 100. The CPU 142, the ROM 144, and the RAM 146 are connected to one another via an internal bus configured with a CPU bus or the like.

[0305] For example, as an example of the imaging unit 102 in the information processing apparatus 100, the input device 154 is configured with input means for a user to input information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, a lever, and an input control circuit that generates an input signal based on the user's input and outputs the input signal to the CPU 142. The user of the information processing apparatus 100 can input various types of data to the information processing apparatus 100 and instruct processing operations by operating the input device 154.

[0306] For example, the output device 156 performs output to devices such as a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, and a lamp as an example of the display unit 112 in the information processing apparatus 100. In addition, the output device 156 can perform sound output to a speaker, a headphone, or the like.

[0307] Storage device 158 is a device for storing data. Storage device 158 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. Storage device 158 stores programs executed by CPU 142 and various types of data.

[0308] The drive 160 is a reader / writer for storage media, and is provided inside the information processing apparatus 100 or externally attached to the information processing apparatus 100. The drive 160 reads out information recorded in a removable storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory placed in the drive 160, and outputs the information to the RAM 146. In addition, the drive 160 can write information to the removable storage medium.

[0309] The connection port 162 is, for example, a bus for connecting to the information processing apparatus or a peripheral device outside the information processing apparatus 100. In addition, the connection port 162 may be a universal serial bus (USB).

[0310] The communication device 164 is, for example, a communication interface configured with a communication device for connecting to a network. Furthermore, the communication device 164 may be an infrared communication-compatible device, a wireless local area network (LAN)-compatible communication device, a long-term evolution (LTE)-compatible communication device, or a wired communication device that performs wired communication.

[0311] 5. Conclusion

[0312] According to the first embodiment of the present disclosure, as described above, it is possible to perform synthesis desired by the user by performing synthesis according to input information, and to improve the operational feeling of synthesis by moving the face mask according to the user's intention.

[0313] According to the second embodiment of the present disclosure, the user can perform operations for synthesis sensorily, and the operability of operations related to synthesis can be improved.

[0314] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is not limited to the above examples. Those skilled in the art may find various alternatives and modifications within the scope of the appended claims, and it should be understood that the modifications and alternatives will naturally fall under the technical scope of the present disclosure.

[0315] While the facial mask, etc., that is the target of synthesis, is displayed on the same screen in the above embodiment, the present technology is not limited to such an example. For example, the facial mask, etc. that is the target of synthesis may be displayed on different screens. For example, a user selects a facial mask, etc. that is used as the source through a mobile terminal such as a smartphone, and moves the selected facial mask, etc. toward a device such as a digital signage on which a facial mask, etc. that is used as the destination is displayed, for example, by sliding the selected facial mask, etc. As a result, the facial mask that is used as the source is provided from the smartphone to the digital signage via communication, and the synthesized facial mask obtained by synthesis using the provided facial masks is superimposed and displayed on the face of the destination displayed on the digital signage.

[0316] Although the above embodiment has described an example in which synthesis is performed on the entire face, synthesis may be performed on a portion of the face, for example, only on a portion of the face such as the eyes, nose, mouth, or eyebrows.

[0317] Although the display unit 112 is described as an example of a display in the foregoing embodiment, the display unit 112 may be a projector. For example, the control unit 108 projects the synthesized face mask toward the face corresponding to the face mask used as dst onto the display unit 112.

[0318] Although the above embodiment has described an example in which the main target of synthesis is a face mask, the target of synthesis may be a head mask. For example, the information processing device 100 may generate a model of the entire head, that is, a head mask, and perform synthesis of the generated head mask.

[0319] Although the aforementioned embodiments have described examples in which the information processing device 100 is a smartphone, tablet terminal, or digital signage, the information processing device 100 may be a so-called optical see-through or video see-through head-mounted display (HMD) or a personal computer (PC).

[0320] The information processing device 100 can change the synthesized facial mask based on the location information of a person related to the facial mask used as the source. Specifically, the information processing device 100 changes the synthesized facial mask according to the distance between the location indicated by the location information and a specific location. For example, when a user approaches a live music concert hall, the information processing device 100 makes the facial feature information of the synthesized facial mask, obtained by synthesizing the facial mask of the user's face with the facial mask of the face of the artist in the live concert, closer to the facial feature information of the artist's facial mask. If the information processing device 100 is an HMD, the synthesized facial mask is superimposed and displayed on the people around the user. Of course, the synthesized facial mask can also be superimposed and displayed on the artist's face. In this case, the user can enjoy the experience even when visiting a specific location such as a live music concert hall.

[0321] Information processing device 100 can change the synthesized facial mask based on whether the location indicated by the location information is within a specific range. For example, if the user is in a theme park, information processing device 100 can superimpose and display a synthesized facial mask obtained by synthesizing the facial masks of other people in the theme park with the user's facial mask on the other people's faces. If the user leaves the theme park, information processing device 100 stops superimposing and displaying the synthesized facial mask. In this case, the sense of reality in a specific area, such as a theme park, can be improved.

[0322] While some application examples have been described so far, the configurations of the various embodiments of the present disclosure can of course be applied to other fields, business categories, products, applications, and the like. For example, the configurations can be applied to the field of cosmetic surgery. In such a case, for example, a post-operative face can be superimposed on the user's face as a synthetic face mask.

[0323] In addition, the effects described in this specification are merely illustrative or exemplary effects, and are not restrictive. That is, together with or in place of the above-mentioned effects, the technology according to the present disclosure can achieve other effects that are clear to those skilled in the art from the description of this specification.

[0324] Furthermore, the present invention includes not only processing in which the steps shown in the flowcharts of the above-described embodiments are executed in a time-series manner according to the order described, but also processing in which the steps are not necessarily processed in a time-series manner but are executed in parallel or individually. Furthermore, it goes without saying that even the steps processed in a time-series manner can be appropriately changed in order according to circumstances.

[0325] Furthermore, the present technology can also be configured as follows.

[0326] (1) An information processing device comprising:

[0327] A control unit configured to

[0328] generating a composite object model by synthesizing the first object model and the second object model, and

[0329] causing the generated synthetic object model to be displayed in association with an image corresponding to the first object model or the second object model,

[0330] The control unit controls the synthesis mode based on input information.

[0331] (2) The information processing device according to (1),

[0332] Wherein, the first object model includes a first face model,

[0333] The second object model includes a second face model, and

[0334] The synthesized object model includes a synthesized face model.

[0335] (3) The information processing device according to (2),

[0336] The modes include a first mode and a second mode, wherein, in the first mode, the synthetic facial model is generated based on the mesh information of the first facial model, and in the second mode, the synthetic facial model is generated based on the mesh information of the second facial model.

[0337] (4) The information processing device according to (3),

[0338] Wherein, in the first mode, the synthesized facial model is generated based on texture information of the second facial model, and

[0339] In the second mode, the synthesized facial model is generated based on texture information of the first facial model.

[0340] (5) The information processing device according to any one of (2) to (4),

[0341] The facial feature information includes the mesh information and texture information of the facial model, as well as

[0342] The modes include a third mode in which the synthetic facial model is generated based on mesh information of the first facial model, mesh information of the second facial model, texture information of the first facial model, and texture information of the second facial model.

[0343] (6) The information processing device according to any one of (2) to (5),

[0344] The control unit displays the synthesized facial model in association with an image specified based on a user input from among images corresponding to the first facial model and the second facial model.

[0345] (7) The information processing device according to any one of (2) to (6),

[0346] The input information includes information related to an image corresponding to the first face model or the second face model, used as input for a process performed by the control unit.

[0347] (8) The information processing device according to (7),

[0348] The information related to the image includes information related to the attributes of the image or the content of the image.

[0349] (9) The information processing device according to any one of (2) to (8),

[0350] The input information includes information related to the user's input operation.

[0351] (10) The information processing device according to (9),

[0352] The user's input operation includes an operation of selecting the first facial model or the second facial model, and

[0353] The control unit specifies the mode according to a selection position of the first face model or the second face model.

[0354] (11) The information processing device according to (10),

[0355] The control unit displays the selected position of the first face model or the second face model corresponding to each of the modes in a visually distinguishable form.

[0356] (12) The information processing device according to any one of (2) to (11),

[0357] Wherein, the control unit uses facial feature information of other facial models when the facial feature information used to generate the synthetic facial model is insufficient.

[0358] (13) The information processing device according to (12),

[0359] The other facial models include past facial models or the first facial model or the second facial model used as a synthesis object.

[0360] (14) The information processing device according to any one of (2) to (13),

[0361] The control unit changes facial feature information of the synthesized facial model based on the input information.

[0362] (15) The information processing device according to (14),

[0363] The control unit changes the facial feature information of the synthesized facial model between the facial feature information of the first facial model and the facial feature information of the second facial model based on an input operation of the user.

[0364] (16) The information processing device according to (14) or (15),

[0365] The control unit changes the mesh information of the synthesized facial model based on the attribute information of the first facial model or the second facial model.

[0366] (17) The information processing device according to any one of (14) to (16),

[0367] The control unit changes texture information of the synthesized facial model based on the first facial model, the second facial model, or an image associated with the synthesized facial model.

[0368] (18) The information processing device according to any one of (2) to (17),

[0369] At least one of the images corresponding to the first face model and the second face model includes an image temporarily recorded by imaging.

[0370] (19) An information processing method comprising: a control unit,

[0371] generating a composite object model by synthesizing the first object model and the second object model;

[0372] causing the generated synthetic object model to be displayed in association with an image corresponding to the first object model or the second object model; and

[0373] A mode of the synthesis is controlled based on input information.

[0374] (20) A program that causes a computer to implement the following functions:

[0375] A function of generating a composite object model by synthesizing the first object model and the second object model;

[0376] a function of displaying the generated synthetic object model in association with an image corresponding to the first object model or the second object model; and

[0377] A function of the synthesized mode is controlled based on input information.

[0378] Furthermore, the present technology can also be configured as follows.

[0379] (1) An information processing device comprising:

[0380] A control unit configured to

[0381] generating a composite object model by synthesizing the first object model and the second object model, and

[0382] causing the generated synthetic object model to be displayed in association with an image corresponding to the first object model or the second object model,

[0383] The control unit moves the first object model or the second object model used for the synthesis in response to an input based on a user's motion relative to a screen on which an object model or an image corresponding to the object model is displayed.

[0384] (2) The information processing device according to (1),

[0385] Wherein, the first object model includes a first face model,

[0386] The second object model includes a second face model, and

[0387] The synthesized object model includes a synthesized face model.

[0388] (3) The information processing device according to (2),

[0389] After completing the movement from one of the first facial model and the second facial model to the other, the control unit displays the synthesized facial model in association with an image corresponding to the other facial model.

[0390] (4) The information processing device according to (2) or (3),

[0391] Wherein, when the input based on the user's motion is interrupted, the control unit does not perform the synthesis.

[0392] (5) The information processing device according to any one of (2) to (4),

[0393] The input based on the user's motion includes an operation of specifying both a start point and an end point or an operation of specifying only the start point.

[0394] (6) The information processing device according to any one of (2) to (5),

[0395] The input based on the user's movement includes gesture operation.

[0396] (7) The information processing device according to any one of (2) to (6),

[0397] When the synthesized facial model is displayed, the control unit causes the first facial model or the second facial model corresponding to the image associated with the synthesized facial model to be displayed adjacent to the synthesized facial model.

[0398] (8) The information processing device according to any one of (2) to (7),

[0399] The first face model or the second face model to be moved includes a face model arranged in a face list displayed on a screen or a face model generated from an image arranged in the face list.

[0400] (9) The information processing device according to (8),

[0401] The control unit selects the face model or image arranged in the face list based on the attributes of the face represented by the first face model or the second face model.

[0402] (10) The information processing device according to (8) or (9),

[0403] The control unit adds the first face model or the second face model moving toward the face list to the face list.

[0404] (11) The information processing device according to any one of (8) to (10),

[0405] The control unit performs a preliminary display of a second synthesized face model related to the synthesized face model.

[0406] (12) The information processing device according to (11),

[0407] The pre-display of the second synthesized facial model includes changing the first facial model or the second facial model being moved into the second synthesized facial model.

[0408] (13) The information processing device according to (12),

[0409] The second synthetic facial model includes the following synthetic facial mask: facial feature information of the synthetic facial model changes according to information related to the movement of the first facial model or the second facial model being moved.

[0410] (14) The information processing device according to any one of (11) to (13),

[0411] The pre-display of the second synthesized face model includes changing a face model arranged in the face list or a face model generated from an image arranged in the face list to the second synthesized face model.

[0412] (15) The information processing device according to (14),

[0413] The control unit changes a face model selected from the face models arranged in the face list or a face model generated from an image arranged in the face list, or all of the face models, to the second synthesized face model.

[0414] (16) The information processing device according to any one of (2) to (15),

[0415] In which, when multiple facial models or images corresponding to the facial model are displayed on the screen, the control unit generates each of the synthetic facial models based on the moved facial model and the multiple facial models or the images corresponding to the facial models.

[0416] (17) The information processing device according to (16),

[0417] The control unit performs the synthesis when the face model is moved to one of the plurality of face models or one of the images corresponding to the face model.

[0418] (18) The information processing device according to any one of (2) to (17),

[0419] Wherein, at least one of the first facial model and the second facial model includes a facial model of a user.

[0420] (19) An information processing method comprising: using a control unit,

[0421] generating a composite object model by synthesizing the first object model and the second object model;

[0422] causing the generated synthetic object model to be displayed in association with an image corresponding to the first object model or the second object model; and

[0423] The first object model or the second object model used for the synthesis is moved in response to an input based on a user's motion relative to a screen on which an object model or an image corresponding to the object model is displayed.

[0424] (20) A program that causes a computer to implement the following functions:

[0425] A function of generating a composite object model by synthesizing the first object model and the second object model;

[0426] a function of displaying the generated synthetic object model in association with an image corresponding to the first object model or the second object model; and

[0427] A function of causing the first object model or the second object model used for the synthesis to be moved in response to an input based on a user's motion relative to a screen on which an object model or an image corresponding to the object model is displayed.

[0428] Reference Signs List

[0429] 100 Information Processing Device

[0430] 102 Imaging Unit

[0431] 104 Facial Recognition Unit

[0432] 106 Operation detection unit

[0433] 108 control unit

[0434] 110 storage units

[0435] 112 display units

Claims

1. A mobile device, comprising: an imaging unit configured to capture a through-image; a display unit configured to display the through image; as well as The processor is configured to: acquiring at least one grid data of the first facial image based on feature points of the first facial image included in the through image; acquiring texture data of at least one second facial image different from the first facial image; determining whether the through image includes a plurality of the first facial images; controlling the display unit to display the texture data of the second facial image on each of the plurality of first facial images to correspond to the grid data of the plurality of first facial images based on a determination that the through image includes the plurality of first facial images; Recording the grid data; as well as Even when the number of feature points of the first facial image decreases, the display unit is controlled to display the texture data of the second facial image based on the recorded mesh data.

2. A mobile device comprising: storage unit; an imaging unit configured to capture a through-image; a display unit configured to display the through image; as well as The processor is configured to: acquiring at least one grid data of the first facial image based on feature points of the first facial image included in the through image; acquiring texture data of at least one second facial image different from the first facial image; controlling the display unit to display the texture data of the second facial image on the through image to correspond to the grid data of the first facial image; recording the grid data in the storage unit; as well as Even when the number of feature points of the first facial image decreases, the display unit is controlled to display the texture data of the second facial image based on the recorded mesh data.

3. A mobile device comprising: an imaging unit configured to capture a through-image; a display unit configured to display the through image; as well as The processor is configured to: acquiring at least one grid data of the first facial image based on feature points of the first facial image included in the through image; acquiring texture data of at least one second facial image different from the first facial image; controlling the display unit to display the texture data of the second facial image on the through image to correspond to the grid data of the first facial image; Recording the grid data; controlling the display unit to display texture data of the second facial image based on the recorded mesh data even when the number of feature points of the first facial image decreases; as well as The display unit is controlled to display a plurality of the second facial images that are different from each other together with the first facial image.

4. An information processing method, comprising: acquiring a through-image captured by an imaging unit of the mobile device; acquiring at least one grid data of the first facial image based on feature points of the first facial image included in the through image; acquiring texture data of at least one second facial image different from the first facial image; determining whether the through image includes a plurality of the first facial images; controlling a display unit of the mobile device to display texture data of the second facial image on each of the plurality of first facial images to correspond to grid data of the plurality of first facial images based on a determination that the through image includes the plurality of first facial images; Recording the grid data; as well as Even when the number of feature points of the first facial image decreases, the display unit is controlled to display the texture data of the second facial image based on the recorded mesh data.

5. A non-transitory computer-readable medium having a program embodied thereon, the program, when executed by a mobile computing device, causing the mobile computing device to perform a method comprising: acquiring a through-image captured by an imaging unit of the mobile computing device; acquiring at least one grid data of the first facial image based on feature points of the first facial image included in the through image; acquiring texture data of at least one second facial image different from the first facial image; determining whether the through image includes a plurality of the first facial images; controlling a display unit of the mobile computing device to display texture data of the second facial image on each of the plurality of first facial images to correspond to mesh data of the plurality of first facial images based on a determination that the through-image includes the plurality of first facial images; Recording the grid data; as well as Even when the number of feature points of the first facial image decreases, the display unit is controlled to display the texture data of the second facial image based on the recorded mesh data.

Images