Image processing device, image processing method, and program

Abstract

To provide a photograph creation game machine, an image processing method, and a program, with which an image for synthesis can be synthesized in a more natural state with respect to a specific portion of a user in a captured image.SOLUTION: A photograph creation game machine of one aspect of the present technique generates three-dimensional information representing a three-dimensional shape of a specific portion of each of a plurality of users photographed in a captured image, and synthesizes an image for synthesis deformed based on the three-dimensional information of the user to the specific portion of the user to be synthesized. The present technique can be applied to a photograph creation game machine for providing a series of games for printing a captured image on seal paper.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] This technology is Image Processing Device The present invention relates to an image processing method and a program, and in particular, to a method for synthesizing a composite image in a more natural state for a specific part of a user that appears in a photographed image. Image Processing Device , an image processing method, and a program. [Background technology]

[0002] Photo sticker creation devices are known in the art. These devices photograph a user, allow the user to edit the photographed image, and print the edited image on sticker paper. These photo sticker creation devices are installed in amusement facilities and the like.

[0003] The flow of a game provided by the photo sticker creation device usually involves taking a photo of the user in the shooting space as the subject, then moving the user to the editing space, editing the image according to the operations performed in the editing space, and printing the edited image on sticker paper.

[0004] After moving from the shooting space to the editing space, the captured image is edited using an editing screen displayed on a display in the editing space. The editing screen displays the captured image to be edited, as well as buttons used to select various editing tools.

[0005] Patent Document 1 describes a photography game machine that has automatic and manual modes for doodling makeup. In automatic mode, the facial area and specific organ areas are identified, and makeup images such as "lips," "cheeks," and "colored contact lenses" are automatically superimposed on the identified areas. For example, an image of "cheeks" is automatically superimposed and synthesized on the cheek area of ​​the face. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-181280 Summary of the Invention [Problem to be solved by the invention]

[0007] If a flat image for makeup is simply synthesized, the flatness of the image for makeup may stand out depending on the direction of the face, which may create a sense of incongruity, since the human face is three-dimensional.

[0008] The present technology has been developed in light of such circumstances, and makes it possible to synthesize a synthesis image in a more natural state for a specific part of a user that appears in a photographed image. [Means for solving the problem]

[0009] A photo creation game machine according to one aspect of the present technology includes an image processing unit that generates three-dimensional information for each user that represents the three-dimensional shape of a specific part of each of multiple users appearing in a captured image, and combines a composite image that has been transformed based on the three-dimensional information of the user with the specific part of the user that is the target of the composite.

[0010] In one aspect of this technology, three-dimensional information representing the three-dimensional shape of specific parts of each of multiple users appearing in a captured image is generated for each user, and a synthesis image transformed based on the user's three-dimensional information is synthesized with the specific part of the user to be synthesized. [Effects of the Invention]

[0011] According to the present technology, it is possible to synthesize a composite image in a more natural state for a specific part of a user that appears in a photographed image. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a perspective view showing an external appearance of a photo sticker creation device according to an embodiment of the present technology; [Figure 2] FIG. 2 is a perspective view showing an example of the front configuration of the imaging unit. [Figure 3] 1 is a block diagram showing an example of the configuration of a photo sticker creation device; [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of the photo sticker creating device. [Figure 5] FIG. 10 is a diagram showing an example of a captured image onto which face sticker content is combined. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an edit processing unit. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of a portable image generating unit. [Figure 8] FIG. 10 is a diagram illustrating an example of generating a face model. [Figure 9] FIG. 10 is a diagram illustrating another example of generating a face model. [Figure 10] FIG. 10 is a diagram illustrating yet another example of generating a face model. [Figure 11] FIG. 10 is a diagram showing an example of conversion of content for a face sticker. [Figure 12] 10 is a flowchart illustrating the operation of the photo sticker creating device. [Figure 13] 13 is a flowchart illustrating the editing process performed in step S3 of FIG. 12. [Figure 14] 14 is a flowchart continuing from FIG. 13, illustrating the editing process performed in step S3 of FIG. 12. [Figure 15] FIG. 10 is a diagram showing a display example of a mobile graffiti screen. [Figure 16] FIG. 10 is a diagram showing an example of content for face stickers. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present technology will be described in the following order. 1.Configuration of the photo sticker creation device 2. Internal structure of the photo sticker creation device 3. Example of control unit function configuration 4. Operation of the photo sticker creation device 5. Variations

[0014] <Configuration of the photo sticker creation device> FIG. 1 is a perspective view showing the appearance of a photo sticker creating device according to an embodiment of the present technology.

[0015] The photo sticker creation device 1 is a game machine that provides a photo creation game, and is installed in a store such as a game center.

[0016] Users of the photo sticker creation device 1 are mainly high school girls and young women. Multiple users, usually two or three people per group, can enjoy the game with the photo sticker creation device 1. Of course, a single user can also enjoy the game with the photo sticker creation device 1.

[0017] In the photo creation game of the photo sticker creation device 1, the user takes a photo of themselves as the subject. The user then edits the captured image to combine it with composite images such as handwritten characters and stamp images, and edits the captured image into a colorful image. The user then receives a sticker sheet on which the edited captured image is printed, completing the series of games.

[0018] A pre-operation unit 11 is provided on the side of the photo sticker creation device 1, which has a box-shaped housing. The pre-operation unit 11 is a device used for operations before photography, such as selecting the number of users to play with and selecting the photography course. The pre-operation unit 11 is provided with a touch panel monitor 11A. Below the touch panel monitor 11A, a coin slot 11B is provided for paying the play fee.

[0019] A screen used for various selections is displayed on the touch panel monitor 11A. After making various selections according to the display on the touch panel monitor 11A, the user enters the photography space through the entrance / exit G and begins photography. A photography unit 21 is provided in front of the photography space. The space between the photography unit 21 and the background unit 22 constitutes the photography space. A box-shaped camera unit 31 with a built-in camera is provided in the photography unit 21.

[0020] An editing unit 51 is provided to the left of the pre-operation unit 11. The editing unit 51 is a device used to edit captured images obtained by taking pictures with a camera. A user who has taken pictures in the shooting space moves to the editing space in front of the editing unit 51 and edits the captured images. A tablet-integrated monitor 61, which is a monitor for editing the captured images, is provided on a slope formed above the editing unit 51. A screen used for editing the captured images is displayed on the tablet-integrated monitor 61.

[0021] Two editing spaces are prepared, one on either side of the housing of the editing unit 51. An editing space is formed in the space in front of the editing unit 51 shown in Fig. 1, and the other on the back side of the editing unit 51. This allows a group of users who have finished shooting in the shooting space to move to the vacant editing space, and edit in parallel in the two editing spaces.

[0022] 2 is a perspective view showing an example of the front configuration of the imaging unit 21. The imaging unit 21 is configured so as to be surrounded by a side panel 21A, a side panel 21B, and a front panel 21C.

[0023] A camera unit 31 is provided at a position slightly above the center of the front of the photographing section 21. The camera unit 31 is provided with a camera 41 and a touch panel monitor .

[0024] Camera 41 is, for example, a single-lens reflex camera, and is attached inside camera unit 31 so that the lens is exposed. Camera 41 has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures images of users in the imaging space. Camera 41 functions as an imaging unit that captures images of users as subjects.

[0025] Moving images (hereinafter also referred to as live view images) captured by the camera 41 are displayed in real time on the touch panel monitor 42. Still images captured by the camera 41 at a predetermined timing, such as when a shooting instruction is issued, are saved as captured images.

[0026] The touch panel monitor 42 is provided below the camera 41. The touch panel monitor 42 is composed of a monitor such as an LCD and a touch panel stacked on it. The touch panel monitor 42 has a function as a live view monitor that displays moving images captured by the camera 41, and a function to display various GUIs and accept selection operations by the user.

[0027] An upper strobe unit 32 with a curved light-emitting surface facing the user is provided above the camera unit 31. The upper strobe unit 32 irradiates light from above and in front of the user, mainly toward the user's face and upper body.

[0028] A lower box strobe unit 33 that irradiates the user with light from below is provided below the camera unit 31. A strobe light source is provided in each of the upper strobe unit 32 and the lower box strobe unit 33. The strobe light source emits light in synchronization with the timing of capturing a still image, irradiating the user, who is the subject, with strobe light.

[0029] A box-shaped portion 34L is provided between the lower box strobe unit 33 and the side panel 21A. A box-shaped portion 34R is provided between the lower box strobe unit 33 and the side panel 21B. The top surfaces of the box-shaped portions 34L and 34R are substantially horizontal and are used as a luggage storage area for users performing photography work to place their baggage and the like.

[0030] <Internal configuration of the photo sticker creation device> Fig. 3 is a block diagram showing an example of the configuration of the photo sticker creating device 1. In Fig. 3, the same components as those described above are given the same reference numerals, and duplicated explanations will be omitted where appropriate.

[0031] 3, the photo sticker creation device 1 is configured by connecting a pre-operation unit 11, a photographing unit 21, an editing unit 51, and a printing unit 103 to a PC (Personal Computer) unit 101 via an external input / output interface 102. The PC unit 101 is housed inside the housing of the editing unit 51, for example.

[0032] The PC section 101 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113, which are interconnected by a bus 114.

[0033] The CPU 111 executes a predetermined program and controls the overall operation of the photo sticker creating device 1. The ROM 112 stores the programs and data executed by the CPU 111. The RAM 112 temporarily stores the data and programs processed by the CPU 111.

[0034] An input / output interface 115 is further connected to the bus 114. To the input / output interface 115, a storage unit 116, a communication unit 117, and a drive 118 are connected.

[0035] The storage unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The storage unit 116 stores various setting information and the like supplied from the CPU 111. The information stored in the storage unit 116 is read out by the CPU 111 as appropriate.

[0036] The communication unit 117 is an interface for a network such as the Internet. The communication unit 117 communicates with external devices under the control of the CPU 111. The communication unit 117 transmits an image selected by a user to, for example, a server managed by the manufacturer of the photo sticker creating device 1. The image transmitted from the communication unit 117 is allocated a predetermined storage area and saved, and is then displayed on or downloaded to a mobile terminal that has accessed the server.

[0037] Removable media 119, such as an optical disk or semiconductor memory, is appropriately attached to drive 118. Programs and data read from removable media 119 by drive 118 are supplied to CPU 111, where they are stored in storage unit 116 or installed.

[0038] The input / output interface 115 is connected to an external input / output interface 102. The PC unit 101 controls each unit via the external input / output interface 102.

[0039] The pre-operation unit 11 performs a pre-selection process for the user before photographing. The pre-operation unit 11 is composed of a touch panel monitor 11A, a speaker 121, and a coin processing unit 122.

[0040] The touch panel monitor 11A displays various selection screens and accepts user operations on the selection screens under the control of the PC unit 101. An input signal representing the content of the user operation is supplied to the PC unit 101, and various settings are made.

[0041] The coin processing unit 122 detects the insertion of coins into the coin insertion slot 11B. When the coin processing unit 122 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the start of the game.

[0042] The photographing unit 21 performs photographing processing of a user in a photographing space. The photographing unit 21 is composed of a lighting device 131, a camera 41, a touch panel monitor 42, and a speaker 132.

[0043] The lighting devices 131 are strobe units in the shooting space, and emit light in accordance with a lighting control signal supplied from the PC section 101. The lighting devices 131 include strobe units provided in the shooting space, such as the upper strobe unit 32 and the lower box strobe unit 33.

[0044] The camera 41 takes a photograph in accordance with shutter control by the PC section 101, and outputs image data obtained by the photograph to the PC section 101.

[0045] The editing unit 51 performs post-shooting editing processing. The editing unit 51 is composed of a tablet built-in monitor 61, touch pens 141A and 141B, and a speaker 142. The tablet built-in monitor 61 and the touch pens 141A and 141B are provided as a set on two sides of the housing of the editing unit 51.

[0046] The tablet built-in monitor 61 displays an editing screen and accepts user operations on the editing screen under the control of the PC unit 101. An input signal representing the content of the user's operation is supplied to the PC unit 101, and the captured image to be edited is edited.

[0047] The printing unit 103 is configured to include a printer 151. A sticker paper unit 152 is attached to the printer 151.

[0048] Printer 151 prints an image on a sticker paper stored in sticker paper unit 152 based on print data supplied from PC section 101. Printer 151 ejects the sticker paper on which the image is printed to a sticker paper ejection port.

[0049] <Example of functional configuration of photo sticker creation device> 4 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. At least some of the functional units shown in FIG. 4 are realized by the CPU 111 in FIG. 3 executing a predetermined program.

[0050] A control unit 201 is realized in the photo sticker creating device 1. The control unit 201 is made up of a pre-selection processing unit 211, a photography processing unit 212, an edit processing unit 213, a print processing unit 214, and a transmission processing unit 215.

[0051] The pre-selection processing unit 211 executes pre-selection processing by controlling each unit of the pre-operation unit 11. The pre-selection processing involves the selection of the number of users, the selection of a photography course, etc. Information representing the selection made by the user is supplied to the photography processing unit 212.

[0052] The shooting processing unit 212 performs shooting processing by controlling each unit of the shooting unit 21. In the shooting processing, multiple shot images, such as six shot images, are taken. The shooting processing unit 212 outputs data of the multiple shot images obtained by the shooting processing to the editing processing unit 213.

[0053] The editing processing unit 213 performs editing processing by controlling each unit of the editing unit 51. The editing processing includes editing the captured image, printing the image on sticker paper, and sending the image to an external image management server.

[0054] The images sent to the image management server can be viewed or downloaded by the user on a mobile device such as a smartphone after the user has finished playing the game on the photo sticker creation device 1.

[0055] Editing the photographed image includes a process of generating an image to be printed on sticker paper based on the photographed image, and a process of generating an image to be sent to the image management server based on the photographed image.

[0056] The image to be printed on the sticker paper is generated by having the user select a sticker design and then performing image processing according to the sticker design selected by the user. When generating the image to be printed on the sticker paper, it is possible to combine a design image with the sticker design.

[0057] On the other hand, portable images, which are images to be sent to the image management server, are generated by applying image processing to the photographed image to be edited in accordance with operations performed using a tool selected by the user from pre-prepared editing tools, such as a pen tool or stamp tool. By using the pen tool or stamp tool, a pen image or stamp image, which serves as a composite image, is composited into the photographed image to be edited. The user selects their preferred editing tool and selects the position on the photographed image to be edited where the image processing will be applied, thereby proceeding with editing the photographed image.

[0058] In this way, editing of a photographed image, particularly for creating a portable image, is carried out by the user using the touch pen 141 as if doodling on the photographed image. Hereinafter, editing will be referred to as doodling where appropriate. Editing of a photographed image to create an image to be printed on sticker paper will be referred to as doodling for stickers, and editing of a photographed image to create a portable image will be referred to as doodling for mobile phones.

[0059] As will be described in detail later, a face sticker tool is provided as a mobile graffiti tool. The face sticker tool is a function used to superimpose a pre-prepared image onto a face. As shown by the dotted line in Figure 5, multiple types of highly designed images are provided as content to be superimposed on areas such as the cheeks of the face. Hereinafter, the superimposed images prepared as content for face stickers will be referred to as face sticker content, where appropriate.

[0060] In the example of Figure 5, face sticker content C1 is composited around the eyes of the person on the left, and face sticker content C2 is composited around the cheeks of the person on the right. Face sticker content C1 is an image made up of multiple small heart-shaped images. Face sticker content C2 is an image made up of cherry-shaped images. The composite position is set in advance for each type of face sticker content.

[0061] When a user selects the face sticker tool and the type of content to be combined, the content is automatically combined with the cheeks and other areas of the user's face in the captured image. In this way, content is combined automatically without the user having to specify the destination position.

[0062] The edit processing unit 213 outputs the print data generated by the process of the graffiti sticker to the print processing unit 214, and outputs the data of the portable image to the transmission processing unit 215.

[0063] The print processing unit 214 outputs the print data to the printer 151, which prints the image on the sticker paper.

[0064] The transmission processing unit 215 controls the communication unit 117 to communicate with the image management server, and transmits (uploads) portable images to the image management server.

[0065] FIG. 6 is a block diagram showing an example of the configuration of the edit processing unit 213.

[0066] The editing processing unit 213 is made up of an image analysis unit 221, a sticker image generation unit 222, a sticker design data storage unit 223, a portable image generation unit 224, and a display control unit 225. The captured image data output from the capture processing unit 212 is input to the image analysis unit 221 and the display control unit 225.

[0067] The image analysis unit 221 analyzes the features of the user appearing in each captured image and recognizes the positions of specific body parts of the user appearing in the captured image. The positions of specific body parts recognized by the image analysis unit 221 include, for example, the head, eyes, nose, mouth, and ears. Information indicating the positions recognized by the image analysis unit 221 is supplied to the sticker image generation unit 222 and the portable image generation unit 224 together with the captured image data.

[0068] The sticker image generating unit 222 reads information about the sticker design selected by the user from the sticker design data storage unit 223, and performs image processing based on the read information on the captured image. For example, an image for a sticker is generated by combining an image of the design selected by the user.

[0069] Print data for the sticker image is supplied to the print processing unit 214. Information indicating the content of image processing by the sticker image generation unit 222 is supplied to the display control unit 225 as appropriate and used for screen display. The image processing by the sticker image generation unit 222 is processing for creating an image to be printed on sticker paper.

[0070] The portable image generation unit 224 performs image processing on each captured image according to the portable graffiti made by the user to generate a portable image. The portable graffiti is graffiti used to create an image to be sent to the image management server. The image with the portable graffiti is a transmission image used to send to the image management server.

[0071] The portable image generated by the portable image generation unit 224 is supplied to the transmission processing unit 215. Information indicating the content of image processing by the portable image generation unit 224 is supplied to the display control unit 225 as appropriate and used for screen display.

[0072] The display control unit 225 displays various screens including a sticker doodle screen and a mobile doodle screen on the tablet built-in monitor 61. The display control unit 225 controls the display of the screen in response to the user's operation.

[0073] Fig. 7 is a block diagram showing an example of the configuration of portable image generation unit 224. Note that Fig. 7 shows a configuration related to the function of the face seal tool. In reality, portable image generation unit 224 is also provided with configurations related to other functions realized using the display of the portable doodle screen.

[0074] The portable image generation unit 224 is made up of a retouching processing unit 231, a face seal processing unit 232, a content storage unit 233, and a synthesis unit 234. The captured image output from the image analysis unit 221 is input to the retouching processing unit 231.

[0075] The retouching processor 231 performs retouching on the face of the user appearing in each captured image. Examples of retouching include a process to make the face smaller and a process to make the eyes larger. The retouching processor 231 performs retouching such as adjusting the size of the face or the size of the eyes in accordance with operations performed by the user using a retouching screen displayed on the tablet built-in monitor 61. The retouching may be performed automatically without user operations.

[0076] Each captured image retouched by the retouching processor 231 is supplied to the face sticker processor 232 and the synthesizer 234. The retouching is performed, for example, before starting mobile doodling using the mobile doodling screen. The retouched image is used as the target for mobile doodling.

[0077] The retouching process is also performed after the start of the mobile doodling. The mobile doodling screen is provided with displays such as buttons to be operated when performing the retouching process.

[0078] In this way, a captured image is subjected to retouching processing at a predetermined timing to change the shape and size of the entire face and the shape and size of parts of the face, such as the eyes. Only the change in the shape of the entire face or the shape of parts of the face may be performed as retouching processing. Also, only the change in the shape or size of the face may be performed as retouching processing.

[0079] The retouching process may be performed not only on the eyes but also on other parts such as the nose, eyebrows, ears, mouth, etc. The part to be retouched may be one part or multiple parts.

[0080] The face seal processing unit 232 generates content for a face seal in response to a user's operation using the face seal tool. The face seal processing unit 232 is composed of a 3D model generation unit 241 and a conversion unit 242.

[0081] The 3D model generation unit 241 identifies the facial area of ​​the user appearing in the captured image supplied from the retouching processing unit 231, and generates a facial model, which is a 3D model representing the three-dimensional shape (stereoscopic shape) of the face. The facial model is configured, for example, from a plurality of meshes separated by lines connecting facial feature points arranged in a virtual three-dimensional space. The facial area of ​​the user appearing in the captured image is identified based on the analysis results by the image analysis unit 221.

[0082] FIG. 8 is a diagram showing an example of generating a face model.

[0083] When two users, user U1 and user U2, appear in a captured image, a face model M1 representing the three-dimensional shape of the face of user U1 and a face model M2 representing the three-dimensional shape of the face of user U2 are generated, as indicated by the arrows at the top of the arrows. In other words, a face model is generated for each user appearing in the captured image.

[0084] Face model M1 is used when synthesizing face sticker content onto the face of user U1, and face model M2 is used when synthesizing face sticker content onto the face of user U2. As will be described later, the face models are used to convert the face sticker content. Changes to the shape, size, color, etc. of the face sticker content are made based on the face models.

[0085] The generation of the face model is performed, for example, by using an inference device that receives a face image as input and outputs a 3D model. The 3D model generation unit 241 is provided with an inference device for generating a 3D face model, which is generated by performing machine learning using the face image and the 3D model as learning data.

[0086] The 3D model generation unit 241 identifies the facial area of ​​each user appearing in the captured image based on the analysis results by the image analysis unit 221, and generates a facial model for each user by inputting the facial image into an inference device.

[0087] The facial model may be generated using other tools instead of an inference device generated by machine learning. For example, the facial model may be generated by analyzing a facial image and arranging feature points identified in a three-dimensional space. Various techniques for estimating the three-dimensional shape of a subject captured in a two-dimensional image may be employed.

[0088] Instead of a 3D model configured by meshes, other three-dimensional information representing the three-dimensional shape of the face may be generated for each user.

[0089] FIG. 9 is a diagram showing another example of generating a face model.

[0090] As shown in Fig. 9, face models are generated for each captured image. In the example of Fig. 9, a face model M1-1 representing the three-dimensional shape of the face of user U1 and a face model M2-1 representing the three-dimensional shape of the face of user U2 are generated based on the first captured image, and a face model M1-2 representing the three-dimensional shape of the face of user U1 and a face model M2-2 representing the three-dimensional shape of the face of user U2 are generated based on the second captured image.

[0091] Similarly, for the third and subsequent captured images, a face model is generated for each user based on each captured image. Each face model is used when it is selected to synthesize face sticker content using the face of the person from which the face model was generated. For example, if the third captured image is selected as the target for graffiti, and it is selected to synthesize face sticker content using user U1, face model M1-3 is used.

[0092] A single face model for a user may be generated based on multiple face models generated based on the face image of the same user.

[0093] For example, if six photographs are taken, one face model for user U1 is generated based on face models M1-1 to M1-6 representing the three-dimensional shape of user U1's face, which are generated based on each of the six photographed images. Similarly, one face model for user U2 is generated based on face models M2-1 to M2-6 representing the three-dimensional shape of user U2's face. For example, one face model for each user is generated by averaging the six face models.

[0094] By generating one face model based on six face models, it is possible to generate a more accurate face model for each user.

[0095] Such generation of a face model is repeated every time a retouching process is performed.

[0096] FIG. 10 is a diagram showing yet another example of generating a face model.

[0097] As shown in the upper part of Fig. 10, when face models M1 and M2 are generated based on the first captured image, and retouching is performed on the faces of users U1 and U2 appearing in the captured image, face models M1' and M2' are generated based on the retouched captured image, as shown in the lower part of Fig. 10. In the captured image in the lower part of Fig. 10, the colored contours of the faces of users U1 and U2 indicate that the contours have been distorted by retouching.

[0098] Facial model M1' is a facial model that represents the three-dimensional shape of the face of user U1 that appears in the first captured image after retouching.Facial model M2' is a facial model that represents the three-dimensional shape of the face of user U2 that appears in the first captured image after retouching.

[0099] If multiple users are in the photo and retouching is performed on only one user, only the face model of the retouched user will be remade.

[0100] By recreating the face model each time retouching is performed, it becomes possible to appropriately synthesize face sticker content according to the three-dimensional shape of the face after retouching. For example, if retouching is performed after face sticker content has been synthesized once, face sticker content converted using a newly generated face model is synthesized in place of the synthesized face sticker content. The display of the face sticker content synthesized on the face changes in accordance with the retouching.

[0101] Information about the face model generated by the 3D model generation unit 241 is supplied to the conversion unit 242 in FIG.

[0102] The conversion unit 242 reads and acquires the face sticker content selected by the user from the content storage unit 233. The conversion unit 242 converts the face sticker content read from the content storage unit 233 based on a face model of the user to be composited, which is generated based on the photographed image that is the subject of graffiti.

[0103] FIG. 11 is a diagram showing an example of conversion of content for a face sticker.

[0104] The following describes a case where the face sticker content C1 shown on the left side of Fig. 11 is selected to be superimposed on the face of user U1 appearing in the photographed image that is the subject of graffiti. In this case, as indicated by the tips of arrows #1 and #2, the face sticker content C1 is converted based on a face model M1 of user U1 that was generated based on the photographed image that is the subject of graffiti.

[0105] When the center position P of the face is set as the synthesis position of the face sticker content C1, the face sticker content C1 is transformed so as to be deformed according to the three-dimensional shape around the position P. Not only is a planar transformation performed, but a depth transformation is also performed.

[0106] The shape of the face sticker content C1 after conversion will be the same as when the default face sticker content C1 is pasted at position P on the face model M1, as indicated by the tip of arrow #3. In accordance with the change in shape, the color, density, transparency, etc. of the face sticker content C1 are also changed as appropriate.

[0107] The conversion unit 242 outputs the converted face sticker content to the synthesis unit 234.

[0108] The content storage unit 233 stores various types of content used in the mobile graffiti app. The content stored in the content storage unit 233 includes face sticker content as well as various types of content such as images for a pen tool and an image for a stamp tool.

[0109] The combining unit 234 combines the face sticker content supplied from the converting unit 242 with the face of the user appearing in the photographed image of the graffiti target.

[0110] Furthermore, the composition unit 234 composes content onto the photographed image of the graffiti target in response to a user's operation using a tool other than the face sticker tool. The composition unit 234 generates a portable image by appropriately performing image processing such as filtering. The portable image generated by the composition unit 234 is supplied to the display control unit 225 (FIG. 6) and the like.

[0111] <Operation of the photo sticker creation device> Here, the operation of the photo sticker creating device 1 that provides the photo sticker creating game will be described with reference to the flowchart in Fig. 12. The process in Fig. 12 starts when coins worth a predetermined amount, such as 400 yen, are inserted.

[0112] In step S1, the pre-selection processing unit 211 performs a pre-selection process, which includes selecting the number of users, the game course, and the like.

[0113] In step S2, the photography processing unit 212 controls the photography unit 21 to perform photography processing. Specifically, the photography processing unit 212 displays the live view image captured by the camera 41 on the touch panel monitor 42. When the photography timing arrives, the photography processing unit 212 photographs the user in the photography space as a subject and generates a photographed image that is a close-up image. The close-up image is a still image in which the upper half of the user's body is shown in a large size. The photography of the close-up image is repeated a predetermined number of times, such as six times.

[0114] In step S3, the editing processing unit 213 performs editing processing by controlling the configuration of the editing unit 51 corresponding to the editing space to which the user has moved after completing the shooting processing.

[0115] Editing process Next, the editing process performed in step S3 of FIG. 12 will be described with reference to the flowcharts of FIGS.

[0116] In step S11, the retouching processor 231 performs retouching on the user's face appearing in the captured image. A retouching screen is displayed on the tablet built-in monitor 61, and retouching is performed on the user's face appearing in each captured image in response to an operation by the user.

[0117] In step S12, the 3D model generation unit 241 of the face seal processing unit 232 identifies the facial area of ​​the user appearing in the photographed image after the retouching process and generates a face model.

[0118] In step S13, display control unit 225 of edit processing unit 213 displays a sticker doodle screen. The sticker doodle screen displays various buttons, such as a button to be operated when selecting a sticker design and a button to be operated when selecting an image to be combined with the photographed image, along with the photographed image to be doodled.

[0119] In step S14, the sticker image generating unit 222 performs image processing on the photographed image in accordance with the user's operation, and generates a sticker image.

[0120] In step S15, the sticker image generating unit 222 determines whether the sticker graffiti has ended. If it is determined in step S15 that the sticker graffiti has not ended, the process returns to step S13, and the sticker graffiti continues. For example, if a preset time limit has elapsed, the sticker graffiti ends.

[0121] On the other hand, if it is determined in step S15 that the sticker doodling has ended, in step S16 of Fig. 14, the display control unit 225 displays the mobile doodling screen on the tablet built-in monitor 61. The user uses the mobile doodling screen to doodling on the captured image.

[0122] FIG. 15 is a diagram showing a display example of the mobile doodling screen.

[0123] 15, the mobile scribble screen is basically configured by providing each display symmetrically on either side of the central thumbnail display area 303. The left area is used by a user standing on the left side facing the tablet-built-in monitor 61. The right area is used by a user standing on the right side facing the tablet-built-in monitor 61.

[0124] It should be noted that the rectangular dashed lines surrounding each area on the mobile doodling screen shown in FIG. 15 are not actually displayed on the screen.

[0125] The thumbnail display area 303, located in the center of the mobile doodle screen, is a display area for thumbnail images representing captured images. The user uses the thumbnail images displayed in the thumbnail display area 303 to select a captured image on which to doodle.

[0126] Thumbnail images representing all captured images are displayed in the thumbnail display area 303. In the example of Fig. 15, thumbnail images of six close-up images are displayed. In the example of Fig. 15, the user on the left has selected the thumbnail image on the top left, and the user on the right has selected the thumbnail image on the bottom right.

[0127] A doodle input area 301L is provided approximately in the center of the left-hand area. The doodle input area 301L is a display area for a captured image that is selected using a thumbnail image and is to be doodled on. The user can select an editing tool using the touch pen 141 and edit the captured image displayed in the doodle input area 301L. The edited content is reflected in the display in the doodle input area 301L.

[0128] To the right of doodle input area 301L is provided tool area 302L, which contains an array of buttons such as a button to be pressed to select the eraser tool, a button to be pressed to reset the doodle, and a button to be pressed to return to the previous operation.

[0129] A palette area 304L is provided below the graffiti input area 301L. The palette area 304L is an area that displays buttons and the like used to select various editing tools to be combined with the captured image displayed in the graffiti input area 301L.

[0130] Tabs used to select categories of editing tools are provided above palette area 304L. The user can use the tabs to switch between editing tools displayed in palette area 304L. In the example of FIG. 15, tabs for face stickers, eye makeup, eye size, small face, lips, color contact lenses, three-dimensional effect, and eyelashes are provided, and the face sticker tab is selected.

[0131] When the face seal tab is selected, the face seal tool is selected. When the face seal tool is selected, buttons 311-1 to 311-5 and the like used to select face seal content are displayed in palette area 304L. Various face seal content such as those described with reference to FIG. 5 is selected using the display in palette area 304L.

[0132] The right area used by the right user has the same configuration as the left area.

[0133] Returning to the description of FIG. 14, in step S17, the portable image generating section 224 determines whether or not the face seal tool has been selected.

[0134] If it is determined in step S17 that the face sticker tool has been selected, in step S18, the conversion unit 242 converts the content for the face sticker based on the face model of the user selected as the target for compositing from among the users appearing in the photographed image that is the target of the graffiti.

[0135] In step S19, composition unit 234 composites the converted face sticker content, so that the face sticker content is composited and displayed on the face of the user appearing in the captured image displayed in doodle input areas 301L and 301R.

[0136] In step S20, the retouching processor 231 determines whether retouching has been performed. Retouching is performed when the eye size tab or small face tab provided in the palette areas 304L and 304R is selected.

[0137] If it is determined in step S20 that retouching has been performed by the user, then in step S21 the retouching processor 231 performs retouching in response to the user's operation.

[0138] In step S22, the 3D model generation unit 241 identifies the facial area of ​​the user appearing in the photographed image after the retouching process, and generates a new facial model.

[0139] In step S23, the conversion unit 242 converts the face sticker content that is the same as the one that has already been synthesized, based on the new face model.

[0140] In step S24, composition unit 234 composites the converted face sticker content. As a result, the composite face sticker content displayed in doodle input areas 301L and 301R is displayed deformed to follow the three-dimensional shape of the retouched face.

[0141] In step S25, the portable image generation unit 224 performs image processing on the photographed image of the graffiti target in response to an operation by the user using a tool other than the face seal tool, and generates a portable image. Graffiti using another tool is further applied to the photographed image of the graffiti target that has been processed with the face seal tool.

[0142] Similarly, if it is determined in step S17 that the face seal tool has not been selected, or if it is determined in step S20 that retouching has not been performed, a portable image is generated in step S25.

[0143] In step S26, the portable image generating unit 224 determines whether the portable doodle has ended. If it is determined in step S26 that the portable doodle has not ended, the process returns to step S16, and the portable doodle continues. For example, if a preset time limit has elapsed, the portable doodle ends.

[0144] If it is determined in step S26 that the mobile graffiti has been completed, in step S27, the print processing unit 214 starts printing the sticker image generated by the sticker graffiti. The timing to start printing can be any timing as long as it is after the sticker graffiti has been created.

[0145] In step S28, the transmission processing unit 215 transmits the portable image to the image management server. When the transmission of the portable image is completed, the process returns to step S3 in Fig. 12, and the series of processes of the photo sticker creation game is completed.

[0146] The above process allows for the synthesis of three-dimensional face sticker content onto the user's face in the captured image. Because the conversion is performed according to the three-dimensional shape of the destination position, it is possible to synthesize the face sticker content in a more natural state without any sense of incongruity.

[0147] Furthermore, by recreating the face model each time retouching is performed, it is possible to maintain a more natural appearance even if the face is deformed by retouching.

[0148] The captured image used to generate the face model is an image taken using a strobe light to ensure sufficient brightness, making it suitable for image processing. A highly accurate face model can be generated based on the image suitable for image processing and used to convert face sticker content.

[0149] FIG. 16 is a diagram showing another example of content for a face sticker.

[0150] 16 is an image in which the word "LOVE" is placed inside a speech bubble. In this way, an image including text is also prepared as face sticker content.

[0151] When a photographed image is the subject of graffiti and the face of user U1 is selected as the target for compositing face sticker content, the face sticker content C11 is converted and composited to have a sense of depth based on the face model of user U1, as shown at the tip of arrow #11.

[0152] As indicated by the tip of arrow #12, the face sticker content C11 after synthesis is displayed in a natural manner, as if stickers were stuck along the cheeks of the face of the user U1.

[0153] <Modification> ·Facial model generation The generation of a face model may be performed in an external server rather than in the photo sticker creating device 1. In this case, the 3D model generating unit 241 (FIG. 7) transmits a face image extracted from a photographed image to the external server, and acquires the face model transmitted from the external server.

[0154] - Composite position of face seal content Although the combining position is set in advance for each type of face sticker content, it may be combined at a position selected by the user. In this case, the conversion unit 242 converts the face sticker content according to the selected position when the combining position of the face sticker content is selected.

[0155] 3D model examples A 3D model representing the three-dimensional shape of a specific body part other than the face may be generated and used for converting the content. For example, 3D models may be generated for various body parts such as the head, neck, arms, and legs as specific body parts.

[0156] Although the conversion of the face sticker content is performed based on a 3D model as three-dimensional information generated for each user, it may also be performed based on other three-dimensional information that represents the three-dimensional shape of each user's face.

[0157] ·others Although the face sticker tool is provided as an editing tool for doodling on a mobile device, it may also be provided as an editing tool for doodling on a sticker. That is, as output destinations for images generated by a game, an output destination for outputting to a mobile device as a mobile image and an output destination for outputting to sticker paper as a sticker image are provided, and doodling using the face sticker tool may be performed when creating the latter output destination for the sticker image. It is possible to doodling using the face sticker tool on at least one of the mobile image and the sticker image.

[0158] The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0159] The program to be installed is provided by being recorded on removable media such as an optical disc (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in a ROM or memory unit.

[0160] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0161] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0162] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology. [Explanation of symbols]

[0163] 1 Photo sticker creation device, 201 control unit, 211 pre-selection processing unit, 212 photography processing unit, 213 editing processing unit, 214 print processing unit, 215 transmission processing unit, 221 image analysis unit, 222 sticker image generation unit, 223 sticker design data storage unit, 224 portable image generation unit, 225 display control unit, 231 retouch processing unit, 232 face sticker processing unit, 233 content storage unit, 234 synthesis unit, 241 3D model generation unit, 242 conversion unit

Claims

1. a generating unit that generates, for each of a plurality of users, three-dimensional information representing a three-dimensional shape of a specific part of each of the users appearing in the photographed image; a display control unit that displays a first screen for a first image processing, and, after the user has completed the operation using the first screen, displays a second screen for a second image processing, the second screen including information used for retouching the specific portion; a first image processing unit that performs the first image processing on the captured image in response to an operation by the user on the first screen; a second image processing unit that performs the second image processing on the captured image in response to an operation by the user on the second screen, the second image processing including a retouching process on the specific part and a process of compositing the compositing image transformed based on the three-dimensional information onto the specific part of the user that is to be composited; An image processing device comprising:

2. the display control unit displays a third screen used for retouching the specific part; the second image processing unit retouches the specific portion in response to an operation by the user on the third screen; The generating unit generates the three-dimensional information based on the retouched captured image. The image processing device according to claim 1 .

3. The display control unit displays the first screen after the user completes the operation using the third screen. The image processing device according to claim 2 .

4. The generating unit generates the three-dimensional information using the face of the user as the specific part. The image processing device according to claim 1 .

5. The second image processing unit transforms the compositing image selected by the user based on the three-dimensional information of the user selected by the user as a composite image. The image processing device according to claim 1 .

6. The generating unit generates the three-dimensional information representing the three-dimensional shape of the specific part after retouching, each time the specific part is retouched in response to an operation by the user on the second screen. The image processing device according to claim 1 .

7. When retouching is performed on any one of the users appearing in the photographed image, the generation unit generates the three-dimensional information representing the three-dimensional shape of the specific part of the user that has been retouched. The image processing device according to claim 6 .

8. The generation unit generates the three-dimensional information for each user based on each of the plurality of captured images obtained by capturing images a plurality of times.

8. The image processing device according to claim 1.

9. The image processing device generating three-dimensional information representing the three-dimensional shape of a specific part of each of a plurality of users appearing in the photographed image for each of the users; displaying a first screen for a first image processing; performing the first image processing on the captured image in response to an operation by the user on the first screen; after the user has completed the operation using the first screen, displaying a second screen for second image processing, the second screen including information used for retouching the specific portion; The second image processing, which includes a process of retouching the specific part and a process of compositing the compositing image transformed based on the three-dimensional information onto the specific part of the user as a compositing destination, is performed on the captured image in response to an operation by the user on the second screen. Image processing methods.

10. On the computer, generating three-dimensional information representing the three-dimensional shape of a specific part of each of a plurality of users appearing in the photographed image for each of the users; displaying a first screen for a first image processing; performing the first image processing on the captured image in response to an operation by the user on the first screen; after the user has completed the operation using the first screen, displaying a second screen for second image processing, the second screen including information used for retouching the specific portion; The second image processing, which includes a process of retouching the specific part and a process of compositing the compositing image transformed based on the three-dimensional information onto the specific part of the user as a composite destination, is performed on the captured image in response to an operation by the user on the second screen. A program for executing a process.

Images