Image processing device, image processing method, and program

Abstract

To make it possible to adjust the reflection of a bangs part shown in a captured image.SOLUTION: An image processing device according to one aspect of the present technique clips, as an image for synthesis, an image of a range according to a selection operation performed by a user, within a selectable range set in part of an area of a person shown as a subject in a captured image, and synthesizes and displays the image for synthesis at an arbitrary position on the captured image selected by the user. The present technique can be applied to a photo-creating game machine that provides a series of games in which captured images are printed on stickers.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to an image processing device, an image processing method, and a program that are capable of adjusting the appearance of a bangs portion appearing in a captured image. [Background technology]

[0002] Photo sticker creation devices are known in the art. These devices photograph a user, allow the user to edit the photographed image, and print the edited image on sticker paper. These photo sticker creation devices are installed in amusement facilities and the like.

[0003] The flow of a game provided by the photo sticker creation device typically involves taking a photo of the user in the photography space as the subject, then moving the user to the editing space, editing the image according to the operations performed in the editing space, and printing the edited image on sticker paper.

[0004] After moving from the shooting space to the editing space, the captured image is edited using an editing screen displayed on a display in the editing space. The editing screen displays the captured image to be edited, as well as buttons used to select various editing tools.

[0005] For example, Patent Document 1 describes a technology that allows an arbitrary range to be specified using a copy range frame, and allows multiple composite stamp images included in the specified range to be copied to another location as a single stamp image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-109579 Summary of the Invention [Problem to be solved by the invention]

[0007] In the technology described in Patent Document 1, the content that can be copied is a stamp image. A user cannot copy any part of a person in a photographed image and paste it in another position.

[0008] The present technology has been developed in view of such circumstances, and makes it possible to adjust the appearance of the bangs in a photographed image. [Means for solving the problem]

[0009] An image processing device according to one aspect of the present technology includes a cutout unit that cuts out an image of a range corresponding to a selection operation performed by a user within a selectable range set in a part of the area of ​​a person appearing as a subject in a captured image as an image to be synthesized, and a display control unit that synthesizes and displays the image to be synthesized at any position on the captured image selected by the user.

[0010] In one aspect of the present technology, an image within a range corresponding to a selection operation performed by a user within a selectable range set in a portion of the area of ​​a person appearing as a subject in a captured image is cut out as an image to be synthesized, and the image to be synthesized is synthesized at any position on the captured image selected by the user. [Effects of the Invention]

[0011] According to the present technology, it is possible to adjust the appearance of the bangs in a photographed image. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a perspective view showing an appearance of a photo sticker creation device according to an embodiment of the present technology; [Figure 2] FIG. 2 is a perspective view showing an example of the front configuration of the imaging unit. [Figure 3] 1 is a block diagram showing an example of the configuration of a photo sticker creation device; [Figure 4]FIG. 2 is a block diagram illustrating an example of a functional configuration of the photo sticker creating device. [Figure 5] FIG. 10 is a diagram illustrating a bangs retouch function. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an edit processing unit. [Figure 7] FIG. 2 is a block diagram illustrating an example of the configuration of a bangs retouching processing unit. [Figure 8] 10 is a flowchart illustrating the operation of the photo sticker creating device. [Figure 9] 9 is a flowchart illustrating the editing process performed in step S3 of FIG. 8. [Figure 10] FIG. 10 is a diagram showing a display example of a mobile graffiti screen. [Figure 11] FIG. 11 is an enlarged view of the palette area of ​​FIG. 10. [Figure 12] 10A and 10B are diagrams illustrating an example of an operation using a clipping pen for bangs retouching [Figure 13] 10A and 10B are diagrams illustrating another example of an operation using the clipping pen for bangs retouching [Figure 14] FIG. 10 is an enlarged view of a bangs image. [Figure 15] FIG. 10 is a diagram illustrating reduction and rotation of a bangs image. [Figure 16] 10A and 10B are diagrams illustrating examples of selecting a position to paste a bangs image. [Figure 17] FIG. 10 is a diagram illustrating an example of combining a bangs image. [Figure 18] FIG. 10 is a diagram showing another example of selection of the cutout range. [Figure 19] 10A and 10B are diagrams illustrating an example of an operation using a dedicated eraser. [Figure 20] FIG. 10 is a diagram showing an example of a photographed image on which a stamp image is combined. [Figure 21] 10A and 10B are diagrams illustrating an example of an operation using a normal eraser. [Figure 22] 10A and 10B are diagrams illustrating an example of an operation using a dedicated eraser. [Figure 23] FIG. 10 is a diagram illustrating an example of a message. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Configuration of the photo sticker creation device> FIG. 1 is a perspective view showing the appearance of a photo sticker creating device according to an embodiment of the present technology.

[0014] The photo sticker creation device 1 is a game machine that provides a photo creation game, and is installed in a store such as a game center.

[0015] Users of the photo sticker creation device 1 are mainly high school girls and young women. Multiple users, usually two or three people per group, can enjoy the game with the photo sticker creation device 1. Of course, a single user can also enjoy the game with the photo sticker creation device 1.

[0016] In the photo creation game of the photo sticker creation device 1, the user takes a photo of themselves as the subject. The user then edits the captured image to combine it with composite images such as handwritten characters and stamp images, and edits the captured image into a colorful image. The user then receives a sticker sheet on which the edited captured image is printed, completing the series of games.

[0017] A pre-operation unit 11 is provided on the side of the photo sticker creation device 1, which has a box-shaped housing. The pre-operation unit 11 is a device used for operations before photography, such as selecting the number of users to play with and selecting the photography course. The pre-operation unit 11 is provided with a touch panel monitor 11A. Below the touch panel monitor 11A, a coin slot 11B is provided for paying the play fee.

[0018] A screen used for various selections is displayed on the touch panel monitor 11A. After making various selections according to the display on the touch panel monitor 11A, the user enters the photography space through the entrance / exit G and begins photography. A photography unit 21 is provided in front of the photography space. The space between the photography unit 21 and the background unit 22 constitutes the photography space. A box-shaped camera unit 31 with a built-in camera is provided in the photography unit 21.

[0019] An editing unit 51 is provided to the left of the pre-operation unit 11. The editing unit 51 is a device used to edit captured images obtained by taking pictures with a camera. A user who has taken pictures in the shooting space moves to the editing space in front of the editing unit 51 and edits the captured images. A tablet-integrated monitor 61, which is a monitor for editing the captured images, is provided on a slope formed above the editing unit 51. A screen used for editing the captured images is displayed on the tablet-integrated monitor 61.

[0020] Two editing spaces are prepared, one on either side of the housing of the editing unit 51. An editing space is formed in the space in front of the editing unit 51 shown in Fig. 1, and the other on the back side of the editing unit 51. This allows a group of users who have finished shooting in the shooting space to move to the vacant editing space, and edit in parallel in the two editing spaces.

[0021] 2 is a perspective view showing an example of the front configuration of the imaging unit 21. The imaging unit 21 is configured so as to be surrounded by a side panel 21A, a side panel 21B, and a front panel 21C.

[0022] A camera unit 31 is provided at a position slightly above the center of the front of the photographing section 21. The camera unit 31 is provided with a camera 41 and a touch panel monitor .

[0023] Camera 41 is, for example, a single-lens reflex camera, and is attached inside camera unit 31 so that the lens is exposed. Camera 41 has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures images of users in the imaging space. Camera 41 functions as an imaging unit that captures images of users as subjects.

[0024] Moving images (hereinafter also referred to as live view images) captured by the camera 41 are displayed in real time on the touch panel monitor 42. Still images captured by the camera 41 at a predetermined timing, such as when a shooting instruction is issued, are saved as captured images.

[0025] The touch panel monitor 42 is provided below the camera 41. The touch panel monitor 42 is composed of a monitor such as an LCD and a touch panel stacked on it. The touch panel monitor 42 has a function as a live view monitor that displays moving images captured by the camera 41, and a function that displays various GUIs (Graphical User Interfaces) and accepts selection operations by the user.

[0026] An upper strobe unit 32 with a curved light-emitting surface facing the user is provided above the camera unit 31. The upper strobe unit 32 irradiates light from above and in front of the user, mainly toward the user's face and upper body.

[0027] A lower box strobe unit 33 that irradiates the user with light from below is provided below the camera unit 31. A strobe light source is provided in each of the upper strobe unit 32 and the lower box strobe unit 33. The strobe light source emits light in synchronization with the timing of capturing a still image, irradiating the user, who is the subject, with strobe light.

[0028] A box-shaped portion 34L is provided between the lower box strobe unit 33 and the side panel 21A. A box-shaped portion 34R is provided between the lower box strobe unit 33 and the side panel 21B. The top surfaces of the box-shaped portions 34L and 34R are substantially horizontal and are used as a luggage storage area for users performing photography work to place their baggage and the like.

[0029] <Internal configuration of the photo sticker creation device> Fig. 3 is a block diagram showing an example of the configuration of the photo sticker creating device 1. In Fig. 3, the same components as those described above are given the same reference numerals, and duplicated explanations will be omitted where appropriate.

[0030] 3, the photo sticker creation device 1 is configured by connecting a pre-operation unit 11, a photographing unit 21, an editing unit 51, and a printing unit 103 to a PC (Personal Computer) unit 101 via an external input / output interface 102. The PC unit 101 is housed inside the housing of the editing unit 51, for example.

[0031] The PC section 101 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113, which are interconnected by a bus 114.

[0032] The CPU 111 executes a predetermined program and controls the overall operation of the photo sticker creating device 1. The ROM 112 stores the programs and data executed by the CPU 111. The RAM 112 temporarily stores the data and programs processed by the CPU 111.

[0033] An input / output interface 115 is further connected to the bus 114. To the input / output interface 115, a storage unit 116, a communication unit 117, and a drive 118 are connected.

[0034] The storage unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The storage unit 116 stores various setting information and the like supplied from the CPU 111. The information stored in the storage unit 116 is read out by the CPU 111 as appropriate.

[0035] The communication unit 117 is an interface for a network such as the Internet. The communication unit 117 communicates with external devices under the control of the CPU 111. The communication unit 117 transmits an image selected by a user to, for example, a server managed by the manufacturer of the photo sticker creating device 1. The image transmitted from the communication unit 117 is allocated a predetermined storage area and saved, and is then displayed on or downloaded to a mobile terminal that has accessed the server.

[0036] Removable media 119, such as an optical disk or semiconductor memory, is appropriately attached to drive 118. Programs and data read from removable media 119 by drive 118 are supplied to CPU 111, where they are stored in storage unit 116 or installed.

[0037] The input / output interface 115 is connected to an external input / output interface 102. The PC unit 101 controls each unit via the external input / output interface 102.

[0038] The pre-operation unit 11 performs a pre-selection process for the user before photographing. The pre-operation unit 11 is composed of a touch panel monitor 11A, a speaker 121, and a coin processing unit 122.

[0039] The touch panel monitor 11A displays various selection screens and accepts user operations on the selection screens under the control of the PC unit 101. An input signal representing the content of the user operation is supplied to the PC unit 101, and various settings are made.

[0040] The coin processing unit 122 detects the insertion of coins into the coin insertion slot 11B. When the coin processing unit 122 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the start of the game.

[0041] The photographing unit 21 performs photographing processing of a user in a photographing space. The photographing unit 21 is composed of a lighting device 131, a camera 41, a touch panel monitor 42, and a speaker 132.

[0042] The lighting devices 131 are strobe units in the shooting space, and emit light in accordance with a lighting control signal supplied from the PC section 101. The lighting devices 131 include strobe units provided in the shooting space, such as the upper strobe unit 32 and the lower box strobe unit 33.

[0043] The camera 41 takes a photograph in accordance with shutter control by the PC section 101, and outputs image data obtained by the photograph to the PC section 101.

[0044] The editing unit 51 performs editing processing after shooting. The editing unit 51 is composed of a tablet built-in monitor 61, touch pens 141A and 141B, and a speaker 142. A set of the tablet built-in monitor 61 and the touch pens 141A and 141B is provided on two sides of the housing of the editing unit 51. When there is no need to distinguish between the touch pen 141A and the touch pen 141B, they will be collectively referred to as touch pen 141 as appropriate.

[0045] The tablet built-in monitor 61 displays an editing screen and accepts user operations on the editing screen under the control of the PC unit 101. An input signal representing the content of the user's operation is supplied to the PC unit 101, and the captured image to be edited is edited.

[0046] The printing unit 103 is configured to include a printer 151. A sticker paper unit 152 is attached to the printer 151.

[0047] Printer 151 prints an image on a sticker paper stored in sticker paper unit 152 based on print data supplied from PC section 101. Printer 151 ejects the sticker paper on which the image is printed to a sticker paper ejection port.

[0048] <Example of control unit functional configuration> 4 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. At least some of the functional units shown in FIG. 4 are realized by the CPU 111 in FIG. 3 executing a predetermined program.

[0049] A control unit 201 is realized in the photo sticker creating device 1. The control unit 201 is made up of a pre-selection processing unit 211, a photography processing unit 212, an edit processing unit 213, a print processing unit 214, and a transmission processing unit 215.

[0050] The pre-selection processing unit 211 executes pre-selection processing by controlling each unit of the pre-operation unit 11. The pre-selection processing involves the selection of the number of users, the selection of a photography course, etc. Information representing the selection made by the user is supplied to the photography processing unit 212.

[0051] The shooting processing unit 212 performs shooting processing by controlling each unit of the shooting unit 21. In the shooting processing, multiple shot images, such as six shot images, are taken. The shooting processing unit 212 outputs data of the multiple shot images obtained by the shooting processing to the editing processing unit 213.

[0052] The editing processing unit 213 performs editing processing by controlling each unit of the editing unit 51. The editing processing includes editing the captured image, printing the image on sticker paper, and sending the image to an external image management server.

[0053] The images sent to the image management server can be viewed or downloaded by the user on a mobile device such as a smartphone after the user has finished playing the game on the photo sticker creation device 1.

[0054] Editing the photographed image includes a process of generating an image to be printed on sticker paper based on the photographed image, and a process of generating an image to be sent to the image management server based on the photographed image.

[0055] The image to be printed on the sticker paper is generated by having the user select a sticker design and then performing image processing according to the sticker design selected by the user. When generating the image to be printed on the sticker paper, it is possible to combine a design image with the sticker design.

[0056] On the other hand, portable images, which are images to be sent to the image management server, are generated by applying image processing to the photographed image to be edited in accordance with operations performed using a tool selected by the user from pre-prepared editing tools, such as a pen tool or stamp tool. By using the pen tool or stamp tool, a pen image or stamp image, which serves as a composite image, is composited into the photographed image to be edited. The user selects their preferred editing tool and selects the position on the photographed image to be edited where the image processing will be applied, thereby proceeding with editing the photographed image.

[0057] In this way, editing of a photographed image, particularly for creating a portable image, is carried out by the user using the touch pen 141 as if doodling on the photographed image. Hereinafter, editing will be referred to as doodling where appropriate. Editing of a photographed image to create an image to be printed on sticker paper will be referred to as doodling for stickers, and editing of a photographed image to create a portable image will be referred to as doodling for mobile phones.

[0058] As will be explained in detail later, a bangs retouching function is provided as a mobile doodling tool. The bangs retouching function is used to adjust the bangs of a person in a photograph. By using the bangs retouching function, it is possible to add more bangs or lengthen them, as shown in the dashed line at the bottom of Figure 5.

[0059] The bangs are retouched using a touch pen 141. The bangs retouch function may be used by touching the screen with a finger.

[0060] The edit processing unit 213 in FIG. 4 outputs the print data generated by the process of the graffiti sticker to the print processing unit 214, and outputs the data of the portable image to the transmission processing unit 215.

[0061] The print processing unit 214 outputs the print data to the printer 151, which prints the image on the sticker paper.

[0062] The transmission processing unit 215 controls the communication unit 117 to communicate with the image management server, and transmits (uploads) portable images to the image management server.

[0063] FIG. 6 is a block diagram showing an example of the configuration of the edit processing unit 213.

[0064] The editing processing unit 213 is made up of an image analysis unit 221, a sticker image generation unit 222, a sticker design data storage unit 223, a portable image generation unit 224, and a display control unit 225. The captured image data output from the capture processing unit 212 is input to the image analysis unit 221 and the display control unit 225.

[0065] The image analysis unit 221 analyzes the features of the user that appear in each captured image and recognizes the position of each part of the user that appears in the captured image. For example, the positions of the head, eyes, nose, mouth, and ears are recognized. The position of the contour points of each part is also recognized. The shape of each part is identified based on the positions of the contour points. Information representing the recognition results by the image analysis unit 221 is supplied to the sticker image generation unit 222 and the portable image generation unit 224 together with the captured image data.

[0066] The sticker image generating unit 222 reads information about the sticker design selected by the user from the sticker design data storage unit 223, and performs image processing based on the read information on the captured image. For example, an image for a sticker is generated by combining an image of the design selected by the user.

[0067] Print data for the sticker image is supplied to the print processing unit 214. Information indicating the content of image processing by the sticker image generation unit 222 is supplied to the display control unit 225 as appropriate and used for screen display. The image processing by the sticker image generation unit 222 is processing for creating an image to be printed on sticker paper.

[0068] The portable image generation unit 224 performs image processing on each captured image according to the portable graffiti made by the user to generate a portable image. The portable graffiti is graffiti used to create an image to be sent to the image management server. The image with the portable graffiti is a transmission image used to send to the image management server.

[0069] The portable image generated by the portable image generation unit 224 is supplied to the transmission processing unit 215. Information indicating the content of image processing by the portable image generation unit 224 is supplied to the display control unit 225 as appropriate and used for screen display.

[0070] The portable image generating unit 224 includes a bangs retouching processing unit 231. The bangs retouching processing unit 231 performs image processing for bangs retouching in response to a user's operation using a bangs retouching tool.

[0071] The display control unit 225 displays various screens including a sticker doodle screen and a mobile doodle screen on the tablet built-in monitor 61. The display control unit 225 controls the display of the screen in response to the user's operation.

[0072] FIG. 7 is a block diagram showing an example of the configuration of the bangs retouching processing unit 231. As shown in FIG.

[0073] Bangs retouching processing section 231 is made up of cut-out section 241 and pasting section 242. Data of the photographed image of the graffiti target output from image analysis section 221 is input to cut-out section 241 and pasting section 242, respectively.

[0074] The cropping unit 241 sets a selectable range for each captured image. As will be described in detail later, the selectable range is a range within which a cropping range can be selected. The selectable range is set, for example, to a portion of the area of ​​a person appearing in the captured image based on the characteristics of the person represented by the analysis results of the image analysis unit 221. The portion of the area is an area that includes an area showing bangs.

[0075] If multiple people appear in a single captured image, a selectable range is set for each person's area.

[0076] Furthermore, cutout unit 241 cuts out the front hair portion of a person appearing in the photographed image that is the graffiti target as an image to be synthesized in response to a user's operation using touch pen 141. Here, cutting out the image to be synthesized is a process of copying the image of the front hair portion selected by the user while leaving the original front hair portion appearing in the photographed image as is.

[0077] The compositing image of the bangs portion cut out by the cutout unit 241 is supplied to the pasting unit 242. Hereinafter, the image of the bangs portion cut out as the compositing image will be referred to as the bangs image where appropriate.

[0078] The pasting unit 242 composites the bangs image cut out by the cutting unit 241 onto the photographed image of the graffiti target. After the operation for cutting out the bangs part is performed, the user selects the position where the bangs will be combined.

[0079] <Operation of the photo sticker creation device> Here, the operation of the photo sticker creating device 1 that provides the photo sticker creating game will be described with reference to the flowchart in Fig. 8. The process in Fig. 8 starts when coins worth a predetermined amount, such as 400 yen, are inserted.

[0080] In step S1, the pre-selection processing unit 211 performs a pre-selection process, which includes selecting the number of users, the game course, and the like.

[0081] In step S2, the photography processing unit 212 controls the photography unit 21 to perform photography processing. Specifically, the photography processing unit 212 displays a live view image captured by the camera 41 on the touch panel monitor 42. When the photography timing arrives, the photography processing unit 212 photographs a user in the photography space as a subject and generates a photographed image, for example, a close-up image. The close-up image is a still image in which the upper half of the user's body is shown prominently.

[0082] In step S3, the editing processing unit 213 performs editing processing by controlling the configuration of the editing unit 51 corresponding to the editing space to which the user has moved after completing the shooting processing.

[0083] Editing process Next, the editing process performed in step S3 of FIG. 8 will be described with reference to the flowchart of FIG.

[0084] In step S11, display control unit 225 of edit processing unit 213 displays a sticker doodle screen. The sticker doodle screen displays a photographed image to be doodled, as well as various buttons such as a button to be operated when selecting a sticker design and a button to be operated when selecting an image to be combined with the photographed image.

[0085] In step S12, the sticker image generating unit 222 performs image processing on the photographed image in accordance with the user's operation, and generates a sticker image.

[0086] In step S13, the sticker image generating unit 222 determines whether the sticker graffiti has ended. If it is determined in step S13 that the sticker graffiti has not ended, the process returns to step S11, and the sticker graffiti continues. For example, if a preset time limit has elapsed, the sticker graffiti ends.

[0087] In step S14, the display control unit 225 displays the mobile doodle screen on the tablet built-in monitor 61. The user uses the mobile doodle screen to doodle on the captured image.

[0088] FIG. 10 is a diagram showing a display example of the mobile graffiti screen.

[0089] 10, the mobile scribble screen is basically configured by providing each display symmetrically on either side of the central thumbnail display area 303. The left area is used by a user standing on the left side facing the tablet-built-in monitor 61. The right area is used by a user standing on the right side facing the tablet-built-in monitor 61.

[0090] It should be noted that the rectangular dashed lines surrounding each area on the mobile doodling screen shown in FIG. 10 are not actually displayed on the screen.

[0091] The thumbnail display area 303, located in the center of the mobile doodle screen, is a display area for thumbnail images representing captured images. The user uses the thumbnail images displayed in the thumbnail display area 303 to select a captured image on which to doodle.

[0092] Thumbnail images representing all captured images are displayed in the thumbnail display area 303. In the example of Fig. 10, thumbnail images of six close-up images are displayed. In the example of Fig. 10, the user on the left has selected the thumbnail image on the top left, and the user on the right has selected the thumbnail image on the bottom right.

[0093] Doodle input area 301L is provided approximately in the center of the left area. Doodle input area 301L is a display area for a captured image that is selected using a thumbnail image and is to be doodled on. The user can select an editing tool using touch pen 141 and edit the captured image displayed in doodle input area 301L.

[0094] To the right of doodle input area 301L is provided tool area 302L, which contains an array of buttons such as a button to be pressed to select the eraser tool, a button to be pressed to reset the doodle, and a button to be pressed to return to the previous operation.

[0095] Tool area 302L displays buttons including selection button 302-1 and selection button 302-2. Selection button 302-1 is a button that is pressed to select the eraser tool, and selection button 302-2 is a button that is pressed to select the selection tool.

[0096] The eraser tool (normal eraser) is a tool used, for example, to erase a composite image that has already been combined. The selection tool is a tool used, for example, to select and move a composite image that has already been combined. The composite image is moved using the selection tool by pressing the tip of the stylus 141 against the composite image and moving it in that state, a so-called drag-and-drop operation.

[0097] Below the thumbnail display area 303, selection buttons 305 are displayed that are pressed to select various editing tools. The user can use the selection buttons 305 to switch between the editing tools displayed in the palette areas 304L and 304R. In the example of FIG. 10, selection buttons are provided for the editing tools retouching, makeup, headband, stamp, and pen. Stamp is selected as the editing tool to be displayed in the palette area 304L, and retouching is selected as the editing tool to be displayed in the palette area 304R.

[0098] Palette area 304L is provided below graffiti input area 301L. Palette area 304L is an area where buttons used to select various editing tools to be combined with the captured image displayed in graffiti input area 301L are displayed. In the example of Fig. 10, the button used to select the stamp tool is displayed in palette area 304L.

[0099] Tabs used to select a stamp tool category are provided above the palette area 304L. The user can use the tabs to switch between stamp tool categories displayed in the palette area 304L.

[0100] Similar displays are also provided on the right side of the mobile doodling screen. In the example of Figure 10, buttons used to select a retouching tool are displayed in palette area 304R. When a retouching tool is selected, the bangs retouching function described above becomes available.

[0101] FIG. 11 is an enlarged view of the palette area 304R of FIG.

[0102] Tabs used to select categories are provided above the palette area 304R. The user can use the tabs to switch between tools displayed in the palette area 304R. In the example of Figure 11, tabs for bangs, small face, eye size, and skin tone are provided, and the bangs tab is selected.

[0103] A cutout pen and a dedicated eraser are provided as tools for bangs retouching. Selection button 311 is a button used to select the thickness of the cutout pen. Selection button 312 is a button used to select the thickness of the dedicated eraser.

[0104] In the example of Figure 11, three different thicknesses are provided for the cut-out pen and the dedicated eraser. The cut-out pen is a tool used to select the bangs portion to be cut out as the bangs image. The dedicated eraser is a tool used to erase the bangs image pasted on the captured image.

[0105] An explanation on how to use the bangs retouch function is displayed below the selection buttons 311 and 312. The explanation on how to use the bangs retouch function is displayed, which explains how to use the bangs retouch function by tracing the part you want to cut out with a pen and placing the cut-out image in the desired position. The user will read the explanation on how to use the bangs retouch function and use the cut-out pen and dedicated eraser to retouch the bangs.

[0106] When bangs retouching is performed, image processing is performed on the photographed image selected as the subject of doodles, and the result is reflected in the display of doodle input area 301R. In doodle input area 301R, the photographed image showing the user with the bangs changed is displayed.

[0107] Returning to the explanation of FIG. 9, in step S15, the portable image generating unit 224 determines whether or not bangs retouching has been selected.

[0108] If it is determined in step S15 that bangs retouching has been selected, in step S16, bangs retouching processor 231 performs image processing in response to the user's operation using the bangs retouching tool. Details of the image processing performed in response to the operation using the bangs retouching tool will be described later.

[0109] In step S17, portable image generation unit 224 performs image processing on the photographed image to be graffitied in response to an operation by the user using a tool other than the bangs retouching tool, and generates a portable image. Graffiti is further applied using another tool to the photographed image to be graffitied that has been subjected to the bangs retouching tool.

[0110] If it is determined in step S15 that bangs retouching has not been selected, the process of step S16 is skipped, and in step S17, image processing is performed in accordance with an operation using a tool other than bangs retouching.

[0111] In step S18, the portable image generating unit 224 determines whether the portable doodle has ended. If it is determined in step S18 that the portable doodle has not ended, the process returns to step S14, and the portable doodle continues. For example, if a preset time limit has elapsed, the portable doodle ends.

[0112] If it is determined in step S18 that the mobile graffiti has been completed, in step S19, the print processing unit 214 starts printing the sticker image generated by the sticker graffiti. The timing to start printing can be any timing as long as it is after the sticker graffiti has been created.

[0113] In step S20, the transmission processing unit 215 transmits the portable image to the image management server. When the transmission of the portable image is completed, the process returns to step S3 in Fig. 8, and the series of processes of the photo sticker creation game is completed.

[0114] <Basic bangs retouching functions> FIG. 12 is a diagram showing an example of an operation using a clipping pen for bangs retouching.

[0115] Fig. 12 shows only the head and surrounding areas of a person appearing in a photographed image on which graffiti is to be added. Bangs retouching will be explained using an image different from the image on the graffiti screen in Fig. 10. This also applies to other figures described later. Each function of bangs retouching is operated using the touch pen 141, just like the other functions provided on the graffiti screen.

[0116] 12, the operation of using the cut-out pen for bangs retouching is performed, for example, by pressing the tip of stylus 141 at position p1 and then moving it in that state to position p2 as indicated by arrow #1 in the lower part. In other words, the cut-out operation using the cut-out pen is performed by tracing the bangs part of the person appearing in the photographed image to be doodled with stylus 141.

[0117] When the operation of tracing the front hair portion is performed, the range along the trajectory from position p1 to position p2 and the range according to the thickness of the selected cutting-out pen are selected as the cutting-out range. For example, the middle thickness of the three types (thick, medium, thin) is selected as the default thickness.

[0118] Here, the selectable range that can be selected as the cut-out range is limited to, for example, the range from the forehead to the top of the head. In the example of Fig. 12, the range above the dashed dotted line L in the person area is set as the selectable range.

[0119] When the tip of the touch pen 141 is removed from the screen at position p2, an image of the bangs portion that appears in the range selected as the cut-out range is cut out as bangs image P1 and displayed superimposed on the cut-out range, as shown in FIG. 13.

[0120] 13, bangs image P1 is displayed superimposed on the trajectory from position p1 to position p2, which is the cut-out range, but it may also be displayed at a position shifted from the cut-out range. Note that the outline of bangs image P1 is shown with a dashed line, but the line representing such an outline may or may not be displayed.

[0121] FIG. 14 is an enlarged view of the bangs image P1.

[0122] 14, bangs image P1 is an image of the bangs portion captured in the cutout range selected by the user, and is used as the compositing image.

[0123] A cancel button 321 is displayed in the upper left of the bangs image P1, and a transform button 322 is displayed in the lower right. For example, the cancel button 321 and the transform button 322 are displayed at the top left and bottom right vertices of a rectangular area surrounding the bangs image P1. The cancel button 321 is a button that is operated to stop compositing using the bangs image P1. The transform button 322 is a button used to enlarge, reduce, rotate, etc. the bangs image P1.

[0124] As the user moves the stylus 141 while holding down the transformation button 322, the bangs image P1 is displayed by being enlarged, reduced, rotated, etc., as shown in Fig. 15. Fig. 15A shows the bangs image P1 in a reduced state, and Fig. 15B shows the bangs image P1 in a rotated state.

[0125] The user can adjust the size and orientation of the cut-out bangs image P1 by scaling it or rotating it. In addition to scaling and rotation, other adjustments such as horizontal flipping may also be possible.

[0126] FIG. 16 is a diagram showing an example of selecting the position where the bangs image P1 is pasted.

[0127] After selecting the cut-out range as described above, the user can select a position on bangs image P1 with touch pen 141 and then move it to any position in that state to select a paste position. In the example of Fig. 16, position p11 on bangs image P1 is selected as shown in the upper part, and then bangs image P141 is moved to position p12 on the right as shown in the lower part. The bangs image P1 moves and is displayed in accordance with the movement of bangs image 141.

[0128] When the user removes the tip of touch pen 141 from the screen at position p12, bangs image P1 is composited (pasted) at position p12. The user can also shift the composite position of bangs image P1 by pressing bangs image P1 again with stylus 141 and moving it to another position while keeping it pressed.

[0129] The bangs image P1 may be composited at any position on the photographed image of the graffiti target, or the composite position may be limited to positions within the same range as the selectable range.

[0130] If a plurality of people appear in the photographed image on which the graffiti is to be taken, the composition position may be limited to a position within the range of the person from which the bangs image was cut out.

[0131] In this case, for example, if two people, user A and user B, appear in the photographed image to be graffitied and user A's bangs are cut out, the bangs image can only be composited onto a position on user A, and cannot be composited onto a position on user B. The area where user A appears and the area where user B appears are identified by analyzing the photographed image.

[0132] The appearance of bangs varies from person to person due to differences in hair color and texture, as well as the way the light hits the photo. If it were possible to synthesize bangs from different people, the bangs would sometimes look unnatural, but this can be prevented.

[0133] FIG. 17 is a diagram showing an example of combining a bangs image P1.

[0134] 17, bangs image P1 is composited at position p21, which is approximately the center of the forehead. By composited with bangs image P1 at position p21, the person's bangs appear to have more hair in the captured image.

[0135] In this way, the user can select the cutout area by tracing the bangs of the person in the photographed image in the direction of the flow of the bangs. Because the selection can be made by tracing, the user can cut out the fine details of the bangs.

[0136] By using the bangs retouching function, users can tailor the appearance of their own bangs and that of their friends in the photo.

[0137] Furthermore, if a captured image contains multiple people, a selectable range is set for each person's area, allowing each user to adjust the appearance of their own bangs using a bangs image cut out from the selectable range set for the area in which they appear.

[0138] <Bangs retouching details> -Cut out pre-composited bangs image FIG. 18 is a diagram showing another example of selection of the cutout range.

[0139] By using the cut-out pen to select an area including the bangs image P1 that has been combined once as an area to be cut out, the user can cut out an image including the bangs image P1 as a new bangs image.

[0140] In the example in the top part of Fig. 18, a path starting from position p31 to position p32, which is shown as a continuous, mountain-shaped solid line and passes over bangs image P1, is selected with touch pen 141. In this case, as shown in the bottom part of Fig. 18, bangs image P11 is cut out from the range along the path from position p31 to position p32. Bangs image P11 includes the portion of the bangs that appears in the captured image and the portion of the bangs image P1 that has already been combined.

[0141] 18, an area including the entire bangs image P1 is cut out as bangs image P11, but it is also possible to cut out a part of bangs image P1 as a new bangs image. In this way, by using the cut-out pen, it is possible to cut out a new bangs image that includes part or all of the combined bangs image.

[0142] Special eraser FIG. 19 is a diagram showing an example of an operation using a dedicated eraser.

[0143] As described above, the dedicated eraser is an erasing tool used to erase a bangs image that has been superimposed on a photographed image. The state shown in the upper part of Fig. 19 is a state in which a bangs image P1 has been superimposed at position p21 (Fig. 17) approximately in the center of the forehead.

[0144] In this state, if the user selects the dedicated eraser and moves the touch pen 141 from position p41 to position p42, the area along the trajectory of the touch pen 141 will disappear from the synthesized bangs image P1, as shown in the lower part of Figure 19.

[0145] In the lower part of Fig. 19, the range according to the selected thickness along the path of the touch pen 14 is shown as dashed line range A1. Even within range A1 along the path of the touch pen 141 input while the dedicated eraser is selected, only the part of the synthesized bangs image P1 is erased by the dedicated eraser, and the original part of the bangs that appears in the photographed image is not erased. In the erased part of the bangs image P1, the forehead and other parts that appear in the photographed image are visible.

[0146] In this way, the dedicated eraser is a dedicated tool used to erase the combined bangs image P1. By using the dedicated eraser, the user can adjust the shape of the combined bangs portion.

[0147] The portion of the bangs image P1 included in the range A1 does not disappear in the same way regardless of position, but the degree of disappearance may change as it moves away from the position on the locus of the touch pen 141. The position selected using the dedicated eraser is set as the center, and the blur rate of each position within the range of the dedicated eraser thickness changes according to the distance from the center. The further away from the center, the higher the blur rate is set.

[0148] For example, among the portions of the bangs image P1 included in the range A1, the bangs image disappears completely at the positions on the trajectory of the touch pen 141, and disappears so that the bangs image remains faintly as it approaches the outside of the range A1, thereby making it possible to blend the boundary between the portion erased with the dedicated eraser and the surrounding bangs portion.

[0149] Difference between a dedicated eraser and a regular eraser The normal eraser selected when the selection button 302-2 (FIG. 10) is pressed is an erasing tool used to erase images to be synthesized other than the bangs image, such as a pen image or a stamp image. The pen image is an image to be synthesized that is input using the pen tool. The pen image is synthesized along the trajectory of the touch pen 141. The stamp image is an image to be synthesized that is input using the stamp tool. The stamp image is synthesized at the position specified by the touch pen 141.

[0150] FIG. 20 is a diagram showing an example of a photographed image on which a stamp image is combined.

[0151] The state shown in the upper part of Fig. 20 is a state in which a heart-shaped stamp image P21 is superimposed on the bangs image P1 and composited. The stamp image P21 is an image for composition selected from among a plurality of stamp images displayed when the stamp tool is selected, as displayed in the palette area 304L of Fig. 10. The stamp image P21 is composited on a layer above the layer of the bangs image P1.

[0152] In this state, as shown in the lower part of FIG. 20, a case will be described where the user moves the touch pen 141 from position p41 to position p42 and performs an operation on range A1 according to the eraser thickness.

[0153] If the tool selected by the user is a normal eraser, the stamp image P21 will disappear as shown in Fig. 21. In the example of Fig. 21, the part of the stamp image P21 that is included in the range A1 where the normal eraser was used will disappear, and the bangs image P1 will remain displayed (will not disappear). The same display will occur if a similar operation is performed on a compositing image other than the bangs image, such as a pen image.

[0154] On the other hand, if the tool selected by the user is a dedicated eraser, the part of the bangs image P11 disappears as shown in Fig. 22. In the example of Fig. 22, the part of the bangs image P1 that is included in the range A1 selected using the dedicated eraser disappears, and the stamp image P21 remains displayed (is not erased).

[0155] In this way, when a user composites a pen image or stamp image onto a bangs image, the user uses a regular eraser to erase the composite image such as the pen image or stamp image, and a dedicated eraser to erase the bangs image. By providing a dedicated eraser separate from the regular eraser, the user can fine-tune the appearance of each image even when the stamp image or the like is composited onto the bangs image.

[0156] Similarly, when the cut-out pen is selected instead of the dedicated eraser and a position on the combined pen image or stamp image is traced, the part that reflects the operation is the part of the bangs that is included in the range selected by the tracing operation. In other words, when the cut-out pen is selected and a position on the combined stamp image is traced, the part of the stamp image that is included in the range selected by the tracing operation is not cut out (copied), and the part of the bangs that is included in the same range is cut out as the bangs image.

[0157] ·Information display As described above, the bangs image can be cut out within a selectable range from the forehead to the top of the head. The selectable range is set based on, for example, the features of the person extracted by analyzing the captured image.

[0158] If a cut-out range is selected at a position outside the selectable range, such as near the neck of a person appearing in the captured image, the user may be presented with a guide to select a cut-out range within the selectable range.

[0159] FIG. 23 is a diagram showing an example of a message regarding the cutout range.

[0160] As shown in Fig. 23, a message is displayed to guide the user to select the cropping range within a selectable range, such as the range from the forehead to the top of the head. The message shown in Fig. 23 is displayed as a pop-up by the display control unit 225, for example, superimposed on the captured image of the graffiti target.

[0161] This allows the user to confirm that they are performing the cropping operation at a position outside the selectable range. When bangs retouching is selected, information indicating the selectable range may be displayed superimposed on the captured image of the graffiti target.

[0162] <Modification> About the selection tool The selection tool selected when the selection button 302-2 is pressed can also be used to move the composited bangs image (change the composite position). Moving the bangs image using the selection tool is effective when, for example, the bangs retouching function is selected.

[0163] When the bangs retouch function is not selected and buttons such as the bangs retouch function are not displayed in palette areas 304L and 304R, the images that can be moved using the selection tool are composite images such as pen images and stamp images. When moving a bangs image, the user uses the selection tool with the bangs retouch function selected, and when moving, for example, a stamp image, the user uses the selection tool with the bangs retouch function not selected.

[0164] Example of crop range If the bangs image contains parts other than hair, such as skin on the forehead, image processing may be performed according to the content of the composite image when combining them. For example, the brightness of each pixel in the bangs image is compared with the brightness of the corresponding pixel in the composite image, and the combination is performed so that the darker pixels are given priority (the darker pixels are displayed).

[0165] As a result, if there is bangs at the destination of combining the skin portion included in the bangs image, the combining is performed so that the bangs at the destination of combining are visible.

[0166] When a part of the forehead or other skin other than the hair is included in the cutout range, the part other than the hair may be removed from the bangs image. Removal of the part other than the hair is performed, for example, based on a hair mask image generated based on the captured image. The hair mask image is an image used to identify the area containing the hair, distinguishing it from other areas.

[0167] The process of cutting out the part selected by the tracing operation and combining it at an arbitrary position may be performed on parts other than the bangs. For example, the part of the hair in the shoulder area may be cut out and combined at an arbitrary position.

[0168] Although the selection of the cut-out range is performed by tracing the image using the cut-out pen, a rectangular area of ​​a predetermined size at a position selected by the user may be selected as the cut-out range.

[0169] ·others Although three different thicknesses are provided for the cut-out pen and the dedicated eraser, the user may be able to adjust each thickness using a GUI such as a slide bar.

[0170] A photographed image after retouching using the bangs retouching function and a photographed image before retouching may be displayed side by side so that they can be compared.

[0171] The hair color may be changed, including the bangs created using the bangs retouch function. Image processing may also be applied to the entire hair, including the bangs, to enhance the shine of the hair.

[0172] An image of bangs cut out from one photographed image may be composited with another photographed image.

[0173] About the program The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0174] The program to be installed is provided by being recorded on removable media such as an optical disc (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in a ROM or memory unit.

[0175] The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in this specification, or may be a program in which processing is performed in parallel, or at the required timing when called, etc. Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0176] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology. [Explanation of symbols]

[0177] 1 Photo sticker creation device, 11 Pre-operation unit, 21 Photography unit, 51 Editing unit, 201 Control unit, 211 Pre-selection processing unit, 212 Photography processing unit, 213 Editing processing unit, 221 Image analysis unit, 224 Portable image generation unit, 225 Display control unit, 231 Bangs retouching processing unit, 241 Cutting unit, 242 Pasting unit

Claims

1. In response to a selection operation by a user in an area of ​​the bangs of a person appearing as a subject in a photographed image, an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation is cut out as an image to be synthesized; The image for composition is superimposed on the bangs appearing at the position on the photographed image selected by the user, and the image for composition is composited and displayed. A program that causes a computer to execute a process.

2. Displaying the cut-out composite image in accordance with the user's operation of selecting a position on the screen; The image to be combined is displayed at the position where the user's operation is performed, moving the image away from the screen. The program according to claim 1.

3. The position of the composite image that has already been composited is changed in accordance with an operation by the user. The program according to claim 1 or 2.

4. The cutting out of the image to be synthesized is performed by cutting out an image of the bangs within a predetermined width range selected by the user along a trajectory from the start position to the end position of the selection operation.

4. The program according to claim 1.

5. When the selection operation is performed on a range including the top of the composite image that has already been composited, the cutout of the composite image is performed so as to cut out the part of the bangs included in the range along the trajectory of the selection operation and the part of the composite image. The program according to claim 4.

6. When the selection operation targeting the range including the top of the composite image that has already been composited is performed with the erase tool selected, the display of the portion of the composite image that is included in the range along the path of the selection operation is erased. The program according to claim 4 or 5.

7. According to an operation by the user, another composite image selected from a plurality of types is composited onto the composite image and displayed; When the selection operation for the range including the other compositing image displayed on the compositing image is performed with the erase tool selected, the display of the other compositing image is erased while the display of the other compositing image remains as is. The program according to claim 6.

8. When the selection operation is performed at a position outside the area where the bangs are shown, a guide is displayed to perform the selection operation within the area where the bangs are shown.

8. The program according to claim 1.

9. The size and orientation of the image to be synthesized are changed in accordance with an operation by the user before selecting a synthesis position.

9. The program according to claim 1.

10. The image processing device In response to a selection operation by a user in an area of ​​the bangs of a person appearing as a subject in a photographed image, an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation is cut out as an image to be synthesized; The image for composition is superimposed on the bangs appearing at the position on the photographed image selected by the user, and the image for composition is composited and displayed. Image processing methods.

11. A cutting unit that, in response to a selection operation by a user in an area of ​​the bangs of a person appearing as a subject in a photographed image, cuts out an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation as an image to be synthesized; a display control unit that displays the image to be combined by superimposing the image to be combined on the bangs that appear at the position on the photographed image selected by the user; An image processing device comprising:

Images