Photographing device, information processing method, and program

Abstract

To enable a user to easily perform an operation during imaging without moving from a standing position.SOLUTION: An imaging device in one aspect of the present technology performs imaging of a user staying in an imaging booth as a subject, displays an operation screen in which are arranged a live view image that represents an image being imaged and information for operation by a gesture using the hand, recognizes a gesture operation performed by the user on the basis of the image obtained by imaging the user after guidance to a standing position that is set at a location away from a display unit, and executes processing that corresponds to the gesture operation. The present technology can be applied to a photo-making game machine that provides a series of games for printing an imaged image to seal paper.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] In particular, the present technology relates to an imaging device, an information processing method, and a program that enable operations during imaging to be easily performed without moving from a standing position. [Background technology]

[0002] A photo sticker creation device is known in the art. The photo sticker creation device takes a photo of a user, allows the user to edit the photographed image, and prints the edited image on sticker paper. The photo sticker creation device is installed in amusement facilities and the like.

[0003] The flow of one game provided by the photo sticker creation device usually involves taking a photo of the user in a photo booth, then moving the user to an editing booth, editing the image according to the operations performed in the editing booth, and printing the edited image on sticker paper.

[0004] The photo sticker creation device is operated in a photography booth mainly by operating the screen displayed on a touch panel monitor. In the photography booth, a monitor used for displaying a live view image, which is an image being photographed, is installed near the camera.

[0005] Patent Document 1 discloses a technology that identifies the three-dimensional position of fingers based on image data and constructs a skeletal model of the user's hand and fingers. The user's gestures are recognized based on the skeletal model, and operations such as turning off the lights in a room by waving a hand are accepted. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-037675 Summary of the Invention [Problem to be solved by the invention]

[0007] In order to capture the best possible image, the user is often positioned away from the camera during the shoot. If the user changes their position in response to a guide during the shoot, it can be difficult for the user to operate the monitor display from that position.

[0008] The present technology has been made in view of such circumstances, and makes it possible to easily perform operations during shooting without moving from a standing position. [Means for solving the problem]

[0009] According to one aspect of the present technology, the photography device includes a photography unit that takes a photograph of a user in a photography booth as a subject, a display unit that displays an operation screen on which a live view image showing the image being photographed by the photography unit and information for operation by hand gestures are arranged, a recognition unit that recognizes gesture operations performed by the user based on an image obtained by photographing the user after being guided to a standing position set at a position away from the display unit, and a control unit that executes processing in accordance with the gesture operations.

[0010] In one aspect of the present technology, a user in a photo booth is photographed as a subject, and an operation screen is displayed on which a live view image representing the image being photographed and information for operation by hand gestures are arranged. Also, a gesture operation performed by the user is recognized based on an image obtained by photographing the user after being guided to a standing position set at a position away from the display unit, and processing corresponding to the gesture operation is executed. [Effects of the Invention]

[0011] According to the present technology, a user can easily perform operations during shooting without moving from a standing position. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a perspective view showing an external appearance of a photo sticker creation device according to an embodiment of the present technology; [Figure 2] FIG. 2 is a perspective view showing an example of the front configuration of the imaging unit. [Figure 3] 1 is a block diagram showing an example of the configuration of a photo sticker creation device; [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of the photo sticker creating device. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of an image capture processing unit. [Figure 6] 10 is a flowchart illustrating a photographing process. [Figure 7] FIG. 10 is a diagram showing a display example of a guidance screen. [Figure 8] FIG. 10 is a diagram showing a display example of a live view screen. [Figure 9] FIG. 10 is a diagram showing a gesture operation. [Figure 10] FIG. 10 is a diagram illustrating an example of a gesture operation. [Figure 11] FIG. 10 is a diagram illustrating an example of timing at which a gesture operation is accepted. [Figure 12] FIG. 10 is a diagram illustrating another example of the timing at which a gesture operation is accepted. [Figure 13] 10A and 10B are diagrams illustrating examples of gesture operations performed outside the angle of view. [Figure 14] FIG. 10 is a diagram showing a display example of a live view screen. [Figure 15] FIG. 10 is a diagram showing a display example of a live view screen. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Configuration of the photo sticker creation device> FIG. 1 is a perspective view showing the appearance of a photo sticker creating device according to an embodiment of the present technology.

[0014] The photo sticker creation device 1 is a game machine that provides a photo creation game, and is installed in a store such as a game center.

[0015] Users of the photo sticker creation device 1 are mainly high school girls and young women. Multiple users, usually two or three people per group, can enjoy the game with the photo sticker creation device 1. Of course, a single user can also enjoy the game with the photo sticker creation device 1.

[0016] In the game using the photo sticker creation device 1, the user takes a photo of themselves as the subject. The user edits the captured image to combine it with composite images such as handwritten characters and stamp images, and edits the captured image into a colorful image. The user receives a sticker sheet on which the edited captured image is printed, completing the series of games.

[0017] A pre-operation unit 11 is provided on the side of the photo sticker creation device 1, which has a box-shaped housing. The pre-operation unit 11 is a device used for operations before photography, such as selecting the number of users to play with and selecting the photography course. The pre-operation unit 11 is provided with a touch panel monitor 11A. A coin slot 11B for paying the play fee is provided below the touch panel monitor 11A.

[0018] A screen used for various selections is displayed on the touch panel monitor 11A. After making various selections according to the display on the touch panel monitor 11A, the user enters the photo booth through the entrance G and begins taking photos. A photo section 21 is provided in front of the photo booth. The space between the photo section 21 and the background section 22 forms the photo booth. A box-shaped camera unit 31 with a built-in camera is provided in the photo section 21.

[0019] An editing unit 51 is provided to the left of the pre-operation unit 11. The editing unit 51 is a device used to edit captured images obtained by taking pictures with a camera. A user who has taken pictures in the photo booth moves to the editing booth in front of the editing unit 51 and edits the captured images. A tablet-integrated monitor 61, which is a monitor for editing the captured images, is provided on a slope formed above the editing unit 51. A screen used for editing the captured images is displayed on the tablet-integrated monitor 61.

[0020] Two editing booths are provided, one on either side of the housing of the editing unit 51. An editing booth is formed in the space in front of the editing unit 51 shown in Fig. 1, and the other in the space behind the editing unit 51. The photo sticker creation device 1 can move a group of users who have finished taking photos in the photo booth to an empty editing booth, and allow them to edit in parallel in the two editing booths.

[0021] 2 is a perspective view showing an example of the front configuration of the imaging unit 21. The imaging unit 21 is configured so as to be surrounded by a side panel 21A, a side panel 21B, and a front panel 21C.

[0022] A camera unit 31 is provided at a position slightly above the center of the front of the photographing section 21. The camera unit 31 is provided with a camera 41 and a touch panel monitor .

[0023] Camera 41 is, for example, a single-lens reflex camera, and is attached inside camera unit 31 so that the lens is exposed. Camera 41 has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures images of users in the photo booth. Camera 41 functions as a photography unit that captures images of users as subjects.

[0024] Moving images (hereinafter also referred to as live view images) captured by the camera 41 are displayed in real time on the touch panel monitor 42. Still images captured by the camera 41 at a predetermined timing, such as when a shooting instruction is issued, are saved as captured images.

[0025] The touch panel monitor 42 is provided below the camera 41. The touch panel monitor 42 is composed of a monitor such as an LCD and a touch panel laminated on it. The touch panel monitor 42 has a function as a live view monitor that displays moving images captured by the camera 41, and a function that displays various GUIs (Graphical User Interfaces) and accepts user selection operations.

[0026] An upper strobe unit 32 with a curved light-emitting surface facing the user is provided above the camera unit 31. The upper strobe unit 32 irradiates light from above and in front of the user, mainly toward the user's face and upper body.

[0027] A lower box strobe unit 33 that irradiates the user with light from below is provided below the camera unit 31. A strobe light source is provided in each of the upper strobe unit 32 and the lower box strobe unit 33. The strobe light source emits light in synchronization with the timing of capturing a still image, irradiating the user, who is the subject, with strobe light.

[0028] A box-shaped portion 34L is provided between the lower box strobe unit 33 and the side panel 21A. In addition, a box-shaped portion 34R is provided between the lower box strobe unit 33 and the side panel 21B. The top surfaces of the box-shaped portions 34L and 34R are substantially horizontal and are used as a luggage storage area for users performing photography work to place their baggage, etc.

[0029] As will be described later, in the photo sticker creating device 1, various operations during photography can be performed by gesture operations. The photo sticker creating device 1 is equipped with a function for recognizing gesture operations by the user. Gesture operations are performed using, for example, hands. The movement of the user's hand is detected based on an image captured by the camera 41, and when, for example, a predetermined movement is performed, it is recognized as a gesture operation.

[0030] When gesture operations are possible, information such as buttons and slide bars that can be operated by gesture operations is displayed on the touch panel monitor 42. The user looks at the display on the touch panel monitor 42 and moves their hand to perform operations during shooting by gesture operations.

[0031] <Internal configuration of the photo sticker creation device> Fig. 3 is a block diagram showing an example of the configuration of the photo sticker creating device 1. In Fig. 3, the same components as those described above are given the same reference numerals, and duplicated explanations will be omitted where appropriate.

[0032] 3, the photo sticker creation device 1 is configured by connecting a pre-operation unit 11, a photographing unit 21, an editing unit 51, and a printing unit 103 to a PC (Personal Computer) unit 101 via an external input / output interface 102. The PC unit 101 is housed inside the housing of the editing unit 51, for example.

[0033] The PC section 101 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113, which are interconnected by a bus 114.

[0034] The CPU 111 executes a predetermined program and controls the overall operation of the photo sticker creating device 1. The ROM 112 stores the programs and data executed by the CPU 111. The RAM 112 temporarily stores the data and programs processed by the CPU 111.

[0035] An input / output interface 115 is further connected to the bus 114. To the input / output interface 115, a storage unit 116, a communication unit 117, and a drive 118 are connected.

[0036] The storage unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The storage unit 116 stores various setting information and the like supplied from the CPU 111. The information stored in the storage unit 116 is read out by the CPU 111 as appropriate.

[0037] The communication unit 117 is an interface for a network such as the Internet. The communication unit 117 communicates with external devices under the control of the CPU 111. The communication unit 117 transmits an image selected by a user to, for example, a server managed by the manufacturer of the photo sticker creating device 1. The image transmitted from the communication unit 117 is allocated a predetermined storage area and saved, and is then displayed on or downloaded to a mobile terminal that has accessed the server.

[0038] Removable media 119, such as an optical disk or semiconductor memory, is appropriately attached to drive 118. Programs and data read from removable media 119 by drive 118 are supplied to CPU 111, where they are stored in storage unit 116 or installed.

[0039] The input / output interface 115 is connected to an external input / output interface 102. The PC unit 101 controls each unit via the external input / output interface 102.

[0040] The pre-operation unit 11 performs a pre-selection process for the user before photography is performed. The pre-operation unit 11 is composed of a touch panel monitor 11A, a speaker 121, and a coin processing unit 122.

[0041] The touch panel monitor 11A displays various selection screens and accepts user operations on the selection screens under the control of the PC unit 101. An input signal representing the content of the user operation is supplied to the PC unit 101, and various settings are made.

[0042] The coin processing unit 122 detects the insertion of coins into the coin insertion slot 11B. When the coin processing unit 122 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the start of the game.

[0043] The photographing unit 21 performs photographing processing of a user in a photographing booth. The photographing unit 21 is composed of a lighting device 131, a camera 41, a touch panel monitor 42, and a speaker 132.

[0044] The lighting devices 131 are strobe units in the photo booth, and emit light in accordance with a lighting control signal supplied from the PC section 101. The lighting devices 131 include strobe units provided in the photo booth, such as the upper strobe unit 32 and the lower box strobe unit 33.

[0045] The camera 41 takes a photograph in accordance with shutter control by the PC section 101, and outputs image data obtained by the photograph to the PC section 101.

[0046] The editing unit 51 performs post-shooting editing processing. The editing unit 51 is composed of a tablet built-in monitor 61, touch pens 141A and 141B, and a speaker 142. The tablet built-in monitor 61 and the touch pens 141A and 141B are provided as a set on two sides of the housing of the editing unit 51.

[0047] The tablet built-in monitor 61 displays an editing screen and accepts user operations on the editing screen under the control of the PC unit 101. An input signal representing the content of the user operation is supplied to the PC unit 101, and the captured image to be edited is edited.

[0048] The printing unit 103 is configured to include a printer 151. A sticker paper unit 152 is attached to the printer 151.

[0049] Printer 151 prints an image on a sticker paper stored in sticker paper unit 152 based on print data supplied from PC section 101. Printer 151 ejects the sticker paper on which the image is printed to a sticker paper ejection port.

[0050] <Example of functional configuration of photo sticker creation device> 4 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. At least some of the functional units shown in FIG. 4 are realized by the CPU 111 in FIG. 3 executing a predetermined program.

[0051] An information processing unit 201 is realized in the photo sticker creating device 1. The information processing unit 201 is made up of a pre-selection processing unit 211, a photography processing unit 212, an edit processing unit 213, a print processing unit 214, and a transmission processing unit 215.

[0052] The pre-selection processing unit 211 executes pre-selection processing by controlling each unit of the pre-operation unit 11. The pre-selection processing selects the number of users to play the game together, the photography course, etc. Information indicating the content of the user's selection is supplied to the photography processing unit 212.

[0053] The shooting processing unit 212 performs shooting processing by controlling each unit of the shooting unit 21. In the shooting processing, shooting is performed multiple times and multiple shot images, such as six images, are generated. The shooting processing unit 212 outputs data of the multiple shot images generated by the shooting processing to the editing processing unit 213.

[0054] The editing processing unit 213 performs editing processing by controlling each unit of the editing unit 51. The editing processing includes editing the captured image, printing the image on sticker paper, and sending the image to an external image management server.

[0055] The images transmitted to the image management server are used by the user to view on a mobile terminal such as a smartphone or to download to the mobile terminal after the game on the photo sticker creating device 1 is finished.

[0056] The edit processing unit 213 outputs print data generated by arranging the edited photographed images in a predetermined layout to the print processing unit 214. The edit processing unit 213 also outputs image data for transmission generated based on the edited photographed images to the transmission processing unit 215.

[0057] The print processing unit 214 outputs the print data to the printer 151, causing the printer 151 to print the image on the sticker paper.

[0058] The transmission processing unit 215 controls the communication unit 117 to communicate with the image management server and transmits (uploads) images to be transmitted to the image management server. Images generated in the game of the photo sticker creation device 1 are provided in the form of being printed on sticker paper, and also in the form of data via the image management server.

[0059] FIG. 5 is a block diagram showing an example of the configuration of the image capture processing unit 212.

[0060] The photographing processing unit 212 is made up of a photographing control unit 221, a gesture recognition unit 222, and a control unit 223. Note that Fig. 5 mainly shows the configuration related to gesture operations.

[0061] The photographing control unit 221 performs photographing by controlling the camera 41. For example, the photographing control unit 221 outputs a moving image photographed by the camera 41 to the gesture recognition unit 222 and the display control unit 231 as a live view image.

[0062] Furthermore, the shooting control unit 221 starts shooting images at a predetermined timing. After the start of shooting, multiple shots, such as six, are taken at predetermined intervals. Data for, for example, six shot images generated by the multiple shots is supplied to the display control unit 231, where it is used for display on the screen and also supplied to the editing processing unit 213 (FIG. 4).

[0063] The gesture recognition unit 222 analyzes each frame of the live view image and estimates the skeleton of the user's hand. For example, the gesture recognition unit 222 detects the user's hand movement (gesture using the hand) based on the result of the skeleton estimation, and when a predetermined movement is detected, recognizes the movement as a gesture operation. Recognition of such a gesture operation is performed at an acceptance timing, which is a timing at which the gesture operation can be accepted.

[0064] For example, gestures recognized include the movement of extending the index finger, pushing the hand with the index finger extended forward, and sliding the hand with the index finger extended up, down, left, and right. Additionally, various movements such as spreading the index finger and middle finger to make a peace sign, spreading all five fingers to make a paper sign, and pointing the thumb up are also recognized as gestures.

[0065] In this way, a movement that combines the movement of the hand and the movement of the fingers is recognized as a gesture operation using the hand. Information representing the recognition result by the gesture recognition unit 222 is supplied to the display control unit 231 and the process execution unit 232.

[0066] The control unit 223 is made up of a display control unit 231 and a processing execution unit 232 .

[0067] The display control unit 231 controls the display on the touch panel monitor 42 during shooting. For example, the display control unit 231 causes the touch panel monitor 42 to display a live view screen including a live view image based on the live view image supplied from the shooting control unit 221.

[0068] Furthermore, the display control unit 231 displays an operation screen on which information for gesture operations is arranged on the touch panel monitor 42. The information for gesture operations is sometimes provided on a live view screen, and the live view screen is sometimes used as the operation screen. The display control unit 231 switches the display of the operation screen depending on the content of the gesture operation recognized by the gesture recognition unit 222.

[0069] The display control unit 231 may display an image such as a pointer at a position on the operation screen corresponding to the position of the user's hand or finger performing the gesture operation. This makes it possible to improve visibility during gesture operation.

[0070] The processing execution unit 232 executes various processes according to the content of the gesture operation recognized by the gesture recognition unit 222. The processing execution unit 232 executes various processes other than the processes related to the screen display, such as a process for adjusting the angle of view of the camera 41 and a process for changing the background music during shooting.

[0071] In this way, the control unit 223, which is configured by the display control unit 231 and the processing execution unit 232, functions as a control unit that executes processing in response to a gesture operation by the user.

[0072] <Shooting process flow> The photographing process of the photo sticker creating device 1 will be described with reference to the flowchart of Fig. 6. The process shown in Fig. 6 starts, for example, when a user who has started the game completes the pre-selection work and moves to the photographing booth.

[0073] In step S1, the display control unit 231 of the photography processing unit 212 causes the touch panel monitor 42 to display a guidance screen that provides guidance on where to stand when taking a photograph.

[0074] FIG. 7 is a diagram showing an example of a display of the guidance screen.

[0075] In the example of Figure 7, the message "First, stand on the 'bigger line'" is displayed, and below that, an image showing the positions prepared on the floor of the photo booth is displayed. The user adjusts their position by following the instructions on the guide screen and checking the marks prepared on the floor of the photo booth.

[0076] Footprint marks or the like indicating the positions of the two users are printed on the floor of the photo booth. The positions at which the users stand during photo shooting are set in advance at a position away from the camera 41 so that the angle of view, brightness, etc., will be appropriate. From the position adjusted according to the guidance, it is difficult to touch the touch panel monitor 42 provided near the camera 41 with one's hand and operate buttons and the like displayed on the touch panel monitor 42. In this way, the positions at which the users stand during photo shooting are set at a position that makes it difficult for the users in that position to reach the touch panel monitor 42 even if they stretch out their hands.

[0077] The above-described gesture operation is used so that, when the user has adjusted their standing position according to the guidance, they can operate the display on the touch panel monitor 42 from that standing position. After the user is given guidance on their standing position, capturing of a live view image begins.

[0078] In step S2, the display control unit 231 displays a live view screen based on the live view image supplied from the shooting control unit 221.

[0079] FIG. 8 is a diagram showing a display example of the live view screen.

[0080] As shown in Fig. 8, a live view image is displayed in a display area 301 formed substantially in the center of the live view screen. In the example of Fig. 8, a moving image showing large images of the faces of two users is displayed as the live view image.

[0081] Buttons 311-1 to 311-7, which are operated to select a stamp image, are displayed side by side below the display area 301. More buttons are provided to the left of button 311-1 and to the right of button 311-7. Each circular button displays an image that is a sample of the stamp image that will be selected when the corresponding button is pressed.

[0082] To the right of the display area 301, an OK button 312 is displayed, which is operated when the selection of the stamp image is completed.

[0083] In this way, gesture operations are accepted while information such as buttons used for operation is displayed on the touch panel monitor 42. The timing for accepting gesture operations is after the guidance on the standing position has been given.

[0084] In step S3 of FIG. 6, the gesture recognition unit 222 accepts a gesture operation.

[0085] In step S4, the gesture recognition unit 222 analyzes each frame of the live view image and recognizes a gesture operation.

[0086] In step S5, the control unit 223 executes processing according to the gesture operation. For example, when an operation to select a stamp image is performed by a gesture operation, the display control unit 231 displays the stamp image by combining it with the face of the user reflected in the live view image.

[0087] It should be noted that, when information such as buttons to be operated when adjusting the angle of view of camera 41 is displayed on touch panel monitor 42, the processing of step S5 in response to the gesture operation is performed by processing execution unit 232 as appropriate.

[0088] In step S6, the photography control unit 221 determines whether photography has started. If it is determined in step S6 that photography has not started, the process returns to step S2, and the selection of a stamp image using the live view screen continues.

[0089] FIG. 9 is a diagram showing a user performing a gesture operation.

[0090] In the example of FIG. 9, a user U1 standing on the left side of the camera 41 makes a gesture operation using both hands, and a user U2 standing on the right side of the camera 41 makes a gesture operation using one hand (the right hand).

[0091] In this way, the gesture recognition unit 222 recognizes the gesture operations of each of the multiple users who are playing the game together. Processing such as skeleton estimation is performed for each user.

[0092] It may be possible to recognize only gesture operations of users standing in specific positions, such as gesture operations of a user standing on the left side of the camera 41. The standing position of each user is identified based on an image captured by the camera 41.

[0093] FIG. 10 is a diagram showing an example of a gesture operation.

[0094] 10A, when button 311-4 representing a stamp image of cat ears is pressed by a gesture operation by either user U1 or user U2, the stamp image of cat ears is displayed superimposed on the live view image. For example, the gesture operation of pressing button 311-4 is performed by moving a hand with an index finger extended to a position corresponding to button 311-4 and, in that state, pressing forward (moving the hand toward touch panel monitor 42).

[0095] 10A, the hand H pressing down the button 311-4 does not represent the hand of the user touching and pressing down the button 311-4 with a finger, but is illustrated for convenience to indicate a gesture operation performed in a space away from the touch panel monitor 42. The same applies to the other figures.

[0096] When button 311-4 is pressed, the positions of each user's eyes, nose, and other body parts shown in the live view image are recognized, and a stamp image of cat ears is superimposed on the position of the user's head. Also, a stamp image of cat whiskers is superimposed on the position of each user's nose. Each stamp image continues to be displayed superimposed on the live view image.

[0097] In this way, on the live view screen, the content of the user's selection is reflected in the display of the live view image in real time. When a different stamp image is selected, the type of stamp image to be combined with the live view image is switched.

[0098] 10B, when the OK button 312 is pressed by a gesture operation, the selection of the stamp image is completed. The gesture operation of pressing the OK button 312 is also performed by moving the hand with the index finger extended to a position corresponding to the OK button 312 and then pressing forward in that state. A movement different from the movement of pressing the buttons 311-1 to 311-7 may be recognized as the gesture operation of pressing the OK button 312.

[0099] If it is determined in step S6 of FIG. 6 that the selection of the stamp image has been completed and therefore that shooting should begin, then in step S7 the gesture recognition unit 222 stops accepting gesture operations.

[0100] As described above, the photographed image is an image to be provided to the user after being edited and printed on sticker paper or via a server that provides an image management service. In the game of the photo sticker creating device 1, a gesture operation is accepted before starting to capture the image to be provided.

[0101] In step S8, the photography control unit 221 controls the camera 41 to capture a photographed image. Just before the photograph is taken, a countdown such as "3, 2, 1" is presented by voice or the like, and a still image is taken when the countdown reaches 0. A stamp image is synthesized with the still image obtained by the photograph, and a photographed image is generated.

[0102] In step S9, the photographing control unit 221 determines whether or not photographing has ended. If it is determined in step S9 that photographing has not ended, the process returns to step S7, and photographing of photographed images is repeated.

[0103] If it is determined in step S9 that the shooting has ended because a predetermined number of shots, such as six, have been taken, then in step S10 the display control unit 231 causes an operation screen for gesture operation to be displayed on the touch panel monitor 42. For example, an operation screen is displayed that is used to select a captured image to be used for editing or to select the content of image processing to be performed on a face appearing in the captured image. Information such as buttons used for gesture operation is displayed on the operation screen.

[0104] In step S11, the gesture recognition unit 222 resumes accepting gesture operations.

[0105] In step S12, the gesture recognition unit 222 analyzes the image captured by the camera 41 and recognizes the gesture operation.

[0106] In step S13, the control unit 223 executes processing according to the gesture operation. For example, when an operation to select a photographed image to be edited from among six photographed images is performed by a gesture operation, the processing execution unit 232 outputs the selected photographed image to the editing processing unit 213 (FIG. 4).

[0107] When the operation using the operation screen is completed, the photographing process is completed. In the editing process performed after the photographing process, editing is performed on the photographed image selected by the gesture operation.

[0108] As described above, in the photo booth, various selection operations are performed by gesture operations. Even after the user has adjusted their standing position according to the instructions, they can easily perform operations using the display on the touch panel monitor 42 without moving from that position.

[0109] Furthermore, the timing for accepting gesture operations is set to be before shooting starts or after shooting ends. By not accepting gesture operations during shooting, even if the user is moving their hands to strike a pose, it is possible to prevent this from being recognized as a gesture operation.

[0110] When taking a photograph, the user usually makes various poses with their hands, but if these are recognized as gesture operations, processing that the user does not intend will be performed. By not accepting gesture operations at the timing when taking a pose for a photograph, it is possible to prevent such unintended processing from being performed.

[0111] The following describes each of the gesture operations in detail.

[0112] <1. Booth accepting gesture control> Although the gesture operation is accepted in the photo booth, it may be accepted in another booth formed in the photo sticker creating device 1.

[0113] (1) Advance Booth Gesture operations may be used for operations in a pre-booth, which is a booth in front of the pre-operation unit 11 (FIG. 1). In this case, the pre-operation unit 11 is provided with a camera.

[0114] At the timing of accepting a gesture operation, a screen including information such as buttons for gesture operation is displayed on the touch panel monitor 11A of the pre-operation unit 11. The gesture operation performed by the user while viewing the display on the touch panel monitor 11A is recognized based on an image captured by a camera provided in the pre-operation unit 11, and various selections are made as pre-selections.

[0115] (2) Editing booth Gesture operations may be used in the editing booth, in which case the editing section 51 is equipped with a camera.

[0116] When a gesture operation is accepted, a screen including information such as buttons for gesture operation is displayed on the tablet built-in monitor 61 of the editing unit 51. The gesture operation performed by the user while viewing the display on the tablet built-in monitor 61 is recognized based on an image captured by a camera provided in the editing unit 51, and various selections related to editing of the captured image are made.

[0117] <2. Timing of gesture operation acceptance> As described above, users in the photo booth pose with their hands. Because it is difficult to determine whether a user is posing or performing a gesture operation, the timing for accepting gesture operations is limited to the following times:

[0118] (1) Timing before and after shooting FIG. 11 is a diagram showing an example of timing at which a gesture operation is accepted.

[0119] After the user enters the photography booth, the live view screen starts to be displayed at time t1. Information for gesture operations is prepared on the live view screen, and gesture operations start to be accepted when the live view screen starts to be displayed.

[0120] Shooting starts at time t2, which is the time a certain amount of time has elapsed since the start of display on the live view screen (the time when selection of a stamp image, etc. has finished). Once shooting has started, acceptance of gesture operations is stopped. This suspension of acceptance of gesture operations continues until time t3, when the final shooting has finished and shooting has ended. Gesture operations are not recognized until the final shooting has finished.

[0121] At time t3 when the final image capture is completed, the operation screen is displayed on the touch panel monitor 42, and acceptance of gesture operations is resumed. Information for gesture operations is displayed on the operation screen. Acceptance of gesture operations continues until time t4 when the display of the operation screen ends.

[0122] In this way, gesture operations are accepted before shooting begins and after shooting ends. The timing for accepting gesture operations shown in Fig. 11 is the same as the timing described with reference to the flowchart in Fig. 6. Only the timing before shooting begins may be set as the timing for accepting gesture operations, and gesture operations may not be accepted after shooting ends. For example, selection after shooting ends is performed by directly touching a button or the like displayed on touch panel monitor 42 with a finger.

[0123] Although gesture operations are accepted before the start of the first shooting and after the end of the last shooting, gesture operations may be accepted before and after each shooting. In this case, even after the start of the first shooting, gesture operations are accepted between each shooting, such as between the first and second shooting.

[0124] FIG. 12 is a diagram showing another example of the timing at which a gesture operation is accepted.

[0125] 12, gesture operations are accepted from time t11 to time t12, which is the timing before a certain number of shots are taken. Furthermore, acceptance of gesture operations is stopped at time t12, when a countdown immediately before shooting starts. As described above, before a captured image is taken, a countdown is presented by audio or the like.

[0126] A countdown such as "3, 2, 1" is performed, and after an image is captured at time t13, acceptance of gesture operations is resumed until time t14 when the countdown for the next image capture begins.

[0127] In this way, acceptance of gesture operations may be stopped when the countdown immediately before shooting begins. This also makes it possible to prevent the user's pose from being recognized as a gesture operation. In other words, by posing after the countdown has started, the user can distinguish between gesture operations and hand movements for posing.

[0128] (2) Timing during shooting The user may be able to stop the capture after the first capture has started. For example, if the pause button displayed on the live view screen is pressed, the capture is paused. Gesture operations are accepted while the capture is paused.

[0129] This allows the user to temporarily stop shooting even after shooting has started, allowing them to perform gesture operations for selection, etc. The user can selectively use gesture operations and hand movements for posing.

[0130] <3. Gesture operation position> FIG. 13 is a diagram showing an example of an image being captured while the live view screen is being displayed.

[0131] A frame F shown in Fig. 13 indicates the range displayed on the live view screen as a live view image. The range indicated by frame F is cut out from the entire angle of view of the image captured by camera 41 and used as the live view image as described with reference to Fig. 8. The user capturing the image can only see the range inside frame F.

[0132] When such an image is used as a live view image, hand movements that appear outside the range of the live view image are accepted as gesture movements, and hand movements that appear inside the range of the live view image are recognized as poses (not accepted as gesture movements). In other words, hand movements made outside the field of view of the live view image are accepted as gesture movements, and hand movements made within the field of view of the live view image are not accepted as gesture movements.

[0133] In the example of Fig. 13, two users are moving their hands with their index fingers extended outside the range of frame F. Hand movements captured in such positions are accepted as gesture operation movements. Gesture operations performed outside the range of frame F are accepted even during shooting (after shooting has started), for example.

[0134] This allows the user to perform gesture operations even during shooting. The user can make various selections while viewing the live view image and checking their own appearance and the trimming range.

[0135] <4. Gesture operation recognition> (1) Decision and Choice Various selections and decisions are made by gesture manipulation using information such as buttons displayed on the screen. For example, if the distance between the base of the index finger and the thumb is less than a predetermined distance, the movement is recognized as a gesture manipulation to confirm the selection.

[0136] Furthermore, when a specific finger is kept in the same position for a predetermined period of time, the movement is recognized as a gesture operation that determines the selected content.

[0137] Swiping your hand sideways is recognized as a gesture to switch screens.

[0138] (2) Trimming FIG. 14 is a diagram showing a display example of the live view screen.

[0139] In the example of Fig. 14, the entire range of the angle of view of the image captured by camera 41 is displayed as a live view image in display area 301. The user can select the range to be used as the captured image by a gesture operation. In the example of Fig. 14, the range indicated by frame F indicates the selected range.

[0140] For example, by performing a downward gesture operation as indicated by a thick arrow #1, the user can move the range of the frame F downward, thereby changing the range to be cut out by trimming.

[0141] Such a gesture operation for selecting a trimming range is recognized based on, for example, the skeletal structure of the fingers.

[0142] The gesture operation for selecting the trimming area may be recognized based on the relative positions of the thumb and index finger. The gesture operation for selecting the trimming area may be performed with one hand or with both hands.

[0143] (3) Enlargement / reduction, zoom in / out When a movement is made such that the distance between the pad of the thumb and the pad of the index finger is equal to or less than a predetermined distance, the movement is recognized as a gesture operation for reducing the image.

[0144] When a movement is made such that the distance between the pad of the thumb and the pad of the index finger is equal to or greater than a predetermined distance, the movement is recognized as a gesture operation for enlarging an image.

[0145] Similarly, a pinching motion with two fingers is recognized as a pinch-in gesture, and a spreading motion with two fingers is recognized as a pinch-out gesture.

[0146] In this way, various operations other than selecting an item using a button or the like and deciding on the selected content are performed by gesture operations.

[0147] (4) BGM control The movement of a hand with an index finger extended and moving it in front of the mouth is recognized as a gesture to lower the volume of background music during filming, allowing each user to adjust the volume of the background music.

[0148] (5) Other Various operations related to photography, such as adjusting the angle of the camera 41, starting photography, pausing photography, and setting the self-timer shutter, may be performed by gesture operations.

[0149] For example, the movement of a specific finger, such as an index finger, at a position corresponding to a button displayed on the screen is recognized as a gesture operation, and the operation is performed. By limiting the fingers used for gesture operations to specific fingers, it is possible to reduce erroneous operations. It is also possible to not limit the fingers used for gesture operations.

[0150] Furthermore, the shooting may be temporarily stopped by a gesture operation performed outside the angle of view of the live view image.

[0151] <5. Selection by gesture operation> (1) Choices When multiple options are available, gesture manipulation is used to select one option. For example, gesture manipulation is used to select retouching or makeup. Retouching is image processing such as adjusting the size of eyes or skin color in an image. Makeup is image processing related to facial makeup, such as changing the color of lips or cheeks.

[0152] Furthermore, the operation of selecting content is performed by a gesture operation. The content includes images to be combined, such as the stamp image described with reference to FIG.

[0153] Selections regarding retouching or makeup processing are made, for example, when a live view screen such as that shown in Fig. 15 is displayed, and the selections are reflected in real time in the display of the live view image. In the example of Fig. 15, area 302L, which is an area for the user appearing on the left side, is formed on the left side of display area 301, and area 302R, which is an area for the user appearing on the right side, is formed on the right side of display area 301. Areas 302L and 302R each display buttons, slide bars, and the like that provide information for gesture operations.

[0154] By operating buttons or slide bars with gestures, the retouching and makeup processing options are selected and reflected in the live view image display. In this way, the user can make various selections while viewing the live view image display. Because the selections are reflected in the live view image display in real time, it becomes easier to imagine the finished image that will be generated as a captured image.

[0155] (2) Specifying the location The part of the image to be changed may be selected by a gesture operation of pointing a finger at the live view image displayed on the touch panel monitor 42.

[0156] This allows the user to adjust the appearance of each position on the live view image. In this way, gesture operations can be used to specify positions.

[0157] (3) Accepting individuals one at a time As described with reference to Fig. 15, multiple users may be allowed to perform gesture operations simultaneously, allowing each user to simultaneously make a selection according to their preference.

[0158] (4) Change your selection The selection may be changed depending on the angle of view of the live view image, such as a full-body image or a close-up image. A full-body image is an image that shows the user's entire body. A close-up image is an image that shows a large image of the face including the upper body.

[0159] <6. Gesture operation by multiple people> (1) When performing the same gesture operation simultaneously When a plurality of users simultaneously perform the same gesture operation, the gesture operation is accepted.

[0160] (2) When performing different gesture operations simultaneously When multiple users simultaneously perform different gesture operations, each of the gesture operations is accepted.

[0161] When a plurality of users simultaneously perform different gesture operations, a warning may be output to notify that the gesture operation cannot be accepted.

[0162] (3) When the person who will perform the gesture operation is determined in advance The user who performs the gesture operation may be determined in advance. For example, only gesture operations performed by users who have been determined in advance by the photo sticker creating device 1 are accepted.

[0163] The user performing the gesture operation may be determined based on the standing position, such as accepting a gesture operation from a user standing on the right side. In this case, a message such as "The person standing on the right should perform the operation" is displayed when accepting a gesture operation.

[0164] The user who performs the gesture operation may be determined by the gesture operation, such as accepting a gesture operation of a user who points their palm toward camera 41. In this case, a message such as "Please show us the palm of the operator's hand" is displayed when the gesture operation is accepted. By analyzing the palm of the hand pointed toward camera 41, it becomes easier to detect the skeleton, and it becomes possible to improve the recognition accuracy of subsequent gesture operations.

[0165] (4) Taking a separate photo of your palm before accepting gestures Before accepting a gesture operation, each user may be asked to hold their palm toward camera 41, and the skeleton of each user's palm may be estimated based on the image captured at that time. Estimating the skeleton of each user before accepting a gesture operation makes it possible to improve the accuracy of subsequent recognition.

[0166] <7. Standing position when using gestures> Although gesture operations performed at a standing position set on the floor of the photography booth are accepted in the above embodiment, gesture operations performed at other positions may also be accepted. For example, the position of a chair placed at a position away from the camera 41 is set as the position where the gesture operation is performed. The user will perform the gesture operation while sitting in the chair.

[0167] In this way, by setting the position where the gesture operation is to be performed in advance, the approximate distance from camera 41 to the user can be known, and the recognition accuracy of the gesture operation can be improved.

[0168] <8.Other> Warning output If the gesture operation fails to be recognized or if the gesture operation is too fast, a warning may be output to notify the user of the failure or the fast movement of the gesture operation.

[0169] - Demonstration of gesture operations A gesture introduction screen may be displayed to show the user what movements are recognized as gesture operations. The gesture introduction screen may display, for example, an image of a person performing a gesture that serves as a model for the gesture operation.

[0170] For example, the gesture introduction screen is displayed on the touch panel monitor 11A of the pre-operation unit 11. This allows the user to check outside the photo booth what movements they should make before entering the photo booth and taking a photo.

[0171] A gesture introduction screen may be displayed on a touch panel monitor 42 provided in the photo booth.

[0172] Example of mobile device application operation Gesture operations may be used to operate an application such as a selfie app installed on a mobile device such as a smartphone. The selfie app is an application that allows a user to take a picture of themselves (take a selfie) using the camera provided on the smartphone.

[0173] The user may be allowed to register in advance the contents of each gesture operation, such as what kind of movements are to be used as gesture operations. The user can register the movements that are easy for the user to perform as gesture operations, which allows the user to take pictures and perform various operations smoothly. Furthermore, since the user registers the movements themselves, the user is less likely to forget the pairs of movements and gesture operations.

[0174] Different gesture operations may be used to start capturing video and still images, eliminating the need to operate the screen of a selfie app or otherwise change the capture mode (video mode and still image mode).

[0175] The image may be captured after a predetermined time has elapsed since the gesture operation of pressing the shutter button was accepted. This allows time for the user to strike a pose after the gesture operation. In most selfie apps, the image is captured immediately after the shutter is pressed, making it difficult for inexperienced users to strike a pose. However, even such users can now capture a selfie while striking a pose.

[0176] Gesture operations may be used to perform retouching processing on the screen of the selfie app.

[0177] 1. Eye deformity If the user touches the area around their eyes (for example, under-eye bags) and then moves their index finger and thumb apart as a gesture, a retouching process will be performed to widen the width of the eyes in the image. If the user touches the area around their eyes (for example, under-eye bags) and then moves their index finger and thumb vertically as a gesture, a retouching process will be performed to increase the vertical width of the eyes in the image. If the user performs a gesture operation by touching the area around their eyes (for example, under the eyes), an adjustment bar will be displayed.

[0178] 2. Changing the distance between the left and right eyes When a user gestures by touching the inner corner of their eye, a retouching process is performed to narrow the distance between the left and right eyes in the image. When a user gestures by touching the corner of their eye, a retouching process is performed to increase the distance between the eyes in the image.

[0179] 3. Change eye color When the user blinks (closes their eyes for more than one second) as a gesture operation, the eye color shown in the image is changed.

[0180] 4. Changing the color of other parts of the face When a user performs a gesture operation in which they touch their cheek, a retouching process is performed to change the color of the blush on the cheeks shown in the image. When a user performs a gesture operation by touching their lips, a retouching process is performed to change the lip color of the lips in the image.

[0181] 5. Change hair color When a user gestures by touching their head, a retouching process is performed to change the hair color in the image.

[0182] 6. Other Processing Gesture operations may be used for operations other than retouching processing. When a user gestures by touching their head, image processing is performed to superimpose content onto the area of ​​the head that appears in the image. If the user performs a gesture operation by shaking their head from side to side, the previous operation will be canceled. Example 1) After the eye transformation process, if the user performs a gesture operation by shaking their head from side to side, the eye transformation process will be reset. Example 2) If the user performs a gesture operation by shaking their head from side to side multiple times, all processing will be reset. Example 3) If the user performs a gesture operation of shaking their head from side to side multiple times after the eye transformation process and the cheek color change process, the eye transformation process and the cheek color change process will be reset.

[0183] Examples of subjects to be photographed Gesture operations are recognized by estimating the skeletal structure of fingers and other parts of the body, so it is possible to capture images that do not include a person's face, which increases the freedom of photographic scenes and poses.

[0184] Only the shutter operation may be performed by gesture operation, which allows for smooth shooting of landscapes and the like.

[0185] Examples of imaging devices The above-described gesture operations can be used in various photographic devices with a photographic function, such as photo sticker creation devices, smartphones, and cameras.

[0186] About the program The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0187] The program to be installed is provided by being recorded on removable media 119 shown in FIG. 3, which may be an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 112 or storage unit 116.

[0188] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0189] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0190] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0191] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0192] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0193] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0194] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. [Explanation of symbols]

[0195] 1 Photo sticker creation device, 41 camera, 42 touch panel monitor, 21 photography unit, 201 information processing unit, 212 editing processing unit, 221 photography control unit, 222 gesture recognition unit, 223 control unit, 231 display control unit, 232 processing execution unit

Claims

1. a photography unit that takes a photograph of a user in a photography booth as a subject; a display unit that displays an operation screen on which a live view image showing an image being captured by the image capturing unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged; a recognition unit that recognizes, as a gesture operation, an action of pressing a finger at a position corresponding to any one of the buttons, based on an image obtained by photographing the user after guidance on a standing position set at a position away from the display unit; and a control unit that executes a process in accordance with the gesture operation; An imaging device comprising:

2. The recognition unit recognizes the gesture operation based on a result of estimating the skeleton of the user's hand. The imaging device according to claim 1 .

3. The recognition unit accepts the gesture operation performed before starting to capture an image to be provided to the user.

3. The imaging device according to claim 1.

4. The recognition unit stops accepting the gesture operation while the image to be provided is being captured. The imaging device according to claim 3 .

5. The recognition unit accepts the gesture operation performed outside a field of view of the live view image, and does not accept the gesture operation performed within a field of view of the live view image. The imaging device according to claim 3 .

6. The control unit temporarily suspends capturing of the image to be provided in response to the gesture operation being performed outside an angle of view of the live view image. The imaging device according to claim 5 .

7. The recognition unit recognizes the content of the gesture operation using one hand or both hands.

7. The imaging device according to claim 1.

8. The recognition unit recognizes the gesture operation performed by each of the users who play the game together.

8. The imaging device according to claim 1.

9. The recognition unit recognizes the gesture operation performed by a predetermined user among the plurality of users who play the game together.

8. The imaging device according to claim 1.

10. The image capturing device further includes another display unit provided outside the image capturing booth, which displays a gesture introduction screen including an image serving as a sample of the gesture operation.

10. The imaging device according to claim 1.

11. The display unit displays a gesture introduction screen including an image serving as a sample of the gesture operation.

10. The imaging device according to claim 1.

12. The display unit displays a guide screen that guides the user to a standing position set in the photo booth before displaying the operation screen.

12. The imaging device according to claim 1.

13. An information processing method for a device including an imaging unit, a display unit, and an information processing unit, the photographing unit photographs the user in the photographing booth as a subject; the display unit displays an operation screen on which a live view image showing an image being captured by the image capture unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged; The information processing unit recognizes, based on an image obtained by photographing the user after guidance on a standing position set at a position away from the display unit, an action of pressing a finger at a position corresponding to any of the buttons as a gesture operation, and executes processing according to the gesture operation. Information processing methods.

14. A program executed by a computer that controls a device including an imaging unit, a display unit, and an information processing unit, the photographing unit photographs the user in the photographing booth as a subject; the display unit displays an operation screen on which a live view image showing an image being captured by the image capture unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged; The information processing unit recognizes, based on an image obtained by photographing the user after guidance on a standing position set at a position away from the display unit, an action of pressing a finger at a position corresponding to any of the buttons as a gesture operation, and executes processing according to the gesture operation. A program for executing a process.

Images