Image processing device

Abstract

To provide an editing technique that allows a user to enjoy an effect similar to catchlight through image processing.SOLUTION: An image processing device includes an imaging unit that images a user, a detection unit that detects an iris area of a person included in the captured image, a processing unit that performs processing to superimpose a decorative image representing a reflection within the iris on a part of the iris area in the captured image, an output unit that outputs the processed image, and a reception unit that receives a first user operation for selecting a decorative image that the user wants to apply from among multiple types of decorative image candidates, and a second user operation for moving the decorative image. The output unit displays an enlarged image for receiving the second user operation, and the processing unit changes the position of the decorative image within the iris area according to the second user operation.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to image processing of photographed images, and in particular to eye decoration technology. [Background technology]

[0002] A playful image capture device called Purikura (registered trademark) is extremely popular, especially among female high school students. Users take pictures of themselves and decorate the captured images in various ways on an editing display. The decorated images are then printed on sticker mounts (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-32213 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-184287 Summary of the Invention [Problem to be solved by the invention]

[0004] The fun image capture device allows users to enjoy adding their own unique touches to their captured images using the graffiti function. The graffiti function also allows users to apply makeup to their faces. For example, adding lipstick to the lips gives the user the feeling of wearing makeup.

[0005] Although the standard processing method for the graffiti function is widespread, people are accustomed to it. In photography studios and other locations with specialized equipment, lighting devices called catchlights are used to capture reflective shapes in the eyes. The user demographic of recreational image capture devices, mainly high school girls, has a strong desire to enhance the impression of their eyes.

[0006] The present invention was completed based on the above-mentioned recognition of the problems by the inventors, and its main purpose is to provide an editing technique that allows the enjoyment of an effect similar to a catchlight through image processing. [Means for solving the problem]

[0007] An image processing device in one aspect of the present invention includes a photographing unit that photographs a user, a detection unit that detects the pupil area of ​​a person included in the photographed image, a processing unit that processes the pupil area of ​​the photographed image by overlaying a decorative image that represents a reflection in the pupil, an output unit that outputs the processed image, and a reception unit that receives a first user operation to select a decorative image that the user wants to apply from multiple types of decorative image candidates and a second user operation to move the decorative image, wherein the output unit displays an enlarged image to receive the second user operation, and the processing unit changes the position of the decorative image within the pupil area in accordance with the second user operation. [Effects of the Invention]

[0008] According to the present invention, an editing technique can be provided that allows the enjoyment of a catchlight-like effect through image processing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of a game image capturing system. [Figure 2] FIG. 2 is a perspective view of the appearance of the game image capturing device. [Figure 3] FIG. [Figure 4] 10 is a flowchart showing the flow of play in the game image capturing device. [Figure 5] FIG. 2 is a plan view of the game image capturing device seen from above. [Figure 6] FIG. 2 is a functional block diagram of the play image capturing device. [Figure 7] FIG. 10 is a diagram illustrating an edit screen. [Figure 8] FIG. 10 is a diagram showing a decoration selection area for catchlights. [Figure 9]FIG. 10 is a functional block diagram relating to catch lights. [Figure 10] FIG. 10 is a flowchart showing a process for processing catch lights. DETAILED DESCRIPTION OF THE INVENTION

[0010] The game image capturing device in this embodiment is equipped with a photo booth (photography space) and an editing booth (editing space). Users take photos of themselves in the photo booth. Hereinafter, images captured of the subject (user) in the photo booth will be referred to as "photographed images." In the photo booth, the game image capturing device applies image corrections to the captured images, such as whitening the skin and making the hair shiny (hereinafter, such image corrections will be referred to as "pre-correction"). In this embodiment, the game image capturing device captures not only still images but also moving images in the photo booth. Of the captured images, those acquired as moving images will be referred to as "photographed moving images," and those acquired as still images will be referred to as "photographed still images." In this embodiment, a maximum of seven sets of captured images (sets of captured moving images and captured still images) are captured.

[0011] In this embodiment, the shooting space and the editing space are described as being separate spaces, but the shooting space and the editing space may be the same space. In this case, a predetermined space may be used as the shooting space during shooting and as the editing space during editing.

[0012] The pre-edited captured images are sent to the editing booth. In the editing booth, the user can decorate the captured images as desired, such as by attaching stamp images or drawing with a stylus pen (hereinafter, such image decoration will be referred to as "decoration processing" or "editing"). The captured image to be edited in the editing booth is called the "target image." Among the target images, those that have been edited from captured moving images are called the "target moving images," and those that have been edited from captured still images are called "target still images." The target image after editing is called the "edited image," and among the edited images, those that have been edited from target moving images are called "edited moving images," and those that have been edited from target still images are called "edited still images." In this embodiment, a maximum of seven sets of captured images are acquired, and therefore a maximum of seven sets of edited images (seven edited moving images and seven edited still images) are generated.

[0013] The user selects a mount image that defines the layout and background of the edited still image. The user then composites the edited still image onto the selected mount image. Hereinafter, the final image obtained by embedding the edited still image into the mount image will be referred to as a "layout-completed image."

[0014] The game image capturing device prints the laid-out image onto a sticker mount (external medium). The game image capturing device also uploads edited moving images, edited still images, and laid-out images, which are images in which edited still images are embedded into mount images, to a server (external medium). Users can also download edited images, etc., by accessing the server from a user terminal such as a smartphone. As described above, the game image capturing device in this embodiment has a capturing function, an editing function (decorative processing function), a printing function, and a communication function.

[0015] The user selects a "presentation mode" at the reception of the game image capture device. In this embodiment, the presentation modes are assumed to be "purple mode" and "pink mode." In purple mode, purple is the base color, and in pink mode, pink is the base color. The base color here refers to the color that should form the foundation of the overall image, such as the central color in lighting, or the color that occupies the widest area on the screen. The game image photographing device changes the photographing environment, editing environment, completed image, etc. according to the presentation mode, particularly the color associated with the presentation mode.

[0016] FIG. 1 is a diagram showing the overall configuration of a game image capturing system 200. As shown in FIG. In the game image capturing system 200, the game image capturing device 100 is connected to a server 202 and a user terminal 206 (mobile device) via the Internet 204. The user captures a picture of themselves using the game image capturing device 100. The game image capturing device 100 acquires a captured image including the subject (user), applies pre-editing to the captured image as necessary, and accepts decorative processing by the user.

[0017] The edited image and the laid-out image after the decorative processing are uploaded to the server 202 together with the user ID from the game image photographing device 100. The server 202 stores the user ID, the photographing date and time, the edited image (edited moving image and edited still image), and the laid-out image (a composite image of the edited still image and the mount image) in association with each other. The user can download the edited image, etc. from the server 202 by accessing the server 202 by specifying the user ID.

[0018] FIG. 2 is a perspective view of the appearance of the game image capturing device 100. As shown in FIG. The game image capturing device 100 comprises a reception device 102, a photo booth 104, an editing booth 106, and a printing device 110. A user reserves the use of the game image capturing device 100 at the reception device 102. The user enters a user ID at the reception device 102 and pays a play fee in cash or electronic money. The user selects a presentation mode (purple mode or pink mode) on the reception device 102.

[0019] The user takes a photo in the photo booth 104. After taking the photo, the user moves to the editing booth 106. An editing device 126 is installed in the editing booth 106. The editing device 126 is equipped with an editing monitor 120. A target image (a photographed image to be decorated) is displayed on the editing monitor 120. In this embodiment, the editing monitor 120 is formed by a capacitive touch panel. Two stylus pens 122a and 122b are provided for one editing monitor 120. The user decorates the target image using the stylus pen 122 and their finger. After editing is completed, the printing device 110 prints the laid-out image on a sticker mount. The editing device 126 also uploads the edited image, etc. to the server 202.

[0020] FIG. 3 is a perspective view of the exterior of the photo booth 104. As shown in FIG. The photo booth 104 is shielded from the outside by a curtain 124. A photo taking device 128 is installed in the photo booth 104. The photo taking device 128 includes three lights 116 that illuminate the subject user from three directions, a camera 118 for capturing still images and moving images, a first photo taking monitor 112, and a second photo taking monitor 114.

[0021] In addition, lighting devices are embedded in the ceiling and floor surfaces. The lighting means, light 116, and ceiling and floor lights (not shown), all include R (red), G (green), B (blue), etc. For example, these lighting devices include full-color LEDs (Light Emitting Diodes) and white LEDs. Hereinafter, these three types of lighting devices will be collectively referred to as the "light group." The full-color LEDs can be illuminated with a base color specified by the performance mode. Hereafter, the purple (purple mode) light and pink (pink mode) light that can be set in a full-color LED will be referred to as "colored light."

[0022] The first photographing monitor 112 displays a preview of the user's photographed images (photographed moving images and photographed still images). The second photographing monitor 114 displays photographed images (hereinafter referred to as "sample images") of a person other than the user who is the subject, for example, a model. The user strikes any pose for photography in front of the camera 118 while referring to the sample images displayed on the second photographing monitor 114. Various explanations regarding photographing methods are also displayed on the second photographing monitor 114. As described above, photographed moving images and photographed still images can be acquired in the photographing booth 104. Note that full-color lighting (not shown) that imitates so-called "actress lights" may be provided around the first photographing monitor 112 and the second photographing monitor 114.

[0023] Furthermore, the first capture monitor 112 displays a live view image (full-length mirror image) that allows the user to check his or her own appearance, as well as a countdown image that indicates the timing of capture.

[0024] Fig. 4 is a flowchart showing the flow of play in the game image capturing device 100. Fig. 5 is a plan view of the game image capturing device 100 as seen from above. When a user makes a reservation for use at the reception device 102, the reception device 102 executes reception processing (S10). When receiving the reservation, the user also selects a performance mode. After receiving the reservation, the user enters the photo booth 104. At this time, the light group illuminates the photo booth 104 with colored light based on the color corresponding to the selected performance mode. For example, when the purple mode is selected, the light group illuminates the photo booth 104 with pale purple light. When the pink mode is selected, the photo booth 104 is filled with pale pink light.

[0025] The photographing device 128 acquires photographed images of the user (S12). In the photographing booth 104, the user can acquire a predetermined number of photographed images. Photographed moving images and photographed still images are acquired simultaneously, and a pair of photographed moving images and photographed still images is counted as one set of photographed images. In this embodiment, seven sets of photographed images (sets of photographed moving images and photographed still images) are acquired. The photographing device 128 applies pre-correction to the photographed images in accordance with the selected rendering mode. The photographing device 128 performs pre-correction each time a photographed image is acquired.

[0026] After shooting, instruction information instructing one of the two editing booths 106 is displayed on the first shooting monitor 112. The user moves to one of the two editing booths 106a, 106b, whichever is specified by the first shooting monitor 112. The shot image (target image) is transferred from the shooting device 128 to the editing device 126. The user applies decorative processing to the shot image (target image) transferred in the editing device 126 (S14). The user selects a mount image and creates a laid-out image by laying out the edited image on the mount image.

[0027] After editing, the editing device 126 sends the laid-out image to the printing device 110, which then prints the laid-out image on a sticker mount (S16). The editing device 126 also uploads the edited image and the laid-out image to the server 202 via the Internet 204 (S18). By accessing the server 202, the user can download the edited image, etc. from the server 202 to the user terminal 206.

[0028] Generally, editing takes longer than photographing. In the game image photographing device 100 of this embodiment, two editing booths 106a, 106b (editing devices 126a, 126b) are provided for one photographing booth 104, thereby increasing customer turnover.

[0029] FIG. 6 is a functional block diagram of the play image capturing device 100. As shown in FIG. Each component of the game image capture device 100 is realized by hardware including computing units such as a CPU (Central Processing Unit) and various co-processors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, as well as software stored in the storage devices that supplies processing instructions to the computing units. The computer program may be composed of device drivers, an operating system, various application programs located at higher layers than these, and libraries that provide common functions to these programs. As described above, the game image capture device 100 is configured as a collection of the capture device 128, the reception device 102, the editing device 126, and the printing device 110. Each block described below represents a functional block, not a hardware configuration.

[0030] The game image capture device 100 includes a user interface processor 130, a data processor 132, a communication unit 134, and a data storage unit 136. As described above, the game image capture device 100 also includes a camera 118 and a light group 210. The user interface processing unit 130 is responsible for user interface processing via the reception monitor 108, editing monitor 120, etc. The communication unit 134 is responsible for communication processing with the server 202 via the Internet 204. The communication unit 134 can also communicate directly with the user terminal 206 using short-range wireless communication such as Wi-Fi (registered trademark). For example, the communication unit 134 may directly transmit an edited image to the user terminal 206. The data storage unit 136 stores various data. The data processing unit 132 performs various processes based on the data acquired by the communication unit 134 and the data stored in the data storage unit 136.

[0031] The user interface processing unit 130 includes an input unit 140 that accepts operation inputs from the user, and an output unit 142 that presents information to the user.

[0032] The input unit 140 detects user operation inputs to touch panels installed in the reception device 102, the photographing device 128, the editing device 126, etc. The input unit 140 also detects operation inputs made by the stylus pen 122. The input unit 140 also detects a user ID in the reception device 102.

[0033] The output unit 142 executes various output processes such as image display on the reception device 102, the photographing device 128, the editing device 126, etc., as well as audio output and print output.

[0034] The output unit 142 includes a pause instruction unit 212 , a performance output unit 214 , and an audio output unit 216 . The pose instruction unit 212 selects a pose to be instructed to the user from the sample image. The performance output unit 214 displays various performance screens, such as sample images, on the second shooting monitor 114 or the first shooting monitor 112. The audio output unit 216 outputs audio and music.

[0035] Additionally, during shooting, the output unit 142 displays a live view image (full-length mirror image) of the user (subject) on the first shooting monitor 112. The user performs shooting while checking their own image in the live view image as if they were looking at a mirror. Also, immediately after shooting, the output unit 142 displays a preview of the shot still image on the first shooting monitor 112.

[0036] The data processing unit 132 includes a lighting control unit 218 , a photography processing unit 144 , an editing processing unit 146 , a reception processing unit 148 , and a printing processing unit 150 . The lighting control unit 218 controls the light group 210. Specifically, the lighting control unit 218 controls lighting settings such as turning on / off, light intensity, and color of the light group 210.

[0037] The photography processing unit 144 controls the photography processing in the photography booth 104. If the photography timing is time tX, the photography processing unit 144 records a video from 1.5 seconds before time tX to 0.1 seconds before time tX (hereinafter referred to as the "first half video"), acquires a photographed still image at time tX, and again records a video from 0.1 seconds after time tX to 1.5 seconds after time tX (hereinafter referred to as the "second half video"). The photography processing unit 144 generates a photographed video by connecting the first half video and the second half video. The photography processing unit 144 controls the camera 118 by sending a photography start instruction and a photography stop instruction to the camera 118. The photography processing unit 144 also fires a strobe (not shown) when acquiring a photographed still image. The photographed video does not include a video during the acquisition period of the photographed still image, in other words, a video when the strobe is fired, so the strobe light is not captured.

[0038] The shooting processing unit 144 can also acquire a captured video before the time tX of the shooting timing according to the user's selection, and acquire a captured still image immediately after acquiring the captured video.

[0039] The photography processing unit 144 includes a photography guidance unit 220. The photography guidance unit 220 notifies the user of the photography timing by instructing the output unit 142 to display a countdown and notify by voice.

[0040] The output unit 142 displays the photographed image as a target image on the editing monitor 120 (capacitive touch panel). The editing processing unit 146 decorates the target image in accordance with operational inputs from the user. The reception processing unit 148 manages the payment of play fees by users at the reception device 102. The reception processing unit 148 also accepts the selection of a presentation mode by the user. The reception processing unit 148 guides users who have made reservations to the photo booth 104 every time a photo shoot is completed in the photo booth 104. The print processing unit 150 controls the printing of the edited image on the printing device 110.

[0041] The communication unit 134 includes a receiving unit 230 that receives data and a transmitting unit 232 that transmits data.

[0042] FIG. 7 is a diagram showing the edit screen 240. The editing processing unit 146 creates graphic data for the editing screen 240 and displays the editing screen 240 on the output unit 142. The editing screen 240 is displayed on the editing monitor 120. In FIG. 7, a target image including two users 170a and 170b is the editing target. The editing processing unit 146 divides the editing screen 240 into a first editing area 162a (first editing screen) and a second editing area 162b (second editing screen). A first image display area 166a and a second image display area 166b are set in the first editing area 162a and the second editing area 162b, respectively. Note that two separate editing monitors 120 may be provided corresponding to the first editing area 162a and the second editing area 162b. The target images are displayed individually in the first image display area 166a and the second image display area 166b. In the edit screen mode setting information 160, two users 170a and 170b can decorate two target images individually.

[0043] In FIG. 7, user 170a decorates the target image displayed in first image display area 166a. User 170b decorates the target image displayed in second image display area 166b. Because editing monitor 120 is a capacitive touch panel, users input various operations using stylus pen 122 or their fingers, with the same feel as using a smartphone. User 170a uses stylus pen 122a located to the left of first editing area 162a. User 170b uses stylus pen 122b located to the right of second editing area 162b. Stylus pen 122a is connected to editing device 126 via conductive cord 172a, and user 170b is connected to editing device 126 via conductive cord 172b.

[0044] As described above, up to seven sets of captured images are acquired. One captured image includes a captured still image and a captured video. The output unit 142 displays the captured still image as a target image (target still image) in the image display area 166. Seven target still images are displayed overlapping each other in the image display area 166. The user can select a target still image to be edited by selecting one of seven tabs 168 (tab 168a and tab 168b). Furthermore, when the user touches the play button 190, the output unit 142 plays and displays approximately three seconds of target video corresponding to the target still image. The editing processing unit 146 may also reflect the results of decorative processing applied to the target still image in the target video.

[0045] The editing processing unit 146 manages the editing time limit. The output unit 142 displays a countdown of the remaining time limit in the remaining time display area 174. When the time limit expires, the editing processing unit 146 forcibly terminates the editing work. Even if the time limit has not yet expired, the user can forcibly terminate the editing process by touching the end button 176.

[0046] During editing, the user selects "Stamp," "Pen," "Text," or "Catchlight" from the decoration buttons 180 as appropriate. The output unit 142 switches the display of the decoration selection area 182 depending on the selection of the decoration button 180. Because "Stamp" is selected in the first editing area 162a in FIG. 7, the output unit 142 displays various stamp images in the decoration selection area 182. The user 170a selects a stamp image from the decoration selection area 182 and pastes the stamp image in any location in the first image display area 166a. Specifically, when the user 170a touches one of the stamp images with his or her finger or stylus pen 122a and touches the location in the first image display area 166a where the stamp image should be pasted, the editing processing unit 146 composites the stamp image with the target image.

[0047] The editing monitor 120 is a projected capacitive touch panel, and is capable of so-called multi-touch. This allows the user to hold and operate the stylus pen 122 in their right hand while selecting the decoration button 180 with their left hand. This allows the user to use both hands for editing, making editing more efficient. More efficient editing also reduces editing time, which is useful for increasing customer turnover.

[0048] FIG. 8 is a diagram showing a decoration selection area 182 for catchlights. When the user touches the catchlight decoration button 180, a catchlight decoration selection area 182 is displayed. In catchlight editing, a decorative image that expresses the reflection in the iris of the user's eye is superimposed on a portion of the iris of the user's eye displayed in the captured image (target still image). The decorative image corresponds to the reflection of the catchlight. In other words, the direct light of the catchlight appears to be reflected by the pupil. The decorative image reflects the color of the light source, and since white light is generally used as the light source, the decorative image is also white. However, if the light source is colored, the decorative image will also be colored. Furthermore, the decorative image is a lighter color than the color of the iris area, where direct light is not reflected. In this example, the same processing is performed on all of the user's eyes included in the captured image. The iris here refers to the iris and pupil. Catchlight editing is not processing the entire iris, as occurs when wearing colored contact lenses. It is processing only part of the iris.

[0049] The user selects a button with the decorative image they want to apply from the decorative image buttons. The decorative image buttons display multiple decorative image options. In this example, the decorative image buttons displayed are a heart button 321, a star button 322, a circle button 323, and a ring button 324.

[0050] When the user touches the heart button 321, a heart-shaped decorative image is pasted on a high-brightness area included in the iris area. Here, it is assumed that there is a light in the upper left as viewed from the user, and that the light is shining from the upper left of the user. Therefore, a reflective area appears in the upper right of the iris in the captured image. That is, in FIG. 8, there is a high-brightness area in the upper right of the iris as an image area corresponding to the reflective area. Therefore, a heart-shaped decorative image is drawn in the upper right position within the iris. Conversely, if light is shining from the upper right of the user, a reflective area appears in the upper left of the iris in the captured image. In that case, there is a high-brightness area in the upper left of the iris, and so a heart-shaped decorative image is drawn in the upper left position within the iris. Note that all decorative images represent reflections in the iris, and are set to a high-brightness color (for example, white).

[0051] When the user touches the star button 322, a star-shaped decorative image is placed over the high-brightness area. Therefore, just like in the case of the heart-shaped decorative image, the star-shaped decorative image is pasted at the upper right position within the pupil.

[0052] When the user touches the circle button 323, a circular decorative image is pasted at the bottom left position within the pupil. The position where the circular decorative image is pasted does not coincide with the position of the reflective area. In other words, the circular decorative image is drawn in a location other than the high-brightness area included in the pupil area. There are two ways to determine the bottom left position where the circular decorative image is pasted as in this example. One method is to determine it at a geometrically fixed position, and the other method is to determine it at a relative position based on the high-brightness area. This will be described in more detail below in relation to S26 in FIG. 10.

[0053] When the user touches the ring button 324, a circular decorative image is pasted into the pupil. The position where the circular decorative image is pasted is determined based on the center of the pupil. In other words, the center of the ring and the center of the pupil are made to overlap, or almost overlap.

[0054] In this way, each decorative image is placed at a position inside the iris of the eye. The internal position is not an arbitrary position within the iris of the eye, but a specific position determined by predetermined conditions. Hereinafter, the "internal position of the iris of the eye" where the decorative image is placed may be referred to as the "decorative image pasting position." This is because the decorative image is pasted at this position within the iris of the eye. When describing this position with a focus on the iris of the eye, it is referred to as the "internal position of the iris of the eye," and when describing this position with a focus on the decorative image, it is referred to as the "decorative image pasting position." The pasting position is determined depending on the type of decorative image. For heart-shaped and star-shaped decorative images, the pasting position is determined based on the original reflective high-brightness area, and the decorative image is overlaid to cover the entire high-brightness area. Covering the entire high-brightness area prevents the shape of the decorative image from being distorted. Furthermore, by shining a wider area than the original reflective area, the impression of the eyes can be enhanced. The positioning of the heart-shaped and star-shaped decorative images described here is an example in which the pasting position of the decorative image within the iris of the eye is determined based on pixel values. This is because the high-brightness area is determined by the brightness indicated by the pixel values. If the lighting is colored, for example, if a photograph is taken using pink lighting, a high-brightness area may be determined based on the presence of a pink color component. In this case, the color component is also determined by the pixel value.

[0055] The paste position in the iris area may be a geometric position based on the center of the iris or the outline of the iris. The paste position may be a point a predetermined distance away from the center of the iris in a predetermined direction. Alternatively, the paste position in the iris area may be a point (internal division point) that divides a radius from the center of the iris in a predetermined direction at a predetermined ratio. The position where a circular decorative image is pasted using the circle button 323 may be such a point a predetermined distance away from the center or an internal division point. Alternatively, the paste position in the iris area may be identified by matching using a circular pattern of a size corresponding to the iris. The paste position is geometrically determined within the circular pattern. In other words, the circular pattern is two-dimensional graphic information that includes a circle serving as a reference for alignment and coordinate information for the paste position defined within that circle. The circular pattern is aligned so that the reference circle overlaps the outline of the iris, and the point where the circle overlaps with the paste position within the circular pattern is defined as the internal position of the iris area. The position where a ring is pasted using the ring button 324 may be identified by matching such a circular pattern. The positioning of the circular decorative image and the annular decorative image described here is an example in which the internal position of the iris region is determined based on the geometric positional relationship within the iris region.

[0056] The user can change the position of the decorative image within the pupil area. When the user touches the up arrow button 331, the decorative image moves slightly upward. Similarly, when the right arrow button 332 is touched, the decorative image moves slightly to the right, when the down arrow button 333 is touched, the decorative image moves slightly downward, and when the left arrow button 334 is touched, the decorative image moves slightly to the left. The user can move the decorative image further by successively touching the arrow buttons. However, the decorative image must remain in a position that does not extend beyond the pupil area. Changing the position of the decorative image changes the impression the user makes. For example, aligning the decorative image with the pupil position will result in a light being directly in front of the user, creating a humorous impression as if the user is staring at a light source.

[0057] The user can change the size of the decorative image. When the user touches the large button 341 (enlarge button), the decorative image becomes slightly larger. If the user touches the large button 341 repeatedly, the decorative image can become even larger. When the user touches the small button 342 (reduce button), the decorative image becomes slightly smaller. If the user touches the small button 342 repeatedly, the decorative image can become even smaller. A button for directly specifying the size may also be used. For example, a large button, a medium button, and a small button may be presented, and the size of the decorative image may be determined by the button that is touched. Changing the size of the decorative image changes the impression the user has. For example, if the decorative image is made larger, the shape of the decorative image becomes clearer, making it easier to convey the feelings that were put into that shape.

[0058] The user can change the color of the decorative image. The default color of the decorative image is white, but when the user touches the pink button 352, the decorative image changes to a pinkish color. If the user touches the pink button 352 repeatedly, the pink color of the decorative image becomes even darker. Similarly, when the user touches the blue button 353, the decorative image changes to a bluish color, and when the user touches the yellow button 354, the decorative image changes to a yellowish color. When the user touches the white button 351, the decorative image returns to the default white color. Changing the color of the decorative image changes the impression given to the user. For example, a pink decorative image can express a blushing excitement. A blue decorative image can express a cool gaze.

[0059] The user operations are summarized below. When the user touches the catchlight decoration button 180, a catchlight decoration selection area 182 is displayed. As shown by the heart button 321, star button 322, circle button 323, and ring button 324, the default placement of the decoration image differs depending on the type of decoration image. When any of these buttons is touched, the decoration image is applied to both eyes of all people (except for closed eyes). Since the default placement is fixed, this operation alone can create a catchlight effect. After that, the user can make fine adjustments such as changing the location, size, or color of the decoration image. The user can freely change the impression of the eyes. Since adjustments can be made while viewing the image, it is easy to adjust the decoration image to one's liking.

[0060] FIG. 9 is a functional block diagram relating to catch lights. The input unit 140 includes a receiving unit 401. The receiving unit 401 receives various user operations, such as selecting a decorative image, moving a decorative image, specifying the color and size of a decorative image, and the like.

[0061] The editing processing unit 146 has a detection unit 411 and a processing unit 413. The detection unit 411 detects the iris area of ​​a person included in a photographed image. Specifically, the detection unit 411 detects a high-brightness area included in the iris area of ​​the photographed image. The processing unit 413 performs processing by superimposing a decorative image that expresses the reflection in the iris onto part of the iris area of ​​the photographed image.

[0062] FIG. 10 is a flow chart showing the process of catch light. When the reception unit 401 receives a touch on the catchlight decoration button 180, the output unit 142 displays a catchlight palette on the editing monitor 120 (S20).

[0063] When the reception unit 401 receives the selection of a decorative image by detecting a touch on a decorative image button (heart button 321, star button 322, circle button 323, or ring button 324) on the catchlight palette (Y in S22), processing is performed according to the type of decorative image.

[0064] First, a case where the heart button 321 or the star button 322 is touched will be described. The detection unit 411 detects a high-luminance area in the iris area of ​​each eye (S24). Specifically, the detection unit 411 determines a range showing a luminance equal to or greater than a predetermined reference value as a high-luminance area. Alternatively, the detection unit 411 may determine the area inside a circle of a predetermined radius centered on the pixel with the highest luminance in the iris area of ​​each eye as a high-luminance area. Alternatively, the detection unit 411 may perform edge processing on the image of each eye, and determine the area including the pixel showing the highest luminance among the areas surrounded by the edge line as a high-luminance area. Usually, an area that reflects direct light from a light source has a higher luminance than the white area of ​​the eye, which does not reflect direct light.

[0065] The processing unit 413 specifies the pasting position (S26). The pasting position is the position where a predetermined portion (e.g., the center) of the heart or star shape is superimposed. The pasting position is determined based on the high-luminance area. For example, a reference position such as the center of gravity of the high-luminance area or the point with the highest luminance becomes the pasting position. Then, the processing unit 413 pastes the decorative image so that the predetermined portion (e.g., a reference position such as the center or center of gravity) of the heart or star shape coincides with the pasting position within the pupil (S28).

[0066] Next, a case where the circle button 323 is touched will be described. The processing unit 413 determines the position where the circular decorative image will be pasted (S26). As described above, the pasting position is a point a predetermined distance away from the center of the iris in a predetermined direction, or a point where a radius from the center of the iris in a predetermined direction is divided at a predetermined ratio (internal division point). Therefore, the high-brightness area detection process (S24) may be omitted. Alternatively, the position where the circular decorative image will be pasted may be determined according to the high-brightness area within the iris. In this case, the detection unit 411 detects the high-brightness area in the iris area of ​​each eye (S24), as described above. For example, the processing unit 413 determines the position of the pasting position to be a location on the opposite side of the high-brightness area within the iris, using the center of the iris as a reference (S26). More specifically, the position of the pasting position is a position symmetrical to the center of the high-brightness area, using the center of the iris as a reference. If the high-brightness area is located in the upper left of the iris area, the pasting position is set to the lower right of the iris area. If the high-brightness area is not located in a high-brightness area, the pasting position may be a location other than the opposite side of the high-brightness area. Then, the processing unit 413 pastes a circular decorative image at the paste position (S28).

[0067] Next, a case where the ring button 324 is touched will be described. The detection unit 411 identifies the center position of the iris area of ​​each eye. The processing unit 413 uses the center position of the iris area as a reference for the position where the circular decorative image is to be attached (S26). The processing unit 413 attaches the circular decorative image so that the center of the circular decorative image overlaps the center position of the iris area (S28).

[0068] When the receiving unit 401 receives an instruction to move the decorative image by detecting a touch on each of the arrow buttons 331 to 334 in the catchlight palette (Y in S30), the processing unit 413 moves the decorative image as described above (S32).

[0069] When the reception unit 401 receives an instruction to size the decorative image by detecting a touch on the Enlarge button 341 (enlarge button) or the Small button 342 on the catchlight palette (Y in S34), the processing unit 413 enlarges or reduces the decorative image as described above (S32).

[0070] When the receiving unit 401 receives a color specification for the decorative image by detecting a touch on the color buttons 351 to 354 in the catchlight palette (Y in S38), the processing unit 413 changes the color of the decorative image as described above (S40).

[0071] When the edit processing unit 146 determines that the time has come to end the edit (Y in S42), it ends the illustrated process.

[0072] The processing unit 413 pastes the decorative image excluding the portion that protrudes from the pupil area. In other words, the decorative image is not overlapped with the eyelid or eyelashes.

[0073] [Variations] In the embodiment, an example is shown in which an instruction to move a decorative image is given using arrow buttons, but the output unit 142 may display an enlarged image of the area around the eye, the receiving unit 401 may receive a slide operation on the enlarged image via a touch panel, and the processing unit 413 may move the decorative image according to the slide direction and amount.

[0074] In the embodiment, an example has been shown in which catchlight processing is applied to both eyes of all people included in the captured image. However, catchlight processing may be applied only to both eyes of a person selected by the user. The output unit 142 displays a reduced image of the target still image, and the receiving unit 401 receives the person touched on the reduced image as a target for catchlight processing. The detection unit 411 and the processing unit 413 perform the above-described processing on the eyes of the person to be processed. In this way, different decorative images can be attached for each user. Furthermore, catchlight processing may be applied only to one eye specified by the user for a person selected by the user or for all people. Furthermore, different catchlight processing may be applied to the left and right eyes.

[0075] The processing unit 413 may automatically determine the type and color of the decorative image according to the settings of the user information. The server 202 manages various information related to the user (hereinafter referred to as "user information"). The data storage unit of the server 202 stores the user information in association with the user ID. The user can edit the user information as desired by specifying the user ID from the user terminal 206 and logging in to the server 202. The user information defines the user's attributes, preferences, etc. The user information includes basic information such as the user's age, My Palette information indicating the user's preferred editing settings, and My Catchlight information indicating the user's preferred catchlight settings. The My Catchlight information includes the type of decorative image (e.g., heart-shaped, star-shaped, etc.) and the color of the decorative image (e.g., white, pink, etc.). When a user starts playing with the game image capturing device 100, the ID acquisition unit of the game image capturing device 100 acquires the user ID from the user card held over the reader of the game image capturing device 100 by the user. The ID sending unit of the game image capturing device 100 sends the acquired user ID to the server 202. The ID receiving unit of the server 202 receives the user ID from the game image capturing device 100. The user information sending unit of the server 202 sends user information corresponding to the received user ID to the game image capturing device 100. The user information receiving unit of the game image capturing device 100 receives user information of the user playing from the server 202. Then, the processing unit 413 of the game image capturing device 100 performs the above-mentioned pasting process using a decorative image of the type and color set in the My Catchlight information of the user information.

[0076] The processing unit 413 may automatically determine the type of decorative image according to the rendering mode. In this case, it is assumed that the rendering mode and the type of decorative image are associated with each other. For example, when the rendering mode is the pink mode, the processing unit 413 uses a heart-shaped decorative image corresponding to the pink mode. When the rendering mode is the purple mode, the processing unit 413 uses a star-shaped decorative image corresponding to the purple mode. The processing unit 413 may also automatically determine the color of the decorative image according to the rendering mode. For example, when the rendering mode is the pink mode, the processing unit 413 uses a pink decorative image. When the rendering mode is the purple mode, the processing unit 413 uses a purple decorative image. In this way, the effect of emphasizing the color of the lighting at the time of shooting and reflecting it in the captured image can be obtained.

[0077] The decorative image may have the same shape as a known symbol. Since known symbols have meanings, it is possible to have fun by using the decorative image to suggest the user's feelings. For example, a heart symbol can mean "I like it," a star symbol can mean "It's shiny," and a musical note symbol can mean "It's fun."

[0078] As mentioned above, after taking a photo, users can select from several types of "catchlight images" (decorative images) to add to the eyes. When a user selects a "catchlight image," that "catchlight image" is automatically placed in line with the position of the user's pupil. Within the range of the pupil, the user can fine-tune (change) the position of the "catchlight image." Each "catchlight image" is adjusted to balance well with the "natural catchlight" that appears due to the position of the lighting in the photo booth. By devising a "catchlight image" for post-processing, users can have fun adding "realistic catchlights reflected in the eyes" and "catchlight images" that appear in the best lighting conditions without adversely affecting the lighting conditions at the time of shooting the original image for post-processing purposes, providing a high level of satisfaction in both cases.

[0079] Unlike functions such as color contact lenses that change the pupil color by adding an image inside the eye or functions that emphasize the "natural catchlight" itself, the embodiment and modified examples provide a function that allows the user to add a "catchlight image" in post-processing. Also, the user can change the position of the catchlight in the eye while considering the balance with the "natural catchlight," which adds to the fun factor.

[0080] The user demographic of the play image capture device 100, primarily female high school students, has a strong desire to enhance the appearance of their eyes. This is evident from the fact that many users prefer eye enlargement processing. As described above, decorations that mimic catchlights can enhance the appearance of eyes. Furthermore, by allowing users to select decorative images, users can enjoy expressing their individuality by choosing their favorite decorative image. For example, heart-shaped and star-shaped lights are rare, and it is difficult to actually capture heart-shaped or star-shaped reflections in the eyes. However, the image processing described above makes it possible to generate special images by simulating catchlights, which are difficult to install in reality.

[0081] As mentioned above, the placement of the decorative image may vary depending on its shape. Lights of each shape are assumed to be placed in various locations. In the example above, the heart-shaped and star-shaped lights are assumed to be in the same location as the actual lighting, so the heart-shaped and star-shaped reflection areas are drawn in the same location as the original reflection area. The circular light is assumed to be diagonally opposite the actual lighting, so a circular reflection area is drawn opposite the original reflection area. The ring-shaped light is assumed to be directly in front of the user, so a ring-shaped reflection area is drawn concentric with the pupil. In this way, a variety of lighting environments are assumed, allowing for a variety of processed images to be obtained.

[0082] The present invention is not limited to the above-described embodiments and modifications, and the components can be modified without departing from the spirit of the invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments and modifications. Furthermore, some components can be omitted from all the components shown in the above-described embodiments and modifications.

[0083] In the above description, the game image capturing system 200 is configured by the game image capturing device 100, the server 202, and the user terminal 206, and in particular the game image capturing device 100 and the server 202. However, some of the functions of the game image capturing device 100 may be realized by the server 202, or some of the functions of the server 202 may be assigned to the game image capturing device 100. Also, some of the functions of these devices may be performed by a third device other than the game image capturing device 100 and the server 202. How the multiple functions required to realize the present invention are allocated to one or more pieces of hardware may be determined in consideration of the processing capabilities of each piece of hardware and the specifications required for the game image capturing system 200.

[0084] As described above, editing input can be performed on the editing screen 240 using either the stylus pen 122 or a finger. The editing processing unit 146 can determine whether a touch has been made with the stylus pen 122 or a finger by measuring capacitance. A decoration selection area 182 (editing tool) for input with the stylus pen 122 and a decoration selection area 182 (editing tool) for input with a finger may be provided separately. Furthermore, even when the same stamp image is touched, the display mode (color, shape, character pose, etc.) of the stamp image may be different when touched with the stylus pen 122 and when touched with a finger.

[0085] As described above, in the game image capturing device 100 of this embodiment, the photo booth 104 (photography space) and the editing booth 106 (editing space) are configured separately, but the photo booth 104 and the editing booth 106 may be shared. For example, not only the photo capturing device 128 but also the editing device 126 may be installed in the photo capturing booth 104 shown in FIG. 2. After capturing images using the photo capturing device 128, the user may continue editing using the editing device 126 in the photo capturing booth 104. In this case, the lighting control unit 218 simply controls the light group 210 installed in the photo capturing booth 104 according to the performance mode. [Explanation of symbols]

[0086] 100 game image photographing device, 102 reception device, 104 photographing booth, 106 editing booth, 108 reception monitor, 110 printing device, 112 first photographing monitor, 114 second photographing monitor, 116 light, 118 camera, 120 editing monitor, 122 stylus pen, 124 curtain, 126 editing device, 128 photographing device, 130 user interface processing unit, 132 data processing unit, 134 communication unit, 136 data storage unit, 140 input unit, 142 output unit, 144 photographing processing unit, 146 editing processing unit, 148 reception processing unit, 150 print processing unit, 162a first editing area, 162b second editing area, 166 image display area, 166a first image display area, 166b second image display area, 168 tab, 170a user, 170b User, 172a conductive cord, 172b conductive cord, 174 remaining time display area, 176 end button, 180 decoration button, 182 decoration selection area, 190 play button, 200 game image shooting system, 202 server, 204 Internet, 206 user terminal, 210 light group, 212 pose instruction unit, 214 performance output unit, 216 audio output unit, 218 lighting control unit, 220 shooting guide unit, 230 receiving unit, 232 transmitting unit, 240 editing screen, 321 heart button, 322 star button, 323 circle button, 324 ring button, 331 up arrow button, 332 right arrow button, 333 down arrow button, 334 left arrow button, 341 large button, 342 small button, 351 white button, 352 Pink button, 353 Blue button, 354 Yellow button, 401 Reception unit, 411 Detection unit, 413 Processing unit

Claims

1. an imaging unit that images a user; a detection unit that detects the pupil area of ​​a person included in a photographed image; a processing unit that performs processing to superimpose a decorative image that expresses reflection in the iris on a part of the iris region of the photographed image; an output unit that outputs the processed image; a receiving unit that receives a first user operation for selecting the decorative image that the user wants to apply from among a plurality of types of decorative image candidates, and a second user operation for moving the decorative image, the output unit displays an enlarged image for receiving the second user operation; The processing unit is an image processing device that changes the position of the decorative image within the pupil area in response to the second user operation.

2. The image processing device according to claim 1 , wherein the processing unit overlays the decorative image on only one eye of the person, or overlays different decorative images on the left and right eyes of the person.

3. The image processing device according to claim 1 , wherein the processing unit uses the decorative image according to user information set in relation to the user.

4. The image processing device according to claim 1 , wherein the processing unit uses the decorative image according to a presentation mode selected by a user as a basis for presentation in the image processing device.

Images