Image processing device, image processing program, and mask generation method

Abstract

To remove catchlight from image data.SOLUTION: An image processing apparatus comprises: an identification section that identifies positions of at least one or more specific portions from input image data including at least a part of a face of a person using a first learned model obtained by learning a relationship between a plurality of sets of image data including persons' eyes and position information on specific portions of eyes in the image data; and a mask generation section that generates a catchlight mask for discriminating a catchlight region from other regions for image data for each region identified by the identification section using a second learned model obtained by learning a relationship between the plurality of sets of image data including specific portions of the persons' eyes and the catchlight region in the image data.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing program, and a mask generation method for processing specific parts of a face shown in a captured image.

Background Art

[0002] When taking a photo, if there is a white reflection, i.e., a catch light, in a person's eyes, for example, an effect such as the person's expression becoming vivid can be obtained. For example, there are cases where there is no catch light in at least one eye or the shape is not ideal, and there is a desire to remove an undesirable catch light from the captured photo. There is also a desire to add an ideal catch light to the captured photo.

[0003] There is also a technology that can erase the catch light in a captured image and then draw a new preferable catch light (see, for example, Patent Document 1). However, with the technology described in Patent Document 1, there was also a possibility that an ideal catch light would not be added to each person.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the present invention, a catch light mask used for removing the catch light included in image data is generated.

Means for Solving the Problems

[0006] A first aspect of the image processing apparatus according to the present invention includes: an identification unit that identifies the location of at least one specific part from input image data including at least a part of a person's face, using a first trained model that has learned the relationship between image data including multiple sets of people's eyes and positional information of specific parts of the eyes in the image data; and a mask generation unit that generates a catchlight mask for each region identified by the identification unit, using a second trained model that has learned the relationship between image data including multiple sets of people's eyes and catchlight regions in the image data, thereby distinguishing catchlight regions from other regions.

[0007] This allows for the generation of a catchlight mask, which is used to remove catchlights from people in image data.

[0008] The input image data may further include an image processing unit that removes catchlights by replacing catchlight areas identified by a catchlight mask with colors determined based on the colors of the surrounding areas.

[0009] This makes it possible to remove catchlights from people in image data.

[0010] The system may further include an image correction unit that generates corrected image data by adding a predetermined pseudo-catchlight to image data from which the catchlight has been removed.

[0011] This allows for the addition of an ideal catchlight to the person included in the image data.

[0012] The image correction unit may set the position of the pseudo-catchlight according to the position information of the catchlight area.

[0013] This allows for the addition of ideal catchlights to a person.

[0014] When the catchlight area is not specified for each eye of a person included in the input image data, and when the input image data includes the eyes that are paired for the person, the image correction unit may set the position of a pseudo-catchlight according to the position information of the catchlight area of the paired eyes.

[0015] Thereby, an ideal catchlight can be added to the person.

[0016] The image correction unit may set the position of a pseudo-catchlight according to the position and size of a specific part of the eyes of a person included in the input image data.

[0017] Thereby, an ideal catchlight can be added to the person. <00 enim="0000067">

[0018] The image correction unit may set the color of a pseudo-catchlight according to the color of a specific part of the eyes of a person included in the input image data.

[0019] Thereby, an ideal catchlight can be added to the person.

Advantages of the Invention

[0020] According to the present invention, a catchlight mask used for removing the catchlight included in a photograph can be generated.

Brief Description of the Drawings

[0021] [Figure 1] It is a perspective view of a photo sticker creating device which is an embodiment of the present invention. [Figure 2] It is a perspective view of a photo sticker creating device. [Figure 3] It is a diagram for explaining the spatial movement of a user during a photo sticker creating game. [Figure 4] It is a front view of a preselection unit. [[ID=4)] <000009]]It is a front view of a photographing unit. [Figure 6] It is a front view of the shooting space side of a background unit. [Figure 7] It is a front view of the editing space side of the editing unit. [Figure 8] It is a left side view of the editing unit. [Figure 9] It is a block diagram showing the internal configuration of the photo sticker creating device. [Figure 10] It is a block diagram showing the functional blocks of the photo sticker creating device. [Figure 11A] It is a schematic diagram for explaining the generation of the first learned model. [Figure 11B] It is an example of the input and output of the first learned model. [Figure 11C] It is a schematic diagram for explaining the generation of the second learned model. [Figure 11D] It is an example of the input and output of the second learned model. [Figure 12A] It is an example image of the eye area which is a specific part. [Figure 12B] It is an example of a catchlight mask generated from the image example of FIG. 12A. [Figure 12C] It is an example of an image in which the catchlight is removed from the image of FIG. 12A using the catchlight mask of FIG. 12B. [Figure 12D] It is an example of the first pseudo catchlight mask added to the image example of FIG. 12C. [Figure 12E] It is an example of the second pseudo catchlight mask added to the image example of FIG. 12C. [Figure 12F] It is an example of the third pseudo catchlight mask added to the image example of FIG. 12C. [Figure 12G] It is an example in which the pseudo catchlight masks of FIGS. 12D to 12F are added to the image example of FIG. 12C. [Figure 13] It is a flowchart showing a series of operations related to the photo sticker creation game by the photo sticker creating device. [Figure 14] It is a flowchart showing the details of the synthesis process. [[ID=5,2]] [Figure 15] It is a flowchart showing the details of the pseudo catchlight addition process. [Modes for carrying out the invention]

[0022] Hereinafter, specific embodiments of the present invention will be described with reference to the attached drawings. Components having the same function will be denoted by the same reference numeral, and their descriptions will be omitted as appropriate. Furthermore, the shapes of the components shown in each drawing, as well as dimensions such as length, depth, and width, do not reflect the actual shapes and dimensions, but have been modified as appropriate for clarity and simplification of the drawings.

[0023] The image processing device according to the present invention will be described as a photo sticker creation device as an example. The photo sticker creation device to which the present invention is applied is a game device that allows users to take and edit photos as a game, and provides the users with the taken and edited images as photo stickers or data. The photo sticker creation device 1 is installed, for example, in game centers, shopping malls, and stores in tourist areas.

[0024] In a game provided by a photo sticker creation device, users take photos of themselves or others using the camera installed in the device. Users then edit the captured images by compositing foreground and / or background images, or by using editing functions (doodle editing functions) such as pen image input and stamp image input as editing / compositing images, to design colorful and vibrant images. After the game ends, users receive photo stickers or other items printed with their edited images as their final product. Alternatively, the photo sticker creation device can provide the edited images to the user's mobile device, and the user can receive the final product via their mobile device.

[0025] (Configuration of the photo sticker creation device) Figures 1 and 2 are perspective views showing an example of the external configuration of the photo sticker creation device 1. The photo sticker creation device 1 is a game machine that provides captured and edited images. The photo sticker creation device 1 provides images to users by printing them on sticker paper or by sending the images to a server so that they can be viewed on the user's mobile device. The users of the photo sticker creation device 1 are mainly young women, such as junior high and high school girls. With the photo sticker creation device 1, multiple users, usually two or three people per group, can enjoy the game. Of course, one user can also enjoy the game with the photo sticker creation device 1.

[0026] In the photo sticker creation device 1, the user takes a photograph of themselves. Through the editing process, the user selects an image from the captured images and combines it with other images such as handwritten text or stamp images. This edits the captured image into a more visually appealing image. The user then receives a sticker sheet printed with the edited image, completing the game.

[0027] As shown in Figure 1, the photo sticker creation device 1 is basically configured by installing the shooting unit 11 and the editing unit 12 in contact with each other.

[0028] The shooting unit 11 consists of a pre-selection operation unit 20, a shooting operation unit 21, and a background unit 22. The pre-selection operation unit 20 is installed on the side of the shooting operation unit 21. The space in front of the pre-selection operation unit 20 becomes the pre-selection space where the pre-selection process is performed. The shooting operation unit 21 and the background unit 22 are installed at a predetermined distance apart. The space formed between the shooting operation unit 21 and the background unit 22 becomes the shooting space where the shooting process is performed.

[0029] The pre-selection operation unit 20 performs pre-selection processing, such as providing guidance introducing games offered by the photo sticker creation device 1, and processing to allow the user to select various settings for the shooting process that takes place in the shooting space. The pre-selection operation unit 20 is equipped with a coin slot for the user to insert money and a touch panel monitor used for various operations. The pre-selection operation unit 20 guides users in the pre-selection space to the shooting space as appropriate, depending on the availability of the shooting space.

[0030] The shooting control unit 21 is a device for photographing the user as the subject. The shooting control unit 21 is located in front of the user when they enter the shooting space. The front of the shooting control unit 21 facing the shooting space is equipped with a camera and a touch panel monitor used for various operations. If the left side is defined as the left side and the right side as viewed from the perspective of a user facing forward in the shooting space, the left side of the shooting control unit 21 is composed of a side panel 41A, and the right side is composed of a side panel 41B. Furthermore, the front of the shooting control unit 21 is composed of a front panel 42. The pre-selection control unit 20 described above is installed on the side panel 41A. The pre-selection control unit 20 may be installed on the side panel 41B, or on both side panels 41A and 41B.

[0031] The background section 22 consists of a rear panel 51, a side panel 52A, and a side panel 52B. The rear panel 51 is a plate-shaped member located on the back side of a user facing forward. The side panel 52A is attached to the left end of the rear panel 51 and is a plate-shaped member that is narrower in width than the side panel 41A. The side panel 52B is attached to the right end of the rear panel 51 and is a plate-shaped member that is narrower in width than the side panel 41B.

[0032] Side panels 41A and 52A are installed in substantially the same plane. The upper parts of side panels 41A and 52A are connected by a connecting part 23A, which is a plate-shaped member. The lower parts of side panels 41A and 52A are connected by a connecting part 23A', which is a metal member installed on the floor surface. Similarly, side panels 41B and 52B are installed in substantially the same plane. The upper parts of side panels 41B and 52B are connected by a connecting part 23B. The lower parts of side panels 41B and 52B are connected by a connecting part 23B'.

[0033] Furthermore, a sheet for chroma keying, such as a green sheet, is attached to the side of the rear panel 51 facing the shooting space. The photo sticker creation device 1 performs chroma keying during the shooting and editing processes by shooting with the chroma keying sheet as the background. As a result, the background image desired by the user is superimposed onto the sheet.

[0034] The opening formed by the side panel 41A, the connecting section 23A, and the side panel 52A serves as an entrance and exit to the shooting space. Additionally, the opening formed by the side panel 41B, the connecting section 23B, and the side panel 52B also serves as an entrance and exit to the shooting space.

[0035] Above the shooting space, a ceiling is formed, surrounded by the front of the shooting control unit 21, the connecting section 23A, and the connecting section 23B. A ceiling strobe unit 24 is installed in a part of this ceiling. One end of the ceiling strobe unit 24 is fixed to the connecting section 23A, and the other end is fixed to the connecting section 23B. The ceiling strobe unit 24 has a built-in strobe that illuminates the shooting space in accordance with the shooting. Inside the ceiling strobe unit 24, in addition to the strobe, a fluorescent lamp is also installed. As a result, the ceiling strobe unit 24 also functions as lighting for the shooting space.

[0036] The editing unit 12 is a device for editing captured images. The editing unit 12 is connected to the shooting unit 11 such that one side of the editing unit 12 is in contact with the front panel 42 of the shooting operation unit 21.

[0037] If the configuration of the editing unit 12 shown in Figure 1 is considered the front side configuration, then the editing unit 12 has components for use in editing work on both its front and back sides. This configuration allows two sets of users to perform editing work simultaneously.

[0038] The front side of the editing unit 12 consists of a surface 61 and a sloped surface 62 formed above surface 61. Surface 61 is perpendicular to the floor and is approximately parallel to the side panel 41A of the shooting operation unit 21. The sloped surface 62 is equipped with a tablet-integrated monitor and a stylus pen for use in editing work. A columnar support 63A is provided on the left side of the sloped surface 62 to support one end of the lighting device 64. A columnar support 63B is provided on the right side of the sloped surface 62 to support the other end of the lighting device 64. A support 65 for supporting the curtain rail 26 is provided on the upper surface of support 63A.

[0039] A curtain rail 26 is installed above the editing unit 12. The curtain rail 26 is composed of three rails 26A, 26B, and 26C. The three rails 26A, 26B, and 26C are combined so that when viewed from above, they form a U-shape. One end of rails 26A and 26B, which are installed parallel to each other, is fixed to connecting parts 23A and 23B, respectively, and the other ends of rails 26A and 26B are joined to both ends of rail 26C, respectively.

[0040] Curtains are attached to the curtain rail 26 so that the front front and rear front spaces of the editing unit 12 are not visible from the outside. The front front and rear rear spaces of the editing unit 12, enclosed by these curtains, become the editing space where the user performs their editing work.

[0041] As will be described later, the left side of the editing unit 12 is provided with an outlet from which the printed sticker paper is ejected. The space in front of the left side of the editing unit 12 becomes a waiting area where users wait for the printed sticker paper to be ejected.

[0042] (Personal movement) Here, we will explain the flow of the image creation game and the user's movement during it. Figure 3 is a diagram illustrating the user's spatial movement during the photo sticker creation game.

[0043] First, the user inserts the payment into the coin slot in the pre-selection space A0, which is the space in front of the pre-selection operation unit 20. Next, the user makes various settings according to the screen displayed on the touch panel monitor. For example, the user makes pre-selection tasks such as selecting a course related to the shooting to be performed in the shooting space.

[0044] After completing the pre-selection process, the user enters the shooting space A1 formed between the shooting control unit 21 and the background unit 22 through the entrance / exit G1 between the side panel 41A and the side panel 52A, as indicated by the white arrow #1. The user then performs the shooting operation using the camera and touch panel monitor provided on the shooting control unit 21.

[0045] After completing the shooting process, users can either exit shooting space A1 through entrance / exit G1 as indicated by the white arrow #2 and move to editing space A2-1, or exit shooting space A1 through entrance / exit G2 as indicated by the white arrow #3 and move to editing space A2-2.

[0046] Editing space A2-1 is the editing space on the front side of the editing unit 12. On the other hand, editing space A2-2 is the editing space on the rear side of the editing unit 12. Users are guided to which editing space A2-1 or editing space A2-2 to move to by the screen display on the touch panel monitor of the shooting operation unit 21. For example, the user is guided to move to the available of the two editing spaces. Once a user moves to editing space A2-1 or editing space A2-2, they begin editing work. Users in editing space A2-1 and users in editing space A2-2 can perform editing work simultaneously.

[0047] After the editing process is complete, printing of the edited image will begin. Once printing starts, users who have finished editing in editing space A2-1 will move from editing space A2-1 to the print waiting space A3, as indicated by the white arrow #4. Similarly, users who have finished editing in editing space A2-2 will move from editing space A2-2 to the print waiting space A3, as indicated by the white arrow #5.

[0048] The user moves to the printing waiting area A3 and waits for the image printing to finish. Once printing is complete, the user receives the sticker paper ejected from the output slot on the right side of the editing unit 12, thus completing the image creation game.

[0049] (Configuration of the pre-selection section) Next, the configuration of each device will be described. Figure 4 is a front view of the pre-selection operation unit 20.

[0050] A touch panel monitor 71 is provided above the pre-selection operation unit 20. The touch panel monitor 71 consists of a monitor such as an LCD (Liquid Crystal Display) and a touch panel stacked on it. The touch panel monitor 71 has the function of displaying various GUIs (Graphical User Interfaces) and accepting selection operations from the user. The touch panel monitor 71 displays screens used for pre-selection processing, such as selecting a course for shooting, selecting a background image for the image to be edited, arranging the layout of the created image, selecting BGM (Background Music) to be played during the image creation game, selecting at least one of sound and voice, and entering the user's name.

[0051] The photography courses offered include a two-person course for two people and a larger group course for three or more people. A couples course for male-female couples may also be available.

[0052] A speaker 72 is provided below the touch panel monitor 71. The speaker 72 outputs voice guidance for pre-selection processing, background music, sound effects, etc. Adjacent to the speaker 72 is a coin slot 73 where users insert and return coins.

[0053] (Composition of the filming department) Figure 5 is a front view of the shooting control unit 21, which functions as the shooting unit. The shooting control unit 21 is configured to be surrounded by side panel 41A, side panel 41B, and front panel 42.

[0054] A camera unit 81 is provided in the center of the front panel 42. The camera unit 81 consists of a camera 91 and a touch panel monitor 92 which serves as a display unit.

[0055] Camera 91 is, for example, a single-lens reflex camera and is mounted inside camera unit 81 so that the lens is exposed. Camera 91 has an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor and takes pictures of the user in shooting space A1. Moving images captured by camera 91 (hereinafter also referred to as live view display images) are displayed in real time on touch panel monitor 92. Still images captured by camera 91 at predetermined timings, such as when a shot is instructed, are saved as captured images.

[0056] The touch panel monitor 92 is located below the camera 91. The touch panel monitor 92 consists of a monitor such as an LCD and a touch panel stacked on it. The touch panel monitor 92 has the function of a live view monitor that displays moving images captured by the camera 91, and the function of displaying various GUIs and accepting selection operations from the user. Specific examples of such selection operations include selecting a shooting course, giving instructions to start and end shooting (shooting control instructions), selecting the degree of eye deformation and color, and the degree of skin color correction, selecting an image to be used as the background image in the created image, and selecting background music (sound / voice) during the image creation game. Moving images (live view images) and still images (captured images) captured by the camera 91 are displayed on the touch panel monitor 92.

[0057] An upper strobe 82 is installed above the camera unit 81, with its curved light-emitting surface facing the user. The upper strobe 82 illuminates the user's face and upper body from directly above.

[0058] Furthermore, a foot-level strobe 85 is provided below the camera unit 81 to illuminate the user's lower body and feet. The upper strobe 82 and the foot-level strobe 85 each contain a strobe and a fluorescent lamp.

[0059] Although not shown in Figures 1 and 5, a speaker 93 is provided on the front panel 42, for example, near the ceiling. The speaker 93 outputs voice guidance for the shooting process, background music, sound effects, etc., based on the audio signal output from the control unit 201.

[0060] (Composition of the background section) Figure 6 is a front view of the background section 22 from the shooting space A1 side.

[0061] An upper rear strobe 101 is installed above the rear panel 51. The upper rear strobe 101 illuminates the user from above the rear.

[0062] In the diagram, a rear left strobe 102 is installed to the left of the rear panel 51. The rear left strobe 102 illuminates the user from the rear right. In the diagram, a rear right strobe 103 is installed to the right of the rear panel 51. The rear right strobe 103 illuminates the user from the rear left.

[0063] Additionally, a chroma key sheet 121 may be attached to the side of the rear panel 51 facing the shooting space A1 (the front side in the diagram). The color of the chroma key sheet 121 is, for example, green.

[0064] Although not shown in the diagram, chroma key sheets may also be attached to the sides of the side panels 52A and 52B facing the shooting space A1, similar to the chroma key sheet 121.

[0065] (Composition of the editorial unit) Figure 7 is a front view of the editing unit 12 on the editing space A2-1 side.

[0066] A tablet-integrated monitor 131 is provided near the center of the slope 62. A stylus pen 132A is provided to the left of the tablet-integrated monitor 131. A stylus pen 132B is provided to the right of the tablet-integrated monitor 131. The tablet-integrated monitor 131 is configured such that the tablet's display is exposed. The tablet allows for operation input using either the stylus pen 132A or the stylus pen 132B. The tablet-integrated monitor 131 displays, for example, an editing screen used for editing work. For example, when two users are performing editing work simultaneously, the stylus pen 132A is used by the user on the left side of the tablet-integrated monitor 131, and the stylus pen 132B is used by the user on the right side of the tablet-integrated monitor 131.

[0067] Figure 8 is a left side view of the editing unit 12.

[0068] A sticker paper output port 161 is provided on the lower left side of the editing unit 12. A printer is provided inside the editing unit 12 as an output unit. The printer prints an image of the user in editing space A2-1, or an image of the user in editing space A2-2, onto sticker paper in a predetermined layout, and the printed image is ejected from the sticker paper output port 161.

[0069] (Internal configuration of a photo sticker creation device) Figure 9 is a block diagram showing an example of the internal configuration of the photo sticker manufacturing device 1. In Figure 9, components identical to those described above are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0070] The control unit 201 is composed of a CPU (Central Processing Unit) and other components. The control unit 201 executes programs stored in the ROM (Read Only Memory) 206 and the memory unit 202, and controls the overall operation of the photo sticker creation device 1. The memory unit 202, communication unit 203, drive 204, ROM 206, and RAM (Random Access Memory) 207 are connected to the control unit 201. The pre-selection operation unit 20, shooting operation unit 21, background unit 22, editing operation units 27A, 27B, and printing operation unit 28 are also connected to the control unit 201.

[0071] The memory unit 202 is a non-volatile storage medium such as a hard disk or flash memory. The memory unit 202 stores various setting information supplied from the control unit 201. The information stored in the memory unit 202 is read out by the control unit 201 as needed.

[0072] The communication unit 203 is an interface to a network such as the internet. The communication unit 203 communicates with external devices according to the control of the control unit 201. For example, the communication unit 203 sends captured images or edited images selected by the user to the server. Images sent from the communication unit 203 are stored in a predetermined storage area on the server and are displayed or downloaded to mobile terminals that access the server.

[0073] The drive 204 is appropriately fitted with removable media 205, such as optical discs or semiconductor memory. Programs and data read from the removable media 205 by the drive 204 are supplied to the control unit 201 and stored or installed in the storage unit 202.

[0074] ROM206 stores programs and data executed by the control unit 201. RAM207 temporarily stores data and programs processed by the control unit 201.

[0075] The pre-selection operation unit 20 implements pre-selection processing for users in the pre-selection space A0. The pre-selection operation unit 20 consists of a touch panel monitor 71, a speaker 72, and a coin processing unit 74.

[0076] The touch panel monitor 71 displays various selection screens according to the control unit 201 and accepts user input on the selection screens. Input signals representing the user's input are supplied to the control unit 201, and various settings are configured.

[0077] The coin processing unit 74 detects when coins are inserted into the coin slot 73. When the coin processing unit 74 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the control unit 201 to instruct the game to start.

[0078] The shooting operation unit 21 performs shooting processing targeting users in shooting space A1. The shooting unit 220 consists of an upper strobe 82, a left strobe 83, a right strobe 84, a foot strobe 85, a camera 91, a touch panel monitor 92, and a speaker 93.

[0079] The upper strobe 82, left strobe 83, right strobe 84, and foot strobe 85 are positioned within the shooting space A1 and emit light according to the lighting control signals supplied from the control unit 201.

[0080] The camera 91 takes pictures according to the shutter control by the control unit 201 and outputs the captured image (image data) obtained by the shooting to the control unit 201.

[0081] The editing operation unit 27A implements editing processing for users in editing space A2-1. The editing operation unit 27A consists of a tablet-integrated monitor 131, a stylus pen 132A, 132B, and a speaker 133. The editing operation unit 27B implements editing processing for users in editing space A2-2 and has the same configuration as the editing operation unit 27A. Hereafter, unless otherwise specified, the editing operation units 27A and 27B will simply be referred to as the editing operation unit 27.

[0082] The tablet's built-in monitor 131 displays the editing screen according to the control unit 201 and accepts user input on the editing screen. Input signals representing the user's input are supplied to the control unit 201, and the captured image to be edited is edited.

[0083] The print operation unit 28 implements a printing process that provides users in the print waiting space A3 with sticker paper printed with the created image. The print operation unit 28 is configured to include a printer 140. A sticker paper unit 141 is attached to the printer 140.

[0084] The printer 140 prints the edited image onto the sticker paper 142 stored in the sticker paper unit 141 based on the print data supplied from the control unit 201, and then ejects it to the sticker paper output port 161.

[0085] (Functional block of the photo sticker creation device) Figure 10 is a block diagram showing the functional blocks of the photo sticker creation device 1. The photo sticker creation device 1 functions as a pre-selection unit 210, a shooting unit 220, an editing unit 230, and a printing unit 240. The control unit 201 also functions as a pre-selection processing unit 301, a shooting processing unit 302, a reception unit 303, a specific unit 304, a mask generation unit 305, an image correction unit 306, a synthesis unit 307, an editing processing unit 308, and a printing processing unit 309 by executing the photo sticker creation program of the present invention.

[0086] The pre-selection unit 210 comprises the pre-selection operation unit 20 described above and the pre-selection processing unit 301. The pre-selection processing unit 301 performs pre-selection processing by controlling the touch panel monitor 71, speaker 72, and coin processing unit 74 of the pre-selection operation unit 20. The pre-selection processing unit 301 displays selection screens, etc., for selecting a course related to shooting to be performed in shooting space A1 on the touch panel monitor 71. The pre-selection processing unit 301 also accepts user operation input to the touch panel monitor 71. Specifically, the pre-selection processing unit 301 accepts selection operation input for the selection screen displayed on the touch panel monitor 71, as well as input of the user's name, etc. The pre-selection processing unit 301 also controls the output of guidance that explains various selection operations. The pre-selection processing unit 301 displays screens that explain various selection operations on the touch panel monitor 71 and outputs audio that explains various selection operations from the speaker 72.

[0087] The imaging unit 220 comprises the imaging operation unit 21 described above and the imaging processing unit 302. The imaging processing unit 302 performs imaging processing by controlling the camera 91, touch panel monitor 92, and speaker 93 in the imaging operation unit 21.

[0088] The imaging processing unit 302 receives user input to the touch panel monitor 92. For example, imaging processing in imaging space A1 is started when the imaging processing unit 302 receives input from the user by touching the touch panel monitor 92.

[0089] The shooting processing unit 302 controls the camera 91 to take pictures of the user as the subject. The pictures taken can be moving images or still images. The shooting processing unit 302 controls the display on the touch panel monitor 92 to display moving images captured by the camera 91 as a live view on the touch panel monitor 92, or to display still images as captured images.

[0090] Furthermore, the shooting processing unit 302 displays an instruction screen on the touch panel monitor 92 that explains the number of shots to take, the user's position, sample poses, messages about eye direction, and shooting timing. In addition, it outputs narration audio and background music corresponding to each instruction screen from the speaker 93.

[0091] The reception unit 303 receives image data of users captured by the shooting unit 220.

[0092] The identification unit 304 uses a first trained model that has learned the relationship between image data containing the eyes of multiple people and the positional information of specific parts of the eyes in the image data to identify the position of at least one specific part from input image data that includes at least a part of a person's face.

[0093] Here, "input image data" is any image data that includes at least a part of a person's face. For example, it may be an image that includes the entire face of a person, or an image from which a part of the entire face of a person has been cropped, in other words, an image from which a part has been cut out. Therefore, the input image data targeted by the identification unit 304 is not limited to the entire image data received by the reception unit 303, but may also be image data from which a range including a specific part has been cropped from the image data received by the reception unit 303.

[0094] The "specific part of the eye" may be, for example, a part identified by at least one of the iris or pupil. Specifically, the specific part of the eye may be the central coordinates of the iris or pupil. Furthermore, the specific part of the eye does not have to be a single point; for example, multiple points such as the coordinates of the outer corner of the eye and the coordinates of the inner corner of the eye may be designated as the specific part.

[0095] The first trained model M1, as shown in Figure 11A, is a model obtained by a learner learning the relationship between "image data" which includes at least a part of a person's face and is a set of input data that is a training dataset, and "position of a specific part of the eye" which is the ground truth data. In addition to the position of the specific part of the eye, the size and shape of the specific part of the eye may also be used for training. For example, a combination of the position and size (radius) of the pupil may be used as the ground truth data. As shown in Figure 11B, when new image data including at least a part of a person's face is input to the first trained model M1 generated in this way, it identifies the position of the specific part of the person's eye from the face in the image data. Here, the first trained model M1 learns the position of the specific part of each eye for each person. When image data including at least a part of the faces of multiple people is input to the identification unit 304, it is preferable that the identification unit 304 identifies the position of the specific part of each person's eye from this image data and temporarily stores the positions of the specific parts of both eyes of each person as a pair in the storage unit 202.

[0096] The mask generation unit 305 uses a second trained model, which has learned the relationship between image data containing specific parts of multiple sets of people's eyes and the catchlight regions in said image data, to extract catchlight regions from the image data of each region identified by the identification unit 304. As a result, the mask generation unit 305 generates a catchlight mask that distinguishes the extracted catchlight regions from the other regions. An example of the catchlight mask shown in Figure 12B distinguishes the catchlight regions as white and the other regions as black.

[0097] The second pre-trained model M2, as shown in Figure 11C, is a model obtained by learning the relationship between the input data, which is "image data" of a region containing at least a specific part of a person's eye and "position information" of that specific part, and the ground truth data, which is the "catchlight region," using a learner. As shown in Figure 12A, the image data of the region containing a specific part of the eye is, for example, a rectangular region containing the specific part of the eye. Although not shown in Figure 12A, the image data is associated with position information of the specific part in the image data, for example, the coordinates of the specific part in the image data. The training dataset used to generate the second pre-trained model includes, for example, multiple sets of image data containing a specific part and position information of that specific part, as shown in Figure 12A, and the catchlight region of a person's eye contained in the image data. As a result, as shown in Figure 11D, when the second pre-trained model receives image data of a region containing a specific part of a person's eye and position information of that specific part as input, it extracts the catchlight region from this image data. Note that if the position of the specific part is fixed in the image data containing the specific part, that position information will be common to all image data. For example, image data may be formed so that a specific part of the body is at the center of the image. Since the positional information of the specific part is common to all images, it is not necessary to include positional information in the training dataset. Also, since the number of catchlights appearing in each eye is not limited, the number of catchlight regions extracted from the image data is also not limited.

[0098] The image correction unit 306 generates corrected image data by adding a predetermined pseudo-catchlight to the image data from which the catchlight has been removed. Specifically, it removes the catchlight from the target image data (Figure 12A) using a catchlight mask (Figure 12B) as shown in Figure 12C. Then, it generates a pseudo-catchlight as shown in Figures 12D to 12F and adds it to the image from which the catchlight has been removed as shown in Figure 12C, thereby generating corrected image data as shown in Figure 12G. The image correction unit 306 can add the pseudo-catchlight according to a predetermined procedure for setting the position and shape of the catchlight and a predetermined procedure for setting the color of the catchlight.

[0099] 《Removal of Catchlight》 The image correction unit 306 removes catchlights from the input image data by replacing the catchlight region identified by the catchlight mask generated by the mask generation unit 305 with a color and / or texture determined based on the color and / or texture of the surrounding region.

[0100] For example, the image correction unit 306 may use the average value of the colors in a predetermined peripheral area of ​​the catchlight as the new color of the catchlight. Specifically, the average value within a predetermined range outside the pixels from the catchlight area can be used as a reference. In this case, the peripheral area does not necessarily have to be within a predetermined range from the catchlight area, but can be an area set for each specific part of the eye. For example, if the catchlight is in the iris, the average value of the iris color can be used as the color of the catchlight area to remove the catchlight. Also, for example, if the catchlight is in the pupil, the average value of the pupil color can be used as the color of the catchlight area to remove the catchlight.

[0101] Determining the position and shape of the catchlight The image correction unit 306 can set the position of the pseudo-catchlight according to the position information of the catchlight area. Specifically, it can determine the position to add the pseudo-catchlight based on the position of the catchlight identified by the catchlight mask. For example, if the original image data contains a catchlight, the position of the original catchlight may be used as a reference, and the center position of the original catchlight may be determined as the center position of the pseudo-catchlight.

[0102] Furthermore, the image correction unit 306 can set the shape and number of catchlights, along with their position, according to the position and size of specific parts of a person's eyes. For example, if the eye size in the image data is large, the shape of the catchlight can be made more complex. On the other hand, if the eye size in the image data is small, the shape of the catchlight can be made simpler or the number of catchlights can be reduced compared to when the eye size is large. More specifically, multiple pseudo-catchlight patterns can be set based on the ideal catchlight shape, and the corresponding pseudo-catchlight can be selected from the multiple patterns according to the size of the original catchlight. When one eye (one eye) contains multiple catchlights, these multiple catchlights can be treated as a set of catchlights. For example, the pseudo-catchlights shown in Figures 12D to 12F constitute a set. Also, since the number of catchlights can be determined according to the size of the original catchlight, the number of catchlights in each set does not need to be the same when there are multiple patterns of pseudo-catchlights. For example, there may be four catchlights when the eye size in the image is large, but two when the eye size is small.

[0103] The image correction unit 306 can set the position of a pseudo-catchlight according to the positional information of the catchlight region of the paired eye when the catchlight region of any of the eyes of a person included in the input image data is not identified in the input image data. For example, if a person does not have a catchlight in their right eye but does have one in their left eye in the original image data, the position of the pseudo-catchlight of the right eye is determined based on the position of the catchlight of the left eye. In this case, the position of the pseudo-catchlight of the right eye may be determined based on the position of the catchlight of the left eye identified by the catchlight mask, or the position of the pseudo-catchlight of the left eye may be determined first, and then the position of the pseudo-catchlight of the right eye may be determined based on the determined position of the pseudo-catchlight of the left eye.

[0104] When the catchlight area cannot be identified in both eyes of a person included in the input image data, the image correction unit 306 can set the position of a pseudo-catchlight according to the position and size of a specific part of the person's eye included in the input image data. For example, the position of the pseudo-catchlight may be set to a predetermined position from the person's pupil in the input image data.

[0105] 《Determining the color of the pseudo-catchlight》 The image correction unit 306 sets the color of the pseudo-catchlight according to the color of a specific part of the person's eye included in the input image data. The color of the pseudo-catchlight can be set according to the color of a specific part of the person's eye in the image, and / or the color of the area surrounding that specific part, similar to the case of catchlight removal. For example, if the image data includes multiple people and each person has a different eye color, more specifically, a different pupil or iris color, the pseudo-catchlight color can be set to a different color for each person. Also, even within the same person, since the pupil and iris are generally different colors, the catchlights added to the pupil and iris can be set to different colors, respectively.

[0106] In addition to correcting catchlights, the image correction unit 306 can perform various correction processes on a person's face and body, such as adjusting the skin tone of the person, applying makeup, and adjusting the size of the eyes themselves.

[0107] The image compositing unit 307 uses the image processed by the image correction unit 306 as a composite image, combines the composite image with the video captured by the camera 91, and displays the composite image as a live view image on the touch panel monitor 92. Therefore, the user can take pictures while checking the finished image in real time.

[0108] The editing unit 230 comprises the editing operation units 27A and 27B described above, and an editing processing unit 308. The editing processing unit 308 performs editing processing by controlling the tablet-integrated monitor 131 and speaker 133 in the editing operation units 27A and 27B. The editing processing unit 308 accepts user input to the tablet-integrated monitor 131 using the stylus pens 132A and 132B.

[0109] Furthermore, the editing processing unit 308 applies predetermined image processing to the captured image as the image to be edited in response to a selection operation on the selection screen displayed on the tablet's built-in monitor 131, and displays it on the tablet's built-in monitor 131. Alternatively, the editing processing unit 308 applies predetermined image processing to the image for synthesis in response to an input operation on the editing screen displayed on the tablet's built-in monitor 131, or generates a new image for synthesis in response to an input operation, synthesizes it with the captured image, and displays it on the tablet's built-in monitor 131.

[0110] Furthermore, the editing processing unit 308 controls the output of guidance explaining how to proceed with editing. For example, the editing processing unit 308 may display a guidance screen explaining how to proceed with editing on the tablet's built-in monitor 131, or output audio of the guidance explaining how to proceed with editing from the speaker 133. The editing processing unit 308 also controls the communication unit 203 and performs processing related to communication via a network such as the Internet. In addition, the editing processing unit 308 may perform printing by controlling the printer 140 of the print operation unit 28.

[0111] The printing unit 240 comprises the printing operation unit 28 and the printing processing unit 309 described above. The printing processing unit 309 receives print data from the editing processing unit 308 and performs printing by controlling the printer 140 of the printing operation unit 28. Here, the printing unit 240, which outputs a photo sticker as print data, was used as an example of an output unit that outputs image data, but the method of outputting image data is not limited to this. For example, a transmission means that transmits the image data itself to an external communication terminal using a network or the like may also be used as an output unit.

[0112] In this way, the photo sticker creation device 1 provides a photo sticker creation game that incorporates various features designed to excite the user (such as the ability to choose a shooting pose, background music, and narration) in exchange for a fee. As a result, the images created by the photo sticker creation device 1 are images that capture the user's joyful expression, or elaborate and vibrant images.

[0113] Furthermore, the photo sticker creation device 1 is equipped with advanced lighting and other facilities, and can perform image transformation processing (such as adjusting the size of the subject's eyes or the length of their legs) and color correction (such as whitening the subject's skin) using sophisticated technology. As a result, the images created by the photo sticker creation device 1 will be images that flatter the user.

[0114] Furthermore, compared to image processing other than that performed by the photo sticker creation device 1 (for example, processing with photo editing applications), editing (doodling on the image) is easier, and the variety of editing options is also richer. From this perspective, it can be said that images created with the photo sticker creation device 1 are relatively more visually appealing than images taken and processed using methods other than the photo sticker creation device 1.

[0115] (How to play the photo sticker creation game) Next, the process flow for a user to play the photo sticker creation game on the photo sticker creation device 1 will be explained using Figure 13. Figure 13 is a flowchart showing the process flow from the start of the game on the photo sticker creation device 1 to the creation of a photo sticker in that game.

[0116] Before the game starts, the control unit 201, which functions as the pre-selection processing unit 301 of the photo sticker creation device 1, displays a message prompting the user to insert a coin on the touch panel monitor 71 of the pre-selection operation unit 20. The control unit 201 also determines whether a coin has been inserted into the coin slot 73 based on the presence or absence of a start signal from the coin processing unit 74, as shown in Figure 13 (S1). If the control unit 201 determines that no coin has been inserted into the coin slot 73 (S1:NO), it continues the process of determining whether a coin has been inserted.

[0117] A user who wishes to start the game inserts a coin into the coin slot 73 in the pre-selection space A0, which is the space in front of the pre-selection operation unit 20. When a coin is inserted into the coin slot 73, the coin processing unit 74 outputs a start signal instructing the start of the game. When the control unit 201 receives the start signal from the coin processing unit 74, it determines that a coin has been inserted into the coin slot 73 (S1: YES) and performs pre-service processing for the user (S2).

[0118] In the pre-service customer processing, the control unit 201 displays messages on the touch panel monitor 71 prompting the user to select a course, enter their name, and select a design. When the user makes various selections or inputs according to the messages displayed on the touch panel monitor 71, the control unit 201 sets the shooting course, name, design, and print layout. The control unit 201 displays multiple types of background images for compositing on the touch panel monitor 92 for the user to select, and allows the user to choose a background image for compositing.

[0119] Once the pre-service processing is complete, the control unit 201 displays a message on the touch panel monitor 71 prompting the user to move to the shooting space A1 and take a picture. The control unit 201, functioning as the shooting processing unit 302, displays a message on the touch panel monitor 92 of the shooting operation unit 21 prompting the user to touch the screen. Alternatively, a start button may be displayed along with or instead of this message. The control unit 201 also outputs a narration from the speaker 93, along with background music, prompting the user to touch the screen. When the user moves to the shooting space A1 and touches the touch panel monitor 92, the control unit 201 reads that the touch panel monitor 92 has been touched and starts the shooting process (S3).

[0120] During the shooting process, the control unit 201 displays guidance on the touch panel monitor 92, for example, regarding lighting, prompting the user to select a lighting level. Once the user selects a lighting level, the control unit 201 sets the lighting level to the selected level.

[0121] Furthermore, the control unit 201 displays an instruction screen explaining the number of photos taken on the touch panel monitor 92 and outputs corresponding narration from the speaker 93. In this embodiment, as an example, the number of photos taken is set to 7.

[0122] Next, the control unit 201 displays an instruction screen on the touch panel monitor 92 to guide the user to their standing position, and outputs corresponding narration from the speaker 93.

[0123] After displaying the instruction screen as described above, the control unit 201 displays a live view image on the touch panel monitor 92 as a live view display, which is a composite of the background image for compositing and the video image acquired by the camera 91, according to the shooting course selected by the user (S4). This allows the user to pose while checking the finished image.

[0124] The control unit 201 displays the live view until just before the countdown for shooting ends, and during that time, the control unit 201 displays a sample pose on the touch panel monitor 92 along with the live view display, or in place of the live view display. The control unit 201 also outputs narration corresponding to the sample pose from the speaker 93.

[0125] The control unit 201 manages the time from the start to the end of the live view display, and when a predetermined time has elapsed, it performs a countdown using the display on the touch panel monitor 92 and the sound from the speaker 93.

[0126] At the end of the countdown, the control unit 201 transmits lighting control signals to the upper strobe 82, left strobe 83, right strobe 84, and foot strobe 85, and also transmits a shutter signal to the camera 91.

[0127] As a result, flashes of light are emitted from the upper strobe 82, left strobe 83, right strobe 84, and foot strobe 85, and the camera 91 acquires a photograph of the illuminated user along with the background. In this embodiment, as an example, the process from step S3 to step S4 is repeated multiple times to acquire seven photographs. The control unit 201 also stores the image to be edited, which is the photographed image with the background image for compositing superimposed on it, in the storage unit 202.

[0128] Although this explanation uses an example where images are acquired at pre-set time intervals, the timing of image acquisition is not limited to this. For example, if the shooting operation unit 21 has an operation button for taking a picture, the image may be acquired when this operation button is pressed.

[0129] After shooting is complete, the control unit 201 displays a guidance screen on the touch panel monitor 92 prompting the user to move to either editing space A2-1 or editing space A2-2, and also outputs voice guidance to the speaker 93.

[0130] Then, the control unit 201 executes an editing process that allows the user to edit the image to be edited (S5). More specifically, the control unit 201 displays the image to be edited on the tablet's built-in monitor 131, and allows the user to draw stamp images, pen images, etc., on this image to be edited using the stylus pens 132A and 132B, thereby creating an edited image.

[0131] Subsequently, the control unit 201 displays a guidance screen on the tablet's built-in monitor 131 prompting the user to move to the printing waiting space A3, which is equipped with a sticker paper output slot 161, and also outputs voice guidance to the speaker 133.

[0132] Furthermore, the control unit 201 creates a printable image by placing the edited image on the print layout selected in the pre-selection operation unit 20, and then performs a printing process to print this printable image onto the sticker paper 142 (S6).

[0133] Once the printing process is complete, the control unit 201 performs the sticker paper 142 ejection process (S7), ejecting the printed sticker paper 142 from the sticker paper ejection port 161 and providing it to the user as a photo sticker, thus ending the game. In this way, the user's captured image created by the photo sticker creation device 1 can be output as a photo sticker. Although a detailed explanation is omitted, the photo sticker creation device 1 of this embodiment can also output the captured image to a mobile terminal or the like via communication.

[0134] (Details of the synthesis process) Next, the synthesis process in step S4 of the flowchart in Figure 13 will be explained with reference to the flowchart in Figure 14. As shown in Figure 14, in the synthesis process in step S4, the control unit 201 receives the image data captured by the imaging unit 220 (S41).

[0135] The control unit 201 uses the first trained model M1 to identify the location of specific parts of a person's face included in the image data received in step S41 (S42). If the captured image includes multiple users, the control unit 201 identifies the location of specific parts of all users.

[0136] The control unit 201 uses the second trained model M2 to extract a catchlight region from the image data that is associated with a specific part identified in step S42, and generates a catchlight mask (S43).

[0137] Furthermore, the control unit 201 removes the catchlight region from the image data using the catchlight mask generated in step S43 (S44).

[0138] The control unit 201 performs a process to add a pseudo-catchlight to the image data from which the catchlight was removed in S44 (S45). This pseudo-catchlight addition process will be described later using the flowchart in Figure 15.

[0139] The control unit 201 determines whether the addition of pseudo-catchlights has been completed for all the identified areas in S42 (S46). If the addition of pseudo-catchlights has not been completed for all identified areas (NO in S46), the control unit 201 returns to step S43 and repeats the process from steps S43 to S46, adding pseudo-catchlights to the identified areas that do not yet have pseudo-catchlights.

[0140] Once the addition of pseudo-catchlights to all specific areas is complete (YES in S46), the control unit 201 performs image processing on the other areas (S47).

[0141] Subsequently, the image data to which the pseudo-catchlight was added in step S45 and the image data obtained in step S48 are combined (S48).

[0142] (Details of the process for adding pseudo-catchlights) Referring to the flowchart shown in Figure 15, the process of adding pseudo-catchlights in step S46 of the flowchart in Figure 14 will be explained. This series of processes for adding pseudo-catchlights is performed for each specific area being targeted. In other words, the control unit 201 adds pseudo-catchlights to each eye individually.

[0143] The control unit 201 determines whether or not the original catchlight is present in the eye, which is the specific target part in the image data (S101). If the original catchlight is present (YES in S101), the control unit 201 determines the position of the pseudo-catchlight based on the position of the original catchlight (S102).

[0144] If there is no original catchlight (NO in S101), the control unit 201 determines whether or not the original catchlight is present in the corresponding eye (S103). If the original catchlight is present in the corresponding eye (YES in S103), the control unit 201 determines the position of the pseudo-catchlight based on the position of the original catchlight in the corresponding eye (S104).

[0145] On the other hand, if the original catchlight is not present in the corresponding eye (NO in S103), the control unit 201 determines the position of the pseudo-catchlight based on a predetermined position, for example, the position of a specific part of the eye (S105).

[0146] Once the position of the pseudo-catchlight is determined in step S102, S104, or S105, the control unit 201 determines the color of the pseudo-catchlight (S106).

[0147] Next, the control unit 201 adds a pseudo-catchlight of the color determined in step S106 to the image data (S107).

[0148] The image processing apparatus of this disclosure can remove catchlights from image data by generating a catchlight mask using the position of a specific part in the image data. Furthermore, by removing catchlights from image data, the image processing apparatus of this disclosure can add ideal catchlights. As a result, the image processing apparatus of this disclosure can provide users with image data with desirable catchlights.

[0149] (Example of implementation using software) The functional blocks of the photo sticker creation device 1 may be implemented by logic circuits (hardware) formed on an integrated circuit (IC chip), or by software using a CPU (Central Processing Unit).

[0150] In the latter case, the photo sticker creation device 1 includes a CPU that executes instructions for a program, which is software that realizes each function; a ROM (Read Only Memory) or storage device (collectively referred to as the "recording medium") on which the program and various data are recorded in a way that can be read by the computer (or CPU); and a RAM (Random Access Memory) for expanding the program. The object of the present invention is achieved when the computer (or CPU) reads the program from the recording medium and executes it. As the recording medium, a "non-temporary tangible medium" such as tape, disk, card, semiconductor memory, or programmable logic circuit can be used. The program may also be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast wave). The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0151] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0152] 1. Photo sticker creation device (image processing device) 301 Pre-selection processing unit 302 Image Processing Unit Room 303 Reception 304 Specific section 305 Mask Generation Unit 306 Image Correction Unit 307 Synthesis section 308 Editing Processing Unit 309 Printing Processing Unit (Output Unit)

Claims

1. An image processing apparatus used for processing image data that includes at least a portion of a person's face, A first trained model that has learned the relationship between image data containing the eyes of multiple people and the positional information of specific parts of the eyes in said image data is used to identify the position of at least one of the specified parts from input image data that includes at least a part of a person's face. A mask generation unit generates a catchlight mask that distinguishes the catchlight region from other regions for the specific eye area of ​​the input image data identified by the identification unit, using a second trained model that has learned the relationship between image data containing specific parts of the eyes of multiple people and the catchlight region in said image data. When a catchlight is not identified in one of the eyes of any person included in the input image data, but the other eye of the same person is included, the image correction unit determines the position of the pseudo-catchlight of the one eye based on the position of the catchlight region of the other eye identified by the catchlight mask, adds the catchlight, and generates corrected image data; An image processing device equipped with the following features.

2. The image processing apparatus according to claim 1, further comprising an image processing unit that removes catchlights by replacing the catchlight region identified by the catchlight mask with a color determined based on the color of the surrounding region in the input image data.

3. The image processing apparatus according to claim 2, wherein the image correction unit generates corrected image data in which a predetermined pseudo-catchlight is added to the image data from which the catchlight has been removed.

4. The image correction unit sets the position of the pseudo-catchlight according to the position information of the catchlight area. The image processing apparatus according to claim 3.

5. The image correction unit sets the position of the pseudo-catchlight according to the position and size of the specific part of the person's eye included in the input image data. The image processing apparatus according to claim 3.

6. The image correction unit sets the color of the pseudo-catchlight according to the color of a specific part of the person's eye included in the input image data. The image processing apparatus according to any one of claims 3 to 5.

7. An image processing program that causes a computer to function as an image processing device according to any one of claims 1 to 6.

8. A first trained model that has learned the relationship between image data containing the eyes of multiple people and the positional information of specific parts of the eyes in said image data, is used to identify the position of at least one of the specific parts from input image data that includes at least a part of a person's face. A second trained model, which has learned the relationship between image data containing specific parts of the eyes of multiple people and the catchlight region in said image data, is used to generate a catchlight mask that distinguishes the catchlight region from other regions for the specific parts of the eyes of the identified input image data. When a catchlight is not identified in one of the eyes of any person included in the input image data, but the other eye of the same person is included, the steps include: determining the position of a pseudo-catchlight in the first eye based on the position of the catchlight region of the other eye identified by the catchlight mask, adding the catchlight, and generating corrected image data; A method for generating a mask that includes [a specific feature / method].

Images