Program, image processing method, and image processing device

Abstract

To appropriately adjust the brightness of a subject appearing in a photographed image.SOLUTION: A program adjusts the brightness of a face area and a skin area other than the face area of a person appearing in a first captured image out of a plurality of captured images based on a retouching parameter corresponding to an adjustment level selected by a user, and reflects an adjustment content of skin brightness for a specific person appearing in the first captured image to the skin brightness of a person identical to the specific person appearing in a second captured image which is different from the first captured image in subject's composition is different from that of the first captured image, in response to a user operation of reflecting the adjustment content of the first captured image in the second captured image. This technique can be applied to a computer installed in a photography studio.SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present technology relates to a program, an image processing method, and an image processing apparatus, and particularly to a program, an image processing method, and an image processing apparatus capable of appropriately adjusting the brightness of a subject shown in a captured image.

Background Art

[0002] In photographing apparatuses such as passport photo machines and print seal machines installed in game centers, users can adjust the appearance of the captured images by themselves. The user can adjust the appearance of each image to be received in a state printed on a sticker paper or the like according to their preference one by one.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Adjustment of the appearance of the image is performed by adjusting the brightness for each region such as the face region.

[0005] The present technology has been made in view of such a situation, and aims to be able to appropriately adjust the brightness of a subject shown in a captured image.

Means for Solving the Problems

[0006] The program of one aspect of the present technology identifies the people shown in each captured image using a model generated by machine learning, detects the face region of the people shown in the captured image and the skin region other than the face region as different skin regions, respectively, adjusts the brightness of the face region and the brightness of the skin region other than the face region of the people shown in the first captured image among a plurality of the captured images based on retouch parameters corresponding to an adjustment level selected by a user, and reflects the adjustment content of the first captured image in response to a user operation of reflecting the adjustment content of the first captured image on a second captured image in which the subject's human composition is different from that of the first captured image, and causes a computer to execute a process of reflecting the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image.

[0007] In one aspect of the present technology, the people shown in each captured image are identified using a model generated by machine learning, and the face region of the people shown in the captured image and the skin region other than the face region are detected as different skin regions, respectively. Further, the brightness of the face region and the brightness of the skin region other than the face region of the people shown in the first captured image among a plurality of the captured images are adjusted based on retouch parameters corresponding to an adjustment level selected by a user, and in response to a user operation of reflecting the adjustment content of the first captured image on a second captured image in which the subject's human composition is different from that of the first captured image, the adjustment content of the skin brightness for a predetermined person shown in the first captured image is reflected on the skin brightness of the same person as the predetermined person shown in the second captured image.

Advantages of the Invention

[0008] According to the present technology, it is possible to appropriately adjust the brightness of the subject shown in the captured image.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Best Mode for Carrying Out the Invention

[0010] <<Workflow in a Photography Studio>> FIG. 1 is a diagram showing the workflow in a photography studio.

[0011] This technology is used, for example, in a photography studio for wedding commemorative photography. This technology is also applicable when taking commemorative photos of various events such as school entrance / graduation commemorative photos, coming-of-age ceremony commemorative photos, and Shichigosan commemorative photos in a photography studio.

[0012] As shown on the left side of FIG. 1, the photography in the photography studio is carried out by a cameraman. In the example of FIG. 1, the bride and groom standing in front of the background screen are being photographed as the subject. The groom is in a tuxedo and the bride is in a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photography by the cameraman is arranged.

[0013] The photography by the cameraman is carried out by changing the composition and taking many still images (photos) such as 50 or 100 images. The composition of the still image is determined by factors such as the poses of the bride and groom, the positional relationship between the bride, groom, and the cameraman, and camera parameters such as the focal length. The photographed image, which is a still image taken by the cameraman, is transferred to a PC (not shown) connected to the camera and saved.

[0014] After the photography is completed, the bride and groom move to the retouch space prepared in the photography studio and, as shown on the right side of FIG. 1, use the retouch PC1 to check the photographed images taken by the cameraman.

[0015] Also, as users of the retouching PC 1, the bridegroom and the bride select, for example, the number of photographed images to be finally printed, and adjust the appearance of each photographed image by retouching them themselves. Retouching is performed on some of the photographed images selected from among the photographed images taken by the cameraman, such as 5 or 10 images. Retouching (processing) includes various image processes for adjusting the appearance of the photographed images.

[0016] One or more retouching PCs 1 are prepared in the retouching space. The photographed images stored in the storage PC are transferred to the retouching PC 1 and displayed on the display 11 of the retouching PC 1. An application for retouching the photographed images taken by the cameraman, the retouching app 1A, is installed on the retouching PC 1.

[0017] Thus, in a photography studio equipped with the retouching PC 1 to which this technology is applied, the retouching of the photographed images taken by the cameraman is performed by the users (the bridegroom and the bride) themselves. The users can use the functions of the retouching app 1A to retouch the photographed images so that they have the appearance they like. The retouching app 1A may be installed on a terminal different from a PC, such as a tablet terminal, and used for retouching.

[0018] FIG. 2 is a diagram showing an example of the provision of photographed images.

[0019] The retouched photographed images by the retouching app 1A are printed by the operating company of the photography studio and delivered to the user at a later date in the form of, for example, an album, as shown by the arrow #1.

[0020] In addition, as shown by arrow #2, the retouched captured images are uploaded from the retouch PC 1 to the image management server 2 and managed by the image management server 2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the operating company of the photography studio or by a business operator different from the operating company of the photography studio.

[0021] The user operates their own mobile terminal 3 such as a smartphone and operates the image management app 3A to view or download the captured images taken with themselves as the subject, as shown by arrow #3. The image management app 3A is an application for viewing the captured images managed by the image management server 2. The image management app 3A is provided, for example, by the business operator that manages the image management server 2.

[0022] The provision of the captured images taken at the photography studio is made in the form of an album and, as appropriate, in the form of data. The provision of the captured images in data form may be made possible at a separate fee from the album fee.

[0023] The provision of the captured images in data form is made to provide not only the captured images retouched by the user but also the captured images not selected by the user as retouch targets. For the captured images not selected by the user as retouch targets, as will be described later, retouching is automatically performed by the retouch app 1A without depending on the user's operation.

[0024] The user can also receive the captured images not selected as retouch targets in the form of data. Instead of using the image management app 3A, the user may access a predetermined website using a browser so as to receive the captured images managed in the image management server 2. For example, by registering as a member of the image providing service provided by the administrator of the image management server 2, the user can receive the captured images managed in the image management server 2 using the image management app 3A or using a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the captured images.

[0026] <<Regarding the retouch app>> <File Management> FIG. 3 is a diagram showing an example of file management in the retouch PC 1.

[0027] As shown in the speech balloon in FIG. 3, two folders, an all folder and an album folder, are prepared in association with the user ID which is the identification information of the user. The all folder is a folder with the folder name of "all", and the album folder is a folder with the folder name of "album".

[0028] The files of the captured images transferred from the saving PC are stored in the all folder. The files of the captured images selected as retouch targets and retouched are stored in the album folder. The files stored in the album folder become the files used for printing.

[0029] When a user who has finished shooting starts retouching, the user ID is input on the screen of the retouch app 1A. Based on the user ID, the captured images are read by the retouch app 1A, and processing using the captured images is performed. For example, all the captured image files stored in the all folder associated with the user ID are read by the retouch app 1A. After the captured images are read, a captured image to be retouched is selected.

[0030] <Screen display> · Image selection screen Figure 4 is a diagram showing an example of an image selection screen.

[0031] Above the image selection screen, a message guiding the selection of five captured images is displayed. The number of captured images to be selected is set, for example, by the staff of the shooting studio before image selection. When the user has applied for a plan to print five captured images and receive them as an album, five is set as the number of selected images.

[0032] On the right side of the image selection screen, a thumbnail display area 21 is formed. The thumbnail display area 21 is an area where thumbnail images of the captured images are displayed. For example, when 50 captured images have been taken, the thumbnail images of each of the 50 captured images are displayed. Various operations using the screen of the retouch app 1A are performed using a mouse, keyboard, etc.

[0033] The captured image selected using the thumbnail image is enlarged and displayed in the display area 22 formed on the left side of the thumbnail display area 21. In the example of Figure 4, the thumbnail image of the captured image P1 displayed at the upper left end of the thumbnail display area 21 is selected, and the captured image P1 is enlarged and displayed in the display area 22.

[0034] A decision button 23 is displayed at the lower right of the display area 22. The decision button 23 is a button that is pressed when determining a captured image displayed in the display area 22 as a retouch target. By selecting a thumbnail image, the user sequentially enlarges and displays each captured image in the display area 22, and when there is a captured image that the user likes, the user presses the decision button 23 to select it as a retouch target.

[0035] Below the display area 22, a selection image display area 24 is formed, where thumbnail images of the captured images selected by the user are displayed.

[0036] When the selection of 5 captured images is completed, the selection of the captured images to be retouched ends. After the selection of the captured images to be retouched ends, retouching is performed on each captured image selected as a retouch target.

[0037] · Retouch screen FIG. 5 is a diagram showing an example of a retouch screen.

[0038] Above the center of the retouch screen, a display area 31 is formed. The display area 31 is an area where the captured image that is currently the retouch target is enlarged and displayed. In the thumbnail area 32 formed on the left side of the display area 31, thumbnail images of the captured images selected by the user are displayed. In the example of FIG. 5, thumbnail images of 5 captured images P1 to P5 are displayed. The captured image P1 selected using the thumbnail image is enlarged and displayed in the display area 31.

[0039] Below the retouch screen, a horizontally long retouch selection area 33 is formed. Using the buttons displayed in the retouch selection area 33, the content of the retouch is selected. Each time the content of the retouch is selected using the buttons displayed in the retouch selection area 33, the selected content is reflected in the display of the display area 31. The user selects the content of the retouch to be applied to the captured image that is currently the retouch target by using the display of the retouch selection area 33 while looking at the display of the display area 31.

[0040] FIG. 6 is a diagram showing an enlarged view of the display of the retouch selection area 33.

[0041] The retouch selection area 33 is formed from an area 33-1 which is a display area for information used for adjusting the entire face and body, an area 33-2 which is a display area for information used for adjusting the eyes and makeup, and an area 33-3 which is a display area for information used for adjusting brightness.

[0042] In the area 33-1 formed on the left side of the retouch selection area 33, adjustment buttons 51 used for adjusting the small face feeling, adjustment buttons 52 used for adjusting the shape of the face, adjustment buttons 53 used for adjusting the skin texture, adjustment buttons 54 used for adjusting the shortness of the chin, and adjustment buttons 55 used for adjusting the styling up are displayed.

[0043] As the small face feeling, it is possible to select one of five levels of size. "0" of the small face feeling represents leaving the size of the face unchanged and keeping it at the original size. By selecting any one of the sizes from "1" to "4", the user can adjust the size of their own face shown in the captured image displayed in the display area 31.

[0044] As the shape of the face, it is possible to select any one of the shapes "round", "egg", and "slim". By selecting any one of the shapes, the user can adjust the shape of their own face shown in the captured image displayed in the display area 31. By selecting "no retouch", it is also possible not to change the shape of the face.

[0045] As the skin texture, it is possible to select one of five levels. "0" of the skin texture represents leaving the skin texture unchanged and keeping it at the original skin texture. By selecting any one of the levels, the user can adjust the degree of smoothness of their own skin texture shown in the captured image displayed in the display area 31.

[0046] The chin can be shortened and three levels can be selected. "0" for shortening the chin means no processing, that is, leaving the original chin unchanged without altering it. By selecting any level, the user can adjust the length of their chin shown in the captured image displayed in the display area 31.

[0047] For styling up, three levels can be selected. "0" for styling up means no processing, that is, leaving the original style unchanged without altering it. By selecting any level, the user can adjust their style shown in the captured image displayed in the display area 31.

[0048] In the area 33-2 formed at the center of the retouch selection area 33, adjustment buttons 56 used for adjusting the eye size, adjustment button 57 used for adjusting the eye shape, adjustment button 58 used for adjusting the makeup intensity, adjustment button 59 used for adjusting the nasal bridge, and adjustment button 60 used for adjusting beard removal are displayed.

[0049] For the eye size, five levels of size can be selected. "0" for the eye size means no processing, that is, leaving the original size unchanged without altering it. By selecting any size, the user can adjust the size of their eyes shown in the captured image displayed in the display area 31.

[0050] For the eye shape, any one of the shapes "round eyes", "downward-slanting eyes", and "cat eyes" can be selected. By selecting any shape, the user can adjust the shape of their eyes shown in the captured image displayed in the display area 31. By selecting "no processing", it is also possible to leave the eye shape unchanged.

[0051] As the makeup intensity, it is possible to select five levels of intensity. "0" for the makeup intensity indicates leaving the makeup intensity unchanged and keeping it at the original intensity. By selecting any intensity, the user can adjust the makeup intensity of themselves shown in the captured image displayed in the display area 31.

[0052] As the nose bridge highlight, it is possible to select on / off. By selecting on or off, the user can choose whether to emphasize the highlighted part of their own nose bridge shown in the captured image displayed in the display area 31 or keep it in the original state.

[0053] As beard removal, it is possible to select three levels. "0" for beard removal indicates leaving the beard unchanged and keeping it at the original beard. By selecting any level, the user can adjust their own beard shown in the captured image displayed in the display area 31. The beard removal function is effective when the person to be retouched is the groom.

[0054] In the area 33-3 formed on the right side of the retouch selection area 33, adjustment buttons 61 used for adjusting the brightness of only the skin, adjustment buttons 62 used for adjusting the overall brightness, and adjustment buttons 63 used for adjusting the brightness of only the background are displayed.

[0055] As the adjustment of the skin brightness, it is possible to select three levels of brightness. "0" for the skin brightness indicates leaving the skin brightness unchanged and keeping it at the original brightness. By selecting any brightness, the user can adjust the skin brightness of themselves shown in the captured image displayed in the display area 31. Details of the skin brightness adjustment function will be described later.

[0056] As for the adjustment of the overall brightness, it is possible to select three levels of brightness. "0" for the overall brightness indicates keeping the original brightness without changing the overall brightness. By selecting any of the brightness levels, the user can adjust the overall brightness shown in the captured image displayed in the display area 31.

[0057] As for the adjustment of the background brightness, it is possible to select three levels of brightness. "0" for the background brightness indicates keeping the original brightness without changing the background brightness. By selecting any of the brightness levels, the user can adjust the background brightness shown in the captured image displayed in the display area 31.

[0058] Returning to the description of FIG. 5, a batch retouch button 34 used for batch retouch is displayed on the left side of the display area 31. Batch retouch is a function for collectively selecting combinations of various retouch contents as described above.

[0059] In the example of FIG. 5, it is possible to select any one of three levels of batch retouch with different combinations of retouch content levels. By using the batch retouch function, the user can select each retouch content in one operation without using the display in the retouch selection area 33.

[0060] FIG. 7 is a diagram showing a display example when batch retouch is selected.

[0061] When the batch retouch button 34 is operated, a rounded rectangular popup image 71 is displayed overlaid on the retouch screen. In the popup image 71 displayed approximately at the center of the retouch screen, circular buttons used for selecting the range of the image to be batch retouched are displayed together with messages.

[0062] By operating the button displayed in the pop-up image 71, the user can select whether to reflect the content of the batch retouch only on the single image that is currently the retouch target, or on all (5) images including the single image that is currently the retouch target.

[0063] Returning to the description of FIG. 5, a pre-processing button 35 is provided on the right side of the display area 31. The pre-processing button 35 is a button that is operated when switching the display of the display area 31 to the state before retouch. While the pre-processing button 35 is pressed, the captured image in the state before retouch is displayed in the display area 31. The user can compare the state after retouch and the state before retouch by using the pre-processing button 35, and confirm whether the degree of retouch is too strong.

[0064] Below the pre-processing button 35, a person selection button 36 used for selecting the person to be retouched is provided. By using the person selection button 36, the user can select to retouch only the person shown on the left side of the captured image, only the person shown on the right side, or both, and perform retouching. Face recognition etc. is performed on the captured image, and the person to be the target of the retouch image processing is selected according to the user's operation.

[0065] The user can select the content of the retouch to be applied to the captured image displayed in the display area 31 by using such various functions. Thus, the retouch that can be performed using the retouch app 1A at least includes adjustment of the whole face and body, adjustment of the eyes and makeup, and adjustment of brightness.

[0066] The captured image to be retouched is switched using the display of the thumbnail area 32, and the retouch of each of the captured images P1 to P5 is sequentially performed. Each file of the captured images P1 to P5 that have been retouched is stored in the album folder.

[0067] FIG. 8 is a diagram showing a display example when the captured image to be retouched is switched.

[0068] When the captured image to be retouched is switched, a pop-up image 72 in the shape of a speech bubble is displayed overlapping the retouch screen. The pop-up image 72 is displayed at a position slightly below the center of the retouch screen so that the display area 31 formed above the approximate center of the retouch screen is the origin of the speech bubble. On the pop-up image 72, a horizontally long rectangular button used for selecting whether to reflect the retouch content set in the previous photo (captured image) in the captured image after switching is displayed together with a message.

[0069] That is, when the captured image to be retouched is switched, an image with a shape different from the pop-up image 71 displayed when the batch retouch button 34 is operated is displayed as the pop-up image 72.

[0070] By using the display of the pop-up image 72, the user can inherit the retouch content selected for the captured image before switching to the captured image after switching. Details of the retouch content inheritance function will be described later.

[0071] FIG. 9 is a diagram showing an example of the application of retouch parameters.

[0072] As shown in FIG. 9, the retouch content applied to each of the captured images P1 to P5 by the user's operation is reflected in each unselected image stored in the all folder. The unselected image is a captured image that was not selected as a retouch target on the image selection screen. The captured image selected as a retouch target on the image selection screen becomes the selected image.

[0073] Which of the retouched captured images P1 to P5 has its retouch content reflected in each unselected image is selected, for example, based on the state of the face shown in the selected image and the state of the face shown in the unselected image. In this case, each captured image is analyzed, and the state of the face shown in each captured image is recognized. A selected image in which the face is shown in a state close to the state of the face shown in each unselected image is selected, and the retouch content applied to the selected selected image is reflected in the unselected image.

[0074] <<Configuration and Operation of Retouch PC1>> <Configuration of Retouch PC1> FIG. 10 is a block diagram showing a configuration example of the retouch PC1.

[0075] A CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0076] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0077] The input unit 106 is composed of a keyboard, a mouse, etc.

[0078] The output unit 107 is composed of a display 11, etc.

[0079] The storage unit 108 is composed of a hard disk, a non-volatile memory, etc. The storage unit 108 stores programs such as the retouch application 1A executed by the CPU 101 and various types of information such as captured images.

[0080] The communication unit 109 is an interface for networks such as the Internet and LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio and receives a photographed image in which a user who is about to start retouching appears as a subject, which is transmitted from the storage PC. Also, the communication unit 109 communicates with the image management server 2 and transmits the retouched photographed image to the image management server 2.

[0081] The drive 110 controls the writing of data to the removable media 111 and the reading of data from the removable media 111. The capture image from the storage PC may be imported via the removable media 111.

[0082] FIG. 11 is a block diagram showing a functional configuration example of the retouch PC 1.

[0083] As shown in FIG. 11, in the retouch PC 1, the image processing unit 121 is realized by executing the retouch application 1A. The image processing unit 121 is composed of an image acquisition unit 131, a retouch processing unit 132, an output control unit 133, and a display control unit 134.

[0084] The image acquisition unit 131 receives a user's operation on the image selection screen and acquires a photographed image selected by the user as a retouch target from, for example, the storage unit 108. The photographed image to be retouched acquired by the image acquisition unit 131 is supplied to the retouch processing unit 132 and the display control unit 134.

[0085] Also, when the retouching of each photographed image to be retouched is completed, the image acquisition unit 131 acquires unselected images from, for example, the storage unit 108 and outputs them to the retouch processing unit 132.

[0086] The retouch processing unit 132 performs face recognition and recognizes the face shown in the captured image to be retouched. The retouch processing unit 132 analyzes the features of each part of the recognized face and performs various image processes related to retouching according to the user's operation on the retouch screen. The analysis of the features of each part may be performed using an inference model generated by machine learning.

[0087] For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouch processing unit 132 performs image processing such as shifting the position of the eye contour according to the selection content by the user. Also, when an operation related to adjusting the intensity of makeup is performed, the retouch processing unit 132 performs image processing such as synthesizing an image of a cheek with a predetermined color on the cheek part or synthesizing an image of lipstick with a predetermined color on the lip part.

[0088] When an operation related to adjusting the skin texture is performed, the retouch processing unit 132 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the size (oval face effect) or shape of the face is performed, the retouch processing unit 132 performs image processing such as shifting the position of the face contour according to the selection content by the user.

[0089] The information representing the result of the image processing by the retouch processing unit 132 is supplied to the display control unit 134 and used for displaying the retouch screen.

[0090] Also, when the retouch for each captured image to be retouched is completed, the retouch processing unit 132 analyzes each captured image including the unselected image and reflects the retouch content applied to the retouched captured image on the unselected image. The captured image retouched by the retouch processing unit 132 is supplied to the output control unit 133.

[0091] The output control unit 133 saves the captured image retouched according to the user's operation in the album folder.

[0092] In addition, the output control unit 133 outputs the retouched captured images (selected images and unselected images) to the communication unit 109 and causes them to be transmitted to the image management server 2.

[0093] The display control unit 134 controls the display of various screens of the retouch application 1A, such as the image selection screen and the retouch screen.

[0094] <Operation of the Retouch PC 1> With reference to the flowchart of FIG. 12, the retouch process performed by the retouch application 1A will be described.

[0095] In step S1, the display control unit 134 of the image processing unit 121 displays an image selection screen (FIG. 4).

[0096] In step S2, the image acquisition unit 131 accepts the selection of a captured image to be retouched in response to a user operation on the image selection screen. The image acquisition unit 131 acquires the captured image to be retouched.

[0097] In step S3, the image acquisition unit 131 determines whether or not the selection of the captured image to be retouched has ended. If it is determined in step S3 that the selection of the captured image to be retouched has not ended, the process returns to step S1 and the selection of the image continues.

[0098] For example, if five captured images are selected, it is determined in step S3 that the selection of the captured image to be retouched has ended.

[0099] In step S4, the display control unit 134 displays a retouch screen (FIG. 5). The user performs retouching of the captured image using the retouch screen.

[0100] In step S5, the retouch processing unit 132 performs image processing in response to a user operation and retouches the captured image selected as the retouch target.

[0101] In step S6, the retouch processing unit 132 determines whether the retouching has been completed. If it is determined in step S6 that the retouching has not been completed, the process returns to step S4, and the retouching for each captured image selected as the retouch target continues.

[0102] For example, if the retouching for all the captured images selected as the retouch target has been performed, it is determined in step S6 that the retouching is to be ended.

[0103] In step S7, the retouch processing unit 132 reflects the retouching with the same content as that applied to the selected image on each unselected image.

[0104] In step S8, the output control unit 133 saves the unselected images to which the retouching has been applied.

[0105] In step S9, the communication unit 109 transmits the selected images and unselected images to which the retouching has been applied to the image management server 2. After that, the processing of the retouch application 1A is completed.

[0106] Through the above processing, the user can adjust the appearance of the captured images in which the user has been photographed as the subject by himself / herself. The user can receive an album of the captured images with the appearance that the user likes.

[0107] Also, by using the mobile terminal 3, the user can receive, in the form of data, the captured images whose appearances have been manually adjusted and the captured images (unselected images) to which the retouching with the same content as the manually adjusted content has been automatically applied, which are managed in the image management server 2.

[0108] <<Skin Brightness Adjustment Function>> As described with reference to FIG. 5, it is possible to adjust the brightness of the captured images that are the retouch targets. Here, the brightness adjustment function for only the skin, which is performed using the adjustment button 61 in the area 33-3, will be described.

[0109] <Configuration of Image Processing Unit 121> FIG. 13 is a block diagram showing a configuration example of an image processing unit 121 that realizes a brightness adjustment function for only skin. FIG. 13 shows only a part of the configuration of the image processing unit 121.

[0110] In addition to the above-described configuration, a gender determination unit 135 is provided in the image processing unit 121. In the retouch processing unit 132, a face recognition unit 151, a skin region extraction unit 152, and a brightness adjustment unit 153 are realized. The captured image to be retouched is input to the gender determination unit 135 and each unit of the retouch processing unit 132.

[0111] The gender determination unit 135 determines the gender of the person shown in the captured image. The determination by the gender determination unit 135 is performed using, for example, an inference model that takes the captured image as input and outputs the gender of each person shown in the captured image. In this case, a gender determination inference model generated by machine learning is prepared in advance in the gender determination unit 135. The gender determination may be performed based on the characteristics of the person obtained by analyzing the captured image to be retouched.

[0112] The determination result by the gender determination unit 135 is supplied to the brightness adjustment unit 153 of the retouch processing unit 132. For example, when only the bride is shown in the captured image to be retouched, information indicating that the person shown in the captured image is female is supplied as the determination result. Also, when only the groom is shown in the captured image to be retouched, information indicating that the person shown in the captured image is male is supplied as the determination result. When both the bride and the groom are shown in the captured image to be retouched, information indicating that each person shown in the captured image is female and male is supplied as the determination result.

[0113] The face recognition unit 151 of the retouch processing unit 132 recognizes the face of the person shown in the captured image and extracts the face region. The extraction of the face region by the face recognition unit 151 may be performed using an inference model generated by machine learning, or may be performed based on features such as the contour points of each part obtained by analyzing the captured image. The information on the face region extracted by the face recognition unit 151 is supplied to the skin region extraction unit 152 and the brightness adjustment unit 153.

[0114] The skin region extraction unit 152 extracts the skin region of the person shown in the captured image. For example, the skin region extraction unit 152 is provided with an inference model that takes the captured image as input and outputs information on the skin region of the person shown in the captured image. The skin region extraction unit 152 extracts the skin region other than the face region based on the skin region extracted using the inference model and the face region extracted by the face recognition unit 151.

[0115] FIG. 14 is a diagram showing an example of extraction of the skin region by the skin region extraction unit 152.

[0116] For example, a case where the captured image P11 shown in the upper left of FIG. 14 is selected as the retouch target will be described. In the captured image P11, a person H1 who is a bride in a wedding dress is shown. The skin is exposed at the chest of the person H1 and from the shoulders to the arms (upper arms).

[0117] When such a captured image P11 is the retouch target, in the face recognition unit 151, a face region as shown at the tip of the arrow #11 is extracted.

[0118] The skin region extraction unit 152 inputs the captured image P11 into the skin region extraction model M as shown at the tip of the arrow #12, and extracts the skin region including the face region based on the output of the skin region extraction model M as shown at the tip of the arrow #13. In the captured image P11, the face region and the region from the shoulders to the arms are shown as regions where the skin of the person H1 is exposed.

[0119] Further, as indicated by the arrows #14 and #15, the skin area extraction unit 152 extracts the skin area other than the face area shown by the arrow #16 by excluding the face area extracted by the face recognition unit 151 from the skin area extracted using the skin area extraction model M. The information on the skin area other than the face area extracted in this way is supplied to the brightness adjustment unit 153.

[0120] The brightness adjustment unit 153 in FIG. 13 identifies the face area of the person shown in the captured image to be retouched based on the information supplied from the face recognition unit 151, and identifies the skin area other than the face area based on the information supplied from the skin area extraction unit 152. The brightness adjustment unit 153 adjusts the face area and the skin area other than the face area using different retouch parameters. The retouch parameter is a parameter that defines the amount of brightness adjustment.

[0121] For example, the brightness adjustment unit 153 selects, in response to a user operation using the adjustment button 61, a retouch parameter such that the adjustment amount for the face area is larger than the adjustment amount for the skin area other than the face area, and adjusts the brightness of each area. That is, as the retouch parameter for the brightness of the face area, a retouch parameter that makes it brighter than the retouch parameter for the brightness of the skin area other than the face area is used for image processing. As described with reference to FIG. 5, using the adjustment button 61, one or two types of levels are selected as the brightness level of only the skin.

[0122] As a result, even when the user selects either level 1 or 2 as the brightness level, as shown in FIG. 15A, image processing is performed to make the face area brighter than the skin area other than the face area. For example, when the brightness of the face area and the brightness of the skin area other than the face area are the same, the face area after brightness adjustment becomes brighter than the skin area other than the face area.

[0123] When the person shown in the captured image is a woman (bride), usually, since makeup is applied to the face, even if the face area is brightened, there is no sense of discomfort. By performing image processing to make the face area brighter than the skin area outside the face area, it is possible to prevent the skin area outside the face area from becoming too bright and resulting in an uncomfortable appearance.

[0124] Such image processing that makes the face area brighter than the skin area outside the face area may be performed based on the gender determination result by the gender determination unit 135.

[0125] For example, the image processing that makes the face area brighter than the skin area outside the face area is performed only on the woman shown in the captured image. In this case, for the man shown in the captured image, image processing using retouch parameters that brighten the brightness of the face area and the skin area such as hands outside the face area by the same degree is performed. This makes it possible to prevent only the face area of the man from becoming too bright and resulting in an uncomfortable appearance.

[0126] Also, even when the user selects the same level as the brightness level, as shown in FIG. 15B, retouch parameters that make the skin area (including the face area) of the woman brighter than the skin area of the man may be selected. In this way, different retouch parameters may be used for adjusting the brightness of the skin area of the woman and the brightness of the skin area of the man, respectively.

[0127] <Display of retouch screen> FIG. 16 is a diagram showing a display example of the area 33-3 of the retouch screen.

[0128] After adjusting the brightness of only the skin as described above using the adjustment button 61, the user can further adjust the overall brightness using the adjustment button 62 or adjust the brightness of the background using the adjustment button 63.

[0129] On the other hand, after adjusting the overall brightness using the adjustment button 62, as shown in FIG. 16A, the adjustment of the background brightness using the adjustment button 63 is restricted. In the example on the right side of FIG. 16A, in response to the overall brightness level being selected as "1", the adjustment button 63 is grayed out and displayed so that the selection of the background brightness level becomes unavailable.

[0130] Also, after adjusting the background brightness using the adjustment button 63, as shown in FIG. 16B, the adjustment of the overall brightness using the adjustment button 62 is restricted. In the example on the right side of FIG. 16B, in response to the background brightness level being selected as "2", the adjustment button 62 is grayed out and displayed so that the selection of the overall brightness level becomes unavailable.

[0131] That is, the adjustment of the overall brightness and the adjustment of the background brightness are in an exclusive relationship, and only one of the adjustments can be performed. After one of the adjustments of the overall brightness and the background brightness is performed, the other adjustment is restricted. The adjustment of the overall brightness and the adjustment of the background brightness are also performed by, for example, the brightness adjustment unit 153.

[0132] For example, if it were possible to brighten the background after brightening the whole, there would be a high possibility that image processing for adjusting the brightness would be superimposed and applied to the background, resulting in white bleeding. By enabling only one of the adjustments of the overall brightness and the background brightness, it is possible to prevent such white bleeding from occurring.

[0133] Note that the display of the adjustment button 61 is maintained (unchanged) so that the brightness level of only the skin can still be selected even after adjusting the overall brightness using the adjustment button 62 or after adjusting the background brightness using the adjustment button 63.

[0134] Although adjustment buttons 61 to 63 are displayed as information used for selecting the brightness adjustment level, other selection information such as a slide bar may be displayed.

[0135] <<Function for Transferring Retouch Contents>> As described with reference to FIG. 8, when the captured image to be retouched is switched, it is possible to transfer the retouch contents selected for the captured image before the switch to the captured image after the switch. Here, the function for transferring retouch contents will be described. Assume that the retouch contents set for the previous captured image are selected using the display of the pop-up image 72 to be reflected also in the captured image after the switch.

[0136] <Transfer of Retouch Contents> FIGS. 17 and 18 are diagrams showing examples of the transfer of retouch contents.

[0137] As shown by the arrow #21 in FIG. 17, assume that by the user's operation when the captured image P21 is the first retouch target, retouch parameters such as "small face feeling" being "3", "face shape" being "egg", and "skin texture" being "2",... are set for the captured image P21. The appearance of the retouched captured image P21 shown in the lower part of FIG. 17 is different from the appearance of the captured image P21 before retouch shown in the upper part.

[0138] When the retouch of the captured image P21 is completed and the retouch target is switched from the captured image P21 to the captured image P22, as shown at the tip of the arrow #22 in FIG. 18, the retouch parameters of the retouched captured image P1 are transferred as the default retouch parameters of the captured image P22, which is the second retouch target. In the display area 31, the captured image P22 is displayed in place of the captured image P21 in a state where the same content retouch as the retouched captured image P21 is applied. The display of the retouch selection area 33 also becomes a display corresponding to the content of the retouch applied to the captured image P22.

[0139] The user can start retouching the captured image P22 based on the retouched captured image P22 with the same content as the captured image P21. In the example of FIG. 18, it is adjusted to the content represented by retouching parameters such as "oval face feeling" being "4", "face shape" being "slim", "skin texture" being "1",...

[0140] Similarly, when the retouching of the captured image P22 is completed and the third captured image to be retouched is selected, as shown by the arrow #24, the retouching parameters of the captured image P22 are inherited as the default retouching parameters of the third captured image to be retouched.

[0141] In this way, as the captured image to be retouched is switched, the content of the retouching of the immediately previous captured image to be retouched is inherited by the appearance of the single captured image after the switch. Each time the captured image to be retouched is switched, the inheritance of the appearance of the captured image is repeated. Thereby, the user can efficiently perform retouching of each captured image.

[0142] The inheritance of the retouching content as described above may be performed when the captured image after the switch is an unadjusted image for which retouching has not yet been performed.

[0143] <Regarding the inheritance target> Retouching can be performed for each person by selecting the target person. The inheritance of the retouching content is performed based on the gender determination result by the gender determination unit 135 such that the retouching content for the bride is inherited by the appearance of the bride, and the retouching content for the groom is inherited by the appearance of the groom.

[0144] · Switching from an image showing the groom and the bride to another image showing the groom and the bride FIG. 19 is a diagram showing a first example of inheritance of retouching content.

[0145] FIG. 19 shows an example of carrying over retouching content when the retouching of the photographed image P31 showing the bride person H1 and the groom person H2 is completed and the retouching target is switched from the photographed image P31 to the photographed image P32 also showing the persons H1 and H2.

[0146] In this case, the retouching content applied to the person H1 when the photographed image P31 is selected as the retouching target is carried over to the image of the person H1 shown in the photographed image P32 as indicated by the arrow #31. Based on the determination result by the gender determination unit 135, the retouching content applied to the female person (person H1) shown in the photographed image P31 is carried over to the image of the same female person (person H1) shown in the photographed image P32.

[0147] Also, the retouching content applied to the person H2 when the photographed image P31 is selected as the retouching target is carried over to the image of the person H2 shown in the photographed image P32 as indicated by the arrow #32. Based on the determination result by the gender determination unit 135, the retouching content applied to the male person (person H2) shown in the photographed image P31 is carried over to the image of the same male person (person H2) shown in the photographed image P32.

[0148] · Switching from an image showing the groom and the bride to an image showing only the groom or the bride FIG. 20 is a diagram showing a second example of carrying over retouching content.

[0149] FIG. 20 shows an example of carrying over retouching content when the retouching of the photographed image P31 showing the bride person H1 and the groom person H2 is completed and the retouching target is switched from the photographed image P31 to the photographed image P33 showing only the person H1 or the photographed image P34 showing only the person H2.

[0150] When the retouch target is switched to the captured image P33 in the lower right, the retouch content applied to the person H1 when the captured image P31 was selected as the retouch target is carried over to the image of the person H1 shown in the captured image P33, as indicated by the arrow #33. Based on the determination result by the gender determination unit 135, the retouch content applied to the female person (person H1) shown in the captured image P31 is carried over to the image of the same female person (person H1) shown in the captured image P33.

[0151] In this case, the retouch content applied to the person H2 when the captured image P31 was selected as the retouch target will not be carried over.

[0152] When the retouch target is switched to the captured image P34 in the lower left, the retouch content applied to the person H2 when the captured image P31 was selected as the retouch target is carried over to the image of the person H2 shown in the captured image P34, as indicated by the arrow #34. Based on the determination result by the gender determination unit 135, the retouch content applied to the male person (person H2) shown in the captured image P31 is carried over to the image of the same male person (person H2) shown in the captured image P34.

[0153] In this case, the retouch content applied to the person H1 when the captured image P31 was selected as the retouch target will not be carried over.

[0154] · Switching from an image showing the bridegroom or the bride to an image showing both the bridegroom and the bride FIG. 21 is a diagram showing a third example of carrying over retouch content.

[0155] On the right side of FIG. 21, an example of carrying over the retouch content when the retouch target is switched from the captured image P33 showing only the bride, person H1, to the captured image P31 showing both person H1 and the bridegroom, person H2, after the retouch of the captured image P33 is completed is shown.

[0156] In this case, when the captured image P33 to be retouched is selected, the retouch content applied to the person H1 is carried over to the image of the person H1 shown in the captured image P31 as indicated by the arrow #35. Based on the determination result by the gender determination unit 135, the retouch content applied to the female person (person H1) shown in the captured image P33 is carried over to the image of the same female person (person H1) shown in the captured image P31.

[0157] On the left side of FIG. 21, the retouch of the captured image P34 showing only the person H2 who is the groom is completed, and an example of the carry-over of the retouch content when the retouch target is switched from the captured image P34 to the captured image P31 showing the person H2 and the person H1 who is the bride is shown.

[0158] In this case, when the captured image P34 to be retouched is selected, the retouch content applied to the person H2 is carried over to the image of the person H2 shown in the captured image P31 as indicated by the arrow #36. Based on the determination result by the gender determination unit 135, the retouch content applied to the male person (person H2) shown in the captured image P34 is carried over to the image of the same male person (person H2) shown in the captured image P31.

[0159] This makes it possible to prevent problems such as the retouch content for the bride being carried over to the image of the groom.

[0160] · Example of allowing the user to select whether to carry over the retouch content When the gender composition of the people is different before and after the switching of the captured image to be retouched, information as described with reference to FIG. 8 may be presented. The user may be allowed to select whether to carry over the retouch content.

[0161] For example, as shown on the right side of FIG. 21, when the subject to be retouched is switched from a photographed image P33 showing only women to a photographed image P31 showing both women and men, and thus the gender composition of the people shown in the photographed image changes, the composition changes significantly, and it may not be appropriate to inherit the retouch content. If inappropriate retouch content is inherited, it becomes cumbersome to correct it, but by appropriately making it possible to select not to inherit the retouch content, it becomes possible to save the trouble of correction.

[0162] · Inheritance based on the identification result of a person Although the inheritance of the retouch content is based on the gender determination result, it may also be based on the identification result of a person. By inheriting the retouch content for a certain person to the image of the same person, the retouch content for the bride is inherited to the image of the bride who is the same person, and the retouch content for the groom is inherited to the image of the groom who is the same person.

[0163] FIG. 22 is a block diagram showing another configuration example of the image processing unit 121. The configuration shown in FIG. 22 is different from the configuration of the image processing unit 121 described above in that a person identification unit 136 is provided.

[0164] The person identification unit 136 identifies each person shown in the photographed image. The identification of a person by the person identification unit 136 is performed using, for example, an inference model that takes the photographed image as an input and outputs the ID of the person shown in the photographed image. In this case, an inference model for person identification generated by machine learning is prepared in advance in the person identification unit 136. The identification of a person may also be performed based on the characteristics of the person obtained by analyzing the photographed image of the subject to be retouched.

[0165] The identification result by the person identification unit 136 is supplied to the retouch processing unit 132. For example, when only the bride is shown in the captured image to be retouched, the person ID assigned to the bride shown in the captured image is supplied as the identification result. Also, when only the groom is shown in the captured image to be retouched, the person ID assigned to the groom shown in the captured image is supplied as the identification result. When both the bride and the groom are shown in the captured image to be retouched, the person ID assigned to the bride and the person ID assigned to the groom are supplied as the identification result.

[0166] In the retouch processing unit 132, based on the identification result by the person identification unit 136, retouching is performed so that the content of the retouch applied to the person shown in the captured image before switching the retouch target is carried over to the appearance of the same person shown in the captured image after switching.

[0167] This also makes it possible to prevent problems such as the content of the retouch for the bride being carried over to the appearance of the groom.

[0168] When the composition of the people is different between before and after switching the captured image to be retouched, information as described with reference to FIG. 8 may be presented. The user is made to select whether to carry over the retouch content or not.

[0169] For example, as shown on the right side of FIG. 21, when the retouch target is switched from the captured image P33 showing only person H1 to the captured image P31 showing person H1 and person H2, and thus the composition of the people shown in the captured image changes, since the composition changes significantly, it may not be appropriate to carry over the retouch content. If inappropriate retouch content is carried over, it becomes cumbersome to correct it, but by enabling selection so as not to carry over the retouch content as appropriate, it is possible to save the trouble of correction.

[0170] <<Modification Example>> ·Example of retouch location Retouching is performed using a PC installed in a photography studio, but it may also be possible to perform retouching using other devices such as the mobile terminal 3 or a PC installed in the user's home.

[0171] When retouching using the mobile terminal 3 is enabled, the user who has finished shooting at the photography studio operates the mobile terminal 3 to start the image management app 3A and performs retouching on the screen of the image management app 3A. The image management app 3A is equipped with the same functions as the retouching app 1A.

[0172] On the display of the mobile terminal 3, a screen as described above is displayed by the image management app 3A, and the selection and retouching of the captured image to be retouched are performed on the captured image uploaded to the image management server 2. The captured image retouched using the functions of the image management app 3A is printed and provided to the user in the form of an album, or the retouched captured image is provided to the user in the form of data as it is.

[0173] As a result, the user can perform retouching at various locations such as at home. Also, the user can perform retouching with care without worrying about time.

[0174] ·Regarding the program The above series of processes can be executed by hardware or by software. When the series of processes are executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer or the like.

[0175] The program to be installed is provided by being recorded on a removable medium 111 shown in FIG. 10, which consists of an optical disk (such as CD-ROM (Compact Disc - Read Only Memory), DVD (Digital Versatile Disc)), a semiconductor memory, etc. Further, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in the ROM 102 or the storage unit 108.

[0176] The program executed by the computer may be a program whose processing is performed in time series according to the order described in this specification, or a program whose processing is performed in parallel or at a necessary timing such as when a call is made.

[0177] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0178] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0179] For example, the present technology can take a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0180] In addition, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0181] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

Explanation of Reference Numerals

[0182] 1 Retouch PC, 2 Image management server, 3 Mobile terminal, 121 Image processing unit, 131 Image acquisition unit, 132 Retouch processing unit, 133 Output control unit, 134 Display control unit, 135 Gender determination unit, 136 Person identification unit, 151 Face recognition unit, 152 Skin area extraction unit, 153 Brightness adjustment unit

Claims

1. identifying a person shown in each captured image using a model generated by machine learning, detecting the face area of the person shown in the captured image and the skin area other than the face area as different skin areas, adjusting the brightness of the face area and the brightness of the skin area other than the face area of the person shown in the first captured image among a plurality of the captured images respectively based on retouch parameters corresponding to an adjustment level selected by a user, in response to a user operation for reflecting the adjustment content of the first captured image on a second captured image in which the subject's person composition is different from that of the first captured image, reflecting the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image A program for causing a computer to execute the process.

2. Adjusting the brightness of the face area and the brightness of the skin area other than the face area so that the face area is brighter than the skin area other than the face area The program according to claim 1.

3. The adjustment of the brightness of the face area and the brightness of the skin area other than the face area is performed in response to one operation of selecting an adjustment level of skin brightness The program according to claim 1.

4. An image processing apparatus, identifying a person shown in each captured image using a model generated by machine learning, detecting the face area of the person shown in the captured image and the skin area other than the face area as different skin areas, adjusting the brightness of the face area and the brightness of the skin area other than the face area of the person shown in the first captured image among a plurality of the captured images respectively based on retouch parameters corresponding to an adjustment level selected by a user, in response to a user operation for reflecting the adjustment content of the first captured image on a second captured image in which the subject's person composition is different from that of the first captured image, reflecting the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image An image processing method.

5. An identification unit that identifies a person shown in each captured image using a model generated by machine learning, A detection unit that detects the face area of the person shown in the captured image and the skin area other than the face area as different skin areas, Adjust the brightness of the face area and the brightness of the skin area other than the face area in the first captured image among the plurality of captured images respectively based on retouch parameters corresponding to the adjustment level selected by the user, and in response to the user's operation of reflecting the adjustment content of the first captured image on the second captured image in which the subject's person composition is different from that of the first captured image, the retouch processing unit that reflects the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image An image processing apparatus comprising the same.

Images