Program, image processing method, and image processing device

Abstract

To appropriately adjust the brightness of a subject appearing in a photographed image.SOLUTION: A program adjusts the brightness of a face area and a skin area other than the face area of a person appearing in a first captured image out of a plurality of captured images based on a retouching parameter corresponding to an adjustment level selected by a user, and reflects an adjustment content of skin brightness for a specific person appearing in the first captured image to the skin brightness of a person identical to the specific person appearing in a second captured image which is different from the first captured image in subject's composition is different from that of the first captured image, in response to a user operation of reflecting the adjustment content of the first captured image in the second captured image. This technique can be applied to a computer installed in a photography studio.SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present technology relates to a program, an image processing method, and an image processing apparatus, and particularly relates to a program, an image processing method, and an image processing apparatus that can appropriately adjust the brightness of a subject shown in a captured image.

Background Art

[0002] In imaging devices such as proof photo machines and print seal machines installed in game centers, users can adjust the appearance of the captured images by themselves. The user can adjust the appearance of the images to be received in a printed state on seal paper, etc., one by one according to their preferences.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Adjustment of the appearance of the image is performed by adjusting the brightness for each region such as the face region.

[0005] The present technology has been made in view of such a situation, and aims to be able to appropriately adjust the brightness of a subject shown in a captured image.

Means for Solving the Problems

[0006] The program of one aspect of the present technology identifies a person appearing in each captured image using a model generated by machine learning, detects the face area of the person appearing in the captured image and the skin area other than the face area as different skin areas, respectively, adjusts the brightness of the face area and the brightness of the skin area other than the face area of the person appearing in the first captured image among a plurality of the captured images based on retouch parameters corresponding to an adjustment level selected by the user, and in response to a user operation to reflect the adjustment content of the first captured image on a second captured image in which the subject's person composition is different from that of the first captured image, causes a computer to execute a process of reflecting the adjustment content of the skin brightness for a predetermined person appearing in the first captured image on the skin brightness of the same person as the predetermined person appearing in the second captured image.

[0007] In one aspect of the present technology, a person appearing in each captured image is identified using a model generated by machine learning, and the face area of the person appearing in the captured image and the skin area other than the face area are detected as different skin areas, respectively. Also, the brightness of the face area and the brightness of the skin area other than the face area of the person appearing in the first captured image among a plurality of the captured images are adjusted based on retouch parameters corresponding to an adjustment level selected by the user, and in response to a user operation to reflect the adjustment content of the first captured image on a second captured image in which the subject's person composition is different from that of the first captured image, the adjustment content of the skin brightness for a predetermined person appearing in the first captured image is reflected on the skin brightness of the same person as the predetermined person appearing in the second captured image.

Advantages of the Invention

[0008] According to the present technology, it is possible to appropriately adjust the brightness of the subject appearing in the captured image.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Best Mode for Carrying Out the Invention

[0010] <<Workflow of Operations in a Photography Studio>> FIG. 1 is a diagram showing the workflow of operations in a photography studio.

[0011] This technology is used, for example, in a photography studio for taking commemorative wedding photos. This technology is also applicable when taking commemorative photos of various events such as school entrance / graduation photos, coming-of-age ceremony photos, and Shichigosan photos in a photography studio.

[0012] As shown on the left side of FIG. 1, photography in the photography studio is carried out by a cameraman. In the example of FIG. 1, the bride and groom standing in front of the background screen are being photographed as the subjects. The groom is in a tuxedo and the bride is in a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photography by the cameraman is arranged.

[0013] Photography by the cameraman is carried out by changing the composition to take many still images (photos) such as 50 or 100. The composition of the still images is determined by factors such as the poses of the bride and groom, the positional relationship between the bride, groom, and the cameraman, and camera parameters such as the focal length. The photographed images, which are still images taken by the cameraman, are transferred to a PC (not shown) connected to the camera and saved.

[0014] After the photography is completed, the bride and groom move to the retouch space prepared in the photography studio and, as shown on the right side of FIG. 1, use the retouch PC 1 to check the photographed images taken by the cameraman.

[0015] Also, as users of the retouching PC 1, the bridegroom and the bride select, for example, the number of photographed images to be finally printed, and adjust the appearance of each photographed image by retouching them themselves. Retouching is performed on some of the photographed images, such as 5 or 10 images, selected from the photographed images taken by the cameraman. Retouching (processing) includes various image processes for adjusting the appearance of the photographed images.

[0016] One or a plurality of retouching PCs 1 are prepared in the retouching space. The photographed images stored in the storage PC are transferred to the retouching PC 1 and displayed on the display 11 of the retouching PC 1. An application for retouching the photographed images taken by the cameraman, that is, a retouching app 1A, is installed on the retouching PC 1.

[0017] Thus, in a photography studio equipped with the retouching PC 1 to which this technology is applied, the retouching of the photographed images taken by the cameraman is performed by the users (the bridegroom and the bride) themselves. The users can use the functions of the retouching app 1A to retouch the photographed images so that they have the appearance they like. The retouching app 1A may be installed on a terminal different from a PC, such as a tablet terminal, and used for retouching.

[0018] FIG. 2 is a diagram showing an example of providing photographed images.

[0019] The photographed images retouched by the retouching app 1A are printed by the operating company of the photography studio and delivered to the users at a later date in the form of, for example, an album, as shown by arrow #1.

[0020] In addition, as shown by arrow #2, the retouched captured images are uploaded from the retouch PC 1 to the image management server 2 and managed by the image management server 2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the operating company of the photography studio or by a business operator different from the operating company of the photography studio.

[0021] The user operates their own mobile terminal 3 such as a smartphone and operates the image management app 3A to view or download the captured images taken with themselves as the subject, as shown by arrow #3. The image management app 3A is an application for viewing the captured images managed by the image management server 2. The image management app 3A is provided, for example, by the business operator managing the image management server 2.

[0022] The provision of the captured images taken at the photography studio is made in the form of an album and, as appropriate, in the form of data. The provision of the captured images in data form may be made at a separate charge from the album charge.

[0023] The provision of the captured images in data form is made to provide not only the captured images retouched by the user but also the captured images not selected by the user as retouch targets. For the captured images not selected by the user as retouch targets, as will be described later, retouching is automatically performed by the retouch app 1A without depending on the user's operation.

[0024] The user can also receive the captured images not selected as retouch targets in the form of data. Instead of using the image management app 3A, the user may access a predetermined website using a browser and receive the captured images managed in the image management server 2. For example, by registering as a member of the image providing service provided by the administrator of the image management server 2, the user can receive the captured images managed in the image management server 2 using the image management app 3A or using a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the captured images.

[0026] <<Regarding the retouch app>> <File management> FIG. 3 is a diagram showing an example of file management in the retouch PC 1.

[0027] As shown in the speech bubble of FIG. 3, two folders, an all folder and an album folder, are prepared in association with the user ID which is the identification information of the user. The all folder is a folder with the folder name "all" set, and the album folder is a folder with the folder name "album" set.

[0028] The files of the captured images transferred from the saving PC are stored in the all folder. The files of the captured images selected as retouch targets and retouched are stored in the album folder. The files stored in the album folder become the files used for printing.

[0029] When a user who has finished shooting starts retouching, the user ID is input on the screen of the retouch app 1A. Based on the user ID, the captured image is read by the retouch app 1A, and processing using the captured image is performed. For example, all the captured image files stored in the all folder associated with the user ID are read by the retouch app 1A. After the captured image is read, the captured image to be retouched is selected.

[0030] <Screen display> · Image selection screen FIG. 4 is a diagram showing an example of an image selection screen.

[0031] Above the image selection screen, a message guiding the selection of 5 captured images is displayed. The number of captured images to be selected is set, for example, by the staff of the photography studio before image selection. When the user has applied for a plan to print 5 captured images and receive them as an album, 5 is set as the number of selected images.

[0032] On the right side of the image selection screen, a thumbnail display area 21 is formed. The thumbnail display area 21 is an area where thumbnail images of the captured images are displayed. For example, when 50 captured images have been taken, the thumbnail images of each of the 50 captured images are displayed. Various operations using the screen of the retouch app 1A are performed using a mouse, keyboard, etc.

[0033] The captured image selected using the thumbnail image is enlarged and displayed in the display area 22 formed on the left side of the thumbnail display area 21. In the example of FIG. 4, the thumbnail image of the captured image P1 displayed at the upper left end of the thumbnail display area 21 is selected, and the captured image P1 is enlarged and displayed in the display area 22.

[0034] A decision button 23 is displayed at the lower right of the display area 22. The decision button 23 is a button that is pressed when determining a captured image displayed in the display area 22 as a retouch target. The user sequentially enlarges and displays each captured image in the display area 22 by selecting a thumbnail image, and when there is a captured image that the user likes, the user presses the decision button 23 to select it as a retouch target.

[0035] Below the display area 22, a selection image display area 24 for displaying thumbnail images of the captured images selected by the user is formed.

[0036] When the selection of five captured images is completed, the selection of the captured images to be retouched ends. After the selection of the captured images to be retouched ends, retouching is performed on each captured image selected as a retouch target.

[0037] · Retouch screen FIG. 5 is a diagram showing an example of a retouch screen.

[0038] Above the center of the retouch screen, a display area 31 is formed. The display area 31 is an area where the captured image currently being retouched is enlarged and displayed. In the thumbnail area 32 formed on the left side of the display area 31, thumbnail images of the captured images selected by the user are displayed. In the example of FIG. 5, thumbnail images of five captured images P1 to P5 are displayed. The captured image P1 selected using the thumbnail image is enlarged and displayed in the display area 31.

[0039] Below the retouch screen, a horizontally long retouch selection area 33 is formed. Using the buttons displayed in the retouch selection area 33, the content of retouching is selected. Each time the content of retouching is selected using the buttons displayed in the retouch selection area 33, the selected content is reflected in the display of the display area 31. The user selects the content of retouching to be applied to the captured image currently being retouched using the display of the retouch selection area 33 while looking at the display of the display area 31.

[0040] FIG. 6 is a diagram showing an enlarged view of the display of the retouch selection area 33.

[0041] The retouch selection area 33 is formed from an area 33-1 which is a display area for information used for adjusting the entire face and body, an area 33-2 which is a display area for information used for adjusting the eyes and makeup, and an area 33-3 which is a display area for information used for adjusting brightness.

[0042] In the area 33-1 formed on the left side of the retouch selection area 33, an adjustment button 51 used for adjusting the small face feeling, an adjustment button 52 used for adjusting the shape of the face, an adjustment button 53 used for adjusting the skin texture, an adjustment button 54 used for adjusting the shortness of the chin, and an adjustment button 55 used for adjusting the styling up are displayed.

[0043] As the small face feeling, it is possible to select one of five levels of size. "0" for the small face feeling indicates leaving the size of the face unchanged and keeping it at the original size. By selecting any size from "1" to "4", the user can adjust the size of their own face shown in the captured image displayed in the display area 31.

[0044] As the shape of the face, it is possible to select any one of "round", "oval", and "slim". By selecting any shape, the user can adjust the shape of their own face shown in the captured image displayed in the display area 31. By selecting "no retouch", it is also possible not to change the shape of the face.

[0045] As the skin texture, it is possible to select one of five levels. "0" for the skin texture indicates leaving the skin texture unchanged and keeping it at the original skin texture. By selecting any level, the user can adjust the smoothness degree of their own skin texture shown in the captured image displayed in the display area 31.

[0046] The chin can be shortened and three levels can be selected. "0" for shortening the chin means no processing, that is, leaving the original chin unchanged. By selecting any level, the user can adjust the length of their chin shown in the captured image displayed in the display area 31.

[0047] For styling up, three levels can be selected. "0" for styling up means no processing, that is, leaving the original style unchanged. By selecting any level, the user can adjust their style shown in the captured image displayed in the display area 31.

[0048] In the area 33-2 formed at the center of the retouch selection area 33, adjustment buttons 56 for adjusting the eye size, adjustment button 57 for adjusting the eye shape, adjustment button 58 for adjusting the makeup intensity, adjustment button 59 for adjusting the nasolabial fold, and adjustment button 60 for adjusting beard removal are displayed.

[0049] For the eye size, five levels of size can be selected. "0" for the eye size means no processing, that is, leaving the original size unchanged. By selecting any size, the user can adjust the size of their eyes shown in the captured image displayed in the display area 31.

[0050] For the eye shape, any shape among "round eyes", "downward-slanting eyes", and "cat eyes" can be selected. By selecting any shape, the user can adjust the shape of their eyes shown in the captured image displayed in the display area 31. By selecting "no processing", it is also possible to keep the eye shape unchanged.

[0051] As the makeup intensity, it is possible to select five levels of intensity. "0" for the makeup intensity indicates leaving the makeup intensity unchanged and keeping it at the original intensity. By selecting any intensity, the user can adjust the makeup intensity of themselves shown in the captured image displayed in the display area 31.

[0052] As the nose bridge, it is possible to select on / off. By selecting on or off, the user can choose whether to emphasize the highlighted part of their own nose bridge shown in the captured image displayed in the display area 31 or keep it in the original state.

[0053] As beard removal, it is possible to select three levels. "0" for beard removal indicates leaving the beard unchanged and keeping it at the original beard. By selecting any level, the user can adjust their own beard shown in the captured image displayed in the display area 31. The beard removal function is effective when the person to be retouched is the groom.

[0054] In the area 33-3 formed on the right side of the retouch selection area 33, adjustment buttons 61 used for adjusting the brightness of only the skin, adjustment buttons 62 used for adjusting the overall brightness, and adjustment buttons 63 used for adjusting the brightness of only the background are displayed.

[0055] As the adjustment of the skin brightness, it is possible to select three levels of brightness. "0" for the skin brightness indicates leaving the skin brightness unchanged and keeping it at the original brightness. By selecting any brightness, the user can adjust the skin brightness of themselves shown in the captured image displayed in the display area 31. Details of the skin brightness adjustment function will be described later.

[0056] As for the adjustment of the overall brightness, it is possible to select three levels of brightness. "0" for the overall brightness indicates that the overall brightness remains unchanged and is the same as the original brightness. By selecting any brightness level, the user can adjust the overall brightness shown in the captured image displayed in the display area 31.

[0057] As for the adjustment of the background brightness, it is possible to select three levels of brightness. "0" for the background brightness indicates that the background brightness remains unchanged and is the same as the original brightness. By selecting any brightness level, the user can adjust the background brightness shown in the captured image displayed in the display area 31.

[0058] Returning to the description of FIG. 5, a batch retouch button 34 used for batch retouch is displayed on the left side of the display area 31. Batch retouch is a function for collectively selecting combinations of various retouch contents as described above.

[0059] In the example of FIG. 5, it is possible to select any level from three levels of batch retouch with different combinations of retouch content levels. By using the batch retouch function, the user can select each retouch content in one operation without using the display in the retouch selection area 33.

[0060] FIG. 7 is a diagram showing a display example when batch retouch is selected.

[0061] When the batch retouch button 34 is operated, a rounded rectangular pop-up image 71 is displayed overlapping the retouch screen. In the pop-up image 71 displayed approximately in the center of the retouch screen, circular buttons used for selecting the range of the image to be batch retouched are displayed together with messages.

[0062] The user can select whether to reflect the content of batch retouching only on the single image that is currently the retouching target or on all (five) images including the single image that is currently the retouching target by operating the button displayed in the pop-up image 71.

[0063] Returning to the description of FIG. 5, a pre-processing button 35 is provided on the right side of the display area 31. The pre-processing button 35 is a button that is operated when switching the display of the display area 31 to the state before retouching. While the pre-processing button 35 is pressed, the captured image in the state before retouching is displayed in the display area 31. The user can compare the state after retouching with the state before retouching by using the pre-processing button 35 and confirm whether the degree of retouching is too strong.

[0064] Below the pre-processing button 35, a person selection button 36 used for selecting the person to be retouched is provided. The user can select to retouch only the person shown on the left side of the captured image, only the person shown on the right side, or both persons by using the person selection button 36. Face recognition or the like is performed on the captured image, and the person to be the target of the retouch image processing is selected according to the user's operation.

[0065] The user can select the content of the retouching to be applied to the captured image displayed in the display area 31 by using such various functions. In this way, the retouching that can be performed using the retouch app 1A at least includes adjustment of the whole face and body, adjustment of the eyes and makeup, and adjustment of brightness.

[0066] The captured image to be retouched is switched using the display of the thumbnail area 32, and the retouching of each of the captured images P1 to P5 is sequentially performed. The files of each of the captured images P1 to P5 that have been retouched are stored in the album folder.

[0067] FIG. 8 is a diagram showing a display example when the captured image to be retouched is switched.

[0068] When the captured image to be retouched is switched, a pop-up image 72 with a speech bubble shape is overlaid and displayed on the retouch screen. The pop-up image 72 is displayed at a position slightly below the center of the retouch screen so that the display area 31 formed slightly above the approximate center of the retouch screen is the source of the speech bubble. On the pop-up image 72, a horizontally long rectangular button used for selecting whether to reflect the retouch content set in the previous photo (captured image) in the captured image after switching is displayed together with a message.

[0069] That is, when the captured image to be retouched is switched, an image with a shape different from the pop-up image 71 displayed when the batch retouch button 34 is operated is displayed as the pop-up image 72.

[0070] By using the display of the pop-up image 72, the user can inherit the retouch content selected for the captured image before switching to the captured image after switching. Details of the retouch content inheritance function will be described later.

[0071] FIG. 9 is a diagram showing an example of application of retouch parameters.

[0072] As shown in FIG. 9, the retouch content applied to each of the captured images P1 to P5 by the user's operation is reflected in each unselected image stored in the all folder. The unselected image is a captured image that was not selected as a retouch target on the image selection screen. The captured image selected as a retouch target on the image selection screen becomes the selected image.

[0073] Which of the retouched captured images P1 to P5 has its retouch content reflected on each unselected image is selected based on, for example, the state of the face shown in the selected image and the state of the face shown in the unselected image. In this case, each captured image is analyzed, and the state of the face shown in each captured image is recognized. A selected image in which the face is shown in a state close to the state of the face shown in each unselected image is selected, and the retouch content applied to the selected selected image is reflected on the unselected image.

[0074] <<Configuration and Operation of Retouch PC1>> <Configuration of Retouch PC1> FIG. 10 is a block diagram showing a configuration example of the retouch PC1.

[0075] A CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0076] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0077] The input unit 106 is composed of a keyboard, a mouse, etc.

[0078] The output unit 107 is composed of a display 11, etc.

[0079] The storage unit 108 is composed of a hard disk, a non-volatile memory, etc. The storage unit 108 stores programs such as a retouch application 1A executed by the CPU 101 and various types of information such as captured images.

[0080] The communication unit 109 is an interface for networks such as the Internet and LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a photographed image in which a user who is about to start retouching appears as a subject, which is transmitted from the storage PC. Also, the communication unit 109 communicates with the image management server 2 and transmits the retouched photographed image to the image management server 2.

[0081] The drive 110 controls writing of data to the removable media 111 and reading of data from the removable media 111. The capture images from the storage PC may be imported via the removable media 111.

[0082] FIG. 11 is a block diagram showing a functional configuration example of the retouch PC 1.

[0083] As shown in FIG. 11, in the retouch PC 1, when the retouch application 1A is executed, the image processing unit 121 is realized. The image processing unit 121 is composed of an image acquisition unit 131, a retouch processing unit 132, an output control unit 133, and a display control unit 134.

[0084] The image acquisition unit 131 receives a user's operation on the image selection screen, and acquires a photographed image selected by the user as a retouch target from, for example, the storage unit 108. The photographed image to be retouched acquired by the image acquisition unit 131 is supplied to the retouch processing unit 132 and the display control unit 134.

[0085] Also, when retouching of each photographed image to be retouched is completed, the image acquisition unit 131 acquires unselected images from, for example, the storage unit 108 and outputs them to the retouch processing unit 132.

[0086] The retouch processing unit 132 performs face recognition and recognizes the face shown in the captured image to be retouched. The retouch processing unit 132 analyzes the features of each part of the recognized face and performs various image processes related to retouching according to the user's operation on the retouch screen. The analysis of the features of each part may be performed using an inference model generated by machine learning.

[0087] For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouch processing unit 132 performs image processing such as shifting the position of the eye contour according to the selection content by the user. Also, when an operation related to adjusting the intensity of makeup is performed, the retouch processing unit 132 performs image processing such as synthesizing an image of a cheek with a predetermined color on the cheek part or synthesizing an image of lipstick with a predetermined color on the lip part.

[0088] When an operation related to adjusting the skin texture is performed, the retouch processing unit 132 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the size (oval face effect) or shape of the face is performed, the retouch processing unit 132 performs image processing such as shifting the position of the face contour according to the selection content by the user.

[0089] The information representing the result of the image processing by the retouch processing unit 132 is supplied to the display control unit 134 and used for displaying the retouch screen.

[0090] Also, when the retouch for each captured image to be retouched is completed, the retouch processing unit 132 analyzes each captured image including the unselected image and reflects the retouch content applied to the retouched captured image on the unselected image. The captured image retouched by the retouch processing unit 132 is supplied to the output control unit 133.

[0091] The output control unit 133 saves the captured image retouched according to the user's operation in the album folder.

[0092] In addition, the output control unit 133 outputs the retouched captured images (selected images and unselected images) to the communication unit 109 and causes them to be transmitted to the image management server 2.

[0093] The display control unit 134 controls the display of various screens of the retouch application 1A, such as the image selection screen and the retouch screen.

[0094] <Operation of the Retouch PC 1> With reference to the flowchart of FIG. 12, the retouch process performed by the retouch application 1A will be described.

[0095] In step S1, the display control unit 134 of the image processing unit 121 displays an image selection screen (FIG. 4).

[0096] In step S2, the image acquisition unit 131 accepts the selection of the captured image to be retouched in response to the user's operation on the image selection screen. The image acquisition unit 131 acquires the captured image to be retouched.

[0097] In step S3, the image acquisition unit 131 determines whether or not the selection of the captured image to be retouched has ended. If it is determined in step S3 that the selection of the captured image to be retouched has not ended, the process returns to step S1 and the selection of the image continues.

[0098] For example, if five captured images are selected, it is determined in step S3 that the selection of the captured image to be retouched has ended.

[0099] In step S4, the display control unit 134 displays a retouch screen (FIG. 5). The user performs retouching of the captured image using the retouch screen.

[0100] In step S5, the retouch processing unit 132 performs image processing in response to the user's operation and retouches the captured image selected as the retouch target.

[0101] In step S6, the retouch processing unit 132 determines whether the retouch has been completed. If it is determined in step S6 that the retouch has not been completed, the process returns to step S4, and the retouch on each captured image selected as the retouch target continues.

[0102] For example, if the retouch on all the captured images selected as the retouch target has been performed, it is determined in step S6 that the retouch is to be ended.

[0103] In step S7, the retouch processing unit 132 reflects the retouch of the same content as that applied to the selected image on each unselected image.

[0104] In step S8, the output control unit 133 saves the unselected images to which the retouch has been applied.

[0105] In step S9, the communication unit 109 transmits the selected images and unselected images to which the retouch has been applied to the image management server 2. After that, the processing of the retouch application 1A ends.

[0106] Through the above processing, the user can adjust the appearance of the captured images in which the user himself / herself is the subject. The user can receive an album of the captured images with the appearance he / she likes.

[0107] Also, by using the mobile terminal 3, the user can receive, in the form of data, the captured images whose appearances have been manually adjusted and the captured images (unselected images) to which the retouch of the same content as the manually adjusted content has been automatically applied, which are managed in the image management server 2.

[0108] <<Function for adjusting skin brightness>> As described with reference to FIG. 5, it is possible to adjust the brightness of the captured images that are the retouch targets. Here, the function for adjusting the brightness of only the skin, which is performed using the adjustment button 61 in the area 33-3, will be described.

[0109] <Configuration of Image Processing Unit 121> FIG. 13 is a block diagram showing a configuration example of an image processing unit 121 that realizes a brightness adjustment function for only the skin. FIG. 13 shows only a part of the configuration of the image processing unit 121.

[0110] In addition to the above-described configuration, a gender determination unit 135 is provided in the image processing unit 121. In the retouch processing unit 132, a face recognition unit 151, a skin area extraction unit 152, and a brightness adjustment unit 153 are realized. The captured image to be retouched is input to the gender determination unit 135 and each unit of the retouch processing unit 132.

[0111] The gender determination unit 135 determines the gender of the person shown in the captured image. The determination by the gender determination unit 135 is performed, for example, using an inference model that takes the captured image as an input and outputs the gender of each person shown in the captured image. In this case, the gender determination unit 135 is provided in advance with an inference model for gender determination generated by machine learning. The determination of gender may be performed based on the characteristics of the person obtained by analyzing the captured image to be retouched.

[0112] The determination result by the gender determination unit 135 is supplied to the brightness adjustment unit 153 of the retouch processing unit 132. For example, when only the bride is shown in the captured image to be retouched, information indicating that the person shown in the captured image is female is supplied as the determination result. Also, when only the groom is shown in the captured image to be retouched, information indicating that the person shown in the captured image is male is supplied as the determination result. When both the bride and the groom are shown in the captured image to be retouched, information indicating that each person shown in the captured image is female and male is supplied as the determination result.

[0113] The face recognition unit 151 of the retouch processing unit 132 recognizes the face of the person shown in the captured image and extracts the face region. The extraction of the face region by the face recognition unit 151 may be performed using an inference model generated by machine learning, or may be performed based on features such as the contour points of each part obtained by analyzing the captured image. The information on the face region extracted by the face recognition unit 151 is supplied to the skin region extraction unit 152 and the brightness adjustment unit 153.

[0114] The skin region extraction unit 152 extracts the skin region of the person shown in the captured image. For example, the skin region extraction unit 152 is provided with an inference model that takes the captured image as input and outputs information on the skin region of the person shown in the captured image. The skin region extraction unit 152 extracts the skin region other than the face region based on the skin region extracted using the inference model and the face region extracted by the face recognition unit 151.

[0115] FIG. 14 is a diagram showing an example of the extraction of the skin region by the skin region extraction unit 152.

[0116] For example, the case where the captured image P11 shown in the upper left of FIG. 14 is selected as the retouch target will be described. In the captured image P11, a person H1 who is a bride in a wedding dress is shown. The skin is exposed at the chest of the person H1 and in the part from the shoulder to the arm (upper arm).

[0117] When such a captured image P11 is the retouch target, in the face recognition unit 151, a face region as shown at the tip of the arrow #11 is extracted.

[0118] The skin region extraction unit 152 inputs the captured image P11 into the skin region extraction model M as shown at the tip of the arrow #12, and extracts the skin region including the face region based on the output of the skin region extraction model M as shown at the tip of the arrow #13. In the captured image P11, the face region and the region from the shoulder to the arm are shown as the regions where the skin of the person H1 is exposed.

[0119] In addition, as shown at the tips of arrow #14 and arrow #15, the skin area extraction unit 152 extracts the skin area other than the face area shown at the tip of arrow #16 by excluding the face area extracted by the face recognition unit 151 from the skin area extracted using the skin area extraction model M. The information on the skin area other than the face area extracted in this way is supplied to the brightness adjustment unit 153.

[0120] The brightness adjustment unit 153 in FIG. 13 identifies the face area of the person shown in the captured image to be retouched based on the information supplied from the face recognition unit 151, and identifies the skin area other than the face area based on the information supplied from the skin area extraction unit 152. The brightness adjustment unit 153 adjusts the face area and the skin area other than the face area using different retouch parameters. The retouch parameter is a parameter that defines the amount of brightness adjustment.

[0121] For example, the brightness adjustment unit 153 selects a retouch parameter such that the adjustment amount for the face area is larger than the adjustment amount for the skin area other than the face area in response to a user operation using the adjustment button 61, and adjusts the brightness of each area. That is, as the retouch parameter for the brightness of the face area, a retouch parameter that makes it brighter than the retouch parameter for the brightness of the skin area other than the face area is used for image processing. As described with reference to FIG. 5, using the adjustment button 61, one or two types of levels are selected as the brightness level of only the skin.

[0122] As a result, even when the user selects either level 1 or 2 as the brightness level, as shown in FIG. 15A, image processing is performed to make the face area brighter than the skin area other than the face area. For example, when the brightness of the face area and the brightness of the skin area other than the face area are the same, the face area after brightness adjustment becomes brighter than the skin area other than the face area.

[0123] When the person shown in the captured image is a woman (bride), usually, since makeup is applied to the face, there is no sense of discomfort even if the face area is brightened. By performing image processing to make the face area brighter than the skin area outside the face area, it is possible to prevent the skin area outside the face area from becoming too bright and resulting in an uncomfortable appearance.

[0124] Such image processing that makes the face area brighter than the skin area outside the face area may be performed based on the gender determination result by the gender determination unit 135.

[0125] For example, the image processing that makes the face area brighter than the skin area outside the face area is only performed on the woman shown in the captured image. In this case, for the man shown in the captured image, image processing using retouch parameters that brighten the brightness of the face area and the skin area such as hands outside the face area by the same degree is performed. This makes it possible to prevent only the face area of the man from becoming too bright and resulting in an uncomfortable appearance.

[0126] Also, even when the user selects the same level as the brightness level, as shown in FIG. 15B, retouch parameters may be selected to make the skin area (including the face area) of the woman brighter than the skin area of the man. In this way, different retouch parameters may be used for adjusting the brightness of the skin area of the woman and the brightness of the skin area of the man, respectively.

[0127] <Display of retouch screen> FIG. 16 is a diagram showing an example of the display of the area 33-3 of the retouch screen.

[0128] After adjusting the brightness of only the skin as described above using the adjustment button 61, the user can further adjust the overall brightness using the adjustment button 62 or adjust the brightness of the background using the adjustment button 63.

[0129] On the other hand, after adjusting the overall brightness using the adjustment button 62, as shown in FIG. 16A, the adjustment of the background brightness using the adjustment button 63 is restricted. In the example on the right side of FIG. 16A, the adjustment button 63 is grayed out and displayed so that the selection of the background brightness level becomes unavailable in response to the overall brightness level being selected as "1".

[0130] Also, after adjusting the background brightness using the adjustment button 63, as shown in FIG. 16B, the adjustment of the overall brightness using the adjustment button 62 is restricted. In the example on the right side of FIG. 16B, the adjustment button 62 is grayed out and displayed so that the selection of the overall brightness level becomes unavailable in response to the background brightness level being selected as "2".

[0131] That is, the adjustment of the overall brightness and the adjustment of the background brightness are in an exclusive relationship, and only one of the adjustments can be made. After one of the adjustments of the overall brightness and the background brightness is made, the other adjustment is restricted. The adjustment of the overall brightness and the background brightness is also performed by, for example, the brightness adjustment unit 153.

[0132] For example, if it were possible to brighten the background after brightening the whole, there would be a high possibility that image processing for adjusting the brightness would be superimposed and applied to the background, resulting in white bleeding. By enabling only one of the adjustments of the overall brightness and the background brightness, it is possible to prevent such white bleeding from occurring.

[0133] Note that the display of the adjustment button 61 is maintained (unchanged) so that the brightness level of only the skin can still be selected even after adjusting the overall brightness using the adjustment button 62 or after adjusting the background brightness using the adjustment button 63.

[0134] Although adjustment buttons 61 to 63 are displayed as information used for selecting the brightness adjustment level, other selection information such as a slide bar may be displayed.

[0135] <<Function for inheriting retouch content>> As described with reference to FIG. 8, when the captured image to be retouched is switched, it is possible to inherit the retouch content selected for the captured image before the switch to the captured image after the switch. Here, the function for inheriting the retouch content will be described. It is assumed that the retouch content set in the previous captured image is selected using the display of the pop-up image 72 so as to be reflected also in the captured image after the switch.

[0136] <Inheritance of retouch content> FIGS. 17 and 18 are diagrams showing examples of inheritance of retouch content.

[0137] As shown by the arrow #21 in FIG. 17, when the captured image P21 is the first retouch target, the respective retouch parameters such as “oval face effect” being “3”, “face shape” being “egg”, and “skin texture” being “2” are set for the captured image P21 by the user's operation. The image of the retouched captured image P21 shown in the lower part of FIG. 17 is different from the image of the captured image P21 before retouch shown in the upper part.

[0138] When the retouch of the captured image P21 is completed and the retouch target is switched from the captured image P21 to the captured image P22, as shown at the tip of the arrow #22 in FIG. 18, the retouch parameters of the retouched captured image P1 are inherited as the default retouch parameters of the captured image P22, which is the second retouch target. In the display area 31, the captured image P22 is displayed in place of the captured image P21 in a state where the same retouch as the retouched captured image P21 is applied. The display of the retouch selection area 33 also becomes a display corresponding to the content of the retouch applied to the captured image P22.

[0139] The user can start retouching the captured image P22 based on the retouched captured image P22 with the same content as the captured image P21. In the example of FIG. 18, it is adjusted to the content represented by the retouch parameters such as "oval face feeling" being "4", "face shape" being "slim", "skin texture" being "1", ···.

[0140] Similarly, when the retouching of the captured image P22 is completed and the third captured image to be retouched is selected, as shown by the arrow #24, the retouch parameters of the captured image P22 are inherited as the default retouch parameters of the third captured image to be retouched.

[0141] In this way, as the captured image to be retouched is switched, the content of the retouch of the immediately previous captured image to be retouched is inherited to the appearance of the single captured image after the switch. Every time the captured image to be retouched is switched, the inheritance of the appearance of the captured image is repeated. Thereby, the user can efficiently perform the retouching of each captured image.

[0142] The inheritance of the retouch content may be performed as described above when the captured image after the switch is an unadjusted image for which retouch adjustment has not yet been performed.

[0143] <Regarding the inheritance target> Retouching can be performed for each person by selecting the target person. The inheritance of the retouch content is performed based on the gender determination result by the gender determination unit 135 such that the retouch content for the bride is inherited to the appearance of the bride, and the retouch content for the groom is inherited to the appearance of the groom.

[0144] · Switching from an image showing the groom and the bride to another image showing the groom and the bride FIG. 19 is a diagram showing a first example of inheritance of retouch content.

[0145] FIG. 19 shows an example of carrying over retouching content when the retouching of the photographed image P31 showing the person H1 (the bride) and the person H2 (the groom) is completed, and the retouching target is switched from the photographed image P31 to the photographed image P32 also showing the person H1 and the person H2.

[0146] In this case, the retouching content applied to the person H1 when the photographed image P31 is selected as the retouching target is carried over to the image of the person H1 shown in the photographed image P32, as indicated by the arrow #31. Based on the determination result by the gender determination unit 135, the retouching content applied to the female person (person H1) shown in the photographed image P31 is carried over to the image of the same female person (person H1) shown in the photographed image P32.

[0147] Also, the retouching content applied to the person H2 when the photographed image P31 is selected as the retouching target is carried over to the image of the person H2 shown in the photographed image P32, as indicated by the arrow #32. Based on the determination result by the gender determination unit 135, the retouching content applied to the male person (person H2) shown in the photographed image P31 is carried over to the image of the same male person (person H2) shown in the photographed image P32.

[0148] · Switching from an image showing the groom and the bride to an image showing only the groom or the bride FIG. 20 is a diagram showing a second example of carrying over retouching content.

[0149] FIG. 20 shows an example of carrying over retouching content when the retouching of the photographed image P31 showing the person H1 (the bride) and the person H2 (the groom) is completed, and the retouching target is switched from the photographed image P31 to the photographed image P33 showing only the person H1, or the photographed image P34 showing only the person H2.

[0150] When the retouch target is switched to the captured image P33 in the lower right, the retouch content applied to the person H1 when the captured image P31 was selected as the retouch target is carried over to the image of the person H1 shown in the captured image P33, as indicated by the arrow #33. Based on the determination result by the gender determination unit 135, the retouch content applied to the female person (person H1) shown in the captured image P31 is carried over to the image of the same female person (person H1) shown in the captured image P33.

[0151] In this case, the retouch content applied to the person H2 when the captured image P31 was selected as the retouch target will not be carried over.

[0152] When the retouch target is switched to the captured image P34 in the lower left, the retouch content applied to the person H2 when the captured image P31 was selected as the retouch target is carried over to the image of the person H2 shown in the captured image P34, as indicated by the arrow #34. Based on the determination result by the gender determination unit 135, the retouch content applied to the male person (person H2) shown in the captured image P31 is carried over to the image of the same male person (person H2) shown in the captured image P34.

[0153] In this case, the retouch content applied to the person H1 when the captured image P31 was selected as the retouch target will not be carried over.

[0154] · Switching from an image showing the bride or groom to an image showing both the bride and groom FIG. 21 is a diagram showing a third example of carrying over retouch content.

[0155] On the right side of FIG. 21, an example of carrying over the retouch content when the retouch target is switched from the captured image P33 showing only the bride, person H1, to the captured image P31 showing both person H1 and the groom, person H2, after the retouch of the captured image P33 is completed is shown.

[0156] In this case, when the captured image P33 to be retouched is selected, the retouch content applied to the person H1 is carried over to the image of the person H1 shown in the captured image P31, as indicated by the arrow #35. Based on the determination result by the gender determination unit 135, the retouch content applied to the female person (person H1) shown in the captured image P33 is carried over to the image of the same female person (person H1) shown in the captured image P31.

[0157] On the left side of FIG. 21, the retouch of the captured image P34 in which only the person H2, who is the groom, appears is completed, and an example of carrying over the retouch content when the retouch target is switched from the captured image P34 to the captured image P31 in which the person H2 and the person H1, who is the bride, appear is shown.

[0158] In this case, when the captured image P34 to be retouched is selected, the retouch content applied to the person H2 is carried over to the image of the person H2 shown in the captured image P31, as indicated by the arrow #36. Based on the determination result by the gender determination unit 135, the retouch content applied to the male person (person H2) shown in the captured image P34 is carried over to the image of the same male person (person H2) shown in the captured image P31.

[0159] This makes it possible to prevent problems such as the retouch content for the bride being carried over to the image of the groom.

[0160] · Example of allowing the user to select whether to carry over the retouch content When the gender composition of the people is different before and after switching the captured image to be retouched, information as described with reference to FIG. 8 may be presented. The user may be allowed to select whether to carry over the retouch content.

[0161] For example, as shown on the right side of FIG. 21, when the subject to be retouched is switched from a captured image P33 showing only women to a captured image P31 showing both women and men, and thus the gender composition of the people shown in the captured image changes, the composition may change significantly, making it inappropriate to inherit the retouch content. If inappropriate retouch content is inherited, it becomes cumbersome to correct it. However, by appropriately enabling selection so as not to inherit the retouch content, it is possible to save the effort of correction.

[0162] · Inheritance based on the identification result of a person Although the inheritance of the retouch content has been described as being based on the determination result of gender, it may also be based on the identification result of a person. By inheriting the retouch content for a certain person to the image of the same person, the retouch content for the bride is inherited to the image of the bride who is the same person, and the retouch content for the groom is inherited to the image of the groom who is the same person.

[0163] FIG. 22 is a block diagram showing another configuration example of the image processing unit 1²¹. The configuration shown in FIG. 22 is different from the configuration of the image processing unit 1²¹ described above in that a person identification unit 1³⁶ is provided.

[0164] The person identification unit 1³⁶ identifies each person shown in the captured image. The identification of a person by the person identification unit 1³⁶ is performed using, for example, an inference model that takes the captured image as input and outputs the ID of the person shown in the captured image. In this case, an inference model for person identification generated by machine learning is prepared in advance in the person identification unit 1³⁶. The identification of a person may also be performed based on the characteristics of the person obtained by analyzing the captured image of the subject to be retouched.

[0165] The identification result by the person identification unit 136 is supplied to the retouch processing unit 132. For example, when only the bride appears in the captured image to be retouched, the person ID assigned to the bride appearing in the captured image is supplied as the identification result. Also, when only the groom appears in the captured image to be retouched, the person ID assigned to the groom appearing in the captured image is supplied as the identification result. When both the bride and the groom appear in the captured image to be retouched, the person ID assigned to the bride and the person ID assigned to the groom are supplied as the identification result.

[0166] In the retouch processing unit 132, based on the identification result by the person identification unit 136, retouching is performed so that the content of the retouch applied to the person appearing in the captured image before switching the retouch target is carried over to the appearance of the same person appearing in the captured image after switching.

[0167] This also makes it possible to prevent problems such as the content of the retouch for the bride being carried over to the appearance of the groom.

[0168] When the composition of the people is different before and after switching the captured image to be retouched, information as described with reference to FIG. 8 may be presented. The user is made to select whether to carry over the retouch content or not.

[0169] For example, as shown on the right side of FIG. 21, when the retouch target is switched from the captured image P33 in which only person H1 appears to the captured image P31 in which both person H1 and person H2 appear, and thus the composition of the people appearing in the captured image changes, since the composition changes significantly, it may not be appropriate to carry over the retouch content. If inappropriate retouch content is carried over, it becomes cumbersome to correct it, but by enabling selection so as not to carry over the retouch content as appropriate, it is possible to save the trouble of correction.

[0170] <<Modification Example>> ·Examples of retouch locations Retouching is performed using a PC installed in a photography studio, but it may also be possible to perform retouching using other devices such as a mobile terminal 3 or a PC installed in the user's home.

[0171] When retouching is possible using the mobile terminal 3, the user who has finished taking pictures in the photography studio operates the mobile terminal 3 to start the image management application 3A and performs retouching on the screen of the image management application 3A. The image management application 3A is equipped with the same functions as the retouching application 1A.

[0172] On the display of the mobile terminal 3, a screen as described above is displayed by the image management application 3A, and the selection and retouching of the photographed image to be retouched are performed on the photographed image uploaded to the image management server 2. The photographed image retouched using the functions of the image management application 3A is printed and provided to the user in the form of an album, or the retouched photographed image is provided to the user in the form of data as it is.

[0173] As a result, the user can perform retouching at various locations such as at home. In addition, the user can perform retouching with care without worrying about time.

[0174] ·Regarding the program The above-described series of processes can be executed by hardware or by software. When the series of processes are executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer or the like.

[0175] The program to be installed is provided by being recorded on a removable medium 111 shown in FIG. 10, which consists of an optical disk (such as a CD-ROM (Compact Disc - Read Only Memory), a DVD (Digital Versatile Disc)), a semiconductor memory, etc. Further, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in the ROM 102 or the storage unit 108.

[0176] The program executed by the computer may be a program whose processing is performed in time series according to the order described in this specification, or may be a program whose processing is performed in parallel or at a necessary timing such as when a call is made.

[0177] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0178] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0179] For example, the present technology can adopt a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0180] In addition, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0181] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

Explanation of Reference Numerals

[0182] 1 PC for retouching, 2 image management server, 3 mobile terminal, 121 image processing unit, 131 image acquisition unit, 132 retouching processing unit, 133 output control unit, 134 display control unit, 135 gender determination unit, 136 person identification unit, 151 face recognition unit, 152 skin area extraction unit, 153 brightness adjustment unit

Claims

1. Identifying the people shown in each captured image using a model generated by machine learning, detecting the face region of the people shown in the captured image and the skin region other than the face region as different skin regions, respectively, adjusting the brightness of the face region and the brightness of the skin region other than the face region of the people shown in the first captured image among the plurality of captured images based on retouch parameters corresponding to the adjustment level selected by the user, respectively, In response to a user operation that reflects the adjustment content of the first captured image for a second captured image in which the subject's person composition is different from that of the first captured image, reflecting the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image A program for causing a computer to execute the process.

2. Adjusting the brightness of the face region and the brightness of the skin region other than the face region so that the face region is brighter than the skin region other than the face region The program according to claim 1.

3. The adjustment of the brightness of the face region and the brightness of the skin region other than the face region is performed in response to one operation of selecting an adjustment level of skin brightness The program according to claim 1.

4. An image processing apparatus, identifying the people shown in each captured image using a model generated by machine learning, detecting the face region of the people shown in the captured image and the skin region other than the face region as different skin regions, respectively, adjusting the brightness of the face region and the brightness of the skin region other than the face region of the people shown in the first captured image among the plurality of captured images based on retouch parameters corresponding to the adjustment level selected by the user, respectively, In response to a user operation that reflects the adjustment content of the first captured image for a second captured image in which the subject's person composition is different from that of the first captured image, reflecting the adjustment content of the skin brightness for a predetermined person shown in the first captured image on the skin brightness of the same person as the predetermined person shown in the second captured image An image processing method.

5. An identification unit that identifies the people shown in each captured image using a model generated by machine learning, A detection unit that detects the face region of the people shown in the captured image and the skin region other than the face region as different skin regions, respectively, An image processing apparatus includes a retouch processing unit that adjusts the brightness of the face region and the brightness of the skin region other than the face region of a person captured in a first captured image among a plurality of the captured images based on retouch parameters corresponding to an adjustment level selected by a user, and reflects the adjustment content of the first captured image on a second captured image in which the subject's person composition is different from that of the first captured image in response to an operation of the user to reflect the adjustment content of the first captured image on the second captured image, and reflects the adjustment content of the skin brightness of a predetermined person captured in the first captured image on the skin brightness of the same person as the predetermined person captured in the second captured image. An image processing apparatus comprising the above.

Images