Program, image processing method, and image processing apparatus

Abstract

To appropriately adjust the brightness of a subject photographed in a photographed image.SOLUTION: According to another embodiment of the present disclosure, there is provided a program causing a computer to execute a process including identifying a person appearing in each captured image using a model generated by machine learning, detecting a skin region of the person appearing in the captured image, adjusting brightness of the skin region of the person appearing in a first captured image among a plurality of captured images on the basis of a retouch parameter corresponding to an adjustment level selected by a user, and reflecting an adjustment content of the first captured image in a second captured image according to a user's operation. The adjustment content of the skin brightness of the predetermined person captured in the first captured image is reflected in the skin brightness of the same person as the predetermined person captured in the second captured image. The present technology can be applied to a computer installed in a shooting studio.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present technology relates to a program, an image processing method, and an image processing device, and more particularly to a program, an image processing method, and an image processing device that are capable of appropriately adjusting the brightness of a subject appearing in a captured image. [Background technology]

[0002] In photo taking devices such as ID photo booths and print sticker machines installed in game arcades, users can adjust the appearance of the captured image themselves. Users can adjust the appearance of the image that they receive printed on sticker paper or the like one by one to their liking. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-118329 Summary of the Invention [Problem to be solved by the invention]

[0004] To adjust the appearance of an image, brightness is adjusted for each area, such as a face area.

[0005] The present technology has been made in consideration of such circumstances, and makes it possible to appropriately adjust the brightness of a subject appearing in a captured image. [Means for solving the problem]

[0006] A program according to one aspect of the present technology causes a computer to execute a process of identifying a person appearing in each captured image using a model generated by machine learning, detecting the skin area of ​​the person appearing in the captured image, adjusting the brightness of the skin area of ​​the person appearing in a first captured image among the plurality of captured images based on retouching parameters corresponding to an adjustment level selected by a user, and, in response to a user operation to reflect the adjustment content of the first captured image in a second captured image, reflecting the adjustment content of the skin brightness of a specific person appearing in the first captured image in the skin brightness of a person identical to the specific person appearing in the second captured image.

[0007] In one aspect of the present technology, a person appearing in each captured image is identified using a model generated by machine learning, a skin area of ​​the person appearing in the captured image is detected, and brightness of the skin area of ​​the person appearing in a first captured image of the plurality of captured images is adjusted based on a retouching parameter corresponding to an adjustment level selected by a user. In addition, in response to a user operation to reflect the adjustment content of the first captured image in a second captured image, the adjustment content of the skin brightness of a predetermined person appearing in the first captured image is reflected in the skin brightness of a person identical to the predetermined person appearing in the second captured image. [Effects of the Invention]

[0008] According to the present technology, it is possible to appropriately adjust the brightness of a subject appearing in a captured image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the workflow in a photography studio. [Figure 2] FIG. 10 is a diagram illustrating an example of providing a captured image. [Figure 3] FIG. 10 is a diagram illustrating an example of file management in a retouching PC. [Figure 4] FIG. 10 is a diagram illustrating an example of an image selection screen. [Figure 5] FIG. 10 is a diagram illustrating an example of a retouch screen. [Figure 6] FIG. 10 is an enlarged view showing a display of a retouch selection area. [Figure 7] FIG. 10 is a diagram showing a display example when batch retouching is selected. [Figure 8] 10A and 10B are diagrams illustrating an example of a display when a photographed image to be retouched is switched. [Figure 9] FIG. 10 is a diagram illustrating an example of application of retouching parameters. [Figure 10] FIG. 1 is a block diagram showing an example of the configuration of a retouching PC. [Figure 11] FIG. 2 is a block diagram showing an example of the functional configuration of a retouching PC. [Figure 12] 10 is a flowchart illustrating a retouching process. [Figure 13] FIG. 2 is a block diagram showing an example of the configuration of an image processing unit. [Figure 14] FIG. 10 is a diagram illustrating an example of extraction of a skin region. [Figure 15] FIG. 10 is a diagram illustrating an example of retouching parameters. [Figure 16] FIG. 10 is a diagram showing an example of display of an area used for adjusting brightness. [Figure 17] FIG. 10 is a diagram illustrating an example of taking over retouching details. [Figure 18] FIG. 18 is a diagram following FIG. 17 showing an example of taking over retouching details. [Figure 19] FIG. 10 is a diagram showing a first example of taking over retouching details. [Figure 20] FIG. 10 is a diagram showing a second example of taking over retouching details. [Figure 21] FIG. 10 is a diagram showing a third example of taking over retouching details. [Figure 22] FIG. 10 is a block diagram showing another example of the configuration of the image processing unit. DETAILED DESCRIPTION OF THE INVENTION

[0010] <<Work flow in the photography studio>> FIG. 1 is a diagram showing the workflow in a photography studio.

[0011] This technology is used in photography studios that take commemorative photos for weddings, for example, but can also be applied to photography studios that take commemorative photos for various events, such as entrance / graduation ceremonies, coming-of-age ceremonies, and Shichi-Go-San celebrations.

[0012] As shown on the left side of Figure 1, photography in a photography studio is carried out by a photographer. In the example of Figure 1, the subjects are a bride and groom standing in front of a background screen. The groom is wearing a tuxedo, and the bride is wearing a wedding dress. Around the bride and groom, equipment such as strobes that emit light in time with the photographer's shots are placed.

[0013] The photographer takes pictures by changing the composition and taking as many as 50, 100, or more still images (photographs). The composition of the still images is determined by the poses of the bride and groom, the relative positions of the bride and groom and the photographer, and camera parameters such as focal length. The still images taken by the photographer are transferred to and saved on a PC (not shown) connected to the camera.

[0014] After the photo shoot is completed, the bride and groom move to the retouching space provided in the photography studio and, as shown on the right side of Figure 1, use the retouching PC 1 to check the images taken by the photographer.

[0015] Furthermore, the bride and groom, as users of the retouching PC 1, select the number of photographed images they ultimately want printed, and adjust the appearance of each photographed image by retouching them themselves. Retouching is performed on a portion of the photographed images, such as five or ten images, selected from those taken by the photographer. Retouching (processing) includes various image processing steps to adjust the appearance of the photographed images.

[0016] The retouching space is equipped with one or more retouching PCs 1. Photographed images stored on a storage PC are transferred to the retouching PC 1 and displayed on the display 11 of the retouching PC 1. A retouching application 1A, which is an application used to retouch photographed images taken by a photographer, is installed on the retouching PC 1.

[0017] In this way, in a photography studio where a retouching PC 1 to which the present technology is applied is installed, users (the bride and groom) themselves retouch the images taken by the photographer. The users can use the functions of the retouching application 1A to retouch the images to their liking. The retouching application 1A may be installed on a device other than a PC, such as a tablet device, and used for retouching.

[0018] FIG. 2 is a diagram showing an example of providing a captured image.

[0019] The photographed images retouched by the retouching application 1A are printed by the photography studio management company and later handed over to the user in the form of, for example, an album, as indicated by arrow #1.

[0020] Furthermore, as indicated by arrow #2, the retouched photographed images are uploaded from the retouching PC 1 to the image management server 2, where they are managed. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the photography studio's operating company or by a business entity other than the photography studio's operating company.

[0021] By operating their own mobile terminal 3 such as a smartphone and operating the image management app 3A, users can view and download images that have been taken of them as the subject, as shown by arrow #3. The image management app 3A is an application for viewing the captured images managed in the image management server 2. The image management app 3A is provided, for example, by the business operator that manages the image management server 2.

[0022] The images taken in the photography studio will be provided in the form of an album and, if appropriate, in the form of data. The provision of the images in the form of data may be made possible for a separate fee from the fee for the album.

[0023] The provision of photographed images as data is performed so that not only photographed images that have been retouched by the user but also photographed images that have not been selected by the user as retouching targets are provided. As for photographed images that have not been selected by the user as retouching targets, as will be described later, retouching is automatically performed by the retouching application 1A without any user operation.

[0024] The user can also receive, in the form of data, photographed images that were not selected for retouching. Instead of using the image management app 3A, the user may access a predetermined site using a browser and receive photographed images managed in the image management server 2. For example, by registering as a member of an image providing service provided by the administrator of the image management server 2, the user can receive photographed images managed in the image management server 2 using the image management app 3A or a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the captured images.

[0026] <<About the retouching app>> <File Management> FIG. 3 is a diagram showing an example of file management in the retouching PC 1. As shown in FIG.

[0027] As shown in the speech bubble in Figure 3, two folders, the all folder and the album folder, are prepared and linked to the user ID, which is the user's identification information. The all folder is a folder with the folder name "all," and the album folder is a folder with the folder name "album."

[0028] The image files transferred from the storage PC are stored in the all folder. The image files selected for retouching and retouched are stored in the album folder. The files stored in the album folder are the files used for printing.

[0029] When a user who has finished taking a photo starts retouching, the user inputs a user ID on the screen of retouching application 1A. Based on the user ID, the captured image is read by retouching application 1A, and processing using the captured image is performed. For example, all captured image files stored in an all folder linked to the user ID are read by retouching application 1A. After the captured images are read, the captured images to be retouched are selected.

[0030] <Screen display> Image selection screen FIG. 4 is a diagram showing an example of the image selection screen.

[0031] A message instructing the user to select five captured images is displayed at the top of the image selection screen. The number of captured images to be selected is set, for example, by a staff member at the photography studio before the image selection. If the user has signed up for a plan to print five captured images and receive them as an album, five is set as the number of images to be selected.

[0032] A thumbnail display area 21 is formed on the right side of the image selection screen. The thumbnail display area 21 is an area where thumbnail images of the captured images are displayed. For example, if 50 captured images have been taken, thumbnail images of each of the 50 captured images are displayed. Various operations using the screen of the retouching application 1A are performed using a mouse, keyboard, etc.

[0033] A photographed image selected using a thumbnail image is enlarged and displayed in a display area 22 formed on the left side of the thumbnail display area 21. In the example of Fig. 4, the thumbnail image of photographed image P1 displayed in the upper left corner of the thumbnail display area 21 is selected, and photographed image P1 is enlarged and displayed in the display area 22.

[0034] A confirm button 23 is displayed at the bottom right of the display area 22. The confirm button 23 is a button that is pressed when deciding to confirm the photographed image displayed in the display area 22 as the subject of retouching. The user selects thumbnail images to sequentially enlarge and display each photographed image in the display area 22, and when the user finds a photographed image that he or she likes, he or she presses the confirm button 23 to select it as the subject of retouching.

[0035] Below the display area 22, a selected image display area 24 is formed in which thumbnail images of photographed images selected by the user are displayed.

[0036] When the selection of five photographed images has been completed, the selection of photographed images to be retouched is completed. After the selection of photographed images to be retouched has been completed, retouching is performed on each of the photographed images selected as retouched.

[0037] Retouch screen FIG. 5 is a diagram showing an example of a retouch screen.

[0038] A display area 31 is formed in the upper center of the retouch screen. The display area 31 is an area in which the photographed image currently being retouched is displayed in an enlarged form. A thumbnail area 32 formed on the left side of the display area 31 displays a thumbnail image of the photographed image selected by the user. In the example of FIG. 5, thumbnail images of five photographed images P1 to P5 are displayed. The photographed image P1, selected using the thumbnail image, is displayed in an enlarged form in the display area 31.

[0039] A horizontally long retouching selection area 33 is formed below the retouching screen. The retouching content is selected using the buttons displayed in the retouching selection area 33. Each time a retouching content is selected using the buttons displayed in the retouching selection area 33, the selected content is reflected in the display in the display area 31. While viewing the display in the display area 31, the user uses the display in the retouching selection area 33 to select the retouching content to be applied to the captured image currently being retouched.

[0040] FIG. 6 is an enlarged view of the display of the retouch selection area 33. As shown in FIG.

[0041] The retouching selection area 33 is formed from area 33-1, which is a display area for information used to adjust the entire face and body, area 33-2, which is a display area for information used to adjust the eyes and makeup, and area 33-3, which is a display area for information used to adjust brightness.

[0042] Area 33-1 formed on the left side of retouch selection area 33 displays adjustment button 51 used to adjust the appearance of a smaller face, adjustment button 52 used to adjust the shape of the face, adjustment button 53 used to adjust the skin quality, adjustment button 54 used to adjust the chin length, and adjustment button 55 used to adjust the style.

[0043] The user can select from five different sizes to achieve a smaller face. A smaller face size of "0" indicates that the face size will remain unchanged at its original size. By selecting one of the sizes "1" to "4," the user can adjust the size of their face in the captured image displayed in the display area 31.

[0044] The user can select one of three face shapes: "round," "egg," or "slim." By selecting one of these shapes, the user can adjust the shape of their face that appears in the captured image displayed in the display area 31. By selecting "no retouching," the user can leave the shape of their face unchanged.

[0045] The user can select one of five levels for skin texture. Skin texture "0" indicates that the original skin texture will remain unchanged. By selecting one of the levels, the user can adjust the smoothness of their skin texture in the captured image displayed in the display area 31.

[0046] Three levels of chin shortening can be selected. "0" for chin shortening indicates no processing, i.e., the chin is left unchanged and the original chin is maintained. By selecting one of the levels, the user can adjust the length of their chin that appears in the photographed image displayed in the display area 31.

[0047] There are three levels of style enhancement that can be selected. Style enhancement level "0" indicates no processing, i.e., the original style is left unchanged. By selecting one of the levels, the user can adjust their own style in the captured image displayed in the display area 31.

[0048] An area 33-2 formed in the center of the retouch selection area 33 displays an adjustment button 56 used to adjust the size of the eyes, an adjustment button 57 used to adjust the shape of the eyes, an adjustment button 58 used to adjust the intensity of the makeup, an adjustment button 59 used to adjust the bridge of the nose, and an adjustment button 60 used to adjust the removal of facial hair.

[0049] The eye size can be selected from five sizes. An eye size of "0" indicates no processing, i.e., the eye size is left unchanged at its original size. By selecting one of the sizes, the user can adjust the size of their own eyes that appear in the captured image displayed in the display area 31.

[0050] The user can select one of the following eye shapes: "round eyes," "droopy eyes," and "cat eyes." By selecting one of these shapes, the user can adjust the shape of their eyes that will appear in the photographed image displayed in the display area 31. By selecting "no processing," the user can leave the shape of their eyes unchanged.

[0051] The user can select from five levels of makeup intensity. A makeup intensity of "0" indicates that the makeup intensity will remain unchanged at its original intensity. By selecting one of the levels, the user can adjust the intensity of their makeup in the captured image displayed in the display area 31.

[0052] The nose bridge can be selected as on or off. By selecting on or off, the user can choose whether to emphasize the highlights of the bridge of their nose in the photographed image displayed in the display area 31, or to leave it in its original state.

[0053] Three levels of beard removal can be selected. "0" for beard removal indicates that the original beard is left unchanged. By selecting one of the levels, the user can adjust the beard of the person appearing in the photographed image displayed in the display area 31. The beard removal function is enabled when the person being retouched is the groom.

[0054] An area 33-3 formed on the right side of the retouch selection area 33 displays an adjustment button 61 used to adjust the brightness of only the skin, an adjustment button 62 used to adjust the overall brightness, and an adjustment button 63 used to adjust the brightness of only the background.

[0055] Three levels of brightness can be selected for adjusting the skin brightness. A skin brightness of "0" indicates that the skin brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the brightness of their own skin in the captured image displayed in the display area 31. Details of the skin brightness adjustment function will be described later.

[0056] Three levels of brightness can be selected to adjust the overall brightness. An overall brightness of "0" indicates that the overall brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the overall brightness of the captured image displayed in the display area 31.

[0057] Three levels of brightness can be selected to adjust the background brightness. A background brightness of "0" indicates that the background brightness is not changed and remains at its original brightness. By selecting one of the brightness levels, the user can adjust the brightness of the background in the captured image displayed in the display area 31.

[0058] Returning to the explanation of Figure 5, a batch retouch button 34 used for batch retouching is displayed on the left side of the display area 31. Batch retouching is a function for selecting a combination of various retouching contents as described above all at once.

[0059] In the example of Figure 5, the user can select one of three levels of batch retouching, each with a different combination of retouching levels. By using the batch retouching function, the user can select the retouching content with a single operation, without having to select each retouching content using the display in the retouching selection area 33.

[0060] FIG. 7 is a diagram showing a display example when batch retouching is selected.

[0061] When the batch retouch button 34 is operated, a rounded rectangular pop-up image 71 is displayed superimposed on the retouching screen. The pop-up image 71, which appears approximately in the center of the retouching screen, displays a circular button used to select the range of images to be batch retouched along with a message.

[0062] By operating the button displayed in the pop-up image 71, the user can select whether to apply the batch retouching contents to only the one image currently being retouched, or to all (five) images including the one image currently being retouched.

[0063] Returning to the explanation of FIG. 5, a Before button 35 is provided on the right side of the display area 31. The Before button 35 is a button that is operated to switch the display in the display area 31 to the state before retouching. While the Before button 35 is pressed, the captured image in the state before retouching is displayed in the display area 31. By using the Before button 35, the user can compare the state after retouching with the state before retouching and check whether the degree of retouching is too strong, for example.

[0064] Below the Before button 35, there is provided a Person Selection button 36 used to select a person to be retouched. By using the Person Selection button 36, the user can select and retouch only the person appearing on the left side of the captured image, only the person appearing on the right side, or both people appearing on the right side. Facial recognition and other processes are performed on the captured image, and the person to be the target of image processing for retouching is selected in response to the user's operation.

[0065] Using these various functions, the user can select the retouching content to be applied to the captured image displayed in the display area 31. In this way, retouching that can be performed using the retouching application 1A includes at least adjustments to the face and body as a whole, adjustments to the eyes and makeup, and adjustments to brightness.

[0066] The photographed image to be retouched is switched using the display in the thumbnail area 32, and retouching is performed sequentially on each of the photographed images P1 to P5. The retouched files for each of the photographed images P1 to P5 are stored in the album folder.

[0067] FIG. 8 is a diagram showing a display example when the photographed image to be retouched is switched.

[0068] When the captured image to be retouched is switched, a speech bubble-shaped pop-up image 72 is displayed superimposed on the retouching screen. The pop-up image 72 is displayed slightly below the center of the retouching screen, with the origin of the pop-up image being the display area 31 formed approximately above the center of the retouching screen. The pop-up image 72 displays a horizontally long rectangular button along with a message that can be used to select whether or not the retouching details set on the previous photo (captured image) should be reflected in the switched captured image.

[0069] That is, when the photographed image to be retouched is switched, an image having a different shape from the pop-up image 71 that is displayed when the collective retouch button 34 is operated is displayed as the pop-up image 72.

[0070] The user can carry over the retouching details selected for the previous captured image to the new captured image by displaying the pop-up image 72. The retouching details carrying over function will be described in detail later.

[0071] FIG. 9 is a diagram showing an example of application of retouching parameters.

[0072] As shown in FIG. 9, the retouching content applied to each of the photographed images P1 to P5 by the user's operation is reflected in each of the unselected images stored in the all folder. An unselected image is a photographed image that has not been selected as a target for retouching on the image selection screen. A photographed image that has been selected as a target for retouching on the image selection screen is a selected image.

[0073] The retouching content of which of the retouched photographed images P1 to P5 should be reflected in each unselected image is selected based on, for example, the state of the face appearing in the selected image and the state of the face appearing in the unselected image. In this case, each photographed image is analyzed, and the state of the face appearing in each photographed image is recognized. A selected image in which a face appears in a state similar to the state of the face appearing in each unselected image is selected, and the retouching content applied to the selected selected image is reflected in the unselected image.

[0074] <<Configuration and operation of retouching PC1>> <Configuration of retouching PC1> FIG. 10 is a block diagram showing an example of the configuration of the retouching PC 1. As shown in FIG.

[0075] A CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0076] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0077] The input unit 106 is composed of a keyboard, a mouse, and the like.

[0078] The output unit 107 is configured by a display 11 and the like.

[0079] The storage unit 108 is configured by a hard disk, a non-volatile memory, etc. The storage unit 108 stores programs such as the retouch application 1A executed by the CPU 101, and various types of information such as captured images.

[0080] The communication unit 109 is an interface to a network such as the Internet or a LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a photographed image transmitted from the storage PC, in which the user about to begin retouching appears as a subject. The communication unit 109 also communicates with the image management server 2, and transmits the retouched photographed image to the image management server 2.

[0081] The drive 110 controls writing of data to and reading of data from the removable medium 111. Captured images may be imported from a PC via the removable medium 111 for storage.

[0082] FIG. 11 is a block diagram showing an example of the functional configuration of the retouching PC 1. As shown in FIG.

[0083] 11, in the retouching PC 1, the retouching application 1A is executed to implement an image processing unit 121. The image processing unit 121 is made up of an image acquisition unit 131, a retouching processing unit 132, an output control unit 133, and a display control unit .

[0084] The image acquisition unit 131 accepts a user's operation on the image selection screen and acquires the captured image selected by the user as the target for retouching from, for example, the storage unit 108. The captured image to be retouched acquired by the image acquisition unit 131 is supplied to the retouching processing unit 132 and the display control unit 134.

[0085] Furthermore, when retouching of each photographed image to be retouched is completed, the image acquisition unit 131 acquires an unselected image from, for example, the storage unit 108 and outputs it to the retouching processing unit 132 .

[0086] The retouching processor 132 performs face recognition to recognize faces appearing in captured images to be retouched. The retouching processor 132 analyzes the features of each part of the recognized face and performs various image processing related to retouching in response to user operations on the retouching screen. The analysis of the features of each part may be performed using an inference model generated by machine learning.

[0087] For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouching processor 132 performs image processing such as shifting the position of the eye outline in accordance with the user's selection. Also, when an operation related to adjusting the intensity of makeup is performed, the retouching processor 132 performs image processing such as combining an image of blush of a predetermined color on the cheeks or combining an image of lipstick of a predetermined color on the lips.

[0088] When an operation related to adjusting the skin quality is performed, the retouching processing unit 132 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the face size (feeling of a small face) or face shape is performed, the retouching processing unit 132 performs image processing such as shifting the position of the face outline in accordance with the selection made by the user.

[0089] Information representing the results of image processing by the retouching processor 132 is supplied to the display controller 134 and used to display the retouched screen.

[0090] Furthermore, when retouching of each photographed image to be retouched is completed, the retouching processing unit 132 analyzes each photographed image, including unselected images, and reflects the retouching content applied to the retouched photographed images in the unselected images. The photographed images retouched by the retouching processing unit 132 are supplied to the output control unit 133.

[0091] The output control unit 133 saves the captured image that has been retouched in response to the user's operation in the album folder.

[0092] Furthermore, the output control unit 133 outputs the retouched photographed images (selected image and unselected image) to the communication unit 109, and causes them to be transmitted to the image management server 2.

[0093] The display control unit 134 controls the display of various screens of the retouching application 1A, such as the image selection screen and the retouching screen.

[0094] <Retouching PC1 Operation> The retouching process performed by the retouching application 1A will be described with reference to the flowchart of FIG.

[0095] In step S1, the display control unit 134 of the image processing unit 121 displays the image selection screen (FIG. 4).

[0096] In step S2, the image acquisition unit 131 receives a selection of a photographed image to be retouched in response to a user's operation on the image selection screen. The image acquisition unit 131 acquires the photographed image to be retouched.

[0097] In step S3, the image acquisition unit 131 determines whether or not the selection of the captured images to be retouched has been completed. If it is determined in step S3 that the selection of the captured images to be retouched has not been completed, the process returns to step S1, and image selection continues.

[0098] For example, if five photographed images have been selected, it is determined in step S3 that the selection of photographed images to be retouched has ended.

[0099] In step S4, the display control unit 134 displays the retouch screen (FIG. 5). The user retouches the captured image using the retouch screen.

[0100] In step S5, the retouching processing unit 132 performs image processing in response to a user operation, and retouches the photographed image selected as the target for retouching.

[0101] In step S6, the retouching processor 132 determines whether or not the retouching is completed. If it is determined in step S6 that the retouching has not ended, the process returns to step S4, and retouching continues for each photographed image selected as the retouching target.

[0102] For example, when all of the photographed images selected as retouching targets have been retouched, it is determined in step S6 that retouching is to be completed.

[0103] In step S7, the retouching processor 132 applies the same retouching content as that applied to the selected image to each of the unselected images.

[0104] In step S8, the output control unit 133 saves the retouched unselected image.

[0105] In step S9, the communication unit 109 transmits the retouched selected images and the unselected images to the image management server 2. After that, the processing of the retouching application 1A ends.

[0106] Through the above process, the user can adjust the appearance of the photographed image in which the user is the subject, and can receive an album of the photographed images that are in the user's preferred appearance.

[0107] Furthermore, by using the mobile terminal 3, the user can receive, in the form of data, photographed images whose appearance has been manually adjusted and photographed images (unselected images) whose appearance has been automatically retouched in the same manner as the manual adjustments, which are managed in the image management server 2.

[0108] <<Skin brightness adjustment function>> As explained with reference to Fig. 5, it is possible to adjust the brightness of the photographed image to be retouched. Here, the function of adjusting the brightness of only the skin, which is performed using the adjustment button 61 in area 33-3, will be explained.

[0109] <Configuration of image processing unit 121> 13 is a block diagram showing an example of the configuration of image processing unit 121 that realizes the function of adjusting the brightness of only the skin. In FIG. 13, only a part of the configuration of image processing unit 121 is shown.

[0110] In addition to the components described above, the image processing unit 121 is provided with a gender determination unit 135. The retouching processing unit 132 implements a face recognition unit 151, a skin region extraction unit 152, and a brightness adjustment unit 153. The captured image to be retouched is input to the gender determination unit 135 and each unit of the retouching processing unit 132.

[0111] The gender determination unit 135 determines the gender of people appearing in the captured image. The determination by the gender determination unit 135 is performed, for example, using an inference model that receives the captured image as input and outputs the gender of each person appearing in the captured image. In this case, an inference model for gender determination generated by machine learning is prepared in advance in the gender determination unit 135. The gender determination may be performed based on the characteristics of the person obtained by analyzing the captured image to be retouched.

[0112] The determination result by the gender determination unit 135 is supplied to the brightness adjustment unit 153 of the retouching processing unit 132. For example, if the photographed image to be retouched contains only a bride, information indicating that the person in the photographed image is a woman is supplied as the determination result. Also, if the photographed image to be retouched contains only a groom, information indicating that the person in the photographed image is a man is supplied as the determination result. If the photographed image to be retouched contains both a bride and groom, information indicating that the people in the photographed image are a woman and a man is supplied as the determination result.

[0113] The face recognition unit 151 of the retouching processing unit 132 recognizes the face of a person appearing in a captured image and extracts a face region. The face recognition unit 151 may extract a face region using an inference model generated by machine learning, or may extract a face region based on features such as contour points of each part of the body determined by analyzing the captured image. Information about the face region extracted by the face recognition unit 151 is supplied to the skin region extraction unit 152 and the brightness adjustment unit 153.

[0114] The skin region extraction unit 152 extracts the skin region of a person appearing in a photographed image. For example, an inference model is prepared in the skin region extraction unit 152, which receives the photographed image as input and outputs information about the skin region of the person appearing in the photographed image. The skin region extraction unit 152 extracts a skin region other than the face region based on the skin region extracted using the inference model and the face region extracted by the face recognition unit 151.

[0115] FIG. 14 is a diagram showing an example of extraction of a skin region by the skin region extraction unit 152. In FIG.

[0116] For example, a case will be described in which a photographed image P11 shown in the upper left of Fig. 14 is selected as a target for retouching. The photographed image P11 shows a person H1, a bride in a wedding dress. The person H1's chest and the area from the shoulders to the arms (upper arms) are exposed.

[0117] When such a photographed image P11 is to be retouched, the face recognition unit 151 extracts a face area as indicated by the tip of arrow #11.

[0118] Skin region extraction unit 152 inputs photographed image P11 to skin region extraction model M as indicated by the tip of arrow #12, and extracts, as indicated by the tip of arrow #13, a skin region including a face region based on the output of skin region extraction model M. In photographed image P11, the face region and regions from the shoulders to the arms are captured as regions where the skin of person H1 is exposed.

[0119] Furthermore, as indicated by the tips of arrows #14 and #15, skin region extraction unit 152 extracts skin regions other than the face region, as indicated by the tip of arrow #16, by removing the face region extracted by face recognition unit 151 from the skin region extracted using skin region extraction model M. Information on the skin region other than the face region extracted in this manner is supplied to brightness adjustment unit 153.

[0120] 13 identifies the face area of ​​a person appearing in a photographed image to be retouched based on information supplied from face recognition unit 151, and identifies the skin area other than the face area based on information supplied from skin area extraction unit 152. Brightness adjustment unit 153 adjusts the face area and the skin area other than the face area using different retouching parameters. The retouching parameters are parameters that define the amount of brightness adjustment.

[0121] For example, the brightness adjustment unit 153 adjusts the brightness of each area by selecting retouching parameters that adjust the amount of adjustment for the face area more than the amount of adjustment for the skin area other than the face area in response to a user's operation using the adjustment button 61. That is, the retouching parameters for the brightness of the face area that make the area brighter than the retouching parameters for the skin area other than the face area are used in image processing. As described with reference to FIG. 5, one of levels 1 or 2 is selected as the brightness level for only the skin using the adjustment button 61.

[0122] As a result, regardless of whether the user selects brightness level 1 or 2, image processing is performed to make the face area brighter than the skin area other than the face area, as shown in Fig. 15A. For example, if the brightness of the face area and the brightness of the skin area other than the face area are the same, the face area after brightness adjustment will be brighter than the skin area other than the face area.

[0123] If the person in the photographed image is a woman (bride), she will usually wear makeup, so even if the facial area is brightened, it will not look strange. By applying image processing to brighten the facial area more than the skin area other than the facial area, it is possible to prevent the skin area other than the facial area from becoming too bright and looking strange.

[0124] Such image processing, which brightens the face region more than the skin region other than the face region, may be performed based on the gender determination result by the gender determination unit 135.

[0125] For example, image processing that brightens the face area more than the skin area other than the face area is performed only on women who appear in the photographed image. In this case, image processing is performed on men who appear in the photographed image using retouching parameters that brighten the face area and the skin area other than the face area, such as the hands, to the same extent. This makes it possible to prevent the face area of ​​the man from being too bright and resulting in an unnatural appearance.

[0126] Furthermore, even if the user selects the same brightness level, retouching parameters may be selected that make the skin area (including the face area) of the woman brighter than the skin area of ​​the man, as shown in Fig. 15B. In this way, different retouching parameters may be used to adjust the brightness of the skin area of ​​the woman and the brightness of the skin area of ​​the man.

[0127] <Retouch screen display> FIG. 16 is a diagram showing a display example of the area 33-3 on the retouch screen.

[0128] After adjusting the brightness of only the skin using adjustment button 61 as described above, the user can further adjust the overall brightness using adjustment button 62, or adjust the background brightness using adjustment button 63.

[0129] In contrast, after adjusting the overall brightness using adjustment button 62, adjustment of the background brightness using adjustment button 63 is restricted, as shown in Fig. 16A. In the example on the right side of Fig. 16A, since the overall brightness level has been selected as "1," adjustment button 63 is displayed grayed out so that the background brightness level cannot be selected.

[0130] Furthermore, after adjusting the background brightness using the adjustment button 63, adjustment of the overall brightness using the adjustment button 62 is restricted, as shown in Fig. 16B. In the example on the right side of Fig. 16B, since the background brightness level has been selected as "2," the adjustment button 62 is displayed grayed out so that the overall brightness level cannot be selected.

[0131] In other words, the adjustment of the overall brightness and the adjustment of the background brightness are mutually exclusive, and it is possible to adjust only one of them. After one of the adjustment of the overall brightness and the adjustment of the background brightness has been performed, the adjustment of the other is restricted. The adjustment of the overall brightness and the adjustment of the background brightness are also performed by, for example, the brightness adjustment unit 153.

[0132] For example, if it were possible to brighten the background after brightening the entire image, image processing to adjust the brightness of the background would be applied in a superimposed manner, increasing the likelihood of blown-out highlights. By making it possible to adjust only one of the overall brightness adjustment or the background brightness adjustment, it becomes possible to prevent such blown-out highlights from occurring.

[0133] Even after adjusting the overall brightness using adjustment button 62, or after adjusting the background brightness using adjustment button 63, the display of adjustment button 61 remains (remains unchanged) so that the brightness level of only the skin remains selectable.

[0134] Although the adjustment buttons 61 to 63 are displayed as information used to select the brightness adjustment level, other selection information such as a slide bar may also be displayed.

[0135] <<Retouching content transfer function>> As explained with reference to Fig. 8, when a photographed image to be retouched is switched, the retouching details selected for the photographed image before the switch can be carried over to the photographed image after the switch. Here, the retouching details carrying over function will be explained. It is assumed that the retouching details set for the previous photographed image are selected to be reflected in the photographed image after the switch using the display of the pop-up image 72.

[0136] <Transfer of retouching content> 17 and 18 are diagrams showing an example of taking over retouching details.

[0137] As indicated by arrow #21 in Fig. 17, when photographed image P21 is the first image to be retouched, the user sets the following retouching parameters for photographed image P21: "Small face feeling" to "3," "Facial shape" to "egg," "Skin quality" to "2," etc. The appearance of retouched photographed image P21 shown in the bottom row of Fig. 17 is different from the appearance of photographed image P21 before retouching shown in the top row.

[0138] When retouching of photographed image P21 is completed and the retouching target is switched from photographed image P21 to photographed image P22, the retouching parameters of the retouched photographed image P1 are inherited as the default retouching parameters for photographed image P22, the second photographed image to be retouched, as indicated by arrow #22 in Fig. 18. In display area 31, photographed image P22 is displayed in place of photographed image P21, with the same retouching applied to it as the retouched photographed image P21. The display of retouching selection area 33 also changes depending on the retouching applied to photographed image P22.

[0139] The user can start retouching the photographed image P22 by using the photographed image P22, which has been retouched in the same manner as the photographed image P21, as a reference. In the example of Fig. 18, the adjustments are made to the contents represented by the retouching parameters: "Small face feeling" is "4", "Facial shape" is "Slender", "Skin quality" is "1", etc.

[0140] Similarly, when retouching of photographed image P22 is completed and a third photographed image to be retouched is selected, the retouching parameters of photographed image P22 are inherited as the default retouching parameters for the third photographed image to be retouched, as shown by arrow #24.

[0141] In this way, as the photographed image to be retouched is switched, the retouching details of the photographed image to be retouched immediately before are carried over to the appearance of the next photographed image. Each time the photographed image to be retouched is switched, the carrying over of the appearance of the photographed images is repeated. This allows the user to efficiently retouch each photographed image.

[0142] If the captured image after switching is an unadjusted image that has not yet undergone retouching adjustment, the retouching details may be carried over as described above.

[0143] <About what to transfer> Retouching can be performed for each person by selecting the person to be retouched. The retouching content is passed on based on the gender determination result by the gender determination unit 135, so that the retouching content for the bride is passed on to the bride's photo, and the retouching content for the groom is passed on to the groom's photo.

[0144] Switching from an image with the bride and groom to another image with the bride and groom FIG. 19 is a diagram showing a first example of the transfer of retouching details.

[0145] Figure 19 shows an example of the transfer of retouching content when retouching of a photographed image P31, which shows a bride H1 and a groom H2, is completed and the retouching target is switched from the photographed image P31 to a photographed image P32, which also shows the persons H1 and H2.

[0146] In this case, the retouching details applied to person H1 when photographed image P31 was selected as the retouching target are carried over to the appearance of person H1 in photographed image P32, as indicated by arrow #31. Based on the determination result by gender determination unit 135, the retouching details applied to the female person (person H1) in photographed image P31 are carried over to the appearance of the same female person (person H1) in photographed image P32.

[0147] Furthermore, the retouching details applied to person H2 when photographed image P31 was selected as the retouching target are carried over to the appearance of person H2 in photographed image P32, as indicated by arrow #32. Based on the determination result by gender determination unit 135, the retouching details applied to the male person (person H2) in photographed image P31 are carried over to the appearance of the same male person (person H2) in photographed image P32.

[0148] Switching from an image with the bride and groom to an image with just the groom or just the bride FIG. 20 is a diagram showing a second example of the transfer of retouching details.

[0149] Figure 20 shows an example of the transfer of retouching content when retouching of a photographed image P31 showing a bride H1 and a groom H2 is completed and the retouching target is switched from the photographed image P31 to a photographed image P33 showing only person H1 or to a photographed image P34 showing only person H2.

[0150] When the retouching target is switched to photographed image P33 at the bottom right, the retouching content applied to person H1 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H1 in photographed image P33, as indicated by arrow #33. Based on the determination result by gender determination unit 135, the retouching content applied to the female person (person H1) in photographed image P31 is carried over to the appearance of the same female person (person H1) in photographed image P33.

[0151] In this case, the retouching details applied to person H2 when photographed image P31 was selected as the retouching target will not be carried over.

[0152] When the retouching target is switched to photographed image P34 at the lower left, the retouching content applied to person H2 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P34, as indicated by arrow #34. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P31 is carried over to the appearance of the same male person (person H2) in photographed image P34.

[0153] In this case, the retouching content applied to the person H1 when the photographed image P31 was selected as the retouching target will not be carried over.

[0154] Switching from an image featuring either the groom or the bride to an image featuring both the groom and the bride FIG. 21 is a diagram showing a third example of the transfer of retouching details.

[0155] The right side of Figure 21 shows an example of the transfer of retouching content when retouching of photographed image P33, which shows only person H1, the bride, is completed and the retouching target is switched from photographed image P33 to photographed image P31, which shows person H1 and person H2, the groom.

[0156] In this case, the retouching details applied to person H1 when photographed image P33 was selected as the retouching target are carried over to the appearance of person H1 in photographed image P31, as indicated by arrow #35. Based on the determination result by gender determination unit 135, the retouching details applied to the female person (person H1) in photographed image P33 are carried over to the appearance of the same female person (person H1) in photographed image P31.

[0157] The left side of Figure 21 shows an example of the transfer of retouching content when retouching of photographed image P34, which shows only person H2, the groom, is completed and the retouching target is switched from photographed image P34 to photographed image P31, which shows person H2 and person H1, the bride.

[0158] In this case, the retouching details applied to person H2 when photographed image P34 was selected as the retouching target are carried over to the appearance of person H2 in photographed image P31, as indicated by arrow #36. Based on the determination result by gender determination unit 135, the retouching details applied to the male person (person H2) in photographed image P34 are carried over to the appearance of the same male person (person H2) in photographed image P31.

[0159] This makes it possible to prevent problems such as retouching intended for the bride being carried over to the groom's photo.

[0160] Example of letting the user choose whether to inherit retouching details If the gender composition of people before and after switching of the photographed image to be retouched is different, information such as that described with reference to Fig. 8 may be presented. The user is prompted to select whether or not to carry over the retouching content.

[0161] For example, as shown on the right side of Figure 21, if the retouching target is switched from photographed image P33, which contains only women, to photographed image P31, which contains both women and men, and the composition changes significantly, it may not be appropriate to carry over the retouching content. If inappropriate retouching content is carried over, correcting it can be cumbersome, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to eliminate the effort required for correction.

[0162] Handover based on person identification results Although the transfer of retouching details is based on the gender determination results, it may also be based on the person's identification results. When retouching details intended for a certain person are transferred to a photo of the same person, retouching details intended for the bride are transferred to a photo of the same person, and retouching details intended for the groom are transferred to a photo of the same person, too.

[0163] Fig. 22 is a block diagram showing another example of the configuration of the image processing unit 121. The configuration shown in Fig. 22 differs from the configuration of the image processing unit 121 described above in that a person identification unit 136 is provided.

[0164] The person identification unit 136 identifies each person appearing in the captured image. The person identification unit 136 identifies the person using, for example, an inference model that receives the captured image as input and outputs the ID of the person appearing in the captured image. In this case, an inference model for person identification generated by machine learning is prepared in advance in the person identification unit 136. The person may be identified based on the characteristics of the person determined by analyzing the captured image to be retouched.

[0165] The identification result by the person identification unit 136 is supplied to the retouching processing unit 132. For example, if only the bride appears in the photographed image to be retouched, the person ID assigned to the bride appearing in the photographed image is supplied as the identification result. Also, if only the groom appears in the photographed image to be retouched, the person ID assigned to the groom appearing in the photographed image is supplied as the identification result. If the bride and groom appear in the photographed image to be retouched, the person ID assigned to the bride and the person ID assigned to the groom are supplied as the identification result.

[0166] In the retouching processing section 132, based on the identification result by the person identification section 136, retouching is performed so that the content of retouching performed on a person appearing in a photographed image before switching of the retouching target is carried over to the image of the same person appearing in the photographed image after switching.

[0167] This also makes it possible to prevent problems such as retouching intended for the bride being carried over to the groom's photo.

[0168] If the composition of a person in a photographed image to be retouched differs before and after the image is switched, information such as that described with reference to Fig. 8 may be presented. The user is prompted to select whether or not to carry over the retouching content.

[0169] For example, as shown on the right side of Figure 21, if the composition of people in a photographed image changes, such as when the retouching target is switched from photographed image P33, which contains only person H1, to photographed image P31, which contains people H1 and H2, the composition changes significantly, making it inappropriate to carry over the retouching content. If inappropriate retouching content is carried over, correcting it can be cumbersome, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to eliminate the effort required for correction.

[0170] <<Modifications>> Examples of retouching locations Although retouching is performed using a PC installed in the photography studio, retouching may also be performed using other devices, such as a mobile terminal 3 or a PC installed in the user's home.

[0171] If retouching using the mobile terminal 3 is possible, after a user has finished taking photos in a photography studio, the user operates the mobile terminal 3 to launch the image management application 3A and performs retouching on the screen of the image management application 3A. The image management application 3A has the same functions as the retouching application 1A.

[0172] The image management application 3A displays the above-described screen on the display of the mobile terminal 3, and the photographed images to be retouched are selected and retouched from among the photographed images uploaded to the image management server 2. The photographed images that have been retouched using the functions of the image management application 3A are printed and provided to the user in the form of an album, or the retouched photographed images are provided to the user as digital data.

[0173] This allows the user to carry out retouching in various places such as at home, and the user can carry out retouching to their heart's content without worrying about time.

[0174] About the program The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0175] The program to be installed is provided by being recorded on removable media 111 shown in FIG. 10, which may be an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 102 or storage unit 108.

[0176] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0177] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0178] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0179] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0180] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0181] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. [Explanation of symbols]

[0182] 1 Retouching PC, 2 Image management server, 3 Mobile terminal, 121 Image processing unit, 131 Image acquisition unit, 132 Retouching processing unit, 133 Output control unit, 134 Display control unit, 135 Gender determination unit, 136 Person identification unit, 151 Face recognition unit, 152 Skin region extraction unit, 153 Brightness adjustment unit

Claims

1. displaying a processing screen including first selection information used to select an adjustment level for the brightness of the skin, second selection information used to select an adjustment level for the brightness of the entire captured image, third selection information used to select an adjustment level for the brightness of the background of the person, and person selection information used to select a person to be processed; Detects the skin area of ​​a person in the captured image, adjusting a skin area of ​​the person selected using the person selection information based on a retouching parameter corresponding to a selected adjustment level in response to a user operation on the first selection information; adjusting the brightness of the entire captured image based on a retouching parameter corresponding to the selected adjustment level in response to a user operation on the second selection information; In response to a user's operation on the third selection information, the brightness of the background of the person appearing in the photographed image is adjusted based on a retouching parameter corresponding to the selected adjustment level. A program that causes a computer to perform a process.

2. When one of the selection using the second selection information and the selection using the third selection information is made, the display of the selection information used for the other selection is switched so that the other selection is disabled. The program according to claim 1, which causes a computer to execute the process.

3. The image processing device displaying a processing screen including first selection information used to select an adjustment level for the brightness of the skin, second selection information used to select an adjustment level for the brightness of the entire captured image, third selection information used to select an adjustment level for the brightness of the background of the person, and person selection information used to select a person to be processed; Detects the skin area of ​​a person in the captured image, adjusting a skin area of ​​the person selected using the person selection information based on a retouching parameter corresponding to a selected adjustment level in response to a user operation on the first selection information; adjusting the brightness of the entire captured image based on a retouching parameter corresponding to the selected adjustment level in response to a user operation on the second selection information; In response to a user's operation on the third selection information, the brightness of the background of the person appearing in the photographed image is adjusted based on a retouching parameter corresponding to the selected adjustment level. Image processing methods.

4. a display control unit that displays a processing screen including first selection information used to select an adjustment level for skin brightness, second selection information used to select an adjustment level for brightness of the entire captured image, third selection information used to select an adjustment level for brightness of the background of a person, and person selection information used to select a person to be processed; a detection unit that detects a skin area of ​​a person appearing in a captured image; an adjustment unit that adjusts a skin area of ​​a person selected using the person selection information based on a retouching parameter corresponding to a selected adjustment level in response to a user operation on the first selection information, adjusts brightness of the entire captured image based on a retouching parameter corresponding to the selected adjustment level in response to a user operation on the second selection information, and adjusts brightness of a background of a person appearing in the captured image based on a retouching parameter corresponding to the selected adjustment level in response to a user operation on the third selection information; An image processing device comprising:

Images