Program, image processing method, and image processing device

Abstract

To appropriately adjust the brightness of a subject appearing in a photographed image.SOLUTION: A program according to an embodiment of the present technology detects a face region and a skin region other than the face region of a person appearing in a photographed image as different skin regions, and adjusts the brightness of the face region and the brightness of the skin region other than the face region such that the face region and the skin region other than the face region are brightened in response to a selection of an adjustment level of the brightness of the skin. The present technology can be applied to a computer installed in a photography studio.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present technology relates to a program, an image processing method, and an image processing device, and more particularly to a program, an image processing method, and an image processing device that are capable of appropriately adjusting the brightness of a subject appearing in a captured image. [Background technology]

[0002] In photo taking devices such as ID photo machines and print sticker machines installed in game arcades, the user can adjust the appearance of the captured image by himself / herself. The user can adjust the appearance of the image that is printed on sticker paper or the like and receive it according to his / her preference one by one. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-118329 A Summary of the Invention [Problem to be solved by the invention]

[0004] To adjust the appearance of an image, brightness of each area, such as a face area, is adjusted.

[0005] The present technology has been made in consideration of such circumstances, and makes it possible to appropriately adjust the brightness of a subject appearing in a captured image. [Means for solving the problem]

[0006] A program according to one aspect of the present technology is a program that causes a computer to execute a process of detecting a facial region of a person appearing in a captured image and a skin region other than the facial region as different skin regions, and adjusting the brightness of the facial region and the brightness of the skin region other than the facial region so that the facial region and the skin region other than the facial region are each brighter in response to a selection of a skin brightness adjustment level.

[0007] In one aspect of the present technology, a facial region of a person appearing in a captured image and a skin region other than the facial region are detected as different skin regions, and in response to a selection of a skin brightness adjustment level, the brightness of the facial region and the brightness of the skin region other than the facial region are each adjusted so that the facial region and the skin region other than the facial region are each brighter. Effect of the Invention

[0008] According to the present technology, it is possible to appropriately adjust the brightness of a subject appearing in a captured image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a workflow in a photography studio. [Diagram 2] FIG. 13 is a diagram showing an example of providing a captured image. [Diagram 3] FIG. 13 is a diagram showing an example of file management in a retouching PC. [Figure 4] FIG. 13 is a diagram showing an example of an image selection screen. [Diagram 5] FIG. 13 is a diagram showing an example of a retouch screen. [Figure 6] FIG. 13 is an enlarged view showing a display of a retouch selection area. [Figure 7] FIG. 13 is a diagram showing a display example when batch retouching is selected. [Figure 8] 13A and 13B are diagrams illustrating an example of a display when a photographed image to be retouched is switched. [Figure 9] FIG. 13 is a diagram illustrating an example of application of retouching parameters. [Figure 10]FIG. 2 is a block diagram showing an example of the configuration of a retouching PC. [Figure 11] FIG. 2 is a block diagram showing an example of the functional configuration of a retouching PC. [Figure 12] 11 is a flowchart illustrating a retouching process. [Figure 13] 2 is a block diagram showing an example of the configuration of an image processing unit; [Figure 14] FIG. 13 is a diagram illustrating an example of extraction of a skin region. [Figure 15] FIG. 11 is a diagram illustrating an example of retouching parameters. [Figure 16] 13A and 13B are diagrams illustrating an example of a display of an area used for adjusting brightness. [Figure 17] 13A to 13C are diagrams illustrating an example of taking over retouching contents. [Figure 18] FIG. 18 is a diagram continuing from FIG. 17 showing an example of taking over retouching contents. [Figure 19] FIG. 13 is a diagram showing a first example of taking over retouching contents. [Figure 20] FIG. 13 is a diagram showing a second example of the transfer of retouching contents. [Figure 21] FIG. 13 is a diagram showing a third example of the transfer of retouching contents. [Figure 22] FIG. 13 is a block diagram showing another example of the configuration of the image processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <<Work flow in the photography studio>> FIG. 1 is a diagram showing the flow of work in a photography studio.

[0011] This technology is used in a photography studio for taking commemorative photos of weddings, for example. This technology can also be applied to cases where commemorative photos of various events, such as entrance / graduation photos, coming-of-age ceremonies, and Shichi-Go-San ceremonies, are taken in a photography studio.

[0012] As shown on the left side of Figure 1, a photographer takes pictures in a photography studio. In the example of Figure 1, the subjects of the photo are a bride and groom standing in front of a background screen. The groom is wearing a tuxedo, and the bride is wearing a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photographer's shots is placed.

[0013] The photographer takes many still images (photographs), such as 50 or 100, with different compositions. The composition of the still images is determined by the poses of the bride and groom, the relative positions of the bride and groom and the photographer, and camera parameters such as focal length. The still images taken by the photographer are transferred to and saved in a PC (not shown) connected to the camera.

[0014] After the photo shoot is completed, the bride and groom move to the retouching space prepared in the photography studio and, as shown on the right side of Figure 1, use the retouching PC 1 to check the images taken by the photographer.

[0015] The bride and groom, as users of the retouching PC 1, select the number of photographed images they will ultimately have printed, and adjust the appearance of each photographed image by retouching them themselves. Retouching is performed on a portion of the photographed images, such as five or ten images, selected from the images taken by the photographer. Retouching (processing) includes various types of image processing for adjusting the appearance of the photographed images.

[0016] In the retouching space, one or more retouching PCs 1 are prepared. Photographed images stored in a storage PC are transferred to the retouching PC 1 and displayed on a display 11 of the retouching PC 1. A retouching application 1A, which is an application used to retouch photographed images taken by a photographer, is installed in the retouching PC 1.

[0017] In this way, in a photography studio where a retouching PC 1 to which the present technology is applied is installed, users (the bride and groom) themselves retouch the images taken by the photographer. The users can use the functions of the retouching application 1A to retouch the images so that they look the way they like. The retouching application 1A may be installed on a terminal other than a PC, such as a tablet terminal, and used for retouching.

[0018] FIG. 2 is a diagram showing an example of providing a captured image.

[0019] The photographed images retouched by the retouching application 1A are printed by the photography studio's management company and later handed over to the user in the form of, for example, an album, as indicated by arrow #1.

[0020] Furthermore, as shown by arrow #2, the retouched photographed image is uploaded from the retouching PC 1 to the image management server 2 and managed in the image management server 2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the management company of the photography studio, or by a business entity different from the management company of the photography studio.

[0021] A user can view and download photographed images that have been taken of them as the subject, as shown by arrow #3, by operating their own mobile terminal 3 such as a smartphone and operating image management app 3A. Image management app 3A is an application for viewing photographed images managed in image management server 2. Image management app 3A is provided, for example, by a business entity that manages image management server 2.

[0022] The images taken in the photography studio are provided in the form of an album and, if necessary, in the form of data. The provision of the images in the form of data may be made possible for a fee separate from the fee for the album.

[0023] The provision of the photographed images in the form of data is performed so that not only the photographed images that have been retouched by the user but also the photographed images that have not been selected by the user as the images to be retouched are provided. As for the photographed images that have not been selected by the user as the images to be retouched, they are automatically retouched by the retouching application 1A without any user operation, as described later.

[0024] The user can also receive, in the form of data, photographed images that were not selected as targets for retouching. Instead of using the image management application 3A, a browser may be used to access a specific site and receive photographed images managed in the image management server 2. For example, by registering as a member of an image providing service provided by the administrator of the image management server 2, the user can receive photographed images managed in the image management server 2 using the image management application 3A or a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the captured images.

[0026] <<About the retouching app>> <File Management> FIG. 3 is a diagram showing an example of file management in the retouching PC 1. As shown in FIG.

[0027] As shown in the speech bubble in Figure 3, two folders, the all folder and the album folder, are prepared and linked to the user ID, which is the user's identification information. The all folder is a folder named "all," and the album folder is a folder named "album."

[0028] The captured image files transferred from the storage PC are stored in the all folder. The captured image files that were selected for retouching and have been retouched are stored in the album folder. The files stored in the album folder are the files used for printing.

[0029] When a user who has finished taking a photograph starts retouching, a user ID is input on the screen of the retouching application 1A. Based on the user ID, the captured image is read by the retouching application 1A, and processing is performed using the captured image. For example, the retouching application 1A reads all captured image files stored in an all folder linked to the user ID. After the captured images are read, the captured images to be retouched are selected.

[0030] <Screen display> Image selection screen FIG. 4 is a diagram showing an example of the image selection screen.

[0031] A message is displayed at the top of the image selection screen to guide the user to select five captured images. The number of selected captured images is set, for example, by a staff member of the photography studio before image selection. If the user applies for a plan to print five captured images and receive them as an album, five is set as the number of selected images.

[0032] A thumbnail display area 21 is formed on the right side of the image selection screen. The thumbnail display area 21 is an area in which thumbnail images of the captured images are displayed. For example, when 50 captured images have been captured, thumbnail images of each of the 50 captured images are displayed. Various operations using the screen of the retouching application 1A are performed using a mouse, a keyboard, etc.

[0033] A photographed image selected using a thumbnail image is enlarged and displayed in a display area 22 formed on the left side of the thumbnail display area 21. In the example of Fig. 4, the thumbnail image of the photographed image P1 displayed at the upper left end of the thumbnail display area 21 is selected, and the photographed image P1 is enlarged and displayed in the display area 22.

[0034] A confirm button 23 is displayed at the bottom right of the display area 22. The confirm button 23 is a button that is pressed when deciding on a photographed image displayed in the display area 22 as a subject for retouching. The user selects thumbnail images to sequentially enlarge and display each photographed image in the display area 22, and when the user finds a photographed image that the user likes, the user presses the confirm button 23 to select it as a subject for retouching.

[0035] Below the display area 22, a selected image display area 24 is formed in which a thumbnail image of a photographed image selected by the user is displayed.

[0036] When the selection of the five photographed images has been completed, the selection of the photographed images to be retouched is completed. After the selection of the photographed images to be retouched has been completed, retouching is performed on each of the photographed images selected as the retouched images.

[0037] Retouch screen FIG. 5 is a diagram showing an example of a retouch screen.

[0038] A display area 31 is formed at the upper center of the retouch screen. The display area 31 is an area in which the captured image currently being retouched is enlarged and displayed. A thumbnail image of the captured image selected by the user is displayed in a thumbnail area 32 formed on the left side of the display area 31. In the example of FIG. 5, thumbnail images of five captured images P1 to P5 are displayed. The captured image P1 selected using the thumbnail image is enlarged and displayed in the display area 31.

[0039] A horizontally long retouching selection area 33 is formed below the retouching screen. The buttons displayed in the retouching selection area 33 are used to select the type of retouching. Each time a type of retouching is selected using a button displayed in the retouching selection area 33, the selected type of retouching is reflected in the display in the display area 31. While viewing the display in the display area 31, the user uses the display in the retouching selection area 33 to select the type of retouching to be applied to the captured image currently being retouched.

[0040] FIG. 6 is an enlarged view of the display of the retouch selection area 33. As shown in FIG.

[0041] Retouching selection area 33 is formed from area 33-1, which is a display area for information used for adjusting the entire face and body, area 33-2, which is a display area for information used for adjusting the eyes and makeup, and area 33-3, which is a display area for information used for adjusting brightness.

[0042] Area 33-1 formed on the left side of retouch selection area 33 displays adjustment button 51 used to adjust the appearance of a smaller face, adjustment button 52 used to adjust the shape of the face, adjustment button 53 used to adjust the skin type, adjustment button 54 used to adjust the shorter chin, and adjustment button 55 used to adjust the style.

[0043] It is possible to select five different sizes for the small face effect. The small face effect "0" indicates that the size of the face is not changed and remains the original size. The user can adjust the size of their face in the captured image displayed in the display area 31 by selecting one of the sizes "1" to "4."

[0044] The user can select one of the following shapes for the face: "round," "egg," or "slim." By selecting one of the shapes, the user can adjust the shape of his or her face that appears in the photographed image displayed in the display area 31. By selecting "no retouching," the user can leave the shape of the face unchanged.

[0045] It is possible to select one of five levels for the skin texture. Skin texture "0" indicates that the skin texture is not changed and remains the original skin texture. By selecting one of the levels, the user can adjust the degree of smoothness of the user's skin texture in the captured image displayed in the display area 31.

[0046] It is possible to select three levels for shortening the chin. The value "0" for shortening the chin indicates no processing, i.e., the chin is left unchanged and the original chin is kept as it is. By selecting one of the levels, the user can adjust the length of their chin that appears in the photographed image displayed in the display area 31.

[0047] It is possible to select from three levels of style improvement. Style improvement "0" indicates no processing, i.e., the style is not changed and the original style is kept as it is. By selecting one of the levels, the user can adjust the style of the user that appears in the photographed image displayed in the display area 31.

[0048] In area 33-2 formed in the center of retouch selection area 33, adjustment button 56 used to adjust the size of the eyes, adjustment button 57 used to adjust the shape of the eyes, adjustment button 58 used to adjust the intensity of the makeup, adjustment button 59 used to adjust the bridge of the nose, and adjustment button 60 used to adjust the removal of facial hair are displayed.

[0049] It is possible to select from five different sizes for the eye size. Eye size "0" indicates no processing, i.e., the eye size is left unchanged at its original size. By selecting one of the sizes, the user can adjust the size of their own eyes that appear in the captured image displayed in the display area 31.

[0050] The user can select one of the following eye shapes: "round eyes," "downturned eyes," and "cat eyes." By selecting one of the shapes, the user can adjust the shape of their own eyes that will appear in the photographed image displayed in the display area 31. By selecting "no processing," it is also possible to leave the shape of the eyes unchanged.

[0051] The intensity of makeup can be selected from five levels. The level of makeup "0" indicates that the intensity of makeup is not changed and remains the original intensity. By selecting one of the levels, the user can adjust the intensity of their makeup in the captured image displayed in the display area 31.

[0052] It is possible to select ON / OFF for the nose bridge. By selecting ON or OFF, the user can select whether to emphasize the highlight part of the bridge of the nose that appears in the photographed image displayed in the display area 31, or to leave it in its original state.

[0053] It is possible to select three levels of beard erasure. "0" for beard erasure indicates that the beard is not changed and remains the original beard. By selecting one of the levels, the user can adjust the beard of the user appearing in the photographed image displayed in the display area 31. The beard erasure function is enabled when the person to be retouched is the groom.

[0054] An area 33-3 formed on the right side of the retouch selection area 33 displays an adjustment button 61 used to adjust the brightness of only the skin, an adjustment button 62 used to adjust the overall brightness, and an adjustment button 63 used to adjust the brightness of only the background.

[0055] Three levels of brightness can be selected for adjusting the skin brightness. A skin brightness of "0" indicates that the skin brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the brightness of their own skin that appears in the captured image displayed in the display area 31. The skin brightness adjustment function will be described in detail later.

[0056] The user can select one of three brightness levels to adjust the overall brightness. An overall brightness of "0" indicates that the overall brightness is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the overall brightness of the captured image displayed in the display area 31.

[0057] The brightness of the background can be adjusted by selecting one of three levels of brightness. A background brightness of "0" indicates that the brightness of the background is not changed and remains at the original brightness. By selecting one of the brightness levels, the user can adjust the brightness of the background in the captured image displayed in the display area 31.

[0058] Returning to the explanation of Fig. 5, a batch retouch button 34 used for batch retouching is displayed on the left side of the display area 31. Batch retouching is a function for selecting a combination of various retouching contents as described above all at once.

[0059] In the example of Fig. 5, one of three levels of collective retouching, each of which has a different combination of retouching levels, can be selected. By using the collective retouching function, the user can select the retouching content with a single operation, without having to select each retouching content using the display of the retouching selection area 33.

[0060] FIG. 7 is a diagram showing a display example when batch retouching is selected.

[0061] When the batch retouch button 34 is operated, a rounded rectangular pop-up image 71 is displayed superimposed on the retouch screen. The pop-up image 71, which is displayed approximately in the center of the retouch screen, displays circular buttons used to select the range of images to be subjected to batch retouching together with a message.

[0062] By operating the button displayed in the pop-up image 71, the user can select whether the contents of the batch retouching should be reflected only in the one image currently being retouched, or in all (five) images including the one image currently being retouched.

[0063] Returning to the explanation of FIG. 5, a Before-edit button 35 is provided on the right side of the display area 31. The Before-edit button 35 is a button that is operated when switching the display of the display area 31 to the state before retouching. While the Before-edit button 35 is pressed, the captured image in the state before retouching is displayed in the display area 31. By using the Before-edit button 35, the user can compare the state after retouching with the state before retouching and check whether the degree of retouching is too strong, for example.

[0064] Provided below the Before button 35 is a Person Selection button 36 used to select a person to be retouched. By using the Person Selection button 36, the user can select and retouch only the person appearing on the left side of the photographed image, only the person appearing on the right side, or both people appearing on the right side. Face recognition and the like are performed on the photographed image, and a person to be the subject of image processing for retouching is selected in response to a user operation.

[0065] Using these various functions, the user can select the content of retouching to be applied to the captured image displayed in the display area 31. In this way, retouching that can be performed using the retouching application 1A includes at least adjustment of the face and body as a whole, adjustment of the eyes and makeup, and adjustment of brightness.

[0066] The photographed image to be retouched is switched using the display in the thumbnail area 32, and retouching is performed on each of the photographed images P1 to P5 in sequence. The retouched files of the photographed images P1 to P5 are stored in the album folder.

[0067] FIG. 8 is a diagram showing a display example when the photographed image to be retouched is switched.

[0068] When the captured image to be retouched is switched, a speech bubble-shaped pop-up image 72 is displayed superimposed on the retouch screen. The pop-up image 72 is displayed at a position slightly below the center of the retouch screen, with the speech bubble originating from the display area 31 formed approximately above the center of the retouch screen. The pop-up image 72 displays a horizontally long rectangular button together with a message, which is used to select whether or not the retouching contents set on the previous photo (captured image) should be reflected in the switched captured image.

[0069] That is, when the photographed image to be retouched is switched, an image having a different shape from the pop-up image 71 that is displayed when the collective retouch button 34 is operated is displayed as the pop-up image 72.

[0070] The user can carry over the retouching details selected for the previous captured image to the next captured image by using the display of the pop-up image 72. The retouching details carrying over function will be described in detail later.

[0071] FIG. 9 is a diagram showing an example of application of retouching parameters.

[0072] 9, the retouching contents applied to each of the photographed images P1 to P5 by the user's operation are reflected in each unselected image stored in the all folder. An unselected image is a photographed image that was not selected as a retouching target on the image selection screen. A photographed image selected as a retouching target on the image selection screen becomes a selected image.

[0073] The retouching contents of which of the retouched photographed images P1 to P5 should be reflected in each unselected image is selected based on, for example, the state of the face appearing in the selected image and the state of the face appearing in the unselected image. In this case, each photographed image is analyzed, and the state of the face appearing in each photographed image is recognized. A selected image in which a face appears in a state similar to the state of the face appearing in each unselected image is selected, and the retouching contents applied to the selected selected image are reflected in the unselected image.

[0074] <<Configuration and operation of retouching PC1>> <Configuration of Retouching PC1> FIG. 10 is a block diagram showing an example of the configuration of the retouching PC 1. As shown in FIG.

[0075] A central processing unit (CPU) 101, a read only memory (ROM) 102, and a random access memory (RAM) 103 are interconnected by a bus 104.

[0076] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0077] The input unit 106 includes a keyboard, a mouse, and the like.

[0078] The output unit 107 is composed of a display 11 and the like.

[0079] The storage unit 108 is configured with a hard disk, a non-volatile memory, etc. The storage unit 108 stores programs such as the retouch application 1A executed by the CPU 101, and various information such as captured images.

[0080] The communication unit 109 is an interface to a network such as the Internet or a LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a captured image transmitted from the storage PC, in which the user about to start retouching appears as a subject. The communication unit 109 also communicates with the image management server 2, and transmits the retouched captured image to the image management server 2.

[0081] The drive 110 controls writing of data to the removable medium 111 and reading of data from the removable medium 111. Import of captured images from a PC for storage may be performed via the removable medium 111.

[0082] FIG. 11 is a block diagram showing an example of the functional configuration of the retouching PC 1. As shown in FIG.

[0083] 11, in the retouching PC 1, a retouching application 1A is executed to implement an image processing section 121. The image processing section 121 is made up of an image acquisition section 131, a retouching processing section 132, an output control section 133, and a display control section .

[0084] The image acquisition unit 131 accepts a user's operation on the image selection screen, and acquires the captured image selected by the user as the target for retouching from, for example, the storage unit 108. The captured image to be retouched acquired by the image acquisition unit 131 is supplied to the retouching processing unit 132 and the display control unit 134.

[0085] Furthermore, when retouching of each photographed image to be retouched is completed, the image acquisition unit 131 acquires an unselected image, for example, from the storage unit 108 , and outputs the image to the retouch processing unit 132 .

[0086] The retouching processor 132 performs face recognition to recognize a face appearing in a captured image to be retouched. The retouching processor 132 analyzes the features of each part of the recognized face and performs various image processing related to retouching in response to a user's operation on the retouching screen. The analysis of the features of each part may be performed using an inference model generated by machine learning.

[0087] For example, when an operation for adjusting the size or shape of the eyes is performed, the retouching processor 132 performs image processing such as shifting the position of the eye outline in accordance with the content selected by the user. When an operation for adjusting the intensity of makeup is performed, the retouching processor 132 performs image processing such as compositing an image of a blush of a predetermined color on the cheeks or compositing an image of a lipstick of a predetermined color on the lips.

[0088] When an operation related to adjusting the skin quality is performed, the retouching processor 132 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the face size (feeling of a small face) or face shape is performed, the retouching processor 132 performs image processing such as shifting the position of the face outline according to the content selected by the user.

[0089] Information representing the results of the image processing by the retouching processor 132 is supplied to the display controller 134 and used to display the retouched screen.

[0090] Furthermore, when retouching of each photographed image to be retouched is completed, the retouching processing unit 132 analyzes each photographed image including an unselected image, and reflects the retouching content applied to the retouched photographed image to the unselected image. The photographed images retouched by the retouching processing unit 132 are supplied to the output control unit 133.

[0091] The output control unit 133 saves the captured image that has been retouched in response to a user operation in an album folder.

[0092] Furthermore, the output control unit 133 outputs the retouched photographed images (selected image and unselected image) to the communication unit 109, and causes it to be transmitted to the image management server 2.

[0093] The display control unit 134 controls the display of various screens of the retouching application 1A, such as the image selection screen and the retouching screen.

[0094] <Operation of retouching PC1> The retouching process performed by the retouching application 1A will be described with reference to the flowchart of FIG.

[0095] In step S1, the display control unit 134 of the image processing unit 121 displays the image selection screen (FIG. 4).

[0096] In step S2, the image acquisition unit 131 accepts the selection of a photographed image to be retouched in response to a user's operation on the image selection screen. The image acquisition unit 131 acquires the photographed image to be retouched.

[0097] In step S3, the image acquisition unit 131 determines whether or not the selection of the captured images to be retouched has been completed. If it is determined in step S3 that the selection of the captured images to be retouched has not been completed, the process returns to step S1, and image selection continues.

[0098] For example, when five photographed images have been selected, it is determined in step S3 that the selection of photographed images to be retouched has ended.

[0099] In step S4, the display control unit 134 displays a retouch screen (FIG. 5). The user retouches the captured image using the retouch screen.

[0100] In step S5, the retouching processor 132 performs image processing in response to a user's operation, and retouches the photographed image selected as the target of retouching.

[0101] In step S6, the retouching processor 132 determines whether or not the retouching is complete. If it is determined in step S6 that the retouching is not complete, the process returns to step S4, and retouching continues for each captured image selected as the retouching target.

[0102] For example, when all the photographed images selected as the retouching targets have been retouched, it is determined in step S6 that the retouching is to be ended.

[0103] In step S7, the retouching processor 132 applies the same retouching content as that applied to the selected image to each of the unselected images.

[0104] In step S8, the output control unit 133 stores the retouched unselected image.

[0105] In step S9, the communication unit 109 transmits the retouched selected images and the unselected images to the image management server 2. Thereafter, the processing of the retouching application 1A ends.

[0106] Through the above process, the user can adjust the appearance of the photographed image in which he or she is the subject, and can receive an album of the photographed images that are in the user's preferred appearance.

[0107] Furthermore, by using the mobile terminal 3, the user can receive, in the form of data, photographed images whose appearance has been manually adjusted and photographed images (unselected images) whose appearance has been automatically retouched in the same manner as the manual adjustments, which are managed in the image management server 2.

[0108] <<Skin brightness adjustment function>> As described with reference to Fig. 5, it is possible to adjust the brightness of a photographed image to be retouched. Here, a function for adjusting the brightness of only the skin, which is performed using adjustment button 61 in area 33-3, will be described.

[0109] <Configuration of image processing unit 121> 13 is a block diagram showing an example of the configuration of image processing unit 121 that realizes a function for adjusting the brightness of only the skin. In FIG. 13, only a part of the configuration of image processing unit 121 is shown.

[0110] In addition to the components described above, the image processing unit 121 is provided with a gender determination unit 135. The retouching processing unit 132 realizes a face recognition unit 151, a skin region extraction unit 152, and a brightness adjustment unit 153. The captured image to be retouched is input to the gender determination unit 135 and each unit of the retouching processing unit 132.

[0111] The gender determination unit 135 determines the gender of a person appearing in a photographed image. The determination by the gender determination unit 135 is performed, for example, using an inference model that receives the photographed image as an input and outputs the gender of each person appearing in the photographed image. In this case, an inference model for gender determination generated by machine learning is prepared in advance in the gender determination unit 135. The gender determination may be performed based on the characteristics of a person obtained by analyzing the photographed image to be retouched.

[0112] The determination result by the gender determination unit 135 is supplied to the brightness adjustment unit 153 of the retouching processing unit 132. For example, if only the bride appears in the photographed image to be retouched, information indicating that the person appearing in the photographed image is a woman is supplied as the determination result. Also, if only the groom appears in the photographed image to be retouched, information indicating that the person appearing in the photographed image is a man is supplied as the determination result. If the bride and groom appear in the photographed image to be retouched, information indicating that the people appearing in the photographed image are a woman and a man, respectively, is supplied as the determination result.

[0113] The face recognition unit 151 of the retouch processing unit 132 recognizes the face of a person appearing in a captured image and extracts a face region. The extraction of the face region by the face recognition unit 151 may be performed using an inference model generated by machine learning, or may be performed based on features such as contour points of each part obtained by analyzing the captured image. Information on the face region extracted by the face recognition unit 151 is supplied to the skin region extraction unit 152 and the brightness adjustment unit 153.

[0114] The skin region extraction unit 152 extracts the skin region of a person appearing in a photographed image. For example, an inference model is prepared in the skin region extraction unit 152, which receives a photographed image as an input and outputs information on the skin region of a person appearing in the photographed image. The skin region extraction unit 152 extracts a skin region other than the face region based on the skin region extracted using the inference model and the face region extracted by the face recognition unit 151.

[0115] FIG. 14 is a diagram showing an example of the extraction of a skin region by skin region extraction section 152. As shown in FIG.

[0116] For example, a case will be described in which a photographed image P11 shown in the upper left of Fig. 14 is selected as a target for retouching. A person H1, who is a bride in a wedding dress, is photographed in the photographed image P11. The person H1's chest and the area from the shoulder to the arm (upper arm) are exposed.

[0117] When such a photographed image P11 is to be retouched, the face recognition section 151 extracts a face area as indicated by the tip of an arrow #11.

[0118] Skin region extraction unit 152 inputs photographed image P11 to skin region extraction model M as indicated by arrow #12, and extracts a skin region including a face region as indicated by arrow #13 based on the output of skin region extraction model M. In photographed image P11, the face region and regions from shoulders to arms are captured as exposed skin regions of person H1.

[0119] Furthermore, as indicated by the tips of arrows #14 and #15, skin region extraction unit 152 extracts a skin region other than the face region, as indicated by the tip of arrow #16, by removing the face region extracted by face recognition unit 151 from the skin region extracted using skin region extraction model M. Information on the skin region other than the face region thus extracted is supplied to brightness adjustment unit 153.

[0120] 13 identifies the face region of a person appearing in a captured image to be retouched based on information supplied from face recognition unit 151, and identifies the skin region other than the face region based on information supplied from skin region extraction unit 152. Brightness adjustment unit 153 adjusts the face region and the skin region other than the face region using different retouching parameters. The retouching parameters are parameters that define the amount of brightness adjustment.

[0121] For example, the brightness adjustment unit 153 selects retouching parameters such that the adjustment amount for the face region is greater than the adjustment amount for the skin region other than the face region in response to a user's operation using the adjustment button 61, and adjusts the brightness of each region. That is, a retouching parameter for the brightness of the face region that is brighter than the retouching parameter for the brightness of the skin region other than the face region is used in image processing. As described with reference to Fig. 5, one of levels 1 or 2 is selected as the brightness level of only the skin using the adjustment button 61.

[0122] As a result, regardless of whether the user selects brightness level 1 or 2, image processing is performed to make the face area brighter than the skin area other than the face area, as shown in Fig. 15A. For example, if the brightness of the face area and the brightness of the skin area other than the face area are the same, the face area after the brightness adjustment will be brighter than the skin area other than the face area.

[0123] When the person in the photographed image is a woman (the bride), makeup is usually applied to the face, so brightening the facial area does not look unnatural. By applying image processing to brighten the facial area more than the skin area other than the facial area, it is possible to prevent the skin area other than the facial area from becoming too bright and looking unnatural.

[0124] Such image processing for making the face region brighter than the skin region other than the face region may be performed based on the gender determination result by gender determination section 135.

[0125] For example, image processing that brightens the face area more than the skin area other than the face area is applied only to women who appear in the photographed image. In this case, image processing is applied to men who appear in the photographed image using retouching parameters that brighten the face area and the skin area other than the face area, such as the hands, to the same extent. This makes it possible to prevent only the face area of ​​the man from being too bright, resulting in an unnatural appearance.

[0126] Even if the user selects the same brightness level, retouching parameters may be selected that make the female skin region (including the face region) brighter than the male skin region, as shown in Fig. 15B. In this way, different retouching parameters may be used to adjust the brightness of the female skin region and the male skin region.

[0127] <Retouch screen display> FIG. 16 is a diagram showing a display example of area 33-3 on the retouch screen.

[0128] After adjusting the brightness of only the skin using adjustment button 61 as described above, the user can further adjust the overall brightness using adjustment button 62, or adjust the brightness of the background using adjustment button 63.

[0129] In contrast, after adjusting the overall brightness using adjustment button 62, adjustment of the background brightness using adjustment button 63 is restricted, as shown in Fig. 16A. In the example on the right side of Fig. 16A, in response to the selection of the overall brightness level as "1", adjustment button 63 is displayed grayed out so that the selection of the background brightness level is not possible.

[0130] Furthermore, after adjusting the background brightness using adjustment button 63, adjustment of the overall brightness using adjustment button 62 is restricted, as shown in Fig. 16B. In the example on the right side of Fig. 16B, in response to the background brightness level being selected as "2", adjustment button 62 is displayed in a grayed-out state so that the overall brightness level cannot be selected.

[0131] That is, the adjustment of the overall brightness and the adjustment of the background brightness are mutually exclusive, and only one of them can be adjusted. After one of the adjustment of the overall brightness and the adjustment of the background brightness has been performed, the adjustment of the other is restricted. The adjustment of the overall brightness and the adjustment of the background brightness are also performed by, for example, the brightness adjustment unit 153.

[0132] For example, if it were possible to brighten the background after brightening the entire image, image processing that adjusts the brightness of the background would be applied in a superimposed manner, increasing the possibility of whiteout. By making it possible to adjust only one of the overall brightness adjustment or the background brightness adjustment, it is possible to prevent such whiteout from occurring.

[0133] It should be noted that even after adjusting the overall brightness using adjustment button 62, or adjusting the background brightness using adjustment button 63, the display of adjustment button 61 remains (does not change) so that the brightness level of only the skin remains selectable.

[0134] Although the adjustment buttons 61 to 63 are displayed as information used to select the brightness adjustment level, other selection information such as a slide bar may also be displayed.

[0135] << Transfer of retouching content >> As described with reference to Fig. 8, when a photographed image to be retouched is switched, the retouching contents selected for the photographed image before the switch can be carried over to the photographed image after the switch. Here, the retouching contents carrying over function will be described. It is assumed that the retouching contents set for the previous photographed image are selected to be reflected in the photographed image after the switch using the display of the pop-up image 72.

[0136] <Transfer of retouching content> 17 and 18 are diagrams showing an example of the transfer of retouching contents.

[0137] As shown by arrow #21 in Fig. 17, it is assumed that when the captured image P21 is the first image to be retouched, the following retouching parameters are set for the captured image P21 by user operations when the captured image P21 is the first image to be retouched: "Small face feeling" is "3," "Facial shape" is "egg," "Skin quality" is "2," ... The appearance of the retouched captured image P21 shown in the lower part of Fig. 17 is different from the appearance of the captured image P21 before retouching shown in the upper part.

[0138] When the retouching of the photographed image P21 is completed and the retouching target is switched from the photographed image P21 to the photographed image P22, the retouching parameters of the retouched photographed image P1 are inherited as the default retouching parameters of the photographed image P22, which is the second photographed image to be retouched, as shown at the end of the arrow #22 in Fig. 18. The photographed image P22 is displayed in the display area 31 in place of the photographed image P21, with the same retouching applied to it as the retouched photographed image P21. The display of the retouching selection area 33 also changes depending on the retouching applied to the photographed image P22.

[0139] The user can start retouching the photographed image P22 by using the photographed image P22, which has been retouched in the same manner as the photographed image P21, as a reference image. In the example of Fig. 18, the image has been adjusted to the contents represented by the retouching parameters: "small face feeling" is "4", "face shape" is "slim", "skin quality" is "1", ...

[0140] Similarly, when retouching of the photographed image P22 is completed and a third photographed image to be retouched is selected, the retouching parameters of the photographed image P22 are inherited as the default retouching parameters for the third photographed image to be retouched, as shown by arrow #24.

[0141] In this way, when the captured image to be retouched is switched, the retouching contents of the captured image to be retouched immediately before are carried over to the appearance of the next captured image. Each time the captured image to be retouched is switched, the carrying over of the appearance of the captured images is repeated. This allows the user to efficiently retouch each captured image.

[0142] When the captured image after switching is an unadjusted image that has not yet been retouched, the retouching details may be carried over as described above.

[0143] <About what can be transferred> Retouching can be performed for each person by selecting the person to be retouched. The retouching content is passed on based on the gender determination result by the gender determination unit 135 so that the retouching content for the bride is passed on to the bride's image, and the retouching content for the groom is passed on to the groom's image.

[0144] Switching from an image with the bride and groom to another image with the bride and groom FIG. 19 is a diagram showing a first example of the takeover of retouching contents.

[0145] FIG. 19 shows an example of transfer of retouching content when retouching of a photographed image P31, which shows a bride H1 and a groom H2, is completed and the retouching target is switched from the photographed image P31 to a photographed image P32, which also shows the persons H1 and H2.

[0146] In this case, the retouching content applied to the person H1 when the photographed image P31 was selected as the retouching target is carried over to the appearance of the person H1 in the photographed image P32, as indicated by arrow #31. Based on the determination result by the gender determination unit 135, the retouching content applied to the female person (person H1) in the photographed image P31 is carried over to the appearance of the same female person (person H1) in the photographed image P32.

[0147] Furthermore, the retouching content applied to person H2 when photographed image P31 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P32, as indicated by arrow #32. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P31 is carried over to the appearance of the same male person (person H2) in photographed image P32.

[0148] Switching from an image with the groom and bride to an image with just the groom or just the bride FIG. 20 is a diagram showing a second example of the transfer of retouching contents.

[0149] FIG. 20 shows an example of transfer of retouching contents when retouching of a photographed image P31, which contains a bride H1 and a groom H2, is completed and the retouching target is switched from the photographed image P31 to a photographed image P33, which contains only the person H1, or to a photographed image P34, which contains only the person H2.

[0150] When the retouching target is switched to the photographed image P33 at the lower right, the retouching content applied to the person H1 when the photographed image P31 was selected as the retouching target is carried over to the appearance of the person H1 in the photographed image P33, as indicated by arrow #33. Based on the determination result by the gender determination unit 135, the retouching content applied to the female person (person H1) in the photographed image P31 is carried over to the appearance of the same female person (person H1) in the photographed image P33.

[0151] In this case, the retouching content applied to the person H2 when the photographed image P31 was selected as the retouching target will not be carried over.

[0152] When the retouching target is switched to the photographed image P34 at the lower left, the retouching content applied to the person H2 when the photographed image P31 was selected as the retouching target is carried over to the appearance of the person H2 in the photographed image P34, as indicated by arrow #34. Based on the determination result by the gender determination unit 135, the retouching content applied to the male person (person H2) in the photographed image P31 is carried over to the appearance of the same male person (person H2) in the photographed image P34.

[0153] In this case, the retouching content applied to the person H1 when the photographed image P31 was selected as the retouching target will not be carried over.

[0154] Switching from an image with the groom or the bride to an image with the groom and the bride FIG. 21 is a diagram showing a third example of the transfer of retouching contents.

[0155] The right side of Figure 21 shows an example of the transfer of retouching content when retouching of a photographed image P33, which contains only the bride, person H1, is completed and the retouching target is switched from the photographed image P33 to a photographed image P31, which contains person H1 and the groom, person H2.

[0156] In this case, the retouching content applied to person H1 when photographed image P33 was selected as the retouching target is carried over to the appearance of person H1 in photographed image P31, as indicated by arrow #35. Based on the determination result by gender determination unit 135, the retouching content applied to the female person (person H1) in photographed image P33 is carried over to the appearance of the same female person (person H1) in photographed image P31.

[0157] The left side of Figure 21 shows an example of the transfer of retouching content when retouching of a photographed image P34, which contains only the groom H2, is completed and the retouching target is switched from the photographed image P34 to a photographed image P31, which contains the person H2 and the bride H1.

[0158] In this case, the retouching content applied to person H2 when photographed image P34 was selected as the retouching target is carried over to the appearance of person H2 in photographed image P31, as indicated by arrow #36. Based on the determination result by gender determination unit 135, the retouching content applied to the male person (person H2) in photographed image P34 is carried over to the appearance of the same male person (person H2) in photographed image P31.

[0159] This makes it possible to prevent problems such as retouching intended for the bride being carried over to the groom's photo.

[0160] Example of letting the user choose whether or not to carry over retouching content When the gender composition of people before and after switching of the photographed image to be retouched is different, information as described with reference to Fig. 8 may be presented. The user is prompted to select whether or not to carry over the retouching contents.

[0161] For example, as shown on the right side of Fig. 21, when the gender composition of people in a photographed image changes, such as when the retouching target is switched from a photographed image P33 that contains only women to a photographed image P31 that contains both women and men, the composition changes significantly and it may not be appropriate to carry over the retouching content. If inappropriate retouching content is carried over, it becomes cumbersome to correct it, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to save the effort of correction.

[0162] -Handover based on person identification results Although the transfer of retouching contents is based on the result of gender determination in the above embodiment, it may be based on the result of person identification. When the retouching contents intended for a certain person are transferred to a photograph of the same person, the retouching contents intended for the bride are transferred to the photograph of the same person, and the retouching contents intended for the groom are transferred to the photograph of the same person, the groom.

[0163] Fig. 22 is a block diagram showing another example of the configuration of the image processing unit 121. The configuration shown in Fig. 22 differs from the configuration of the image processing unit 121 described above in that a person identification unit 136 is provided.

[0164] The person identification unit 136 identifies each person appearing in the captured image. The person identification unit 136 identifies the person using, for example, an inference model that receives the captured image as input and outputs the ID of the person appearing in the captured image. In this case, an inference model for person identification generated by machine learning is prepared in advance in the person identification unit 136. The person may be identified based on the characteristics of the person obtained by analyzing the captured image to be retouched.

[0165] The identification result by the person identification unit 136 is supplied to the retouching processing unit 132. For example, if only the bride appears in the photographed image to be retouched, the person ID assigned to the bride appearing in the photographed image is supplied as the identification result. Also, if only the groom appears in the photographed image to be retouched, the person ID assigned to the groom appearing in the photographed image is supplied as the identification result. If the bride and groom appear in the photographed image to be retouched, the person ID assigned to the bride and the person ID assigned to the groom are supplied as the identification result.

[0166] In the retouching processing section 132, based on the identification result by the person identification section 136, retouching is performed so that the contents of retouching performed on a person appearing in a photographed image before switching of the retouching target are carried over to the appearance of the same person appearing in the photographed image after switching.

[0167] This also makes it possible to prevent problems such as retouching intended for the bride being carried over to the groom's image.

[0168] When the composition of a person is different before and after switching of the photographed image to be retouched, information such as that described with reference to Fig. 8 may be presented. The user is prompted to select whether or not to carry over the retouching content.

[0169] For example, as shown on the right side of Fig. 21, when the configuration of people in a captured image changes, such as when the retouching target is switched from a captured image P33 containing only person H1 to a captured image P31 containing people H1 and H2, the composition changes significantly and it may not be appropriate to carry over the retouching content. If inappropriate retouching content is carried over, correcting it becomes cumbersome, but by making it possible to select not to carry over the retouching content as appropriate, it is possible to save the effort of correction.

[0170] <<Modifications>> Examples of retouching locations Although retouching is performed using a PC installed in a photography studio, retouching may also be performed using other devices, such as a mobile terminal 3 or a PC installed in the user's home.

[0171] When retouching using the mobile terminal 3 is possible, a user who has finished shooting in a photography studio operates the mobile terminal 3 to start up the image management application 3A and perform retouching on the screen of the image management application 3A. The image management application 3A has the same functions as the retouching application 1A.

[0172] The above-mentioned screen is displayed on the display of the mobile terminal 3 by the image management application 3A, and the photographed images to be retouched are selected and retouched from among the photographed images uploaded to the image management server 2. The photographed images that have been retouched using the functions of the image management application 3A are printed and provided to the user in the form of an album, or the retouched photographed images are provided to the user as they are in the form of data.

[0173] This allows the user to carry out retouching in various locations, such as at home, and allows the user to carry out retouching to the best of their ability without worrying about time.

[0174] About the Program The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0175] The program to be installed is provided by being recorded on removable medium 111 shown in FIG. 10, which is made up of an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 102 or storage unit 108.

[0176] The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0177] The effects described herein are merely examples and are not limiting, and other effects may also be present.

[0178] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0179] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0180] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0181] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices. [Explanation of symbols]

[0182] 1 Retouching PC, 2 Image management server, 3 Mobile terminal, 121 Image processing section, 131 Image acquisition section, 132 Retouching processing section, 133 Output control section, 134 Display control section, 135 Gender determination section, 136 Person identification section, 151 Face recognition section, 152 Skin region extraction section, 153 Brightness adjustment section

Claims

1. A step of displaying a processing screen including first selection information used to select an adjustment level for skin brightness, second selection information used to select an adjustment level for the brightness of the entire captured image, third selection information used to select an adjustment level for the brightness of the person's background, and person selection information used to select a person to be processed; detecting a face region of a person appearing in a photographed image and a skin region other than the face region as different skin regions; adjusting the brightness of the face area and the brightness of a skin area other than the face area of ​​a person selected using the person selection information in response to a user operation on the first selection information, based on a retouching parameter corresponding to a selected adjustment level; adjusting brightness of the entire captured image based on a retouching parameter corresponding to a selected adjustment level in response to a user operation on the second selection information; adjusting the brightness of the background of the person appearing in the photographed image based on a retouching parameter corresponding to the selected adjustment level in response to a user's operation on the third selection information; A program that causes a computer to execute a process including the steps of:

2. The brightness of the face region and the brightness of the skin region other than the face region are adjusted so that the face region is brighter than the skin region other than the face region. The program according to claim 1.

3. The brightness of the face area and the brightness of the skin area other than the face area are adjusted in response to a single operation of selecting an adjustment level of the skin brightness. The program according to claim 1.

4. and a step of switching display of selection information used for the other selection when one of the selection using the second selection information and the selection using the third selection information is made so that the other selection is not possible. The program according to claim 1.

5. identifying people in each captured image using a model generated by machine learning; adjusting brightness of the face area and brightness of a skin area other than the face area of ​​a person appearing in a first captured image selected by a user as an adjustment target from among a plurality of captured images; a step of reflecting the adjustment content of the first photographed image to the skin brightness of a specific person appearing in the first photographed image in the skin brightness of a person identical to the specific person appearing in the second photographed image when a user performs an operation to reflect the adjustment content of the first photographed image to the skin brightness of a person appearing in the second photographed image in the same manner as the specific person; The program according to claim 1 .

6. An image processing method performed by an image processing device, comprising: displaying a processing screen including first selection information used to select an adjustment level for skin brightness, second selection information used to select an adjustment level for the brightness of the entire captured image, third selection information used to select an adjustment level for the brightness of the background of a person, and person selection information used to select a person to be processed; detecting a face region of a person appearing in a photographed image and a skin region other than the face region as different skin regions; adjusting the brightness of the face area and the brightness of a skin area other than the face area of ​​a person selected using the person selection information in response to a user operation on the first selection information, based on a retouching parameter corresponding to a selected adjustment level; adjusting brightness of the entire captured image based on a retouching parameter corresponding to a selected adjustment level in response to a user operation on the second selection information; adjusting the brightness of the background of the person appearing in the photographed image based on a retouching parameter corresponding to the selected adjustment level in response to a user's operation on the third selection information; An image processing method comprising:

7. A display control unit that displays a processing screen including first selection information used to select an adjustment level for the brightness of the skin, second selection information used to select an adjustment level for the brightness of the entire captured image, third selection information used to select an adjustment level for the brightness of the background of a person, and person selection information used to select a person to be processed; a detection unit that detects a face area of ​​a person appearing in a photographed image and a skin area other than the face area as different skin areas; an adjustment unit that adjusts, in response to a user's operation on the first selection information, a brightness of the face region of a person selected using the person selection information and a brightness of a skin region other than the face region based on a retouching parameter corresponding to a selected adjustment level, respectively, in response to a user's operation on the second selection information, adjusts a brightness of the entire captured image based on a retouching parameter corresponding to the selected adjustment level, and in response to a user's operation on the third selection information, adjusts a brightness of a background of a person appearing in the captured image based on a retouching parameter corresponding to the selected adjustment level; An image processing device comprising:

Images