Program, information processing method, and information processing apparatus

Abstract

To perform image processing appropriately in accordance with the type of clothes of a person appearing in a captured image.SOLUTION: An information processing apparatus determines the type of clothes of a person to be retouched who appears in a captured image, and switches image processing to be applied to a subject person in accordance with the type of clothes of the subject person. This technology is applicable to a computer installed in a photo studio.SELECTED DRAWING: Figure 25

Description

Technical Field

[0001] The present technology particularly relates to a program, an information processing method, and an information processing apparatus that can perform appropriate image processing according to the type of clothing of a person shown in a photographed image.

Background Art

[0002] As a service for commemorative photography of events such as coming-of-age ceremonies and weddings, there is a service that allows users to retouch images taken by a photographer at a photography studio on their own (Patent Document 1). After the shooting, the user will perform retouching using a PC or the like prepared in another space of the photography studio.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Particularly, wedding commemorative photography is carried out with variously dressed people as subjects. For example, the bride-to-be woman wears a white kimono, a colored kimono, a wedding dress, etc., and the groom-to-be man wears a tuxedo, a haori and hakama, etc. From the perspective of how they appear in the image, the same processing may not be suitable for all images.

[0005] The present technology has been made in view of such a situation, and enables appropriate image processing to be performed according to the type of clothing of a person shown in a photographed image.

Means for Solving the Problems

[0006] A program according to one aspect of the present technology causes a computer to execute a process of determining the type of clothing of a person shown in a captured image to be retouched and switching the image processing performed on the subject person according to the type of clothing of the subject person.

[0007] In one aspect of the present technology, the type of clothing of a person shown in a captured image to be retouched is determined, and the image processing performed on the subject person is switched according to the type of clothing of the subject person.

Advantages of the Invention

[0008] According to the present technology, appropriate image processing can be performed according to the type of clothing of a person shown in a captured image.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Mode for Carrying Out the Invention

[0010] <<Workflow of Operations in a Photography Studio>> FIG. 1 is a diagram showing the workflow of operations in a photography studio.

[0011] This technology is used, for example, in a photography studio that conducts commemorative wedding photography. This technology is also applicable when taking commemorative photos of various events such as school entrance / graduation commemorative photos, coming-of-age ceremony commemorative photos, and Shichigosan commemorative photos.

[0012] As shown on the left side of FIG. 1, photography in a photography studio is performed by a cameraman. In the example of FIG. 1, the bride and groom standing in front of a background screen are being photographed as the subjects. The groom is in a tuxedo, and the bride is in a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photography by the cameraman is arranged. Photography of the bride and groom, etc. may also be performed outside the photography studio.

[0013] Shooting by a cameraman is carried out by changing the composition and shooting a large number of still images (photos), such as 50 or 100 images. The composition of the still images is determined by factors such as the poses of the bridegroom and bride, the positional relationship between the bridegroom, bride, and cameraman, and camera parameters such as the focal length. The captured image, which is a still image taken by the cameraman, is transferred to a PC (not shown) connected to the camera and saved.

[0014] After the shooting is completed, the bridegroom and bride move to the retouch space prepared in the shooting studio and use the retouch PC1 shown on the right side of FIG. 1 to check the captured images taken by the cameraman.

[0015] Also, as users of the retouch PC1, the bridegroom and bride select, for example, the number of captured images to be finally printed and adjust the appearance of each captured image by retouching themselves. Retouching is performed on a part of the captured images selected from the captured images taken by the cameraman. For example, a predetermined number of captured images, such as 5 or 10, corresponding to the plan applied for by the user are selected as the images to be retouched. Retouching (processing) includes various image processes for adjusting the appearance of the captured images. A screen used for selecting the number of images to be retouched may be displayed in front of the image selection screen. Any number can be selected as the image to be retouched.

[0016] One or more retouch PCs1 are prepared in the retouch space. The captured images saved in the storage PC are transferred to the retouch PC1 and displayed on the display 11 of the retouch PC1. An application for retouching the captured images taken by the cameraman, namely, the retouch app 1A, is installed on the retouch PC1.

[0017] In this way, in the photography studio where the retouching PC1 applying the present technology is installed, the retouching of the photographed image taken by the cameraman is performed by the users themselves (the bridegroom and the bride). The users can retouch the photographed image to their preferred appearance by using the functions of the retouching app 1A. The retouching app 1A may be installed on a terminal different from a PC, such as a tablet terminal, and used for retouching.

[0018] Figure 2 is a diagram showing an example of providing a photographed image.

[0019] The retouched photographed image by the retouching app 1A is printed by the operating company of the photography studio and delivered to the users at a later date in the form of, for example, an album as indicated by the arrow #1.

[0020] Also, the retouched photographed image is uploaded from the retouching PC1 to the image management server 2 and managed in the image management server 2 as indicated by the arrow #2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the operating company of the photography studio or a business operator different from the operating company of the photography studio.

[0021] The users can operate their own mobile terminal 3 such as a smartphone and operate the image management app 3A to view or download the photographed image in which they are the subjects, as indicated by the arrow #3. The image management app 3A is an application for viewing and the like of the photographed images managed in the image management server 2. The image management app 3A is provided, for example, by the business operator managing the image management server 2.

[0022] The provision of the photographed images taken in the photography studio is performed in the form of an album and, as appropriate, in the form of data. The provision of the photographed images in data form may be made possible at a separate charge from the charge for the album.

[0023] The provision of the photographed images in the data is performed so as to provide not only the photographed images retouched by the user but also the photographed images not selected by the user as retouch targets. For the photographed images not selected by the user as retouch targets, as will be described later, retouching is automatically performed by the retouch app 1A without depending on the user's operation.

[0024] The user can also receive, in the form of data, the photographed images not selected as retouch targets. Instead of using the image management app 3A, the user may access a predetermined site using a browser and receive the photographed images managed in the image management server 2. For example, by registering as a member of the image providing service provided by the administrator of the image management server 2, the user can receive the photographed images managed in the image management server 2 using the image management app 3A or using a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the photographed images. When the provision of the photographed images is performed in data, together with the photographed images which are still images, data of moving images that capture the state of shooting in the shooting studio may be provided.

[0026] <<Regarding the retouch app>> <Screen display> · Retouch screen FIG. 3 is a diagram showing an example of the retouch screen.

[0027] Above the center of the retouch screen, which is a screen used for retouching photographed images, a display area 31 is formed. The display area 31 is an area where one photographed image to be retouched is enlarged and displayed. In the thumbnail area 32 formed on the left side of the display area 31, thumbnail images of the photographed images selected by the user are displayed. In the example of FIG. 3, thumbnail images of five photographed images P1 to P5 are displayed. For example, the photographed image P1 selected using the thumbnail image is enlarged and displayed in the display area 31.

[0028] Below the retouch screen, a horizontally long retouch selection area 33 is formed. Using the buttons displayed in the retouch selection area 33, the content of retouching is selected. Each time the content of retouching is selected using the buttons displayed in the retouch selection area 33, the selected content is reflected in the display of the display area 31. The user will select the content of the retouching to be applied to the photographed image to be retouched using the display of the retouch selection area 33 while looking at the display of the display area 31.

[0029] FIG. 4 is a diagram showing an enlarged view of the display of the retouch selection area 33.

[0030] The retouch selection area 33 is formed from an area 33-1 which is a display area for information used for adjusting the whole face and body, an area 33-2 which is a display area for information used for adjusting the eyes and makeup, and an area 33-3 which is a display area for information used for adjusting brightness.

[0031] In the area 33-1 formed on the left side of the retouch selection area 33, an adjustment button 51 used for adjusting the small face feeling, an adjustment button 52 used for adjusting the shape of the face, an adjustment button 53 used for adjusting the skin texture, an adjustment button 54 used for adjusting the thinness of the arms, and an adjustment button 55 used for adjusting the style (styling up) are displayed.

[0032] The adjustment button 51 is provided with five levels (degrees of adjustment) for a small face effect. "0" for the small face effect indicates leaving the face size unchanged and keeping it at the original size. Also, "1" for the small face effect indicates reducing the face size by one level based on the parameter corresponding to "1". By selecting any level from "1" to "4", the user can adjust the size of their own face shown in the captured image displayed in the display area 31.

[0033] The adjustment button 52 is provided with "round", "egg-shaped", and "slim" for the face shape. By selecting any shape, the user can adjust the shape of their own face shown in the captured image displayed in the display area 31. It is also possible to leave the face shape unchanged by selecting "no retouch".

[0034] The adjustment button 53 is provided with five levels for the skin texture. "0" for the skin texture indicates leaving the skin texture unchanged and keeping it at the original texture. Also, "1" for the skin texture indicates raising the skin texture level by one level based on the parameter corresponding to "1". By selecting any level from "1" to "4", the user can adjust the smoothness degree of their own skin texture shown in the captured image displayed in the display area 31.

[0035] The adjustment button 54 is provided with three levels for the thinness of the arms. "0" for the thinness of the arms indicates no processing, that is, leaving the arm thinness unchanged and keeping it at the original thinness. Also, "weak" for the thinness of the arms indicates performing processing based on the parameter corresponding to "weak" and making the arm one level thinner. "Strong" for the thinness of the arms indicates performing processing based on the parameter corresponding to "strong" and making the arm one level thinner than in the "weak" case. By selecting any level, the user can adjust the thinness of their own arms shown in the captured image displayed in the display area 31. Details of the arm retouch, which is a function for adjusting the arm thinness, will be described later.

[0036] As a styling option, three levels are provided. "0" for styling means no processing, that is, keeping the original style without changing the style. Also, "weak" for styling means performing processing such as making the legs longer based on the parameters corresponding to "weak", and improving the overall balance by one level. The user can adjust their own style shown in the captured image displayed in the display area 31 by selecting any of the levels.

[0037] In the area 33-2 formed at the center of the retouch selection area 33, adjustment buttons 56 used for adjusting the eye size, adjustment buttons 57 used for adjusting the eye shape, adjustment buttons 58 used for adjusting the makeup intensity, adjustment buttons 59 used for adjusting the nasolabial fold, and adjustment buttons 60 used for adjusting beard removal are displayed.

[0038] As for the eye size, five levels are provided. "0" for the eye size means no processing, that is, keeping the original size without changing the eye size. Also, "1" for the eye size means increasing the eye size by one level based on the parameters corresponding to "1". The user can adjust their own eye size shown in the captured image displayed in the display area 31 by selecting any of the levels from "1" to "4".

[0039] As for the eye shape, "round eyes", "downturned eyes", and "cat eyes" are provided. The user can adjust their own eye shape shown in the captured image displayed in the display area 31 by selecting any of the shapes. It is also possible to not change the eye shape by selecting "no processing".

[0040] As the makeup intensity, five levels are provided. "0" for the makeup intensity indicates keeping the original intensity without changing the makeup intensity. Also, "1" for the makeup intensity means increasing the makeup intensity by one level based on the parameter corresponding to "1". The user can adjust the makeup intensity of themselves shown in the captured image displayed in the display area 31 by selecting any level from "1" to "4".

[0041] As the nose bridge, it is possible to select on / off. The user can choose whether to emphasize the highlighted part of their own nose bridge shown in the captured image displayed in the display area 31 or keep it in the original state by selecting on or off.

[0042] As the beard removal, three levels are provided. "0" for the beard removal indicates keeping the original beard without changing it. Also, "weak" for the beard removal means performing processing based on the parameter corresponding to "weak" and thinning the beard by one level. The user can adjust the beard density of themselves shown in the captured image displayed in the display area 31 by selecting any level. The beard removal function is effective when the person to be retouched is the groom.

[0043] In the area 33-3 formed on the right side of the retouch selection area 33, adjustment buttons 61 used for adjusting the skin brightness, adjustment buttons 62 used for adjusting the overall brightness, and adjustment buttons 63 used for adjusting the brightness of the background only are displayed.

[0044] As the skin brightness, three levels are provided. "0" for the skin brightness indicates keeping the original brightness without changing the skin brightness. Also, "1" for the skin brightness means increasing the skin brightness level by one level based on the parameter corresponding to "1". The user can adjust the skin brightness of themselves shown in the captured image displayed in the display area 31 by selecting any level.

[0045] As the overall brightness, three levels are provided. "0" for the overall brightness indicates that the overall brightness remains unchanged and is the original brightness. Also, "1" for the overall brightness indicates that the level of the overall brightness is increased by one level based on the parameter corresponding to "1". By selecting any of the levels, the user can adjust the overall brightness shown in the captured image displayed in the display area 31.

[0046] As the background brightness, three levels are provided. "0" for the background brightness indicates that the background brightness remains unchanged and is the original brightness. Also, "1" for the background brightness indicates that the level of the background brightness is increased by one level based on the parameter corresponding to "1". By selecting any of the levels, the user can adjust the background brightness shown in the captured image displayed in the display area 31.

[0047] Returning to the description of FIG. 3, a batch retouch button 34 used for batch retouch is displayed on the left side of the display area 31. Batch retouch is a function for collectively selecting combinations of various retouch contents as described above.

[0048] In the example of FIG. 3, any level can be selected from three types of levels of batch retouch with different combinations of levels of retouch contents. By using the batch retouch function, the user can select without using the display in the retouch selection area 33 and can select in a single operation.

[0049] A pre - processing button 35 is displayed on the right side of the display area 31. The pre - processing button 35 is a button operated when switching the display in the display area 31 to the state before retouch. While the pre - processing button 35 is pressed, the captured image in the state before retouch is displayed in the display area 31. By using the pre - processing button 35, the user can compare the state after retouch with the state before retouch and check whether the degree of retouch is too strong or not.

[0050] Below the processing button 35, a person selection button 36 used to select a person to be retouched is displayed. By using the person selection button 36, the user can select and retouch only the person shown on the left side of the captured image, only the person shown on the right side, or both of them. Facial recognition or the like is performed on the captured image, and the person to be the target of the retouch image processing is selected according to the user's operation.

[0051] The user can select the content of the retouch to be applied to the captured image displayed in the display area 31 by using such various functions. Thus, the retouch that can be performed using the retouch app 1A at least includes adjustment of the whole face and body, adjustment of the eyes and makeup, and adjustment of brightness.

[0052] The captured image to be retouched is switched using the thumbnail image, and the retouch of each of the captured images P1 to P5 is sequentially performed. When the retouch of each of the captured images P1 to P5 is completed and the end button displayed at the upper right end of the screen is operated, the retouch using the retouch screen is completed.

[0053] <<Overview of Arm Retouch>> FIG. 5 is a diagram showing an overview of arm retouch.

[0054] As shown by the arrow #11 in FIG. 5, skeleton estimation is performed on the person shown in each captured image selected as the retouch target. In FIG. 5, skeleton estimation of the bride shown in the captured image P5 is being performed. Skeleton estimation is a process of estimating the positions of the skeleton points of the person shown in the captured image. For example, skeleton estimation is performed using an inference model that takes the captured image as input and outputs information on the positions of the skeleton. The retouch app 1A is provided with an inference model for skeleton estimation. The skeleton estimation using the inference model may be performed on a server on the network.

[0055] In the example at the center of FIG. 5, the positions of the shoulders, elbows, and wrists of the bride's left and right arms are estimated. The small dot D1L, D2L, D3L represents the positions of the left shoulder, left elbow, and left wrist, respectively. Also, the dots D1R, D2R, D3R represent the positions of the right shoulder, right elbow, and right wrist, respectively. Note that the dots indicating the skeleton points are shown for convenience of explanation and are not information actually displayed on the photographed image. When the arm retouch function is selected, the dots indicating the skeleton points may be displayed on the photographed image.

[0056] When the level of the arm thickness is selected, image processing is performed to narrow the arm as shown at the tip of arrow #12. The photographed image P5 shown on the right side of FIG. 5 is an image after the image processing of the arm retouch. Compared with the image before the image processing, both arms of the bride are slightly thinner. Thus, the image processing of the arm retouch by the retouch app 1A is performed based on the information of the skeleton points which is the result of the skeleton estimation.

[0057] FIG. 6 is a diagram showing an example of the image processing of the arm retouch. In FIG. 6, the vicinity of the bride's right arm in the photographed image P5 shown in FIG. 5 is enlarged and shown.

[0058] As shown on the left side of FIG. 6, the part between the dot D1R and the dot D2R is recognized as the upper arm part, and the part between the dot D2R and the dot D3R is recognized as the forearm part. As shown by the small dots, the outer edges of the upper arm part and the outer edges of the forearm part are detected, and as shown by the arrows on the right side of FIG. 6, image processing is performed to move each position of the edges in the central direction of the arm.

[0059] On the right side of FIG. 6, image processing is performed to slightly move the outer edge of the upper arm part in the central direction of the arm (substantially right direction) represented by the straight line connecting the dot D1R and the dot D2R. Also, image processing is performed to slightly move the outer edge of the forearm part in the central direction of the arm (substantially upper right direction) represented by the straight line connecting the dot D2R and the dot D3R.

[0060] Similar image processing is also performed on the left arm of the bride. That is, image processing is performed to move the outer edge of the upper arm part in the central direction of the arm represented by the straight line connecting dot D1L and dot D2L. Also, image processing is performed to move the outer edge of the forearm part in the central direction of the arm represented by the straight line connecting dot D2L and dot D3L.

[0061] The amount of deformation of the outer edge of the arm is determined according to the level selected by the user using the adjustment button 54 and the like. The amount of deformation is a retouch parameter that defines the amount of movement of the edge position.

[0062] In this way, the image processing for arm retouch is performed by deforming only the outer sides of the upper arm part and the forearm part, and no image processing is performed on the inner side of the arm closer to the body. Instead of deforming both the outer and inner sides of the arm, the image processing for arm retouch is performed by deforming only the outer side as a part of the arm.

[0063] Note that when the subject person is facing the camera frontally and the left arm shown on the right side in the captured image is the object of image processing, the edge portion of the left arm closer to the right end of the image is the outer side of the arm. Also, when the right arm shown on the left side in the captured image is the object of image processing, the edge portion of the right arm closer to the left end of the image is the outer side of the arm.

[0064] Conversely, when the subject person is facing the camera with their back and the right arm shown on the right side in the captured image is the object of image processing, the edge portion of the right arm closer to the right end of the image is the outer side of the arm. Also, when the left arm shown on the left side in the captured image is the object of image processing, the edge portion of the left arm closer to the left end of the image is the outer side of the arm.

[0065] This makes it possible to prevent the body from being deformed as if dragged by the deformation of the arm, resulting in an unnatural appearance. For example, when the outer edge of the upper arm of the right arm is moved to the right, in the portion shown in color in FIG. 7 between the outer line of the upper arm before image processing and the outer line of the upper arm after image processing, image processing is performed to interpolate with the pixels on the outer side of the arm (background pixels in the example of FIG. 7). When the same image processing is performed on the inner side of the arm, the processing is to interpolate the portion between the inner line of the upper arm before image processing and the inner line of the upper arm after image processing with the pixels where the body is shown or the pixels near the body, and the body may be deformed as the arm becomes thinner. By deforming only the outer side of the arm, it is possible to prevent such an unnatural appearance.

[0066] In wedding commemorative photography, the bride often wears a dress with exposed arms or a dress through which the arms can be seen through the lace, and there is a particular need for women to show their arms beautifully. By enabling the arms to be processed to look thinner, it becomes possible to meet such needs.

[0067] FIG. 8 is a diagram showing an example of arm retouch when the bride's clothing is a kimono.

[0068] As shown in FIG. 8, it is possible to utilize the arm retouch function even when the bride wearing a haori is shown. Even when the arms are not visible while wearing a haori, an inference model capable of estimating the skeletal points is prepared in the retouch application 1A.

[0069] Also in the example of FIG. 8, the positions of the shoulders indicated by the dots D1L and D1R of the bride's left and right arms, the positions of the elbows indicated by the dots D2L and D2R, and the positions of the wrists indicated by the dots D3L and D3R are estimated, and the arms are made thinner by deforming the outer sides of the left upper arm and the left forearm, and the outer sides of the right upper arm and the right forearm, respectively. The outer edges of the clothing covering the upper arm and the outer edges of the clothing covering the forearm are detected, and image processing is performed so as to move each position of the edges in the direction of the center of the arm.

[0070] When wearing an uchikake, the body can appear larger depending on the direction of the body, but by using the arm retouching function, it is possible to prevent this appearance. Details of arm retouching will be described later.

[0071] <Reflect all numbers> FIG. 9 is a diagram showing an example of application of retouching parameters.

[0072] As shown in Fig. 9, the retouching contents applied to each of the captured images P1 to P5 by the user's operation are reflected in all unselected images. An unselected image is a captured image that has not been selected as a target for retouching. A captured image selected as a target for retouching on the image selection screen becomes a selected image. For example, when 5 captured images are selected from 100 captured images, the remaining 95 captured images become unselected images.

[0073] That is, the number of photographed images selected as retouching targets according to the plan (selected images) are retouched according to the content directly selected by the user using the retouching screen, whereas the unselected images not selected as retouching targets are retouched so that the retouching content selected using the selected images is automatically reflected. For example, the retouching content of the selected image is reflected on all unselected images.

[0074] In this way, the retouching application 1A is provided with a full reflection function that reflects the retouching content of a selected image to unselected images after the user has retouched the selected image.

[0075] When photographing an event such as a wedding, many shots are usually taken, such as hundreds of shots. It is a large burden for the user to manually retouch all the images taken by the photographer. By applying the retouching content of the selected image to all unselected images, the user can receive all the images taken in a state adjusted to the user's preference.

[0076] Which retouch content of which captured image among the retouched captured images P1 to P5 is reflected in which unselected image is selected according to the respective features of the selected image and the unselected image, such as whether the size of the person's face is shown to be about the same size. For example, in a certain unselected image, the retouch content of the selected image in which the face of the same person as the person shown in the unselected image is shown to be about the same size is reflected.

[0077] In this case, features such as the number of people and the size of the face shown in each captured image (selected image and unselected image) are recognized. For each unselected image, the retouch content of the selected image having the same features is reflected.

[0078] The retouch app 1A is also equipped with a function to recognize such features of the captured image. Not only the size but also other face states such as the orientation of the face may be recognized. Also, other types of features such as the gender, age of the person, the type of clothing the person is wearing, and the shooting scene are recognized.

[0079] <<Processing of Retouch App 1A>> <Overall Processing> With reference to the flowchart of FIG. 10, the processing of the retouch app 1A when providing a service to a certain set of users will be described. The processing shown in FIG. 10 starts when the shooting by the cameraman is completed and the user starts the retouch work. At the start of the retouch work, the folder in which the captured images are saved is specified by the staff of the shooting studio or the like.

[0080] In step S1, the retouch app 1A starts importing the captured images in the specified folder. Attribute determination processing is sequentially performed on the captured images imported from the folder. Details of the captured image import processing will be described later with reference to the flowchart of FIG. 11.

[0081] When the import and attribute determination of all captured images are completed, in step S2, the retouch app 1A displays an image selection screen.

[0082] In step S3, the retouch app 1A accepts the user's operation on the image selection screen and selects the captured image to be retouched. A predetermined number of captured images, such as 5 or 10, are selected as the retouch targets.

[0083] When the selection of the captured image to be retouched is completed, in step S4, the retouch app 1A displays a retouch screen.

[0084] In step S5, the retouch app 1A accepts the user's operation on the retouch screen and performs retouching.

[0085] In step S6, the retouch app 1A determines whether the retouching of all captured images is completed. If it is determined in step S6 that the retouching of all captured images is not completed, the process returns to step S4 and the above-described process is repeated.

[0086] If it is determined in step S6 that the retouching of all captured images is completed, in step S7, the retouch app 1A performs full reflection. When the reflection of the retouch content for all unselected images is completed, the process in FIG. 10 ends.

[0087] <Details of the captured image import process> Referring to the flowchart of FIG. 11, the details of the captured image import process will be described.

[0088] In step S11, the retouch app 1A displays a function on / off setting screen. The function on / off setting screen is a screen used for setting the validity / invalidity of the wrist retouch function. The setting using the function on / off setting screen is performed, for example, by the staff of the photography studio.

[0089] FIG. 12 is a diagram showing an example of a function on / off setting screen.

[0090] In the example of FIG. 12, a message is displayed asking whether to use the wrist attachment function, and below it, buttons to be operated when using the wrist attachment function and buttons to be operated when not using it are displayed. The staff of the photography studio will set whether to use the wrist attachment function according to the user's plan or the like.

[0091] When the plan the user is using is a plan to take wedding commemorative photos, since the user is wearing clothing such as a dress with a lot of arm exposure, the wrist attachment function becomes important. On the other hand, depending on the plan the user is using, such as a plan for adult ceremony commemorative photos taken with clothing with little arm exposure, the wrist attachment function may not be important. By enabling the selection of whether to use the wrist attachment function, it becomes possible to shorten the time required for attribute determination in the case of not using the wrist attachment function.

[0092] When using the wrist attachment function, skeleton estimation of the people appearing in each captured image is performed. When not using the wrist attachment function, by omitting the skeleton estimation process as described later, the time required for attribute determination can be shortened by the time required for the skeleton estimation process.

[0093] The function on / off setting screen of FIG. 12 is displayed when the staff of the photography studio sets to select whether to use the wrist attachment function every time a captured image is imported. For example, it is possible to set from a predetermined menu screen whether to always use the wrist attachment function, always not use it, or select whether to use it every time a captured image is imported. When set to always use the wrist attachment function or always not use the wrist attachment function, the function on / off setting screen is not displayed.

[0094] For each photography studio, the customer base is different, such as a studio with many users using the wedding commemorative photo plan and a studio with many users using the coming-of-age ceremony commemorative photo plan. By enabling the setting of whether the function on / off setting screen is required according to the user's tendency, it becomes possible to improve the convenience for the staff of the photography studio.

[0095] Returning to the description of FIG. 11, in step S12, the retouch app 1A determines whether to use the wrist retouch function. If using the wrist retouch function is selected using the function on / off setting screen, or if using the wrist retouch function is always set using the menu screen, it is determined that the wrist retouch function is to be used.

[0096] If it is determined in step S12 to use the wrist retouch function, the process proceeds to step S13. The attribute determination processes in steps S13 to S16 are performed for each captured image imported from the folder.

[0097] In step S13, the retouch app 1A performs face recognition and face feature determination. Face recognition and face feature determination are processes that recognize the face of a person shown in the captured image and determine the features of the recognized face for each face. Each process of face recognition and face feature determination is performed, for example, by image processing or using an inference model generated by machine learning. Note that the determination of attributes using the inference model is synonymous with the inference of attributes.

[0098] The retouch app 1A is provided with an inference model for face recognition that takes the captured image as input and outputs information on the position of the face shown in the captured image. Also, the retouch app 1A is provided with an inference model for face feature determination that takes the face image, which is the image of the face recognized by face recognition, as input and outputs information on the features of the face. The features of the face include information such as the positions of each part such as eyes, nose, and mouth, the contour points of the face, the orientation of the face, and the size of the face. For each person whose face is recognized, the attribute determination after face feature determination is performed.

[0099] In step S14, the retouch application 1A performs skeleton estimation. Skeleton estimation is a process of estimating the positions of the skeleton points of a person shown in a captured image. The skeleton estimation is performed using, for example, an inference model generated by machine learning. The retouch application 1A is provided with an inference model for skeleton estimation that takes as input a person image, which is an image of a person shown in the captured image, and outputs information on the positions of the skeleton points. The skeleton estimation may be performed by image processing.

[0100] FIG. 13 is a diagram showing an example of skeleton points.

[0101] As shown by the dots in FIG. 13, for example, the positions of the respective skeleton points of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles are estimated. Based on the positions of the skeleton points, attributes such as gender and body orientation are also estimated.

[0102] In step S15 of FIG. 11, the retouch application 1A performs clothing determination. Clothing determination is a process of estimating the type of clothing worn by a person shown in a captured image. The clothing determination is performed using, for example, an inference model generated by machine learning. The retouch application 1A is provided with an inference model for clothing determination that takes as input a person image and outputs information on the type of clothing. The clothing determination may be performed by image processing.

[0103] In step S16, the retouch application 1A determines various attributes of a person shown in the captured image, such as gender, age, and the exposed range of the skin. The gender and age are determined based on, for example, the features of the face. The retouch application 1A is provided with an inference model that takes as input a face image or face features and outputs information on gender and age. An inference model for gender determination and an inference model for age determination may be provided separately. The determination of gender and age may be performed by image processing.

[0104] The exposed area of the skin is determined, for example, by detecting the skin color area through image processing. The skin color area is detected as the exposed area of the skin. An inference model that takes a person image as input and outputs information on the skin area may be prepared in the retouch app 1A. As other attributes, the type of object overlapping and appearing on the person, the shooting scene of the captured image, etc. may be determined.

[0105] In this way, in the retouch app 1A, various attributes of the captured image and various attributes of the person appearing in the captured image are determined using machine learning. Based on these attributes determined at the time of importing the captured image, image processing using each retouch function is performed.

[0106] In step S17, the retouch app 1A saves the results of various attribute determinations such as the determination results of face recognition / face features, the results of skeleton estimation, the results of clothing determination, gender, age, and the determination results of the exposed area of the skin. The results of the attribute determinations are recorded and saved in the management file of each captured image.

[0107] FIG. 14 is a diagram showing an example of the management file.

[0108] As shown in FIG. 14, the management file includes information on the image size, selection flag, and retouched flag. The selection flag is a flag indicating whether the captured image is a selected image. The retouched flag is a flag indicating whether the captured image has been retouched.

[0109] In addition, the management file records the face ID, and in association with each face ID, retouch parameters, gender (gender determined based on face features), age, face size, contour points, face orientation, skeleton point information, gender (gender determined based on skeleton points), body orientation, type of clothing, and information on the exposed area of the skin are recorded.

[0110] The management file records not only retouch parameters indicating the content of retouching for individual persons but also retouch parameters indicating the content of retouching for the entire image (parameters such as the brightness of the image), information on the shooting scene, and the like. The gender of a person may be determined by combining the determination result based on facial features and the determination result based on skeleton points, or may be determined based on only facial features or only skeleton points.

[0111] In step S18 of FIG. 11, the retouch application 1A determines whether the processing of all captured images has been completed. If it is determined in step S18 that the processing of all captured images has not been completed, the process returns to step S13, and the above-described processing is repeatedly performed.

[0112] On the other hand, if it is determined in step S12 not to use the arm retouch function, the process proceeds to step S19. The processing after step S19 is the same as the processing after step S13, except that skeleton estimation is omitted. Redundant explanations will be omitted as appropriate.

[0113] That is, the retouch application 1A performs face recognition and face feature determination in step S19 and clothing determination in step S20. Also, in step S21, the retouch application 1A determines other attributes such as the gender, age, and exposure range of the person shown in the captured image.

[0114] In step S22, the retouch application 1A saves the results of various attribute determinations such as the determination results of face recognition and face features, the result of clothing determination, gender, age, and the determination result of the skin exposure range.

[0115] In step S23, the retouch application 1A determines whether the processing of all captured images has been completed. If it is determined in step S23 that the processing of all captured images has not been completed, the process returns to step S19, and the above-described processing is repeatedly performed.

[0116] When it is determined in step S18 or step S23 that the processing of all captured images has been completed, the processing in FIG. 11 ends.

[0117] As described above, when the function of arm retouch is not used, skeleton estimation is omitted in the attribute determination at the time of importing a captured image. By omitting the processing of skeleton estimation when the function of arm retouch is not used, it is possible to shorten the time required for attribute determination.

[0118] <<Details of Arm Retouch>> Here, the details of arm retouch will be described in the following order. Regarding the control of the image processing of arm retouch below, it is possible to use a combination of one or more. · Switching of a part to be the target of arm retouch · Method of deformation by arm retouch · Example of using the result of gender determination for arm retouch · Example of using the detection result of the exposed skin range for arm retouch · Example of using the estimated age result for arm retouch · Other examples of arm retouch

[0119] <Switching of a part to be the target of arm retouch> · Example 1 of the part to be retouched FIG. 15 is a diagram showing an example of a part to be the target of arm retouch. Hereinafter, when a line of an arbitrary part of a person is shown in a thick line, it indicates that image processing for thinning is performed on that part.

[0120] As shown in FIG. 15A, when the subject person is photographed in a body orientation where the left half of the body is in the foreground and the right half of the body is in the background, the outer side of the upper arm part and the outer side of the forearm part of the left arm indicated by the thick line L1 are the retouch targets.

[0121] On the other hand, as shown in FIG. 15B, when the subject person is photographed in a body orientation where the right half of the body is in the foreground and the left half of the body is in the background, the outer side of the upper arm part and the outer side of the forearm part of the right arm indicated by the thick line L2 are the retouch targets.

[0122] Also, as described with reference to FIG. 5, when facing forward, the outer sides of both arms are the retouch targets as described above. The body orientation of the subject person is specified based on the information included in the management file. Thus, a part of the object of the arm retouch image processing switches according to the way the arm appears as a result of the skeleton estimation.

[0123] That is, as shown in FIG. 15, when one arm appears in front of the other arm, the image processing for arm retouch is performed on the one arm that appears in front. For example, with reference to the front, an orientation serving as a threshold for the body orientation is set. When the body orientation of the subject faces outside the threshold orientation, the image processing for arm retouch is performed only on the one arm that appears in front, and when the body orientation of the subject faces more forward than the threshold orientation, the image processing for arm retouch is performed on both arms.

[0124] As shown in FIG. 15, when only one arm appears and the other arm does not appear (is not visible), the image processing for arm retouch is performed only on the one visible arm. Even if the arm is in front, if it is hidden by another person's body or the like and is not visible, the arm retouch for the one arm that appears in front is not performed.

[0125] · Example 2 of the retouch target part FIG. 16 is a diagram showing another example of the part that is the target of arm retouch.

[0126] As shown in FIG. 16A, when the subject person is facing forward, the outer side of the left arm indicated by the thick line L11 and the outer side of the right arm indicated by the thick line L12 are the retouch targets.

[0127] On the other hand, as shown in FIG. 16B, when the subject person is facing forward and raising the left arm from the shoulder while lowering the right arm, for the right arm, the outer side of the right arm indicated by the thick line L12 is the retouch target as in the case of FIG. 16A. For the left arm, instead of the front side of the left arm indicated by the thick line L11, the back side of the left arm indicated by the thick line L21 that appears on the outside is the retouch target.

[0128] Even when the left arm is raised from the shoulder as shown in FIG. 16B, if the front side of the left arm (the portion indicated by the thick line L11) is deformed in the same manner as when it is lowered, the process involves interpolating the portion between the edge of the upper arm before image processing and the edge of the upper arm after image processing with pixels near the face, and there is a risk that the face will be deformed as the arm becomes thinner. When the arm is raised, it is possible to prevent such unnatural imaging by deforming only the back side portion away from the face.

[0129] In this way, a part of the target for arm retouching switches according to the angle of the arm. For example, with a horizontal angle as the reference (0 degrees), a threshold angle of the arm is set. When the angle of the subject's arm is smaller than the threshold angle, arm retouching image processing is performed on the front side of the arm, and when the angle of the subject's arm is larger than the threshold angle, arm retouching image processing is performed on the back side of the arm.

[0130] Even when switching the amount of deformation used for arm retouching between the case of targeting the front side of the arm and the case of targeting the back side of the arm as shown in FIG. 17. Among the three levels prepared on the adjustment button 54 (FIG. 4), for example, when the "weak" level of arm retouching operation as shown in the balloon is performed, as shown on the right side of FIG. 17, arm retouching image processing is performed using the amount of deformation corresponding to the "weak" level ("Arm retouch: ON").

[0131] In the example of FIG. 17, even for the same upper arm part, the amount of deformation used for arm retouching image processing targeting the upper arm part of the raised left arm is larger (more) than the amount of deformation used for arm retouching image processing targeting the upper arm part of the lowered right arm. On the right side of FIG. 17, the length of the arrow indicates the magnitude of the amount of deformation. The same applies to other figures. In this way, arm retouching image processing is performed using different amounts of deformation according to the angle of the arm.

[0132] <How to Deform by Arm Retouch> ·Example of Deformation 1 FIG. 18 is a diagram showing an example of arm deformation.

[0133] As shown on the left side of FIG. 18, when the subject person is photographed in a body orientation where the left half of the body is forward and the right half of the body is rearward, the outer side of the upper arm of the left arm indicated by the thick line L31 and the outer side of the forearm of the left arm indicated by the thick line L32 are the retouch targets. In FIG. 18, unlike the case of FIG. 15, the portions to be retouched are shown separately for the upper arm and the forearm.

[0134] The amount of deformation used in the arm retouch image processing may be switched as shown in FIG. 18 between the case of targeting the outer side of the upper arm and the case of targeting the outer side of the forearm. For example, when an arm retouch operation at the "weak" level as shown in the balloon is performed, as shown on the right side of FIG. 18, arm retouch is performed using the amount of deformation corresponding to the "weak" level.

[0135] In the example of FIG. 18, the amount of deformation used in the arm retouch image processing for the upper arm of the left arm is larger than the amount of deformation used in the arm retouch image processing for the forearm of the left arm. Thus, the arm retouch image processing is performed using different amounts of deformation according to the part of the arm.

[0136] When image processing is performed using the same amount of deformation for each of the upper arm and the forearm, the balance between the upper arm and the forearm may deteriorate, but by changing the amount of deformation used in each image processing, it becomes possible to improve the balance of each part after the image processing. The amount of deformation used in the arm retouch image processing for the forearm may be made larger than the amount of deformation used in the arm retouch image processing for the upper arm.

[0137] · Example of Deformation 2 FIG. 19 is a diagram showing another example of arm deformation.

[0138] The bride's arm shown on the left side of Fig. 19A appears slightly thicker than the bride's arm shown on the left side of Fig. 19B. The outer side of the upper arm of the left arm indicated by the thick line L31 and the outer side of the forearm of the left arm indicated by the thick line L32 are the retouch targets.

[0139] For example, when an arm retouch operation at the "weak" level as shown in the balloon is performed, as shown on the right side of Fig. 19, arm retouch is performed using the deformation amount corresponding to the "weak" level. In the example of Fig. 19, image processing for arm retouch is performed using different deformation amounts according to the thickness of the arm along with the level selected by the user. The thickness of the arm is calculated based on the distance at a predetermined position of the arm.

[0140] The deformation amount used for the image processing of arm retouch for the bride's left arm shown in Fig. 19A is larger than the deformation amount used for the image processing of arm retouch for the bride's left arm shown in Fig. 19B. Thus, even when the same level is selected as the retouch level using the adjustment button 54, image processing for arm retouch is performed using a larger deformation amount as the thickness of the arm before deformation is thicker. Conversely, image processing for arm retouch is performed using a smaller deformation amount as the thickness of the arm before deformation is thinner. This makes it possible to prevent the arm from becoming too thin after the image processing.

[0141] In the example of Fig. 19, the deformation amount is changed according to the thickness of the arm, but the shape of the arm may be determined and the deformation amount may be changed according to the shape as well. For example, the type of the shape of the arm such as round or linear is determined, and the deformation amount is changed according to the type of the shape of the arm. The type of the shape of the arm is determined based on the thickness of each part of the arm and the like.

[0142] · Example of Deformation 3 Image processing may be performed not only to deform the thickness of the arm but also to extend the length of the arm. The part to be extended can be switched according to the appearance of the arm, such as only the upper arm or only the forearm.

[0143] <Example of Using the Result of Gender Judgment for Arm Retouch> In wedding commemorative photos, the shooting is carried out with a composition in which the bridegroom and the bride are photographed together. When a male figure and a female figure appear in a certain captured image, the male figure is identified as the bridegroom and the female figure is identified as the bride. When importing the captured image, the gender of the people appearing in the captured image is determined as described above. The image processing of arm retouching for a captured image in which the bridegroom and the bride are photographed together is performed with priority given to the processing for the bride.

[0144] · Example 1 with priority given to the bride Figure 20 is a diagram showing an example of deformation when the bridegroom and the bride are photographed.

[0145] The captured image shown on the left side of Figure 20 is an image in which the bridegroom is shown on the left side and the bride is shown on the right side. By using the person selection button 36 (Figure 3) prepared on the retouch screen, the user can select any one of only the bridegroom shown on the left side of the captured image, only the bride shown on the right side, or both of them to perform arm retouching.

[0146] When an arm retouching operation at the "weak" level as shown in the speech bubble is performed in the state of selecting "both of them", as shown on the right side of Figure 20, no arm retouching image processing is performed on the bridegroom ("arm retouching: OFF"), and arm retouching image processing is performed only on the bride using the deformation amount corresponding to the "weak" level ("arm retouching: ON"). Arm retouching image processing according to the user's operation is performed on the bride who is a woman, and no image processing according to the user's operation is performed on the bridegroom who is a man.

[0147] In this way, by making only the arm retouching for the bride effective, it becomes possible to emphasize the thinness of the bride's arm.

[0148] The process of making the arm retouching for the bride effective may be performed only when the bridegroom and the bride are selected as retouching targets using the person selection button 36, or may also be performed when selected individually such as only the bridegroom or only the bride.

[0149] ·Example 2 with Bride Priority FIG. 21 is a diagram showing another example of deformation when the bridegroom and the bride are shown. The captured image shown in FIG. 21 is the same image as the image described with reference to FIG. 20.

[0150] When the operation of arm touch at the "weak" level is performed as shown in the speech bubble, for example, in the state where "both" is selected, as shown on the right side of FIG. 21, for the bridegroom, image processing of arm touch is performed using the deformation amount corresponding to the "weak" level, and for the bride, image processing of arm touch is performed using the deformation amount corresponding to the "strong" level. The image processing of arm touch for the bride, who is female, is performed using a deformation amount larger than the deformation amount used for the image processing of arm touch for the bridegroom, who is male. When comparing before and after the image processing, the deformation amount of the bride's arm becomes larger than the deformation amount of the bridegroom's arm.

[0151] In this way, by switching the deformation amount used for the image processing of arm touch between men and women and applying stronger image processing of arm touch to women, it is also possible to emphasize the thinness of the bride's arm.

[0152] Note that when the operation of arm touch is performed in the state where only the man (only the person on the left in the example of FIG. 21) is selected, it is also possible not to perform the image processing of arm touch for the bridegroom, who is male. For example, the display of the adjustment button 54 may be grayed out so that arm touch is prohibited when only a man is selected as the touch target.

[0153] <Example of Using the Detection Result of the Skin Exposure Range for Arm Touch> In wedding commemorative photos, the bride often wears a dress with the neck, chest, and both arms exposed. When a captured image shows a person with a lot of skin exposure and a person with little skin exposure, the person with a lot of skin exposure is identified as the bride. When the captured image is imported, the skin exposure range is detected as described above.

[0154] The detection result of the exposed area of the skin and the determination result of gender may be combined so that a woman with a large exposed area of the skin is identified as a bride. In this case, a man with a small exposed area of the skin is identified as the groom, and a woman with a small exposed area of the skin is identified as the mother of the bride or the like. The image processing of the arm touch for a photographed image in which the bride and other persons are shown together is performed so as to prioritize the processing for the bride. The processing that prioritizes the arm touch for the bride is the same as in the case where the bride and the groom are shown. Duplicate explanations will be omitted as appropriate.

[0155] · Example 1 of bride priority FIG. 22 is a diagram showing a modified example in the case where a bride and the mother of the bride are shown.

[0156] The photographed image shown on the left side of FIG. 22 is an image in which the bride is shown on the left side and the mother of the bride is shown on the right side. The woman on the left side wearing a dress and having a large exposed area of the skin is identified as the bride, while the woman on the right side wearing a kimono and having a small exposed area of the skin is identified as the mother of the bride. By using the person selection button 36 (FIG. 3) prepared on the retouch screen, the user can perform an arm touch by selecting only the bride shown on the left side of the photographed image, only the mother of the bride shown on the right side, or both of them.

[0157] When an arm touch operation at the level of "weak" is performed as shown in the speech bubble, for example, in the state of selecting "both of them", as shown on the right side of FIG. 22, the arm touch is not performed on the mother of the bride, and only for the bride, the image processing of the arm touch is performed using the deformation amount corresponding to the level of "weak". For the bride with a large exposed area of the skin, the image processing of the arm touch corresponding to the user's operation is performed, and for the mother with a small exposed area of the skin, the image processing corresponding to the user's operation is not performed.

[0158] In this way, by making only the arm touch for the bride effective, it becomes possible to emphasize the thinness of the bride's arm.

[0159] · Example 2 of bride priority FIG. 23 is a diagram showing another example of deformation when the bride and the bride's mother are shown. The captured image shown in FIG. 23 is the same image as the image described with reference to FIG. 22.

[0160] When the operation of arm touch at the level of "weak" is performed as shown in the speech bubble, for example, in the state where "both" is selected, as shown on the right side of FIG. 23, the bride's mother is subjected to arm touch using the deformation amount corresponding to the level of "weak", and the bride is subjected to arm touch using the deformation amount corresponding to the level of "strong". The image processing of arm touch for the bride with a wider skin exposure range is performed using a larger deformation amount than the deformation amount used for the image processing of the mother's arm touch with a narrower skin exposure range. When comparing before and after the image processing, the deformation amount of the bride's arm becomes larger than the deformation amount of the mother's arm.

[0161] In this way, by switching the deformation amount used for the image processing of arm touch according to the skin exposure range and applying stronger image processing of arm touch to a person with a wider skin exposure range, it is also possible to emphasize the thinness of the bride's arm.

[0162] <Example of using the estimated age result for arm touch> When the captured image is imported, the age of the subject person is estimated as described above. The deformation amount used for the image processing of arm touch may be switched based on the estimated age result. For example, the image processing of arm touch for a person belonging to the age group of children or the elderly is performed using a smaller deformation amount than the deformation amount used for the image processing of arm touch for a person belonging to other age groups. By performing the image processing using the deformation amount according to the age, it is possible to prevent the image after arm touch from becoming an unnatural image.

[0163] <Other examples of arm touch> · When multiple people are lined up FIG. 24 is a diagram showing an example of a part to be the target of arm touch.

[0164] The captured image shown in FIG. 24 is an image in which two persons, the bridegroom and the bride, are depicted. The bridegroom is shown on the left side, and the bride is shown on the right side. In this case, the outer side of the bridegroom's right arm indicated by the thick line L51 and the outer side of the bride's left arm indicated by the thick line L61 are the targets for arm attachment. The adjacent arms of the two persons, the bridegroom's left arm and the bride's right arm, are excluded from the targets for arm attachment. Similarly, when a plurality of three or more persons are depicted, the adjacent arms are excluded from the targets for arm attachment.

[0165] In this way, by excluding the adjacent arms from the targets for arm attachment and setting the opposite arms (the bridegroom's right arm and the bride's left arm) shown on the outside as the targets for arm attachment, it is possible to prevent the adjacent arms of each person from being deformed and resulting in an unnatural appearance.

[0166] · When an arm is shown near the face The arm shown near the face is excluded from the targets for arm attachment, and only the arm shown at a position away from the face is the target for arm attachment. For example, when a tall bridegroom and a short bride are shown side by side and one of the bridegroom's arms is shown near the bride's face, among the bridegroom's left and right arms, the arm closer to the bride's face is excluded from the targets for arm attachment, and only the other arm farther from the bride's face is the target for arm attachment.

[0167] For example, an arm shown within a range of a predetermined distance based on the area of the face recognized by face recognition is determined as an arm shown near the face. In this way, by excluding the arm shown near the face from the targets for arm attachment and setting the arm shown at a distant position as the target for arm attachment, it is possible to prevent the face from becoming unnaturally distorted due to the deformation of the arm.

[0168] · When a hand is overlapping an arm in the image The arm with overlapping hands is excluded from the target of arm retouching, and only the arm without overlapping hands becomes the target of arm retouching. For example, in the case of a composition where the bride places her hand on the groom's arm, among the left and right arms of the groom, the arm with the bride's hand is excluded from the target of arm retouching, and only the other arm without the bride's hand becomes the target of arm retouching.

[0169] For example, the part beyond the skeletal point of the wrist is determined as the hand part. In this case, the arm with at least a part of the hand, such as the fingertip, overlapping is excluded from the target of arm retouching. By excluding the arm with overlapping hands from the target of arm retouching in this way, it is possible to prevent the hand overlapping the arm from being deformed along with the deformation of the arm and resulting in an unnatural appearance.

[0170] · When an object overlaps and appears on the arm When there is some object overlapping on the arm, whether the arm with the overlapping object is the target of arm retouching is determined according to the type of the object. In the shooting of wedding commemorative photos, there may be a composition where the bride holds a bouquet or a stuffed toy. Also, there may be a composition where the marriage certificate is spread out and facing the camera.

[0171] When the object overlapping on the arm is a bouquet or a bunch of flowers, that arm becomes the target of arm retouching. On the other hand, when the object overlapping on the arm is a stuffed toy or a marriage certificate, that arm is excluded from the target of arm retouching.

[0172] For example, there are types of objects that do not cause a sense of discomfort even when deformed along with the deformation of the arm, which are set in advance. When the object estimated by object estimation at the time of importing the captured image is an object of the preset type, the arm with the overlapping object becomes the target of arm retouching, and when it is not an object of the preset type, the arm with the overlapping object is excluded from the target of arm retouching.

[0173] In this way, by excluding the arm on which the object appears overlapping from the target of arm retouching according to the type of the object, it is possible to prevent the object from being deformed along with the deformation of the arm and resulting in an unnatural appearance.

[0174] <<Details of Image Processing Using the Result of Clothing Judgment>> At the time of capturing an image, clothing judgment is performed for each person of the subject. Clothing judgment is a process of judging the type of clothing. The types of clothing include Western-style clothing and Japanese-style clothing.

[0175] Rather than large classifications such as Western-style clothing and Japanese-style clothing, the types of clothing belonging to small classifications may be judged. Western-style clothing worn by women includes wedding dresses and colored dresses. Also, Japanese-style clothing worn by women includes white kimonos, colored kimonos, and furisode. On the other hand, Western-style clothing worn by men includes tuxedos and wedding suits. Japanese-style clothing worn by men includes haori and hakama.

[0176] Not only the type of clothing but also the attributes of the clothing such as color, shape, and pattern may be judged. In this case, the judgment results of the clothing attributes are used for various image processes together with the judgment results of the type.

[0177] In the following, mainly the case where Western-style clothing / Japanese-style clothing is obtained as the judgment result will be described. Various image processes such as retouching and full reflection are switched according to whether the clothing of the person of the subject is Western-style clothing or Japanese-style clothing.

[0178] <Example of Using the Result of Clothing Judgment for Image Processing of Arm Retouching> ·Switching of On / Off of Image Processing FIG. 25 is a diagram showing an example of switching of On / Off of image processing.

[0179] The photographed image shown in Fig. 25 is an image of the bride. The bride's clothing shown in Fig. 25A is determined to be Western-style clothing, and the bride's clothing shown in Fig. 25B is determined to be Japanese-style clothing. When the wrist touch operation is performed on each photographed image, as shown at the tip of the arrow in Fig. 25A, for the image in which the type of clothing is determined to be Western-style clothing, image processing for wrist touch is performed, and the arms become thinner.

[0180] On the other hand, as shown at the tip of the arrow in Fig. 25B, for the photographed image in which the type of clothing is determined to be Japanese-style clothing, image processing for wrist touch is not performed. In the example of Fig. 25B, a message notifying that the wrist touch function cannot be used is displayed.

[0181] In this way, when the wrist touch operation is performed, it is possible to perform the image processing for wrist touch according to the user's operation when the clothing of the subject person is Western-style clothing, and not to perform the image processing for wrist touch when the clothing of the subject person is Japanese-style clothing.

[0182] When the clothing of the subject person is Japanese-style clothing, the effect may not be noticeable even if the image processing for wrist touch is performed. By enabling the use of the wrist touch function when the clothing of the subject person is Western-style clothing, it becomes possible to make the effect of the function more noticeable.

[0183] ·Switching of the content of image processing Fig. 26 is a diagram showing an example of switching of the content of image processing.

[0184] The photographed image shown in Fig. 26 is the same image as the photographed image shown in Fig. 26. When the wrist touch operation is performed on each of the photographed image in Fig. 26A in which the clothing of the subject person is determined to be Western-style clothing and the photographed image in Fig. 26B in which the clothing of the subject person is determined to be Japanese-style clothing, as shown at the tip of the arrow in Fig. 26A, for the image in which the type of clothing is determined to be Western-style clothing, image processing for wrist touch is performed, and the arms become thinner.

[0185] On the other hand, as shown at the tip of the arrow in Fig. 26B, for an image in which the type of clothing is determined to be a kimono, image processing for styling up is performed instead of image processing for arm retouch. The image processing for styling up is a process of slimming the entire body. Image processing for slimming is performed not only on the arm part but also on other parts.

[0186] As described above, when the arm retouch operation is performed, when the clothing of the subject person is Western-style clothing, the image processing for arm retouch is performed according to the user's operation, and when the clothing of the subject person is kimono, it is possible to perform image processing on other parts as well using other functions.

[0187] When the clothing of the subject person is kimono, the effect of the arm retouch image processing may not be easily noticeable even if it is performed. By performing image processing for slimming not only the arm part but also other parts when the clothing of the subject person is kimono, it becomes possible to make the effect of the function more noticeable. Different amounts of change may be used according to the part to be the target of the image processing.

[0188] <Example of using the result of clothing determination for image processing of other retouch functions> Switching the on / off of the function or switching the content of the image processing according to the result of the clothing determination is not limited to the arm retouch, and is similarly performed when another retouch function selectable on the retouch screen is selected.

[0189] Fig. 27 is a diagram showing another example of switching the content of the image processing.

[0190] The captured image shown in Fig. 27A is an image in which the clothing of the subject person is determined to be Western-style clothing, and the captured image shown in Fig. 27B is an image in which the clothing of the subject person is determined to be kimono. The image processing when the styling-up retouch function is used for the captured image shown in Fig. 27 and the style is adjusted will be described.

[0191] When a styling operation is performed on each captured image, as shown at the tip of the arrow in Fig. 27A, for an image in which the type of clothing is determined to be Western-style, image processing is performed to narrow the waist. On the other hand, as shown at the tip of the arrow in Fig. 27B, for an image in which the type of clothing is determined to be Japanese-style, image processing is performed to narrow the entire body. Image processing to narrow not only the waist but also other parts is performed.

[0192] Thus, even when the same styling operation is performed, image processing is performed on different parts depending on whether the clothing of the subject person is Western-style or Japanese-style.

[0193] The style adjustment by styling is performed not only by narrowing specific parts of the body but also by appropriately stretching them. Image processing to stretch different parts may be performed depending on whether the clothing of the subject person is Western-style or Japanese-style. For example, when the clothing of the subject person is Western-style, image processing is performed to stretch the part below the waist of the subject person, and when the clothing of the subject person is Japanese-style, image processing is performed to stretch the part below the neck of the subject person.

[0194] As described above, when an operation to deform a predetermined part of the subject person, such as an arm touch operation or a styling operation, is performed by the user, the part to be deformed is switched according to the type of clothing of the subject person. In some cases, it may be preferable to switch the content of the image processing according to whether the clothing of the subject person is Western-style or Japanese-style. By switching the content of the processing as described above, it becomes possible to perform image processing suitable for the type of clothing of the subject person.

[0195] <When the bridegroom and the bride are shown> It is also possible to use the result of clothing determination for image processing targeting a captured image in which the bridegroom and the bride are shown together. Switching the on / off of functions or switching the content of image processing according to the result of clothing determination is also performed when targeting a captured image in which the bridegroom and the bride are shown together.

[0196] FIG. 28 is a diagram showing an example of image processing when the bridegroom and the bride are depicted.

[0197] The photographed image shown on the left side of FIG. 28 is an image in which the bridegroom and the bride are depicted as the subjects of the photograph, and their respective outfits are determined to be traditional Japanese clothing. When photographing in a composition where the bridegroom and the bride wearing traditional Japanese clothing are side by side, the body of the bride may appear large due to the volume of the clothing.

[0198] When a predetermined operation such as a styling operation is performed on a photographed image in which the bridegroom and the bride wearing traditional Japanese clothing are shown side by side, in order to prevent the bride from appearing large, as shown at the tip of the arrow, image processing to slightly reduce the size is performed on the bride. On the right side of FIG. 28, the fact that the entire body of the bride is surrounded by a thick line indicates that the size has been reduced.

[0199] This makes it possible to prevent the bride from appearing large. Instead of reducing the size of the bride, image processing to enlarge the size of the bridegroom may be performed. At least one of the image processing to reduce the size of the bride and the image processing to enlarge the size of the bridegroom is performed.

[0200] Such image processing is performed when the bridegroom and the bride are depicted as the subjects of the photograph and their respective outfits are determined to be traditional Japanese clothing, and is not performed when the outfits of the bridegroom and the bride are determined to be Western-style clothing.

[0201] <Other examples of retouching using the result of clothing determination> ·Examples of using the detection results of the exposure range in combination Retouching image processing may be performed by combining the detection result of the exposed area of the skin and the result of clothing determination. For example, it is determined whether the exposed area of the skin is wide or narrow based on the area set as the threshold value, and depending on the combination of whether the exposed area of the skin is wide or not and the result of clothing determination, the on / off switching of the image processing and the switching of the content of the image processing are performed.

[0202] For example, when the type of the clothing of the subject person is Western-style clothing and the exposed range of the skin is wide, the person is identified as a bride, and is specifically identified as wearing a dress with the skin of the arm part exposed. When an arm touch operation is performed on a bride wearing a dress with the skin of the arm part exposed, image processing for further emphasizing the thinness of the arm is performed. That is, the image processing of the arm touch is performed using a deformation amount larger than the deformation amount used for the arm touch of a bride wearing a dress with no exposed skin on the arm part or the deformation amount used for the arm touch of a bride wearing Japanese traditional clothing.

[0203] ·Example used for scene determination The result of clothing determination may be used for scene determination. Scene determination is a process of determining the type of the shooting scene. The shooting scene includes a wedding scene and a pre-shooting scene. For example, when the clothing of the bride is a white kimono or a white robe, it is determined that the shooting scene is during the wedding ceremony, and when the clothing of the bride is a colored robe or a colored dress, it is determined that it is a pre-shooting scene.

[0204] The parameters used for the image processing of retouching may be switched according to the result of the scene determination. Also, according to the shooting scene determined based on the result of the clothing determination, the shooting image that is the source of the full reflection and the shooting image that is the destination of the reflection may be selected. For example, the shooting image of the same shooting scene as the shooting scene of the shooting image that is the destination of the reflection is selected as the shooting image that is the source of the reflection.

[0205] <Full reflection using the result of clothing determination> As described with reference to FIG. 9, full reflection is a process of reflecting the content of the retouching of the retouched shooting image on the unselected images that were not selected as the retouching target at the time of image selection. For each person shown in the unselected images, it is selected which content of the retouching for which person shown in which retouched shooting image is to be reflected. The result of clothing determination is used for the selection of the person shown in the unselected image that is the destination of the reflection of the retouching content and the person shown in the retouched shooting image that is the source of the reflection.

[0206] · Case 1 FIG. 29 is a diagram showing a first example of full reflection.

[0207] The photographed images P11 to P15 shown in the upper part of FIG. 29 are retouched photographed images. Further, the photographed image P31 shown in the lower part of FIG. 29 is an unselected image selected as a processing target at the time of full reflection. Each unselected image is sequentially selected, and image processing is performed to reflect the content of retouching. The image processing for reflecting the content of retouching is performed by applying parameters that define the content of retouching (performing retouching using the same parameters). For the sake of convenience of explanation, it is assumed that there is one person appearing in all the photographed images.

[0208] In the example of FIG. 29, it is determined that the clothing of the person appearing in each of the photographed images P11 to P15 is Western-style clothing. It is also determined that the clothing of the person appearing in the photographed image P31 is Western-style clothing.

[0209] Also, in the example of FIG. 29, the photographed image P13 is selected from among the photographed images P11 to P15 as an image in which a person satisfying the conditions appears. For example, a condition that the person is of the same gender as the person appearing in the unselected image and has a face of a size closest to the size of the face of the person appearing in the unselected image is set as the condition for the person who is the source of reflection of the content of retouching. The person appearing in the unselected image becomes the person who is the destination of reflection of the content of retouching. For example, for the determination of gender identity, the gender determined based on the facial features is used.

[0210] In this case, as shown by the white arrow, all the content of retouching performed on the source person is reflected on the destination person appearing in the unselected image. The content to be reflected also includes the content of arm retouching. When the arm retouching function is used for the retouching of the photographed image P13, the arm of the Western-style clothing person who is the destination also becomes as thin as the arm of the Western-style clothing person who is the source. Thereby, it becomes possible to make the arm of the Western-style clothing person appearing in the unselected image as thin as the arm of the source person.

[0211] Thus, when the clothing of the people shown in all the captured images is Western-style clothing and the clothing of the people shown in the unselected image is also Western-style clothing, a person who is the source of the retouch content is selected from among the Western-style dressed people shown in the retouched captured images. Also, the content of all retouches, including arm retouches, is reflected in the destination person.

[0212] · Case 2 FIG. 30 is a diagram showing a second example of full reflection. Descriptions that overlap with the above description will be omitted as appropriate. The same applies hereinafter.

[0213] In the example of FIG. 30, it is determined that the clothing of the people shown in the captured image P32, which is the unselected image, is Japanese-style clothing. Here, the captured images in which it is determined that the clothing of the subject person is Japanese-style clothing are shown hatched. The same applies to FIGS. 31 and 32.

[0214] Also, in the example of FIG. 30, from among the captured images P11 to P15, the captured image P13 in which a person of the same gender as the person shown in the unselected image and whose face is shown in the size closest to the size of the face of the person shown in the unselected image is selected as the source of reflection.

[0215] In this case, as shown by the white arrow, the content of the retouch other than the arm retouch performed on the source person is reflected in the destination person shown in the unselected image. Since the content of the arm retouch is excluded from the reflected content, even if the arm retouch function is used in the retouch of the captured image P13, the arms of the Japanese-style dressed destination person will not become thinner unlike the arms of the Western-style dressed source person. If the content of the arm retouch is reflected as it is on the Japanese-style dressed person, the shape of the clothing such as a kimono may be distorted, but it is possible to prevent such a thing.

[0216] Thus, when the clothing of the people shown in all the captured images is Western-style clothing and the clothing of the people shown in the unselected image is Japanese-style clothing, a person who is the source of the retouch content is selected from among the Western-style dressed people shown in the retouched captured images. Also, the content of the retouch except for the arm retouch is reflected in the destination person.

[0217] · Case 3 FIG. 31 is a diagram showing a third example of full reflection.

[0218] In the example of FIG. 31, it is determined that the clothing of the people shown in each of the photographed images P11 to P15 is traditional Japanese clothing. Image processing of wrist attachment is also performed on the people in traditional Japanese clothing shown in the photographed images P11 and P12. On the other hand, it is determined that the clothing of the person shown in the photographed image P33, which is an unselected image, is Western-style clothing.

[0219] Also, in the example of FIG. 31, among the photographed images P11 to P15, the photographed image P13 in which a person of the same gender as the person shown in the unselected image and whose face is shown in the size closest to the size of the face of the person shown in the unselected image is selected as the reflection source.

[0220] In this case, as shown by the white arrow, all the retouching contents performed on the person of the reflection source are reflected on the person of the reflection destination shown in the unselected image. Also, the level 1 wrist retouching that has not been performed on the person shown in the photographed image P13 is reflected. Since the image processing of the wrist retouching is performed using the deformation amount corresponding to level 1, the wrists of the person in Western-style clothing of the reflection destination become thinner.

[0221] The fact that the function of wrist retouching is used in the retouching of the photographed images P11 and P12 in which people in traditional Japanese clothing are shown is considered to indicate that the user has a strong desire to make the wrists thinner. Since the wrists are more prominent in Western-style clothing than in traditional Japanese clothing, even if wrist retouching is not used in the retouching of the reflection source photographed image P13, the content of the wrist retouching is reflected on the photographed image P33 in which the clothing of the subject person is determined to be Western-style clothing. This enables the reflection of the retouching content in a form that meets the user's wishes.

[0222] Thus, when the clothing of the people shown in all the captured images is traditional Japanese clothing and the clothing of the people shown in the unselected image is Western-style clothing, a person whose clothing is traditional Japanese and who appears in the retouched captured image is selected as the source of the retouch content. Also, when arm retouching is used for the retouching of any of the captured images, the content of all retouching, including the arm retouching, is reflected in the target person. The number of captured images (the number of captured images in which arm retouching is used) that serves as the threshold for reflecting the arm retouching may be set.

[0223] · Case 4 FIG. 32 is a diagram showing a fourth example of full reflection.

[0224] In the example of FIG. 32, it is determined that the clothing of the person shown in the captured image P34, which is the unselected image, is traditional Japanese clothing. The other conditions are the same as those shown in FIG. 32. Among the captured images P11 to P15, the captured image P13 in which a person of the same gender as the person shown in the unselected image and whose face is shown in the size closest to the size of the face of the person shown in the unselected image is selected as the source.

[0225] In this case, as shown by the white arrow, the content of the retouching performed on the source person is reflected in the target person shown in the unselected image. Since the content of the arm retouching is excluded from the reflected content, the arms of the target person in traditional Japanese clothing do not become thinner.

[0226] Thus, when the clothing of the people shown in all the captured images is traditional Japanese clothing and the clothing of the people shown in the unselected image is traditional Japanese clothing, a person whose clothing is traditional Japanese and who appears in the retouched captured image is selected as the source of the retouch content. Also, the content of the retouching, excluding the arm retouching, is reflected in the target person.

[0227] As described above, it is possible to use the result of clothing determination for the full reflection image processing.

[0228] When it is possible to identify a person based on facial features, information on skeleton points, etc., the identification result of the person may be used to perform full reflection. In this case, for example, the retouching content for the bride shown in the retouched captured image is reflected in the bride shown in the unselected image, and the retouching content for the groom shown in the retouched captured image is reflected in the groom shown in the unselected image. As a result, the same retouching content is reflected for the same person.

[0229] <<Configuration of Retouch PC>> FIG. 33 is a block diagram showing a configuration example of the retouch PC 1.

[0230] The CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0231] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a recording unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0232] The input unit 106 is composed of a keyboard, a mouse, etc.

[0233] The output unit 107 is composed of a display 11, etc.

[0234] The recording unit 108 is composed of a hard disk, a non-volatile memory, etc. The recording unit 108 records programs such as the retouch application 1A executed by the CPU 101 and various information such as captured images.

[0235] The communication unit 109 is an interface for networks such as the Internet and LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a photographed image in which a user who is about to start retouching appears, which is transmitted from the storage PC. Also, the communication unit 109 communicates with the image management server 2 via the Internet, and transmits the retouched photographed image to the image management server 2.

[0236] The drive 110 controls writing of data to the removable media 111 and reading of data from the removable media 111. The capture image from the storage PC may be imported via the removable media 111.

[0237] FIG. 34 is a block diagram showing a functional configuration example of the retouch PC 1.

[0238] As shown in FIG. 34, in the retouch PC 1, the information processing unit 121 is realized by executing the retouch application 1A. The information processing unit 121 is composed of an image acquisition unit 131, a display control unit 132, a person recognition / attribute determination unit 133, a retouch processing unit 134, and a file management unit 135.

[0239] The image acquisition unit 131 reads and acquires a photographed image from the recording unit 108 at the start of retouching of the photographed image. When the photographing is completed, all the photographed images such as several hundred images in which a set of users are photographed as subjects are transferred from the storage PC and recorded in the recording unit 108. The photographed image acquired by the image acquisition unit 131 is supplied to the display control unit 132 and the person recognition / attribute determination unit 133. The capture of the photographed image in step S1 of FIG. 10 is a process performed by the image acquisition unit 131.

[0240] The display control unit 132 causes an image selection screen to be displayed based on the captured image supplied from the image acquisition unit 131. Further, after five, ten, or other captured images to be retouched are selected using the image selection screen, the display control unit 132 causes a retouch screen to be displayed and controls the display of the retouch screen. While retouching is being performed using the retouch screen, information on the result of image processing is supplied from the retouch processing unit 134. The processes of steps S2, S3, and S4 in FIG. 10 and the process of step S11 in FIG. 11 are processes performed by the display control unit 132.

[0241] The person recognition and attribute determination unit 133 recognizes the faces of the persons shown in each of the captured images captured by the image acquisition unit 131 and determines the attributes of each person. The processes of steps S12 - S16 and steps S19 - S21 in FIG. 11 are processes performed by the person recognition and attribute determination unit 133.

[0242] That is, the person recognition and attribute determination unit 133 recognizes the faces of the persons shown in the captured images and determines attributes such as gender and age for each recognized face (each person). Further, the person recognition and attribute determination unit 133 determines attributes such as skeleton points, clothing type, and exposure range for each person whose face has been recognized. Details of the configuration of the person recognition and attribute determination unit 133 will be described later with reference to FIG. 35. The recognition result and the attribute determination result by the person recognition and attribute determination unit 133 are supplied to the retouch processing unit 134 and the file management unit 135 together with the captured image that is the object of recognition and the like.

[0243] The retouch processing unit 134 performs various image processes related to retouching in response to a user's operation on the retouch screen when retouching a captured image. The processes of steps S5 - S7 in FIG. 10 are processes performed by the retouch processing unit 134. For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouch processing unit 134 performs image processing such as deforming the position of the eye contour according to the selection content by the user.

[0244] Specifically, when any one of the levels of "1", "2", "3", or "4" is selected using the adjustment button 56 (Fig. 4), the retouch processing unit 134 extracts the eye region of the person to be retouched by means of mask processing or the like. The retouch processing unit 134 enlarges the image of the extracted eye region according to the selected level, and performs eye enlargement processing by pasting the enlarged eye region image at the extraction source position.

[0245] Also, when an operation related to adjusting the makeup density is performed, the retouch processing unit 134 performs image processing such as synthesizing an image of a cheek with a predetermined color on the cheek portion or synthesizing an image of lipstick with a predetermined color on the lip portion.

[0246] When an operation related to adjusting the skin texture is performed, the retouch processing unit 134 performs image processing such as adjusting the skin brightness. When an operation related to adjusting the face size (oval face effect) or face shape is performed, the retouch processing unit 134 performs image processing such as shifting the position of the face contour according to the user's selected content.

[0247] Specifically, when any one of the levels of "1", "2", "3", or "4" is selected using the adjustment button 51 (Fig. 4), the retouch processing unit 134 shifts each position of the face contour of the person to be retouched in the direction of the face center according to the selected level, and performs image processing to make the face smaller.

[0248] When an operation related to adjusting the face size (oval face effect) or face shape is performed, the retouch processing unit 134 performs image processing such as shifting the position of the face contour according to the user's selected content. Also, when an operation related to deforming a specific part of a person, such as arm retouch or styling up, is performed, the retouch processing unit 134 performs the image processing of those functions as described above.

[0249] The information representing the result of the image processing by the retouch processing unit 134 is supplied to the display control unit 132 and used for displaying the retouch screen. When the retouch of all the captured images is completed, the retouch parameters, which are the information indicating the content of the retouch, are supplied to the file management unit 135.

[0250] Also, when all are reflected, the retouch processing unit 134 reflects the content of the retouch of the retouched captured image on the unselected image. As described above, which person in which captured image the retouch content is reflected on which person in which unselected image is selected based on the face size, gender, type of clothing, etc. When the all-reflection is completed, the retouch parameters of each unselected image are supplied to the file management unit 135.

[0251] The file management unit 135 generates a management file in which various types of information are recorded based on the information supplied from the person recognition / attribute determination unit 133 and the retouch processing unit 134. The processes in steps S17 and S22 in FIG. 11 are the processes performed by the file management unit 135.

[0252] The file management unit 135 manages by associating the file of each captured image with the management file. The file of the captured image and the management file are recorded in the recording unit 108, for example. The information of the management file managed by the file management unit 135 is appropriately supplied to the display control unit 132 and used for screen display.

[0253] FIG. 35 is a block diagram showing a configuration example of the person recognition / attribute determination unit 133 in FIG. 34.

[0254] The person recognition / attribute determination unit 133 is composed of a face recognition unit 151, a face feature determination unit 152, a skeleton estimation unit 153, a clothing determination unit 154, and a determination unit 155. Each captured image captured by the image acquisition unit 131 is input to each unit.

[0255] The face recognition unit 151 recognizes the face shown in the captured image. Among the processes of steps S13 and S19 in FIG. 11, the face recognition process is the process performed by the face recognition unit 151. An inference model for face recognition is prepared for the face recognition unit 151, which takes the captured image as input and outputs information indicating the position of the face. The recognition result by the face recognition unit 151 is output to each unit.

[0256] The face feature determination unit 152 determines the features of the face recognized by the face recognition unit 151. Among the processes of steps S13 and S19 in FIG. 11, the determination of the face features is the process performed by the face feature determination unit 152. An inference model for face feature determination is prepared for the face feature determination unit 152, which takes the face image as input and outputs information on the face features. The determination result by the face feature determination unit 152 is output to the retouch processing unit 134 and the file management unit 135.

[0257] The skeleton estimation unit 153 performs skeleton estimation on the person whose face is recognized by the face recognition unit 151. The process of step S14 in FIG. 11 is the process performed by the skeleton estimation unit 153. An inference model for skeleton estimation is prepared for the skeleton estimation unit 153, which takes the image of the person as input and outputs information on the positions of the skeleton points. The result of the skeleton estimation by the skeleton estimation unit 153 is output to the retouch processing unit 134 and the file management unit 135.

[0258] The clothing determination unit 154 performs clothing determination on the person whose face is recognized by the face recognition unit 151. The processes of steps S15 and S20 in FIG. 11 are the processes performed by the clothing determination unit 154. An inference model for clothing determination is prepared for the clothing determination unit 154, which takes the image of the person as input and outputs information indicating the type of clothing. The inference model for clothing determination is generated, for example, by learning using a large number of images showing various people in clothing with the information on the type of clothing labeled as learning data. The information on the type of clothing, which is the determination result by the clothing determination unit 154, is output to the retouch processing unit 134 and the file management unit 135.

[0259] The determination unit 155 estimates various attributes of the person whose face has been recognized by the face recognition unit 151. Attributes recorded in the management file other than facial features, skeletal points, and clothing types are estimated by the determination unit 155. The processes of steps S16 and S21 in FIG. 11 are the processes performed by the determination unit 155. Various inference models such as an inference model for gender estimation, an inference model for age estimation, and an inference model for estimating the exposed range of the skin are prepared for the determination unit 155. The determination result by the determination unit 155 is output to the retouch processing unit 134 and the file management unit 135.

[0260] As described above, in each part of the person recognition / attribute determination unit 133, face recognition and determination of various attributes are performed using inference models generated by machine learning. Instead of the processing of each part of the person recognition / attribute determination unit 133 being performed using an inference model, it may be performed by analyzing the image.

[0261] <<Retouch correction tool>> As described above, a user who uses the service of the photography studio can perform retouching by himself / herself using the functions of the retouch app 1A and adjust the appearance of the photographed image. The retouching of the photographed image may also be performed by the staff of the photography studio. Further, the retouching of the photographed image is automatically performed as appropriate by the full reflection function of the retouch app 1A.

[0262] Retouching by manual or full reflection is performed using the determination results of various inference models. However, if there is an error in the determination using this inference model, retouching that results in an unnatural appearance may be performed. A tool for correcting such so-called retouching failures is prepared.

[0263] FIG. 36 is a diagram showing the flow of retouch correction.

[0264] The PC201 is an information processing device used for retouch correction. A correction application 201A, which is a tool for retouch correction, is installed on the PC201. The correction application 201A may be installed on the retouch PC1 so that retouch correction is performed on the retouch PC1.

[0265] As shown by the arrow in FIG. 36, the image to be corrected and the management file are taken into the correction application 201A. The image to be corrected is a captured image for which retouching has failed. For example, before providing the retouched captured images to the user, the staff at the photo studio or the like checks whether the retouching of each captured image is unnatural. The checking operation is, for example, an operation of visually checking one by one on the screen of the retouch PC1.

[0266] The captured images determined to have unnatural retouching are selected as the images to be corrected. In the example of FIG. 36, a plurality of captured images are selected as the images to be corrected. For example, when an image to be corrected is selected on the screen of the retouch application 1A, the file of the image to be corrected and the management file of the image to be corrected are transmitted to the PC201 and taken into the correction application 201A.

[0267] The retouch correction using the correction application 201A may be performed by the staff at the photo studio, or may be performed by the operator of the business that provides the retouch application 1A and the correction application 201A. In the latter case, the file of the image to be corrected selected by the staff at the photo studio is transmitted to the PC201 via a network such as the Internet and taken into the correction application 201A.

[0268] When the file of the image to be corrected and the like are taken in, a retouch correction screen is displayed by the correction application 201A. The retouch correction screen is a screen used for retouch correction. The user of the retouch correction screen is the staff at the photo studio or the operator of the business that provides the retouch application 1A and the correction application 201A.

[0269] FIG. 37 is a screen showing an example of a retouch correction screen.

[0270] The retouch correction screen basically has the same screen configuration as the retouch screen displayed by the retouch app 1A. The retouch is corrected by the same operations as those during retouch. The display in FIG. 37 is the display when an operation for correcting the arm retouch is performed. The correction of the arm retouch will be mainly described.

[0271] A display area 251 is formed above the center of the retouch correction screen. The display area 251 is an area where one captured image to be corrected (correction target image) is enlarged and displayed. A thumbnail image of the correction target image is displayed in a thumbnail area 252 formed on the left side of the display area 251. In the example of FIG. 37, the first captured image selected using the thumbnail image is enlarged and displayed in the display area 251.

[0272] The captured image displayed in the display area 251 is an image in which the groom is shown on the left side and the bride is shown on the right side. Small circular dots indicating skeleton points and an image of a straight line connecting the skeleton points are displayed on the body parts of the groom and the bride.

[0273] FIG. 38 is a diagram showing an enlarged view of the display in the display area 251.

[0274] Dots D11R to D14R are displayed on the right arm of the groom, and dots D11L to D14L are displayed on the left arm. The upper arm part of the right arm is indicated by dot D11R indicating the shoulder position of the right arm and dot D12R indicating the elbow position, and the forearm part of the right arm is indicated by dot D13R indicating the elbow position of the right arm and dot D14R indicating the wrist position. Similarly, the upper arm part of the left arm is indicated by dot D11L indicating the shoulder position of the left arm and dot D12L indicating the elbow position, and the forearm part of the left arm is indicated by dot D13L indicating the elbow position of the left arm and dot D14L indicating the wrist position.

[0275] On one side, dots D21R to D24R are displayed on the bride's right arm, and dots D21L to D24L are displayed on the left arm. The upper arm of the right arm is indicated by dot D21R showing the shoulder position of the right arm and dot D22R showing the elbow position, and the forearm of the right arm is indicated by dot D23R showing the elbow position of the right arm and dot D24R showing the wrist position. Similarly, the upper arm of the left arm is indicated by dot D21L showing the shoulder position of the left arm and dot D22L showing the elbow position, and the forearm of the left arm is indicated by dot D23L showing the elbow position of the left arm and dot D24L showing the wrist position.

[0276] The information such as these skeletal points is displayed based on the information recorded in the management file. The straight lines connecting between the dots indicating each part of the upper arm and forearm of the right arm and the upper arm and forearm of the left arm of the groom and the bride are displayed in different colors respectively. By displaying the information of the skeletal points in different colors, the result of skeletal estimation can be easily confirmed.

[0277] Returning to the description of FIG. 37, below the thumbnail area 252, a person selection button 253 and a skeletal point selection button 254 are displayed. The person selection button 253 is a button used when selecting a person to be the correction target of arm attachment among the people shown in the captured image to be corrected. It is possible to select only the groom shown on the left side, only the bride shown on the right side, or both the groom and the bride. In the example of FIG. 37, both the groom and the bride are selected. The information of the skeletal points of the person selected using the person selection button 253 is displayed on the captured image.

[0278] The skeletal point selection button 254 is a button operated when selecting an added skeletal point. It is possible to add skeletal points in case of an error in skeletal estimation. Each added skeletal point can be selected using the skeletal point selection button 254.

[0279] At the lower left of the retouch correction screen, a retouch correction area 256 is formed, and at the lower right, a skeleton point correction area 257 is formed. Each time the buttons in the retouch correction area 256 and the skeleton point correction area 257 are operated, the content of the user's correction is reflected in the display of the display area 251.

[0280] In the retouch correction area 256, a level correction button 256-1, a correction part selection button 256-2, and a direction selection button 256-3 are displayed.

[0281] The level correction button 256-1 is a button used to select the level of the arm retouch. In the example of FIG. 37, it is possible to select a preferred level from among five levels. That is, at the time of retouching using the retouch screen of FIG. 3, it is possible to select a preferred level from among three levels as the level of the arm retouch, whereas at the time of correction, it is possible to select a preferred level from among the finer five levels.

[0282] The correction part selection button 256-2 is a button used to select the part to correct the arm retouch. It is possible to select the part to be corrected from among seven types: the upper arm parts (both arms) of the left and right arms, the forearm parts (below the elbows) of the left and right arms, the upper arm part of the left arm, the upper arm part of the right arm, the forearm part of the left arm, the forearm part of the right arm, and the whole. That is, at the time of retouching using the retouch screen of FIG. 3, the user cannot select the part to be the target of the image processing of the arm retouch, whereas at the time of correction, it is possible to select.

[0283] The direction selection button 256-3 is a button used to select the part (direction) to correct the arm retouch. It is possible to select the correction direction from among three types: outside, inside, and both sides of the part selected using the correction part selection button 256-2. That is, at the time of retouching using the retouch screen of FIG. 3, only the outside of the arm is the target of the image processing of the arm retouch, whereas at the time of correction, it is possible to select the part to be the target of the image processing.

[0284] By using the display of the retouch correction area 256, finer adjustment of the arm thickness can be made compared to during retouch. For example, by using the correction site selection button 256-2 and the direction selection button 256-3, it becomes possible to select the site and direction for arm retouch from the outside of the upper arm, the inside of the upper arm, the outside of the forearm, and the inside of the forearm of the subject person's arm.

[0285] In the skeleton point correction area 257, a movement button 257-1, an addition button 257-2, and a deletion button 257-3 are displayed. Also displayed in the skeleton point correction area 257 are buttons operated when moving the display range of the display area 251, buttons operated when returning the state of the skeleton point to the state before correction, and the like.

[0286] The movement button 257-1 is a button operated when moving a skeleton point. For example, after operating the movement button 257-1, by pressing and sliding an arbitrary dot on the display area 251, it becomes possible to move the position of the skeleton point.

[0287] The addition button 257-2 is a button operated when adding a skeleton point. For example, after operating the addition button 257-2, by pressing an arbitrary position on the display area 251, it becomes possible to add a skeleton point at an arbitrary position.

[0288] The deletion button 257-3 is a button operated when deleting a skeleton point. For example, after operating the deletion button 257-3, by pressing an arbitrary dot on the display area 251, it becomes possible to delete a skeleton point at an arbitrary position.

[0289] By using the display of the skeleton point correction area 257, it becomes possible to correct the error in skeleton estimation and accordingly correct the arm retouch.

[0290] On the right side of the display area 251, various buttons such as an on / off button 258 that is operated when switching the display of the skeleton points on and off, and a determination result display button 259 that is operated when displaying the determination result of the captured image to be corrected are displayed. For example, when the determination result display button 259 is operated, the attribute information recorded in the management file is listed and displayed on the retouch correction screen. In a state where the attribute information is listed and displayed, it is possible to check and correct the determination results of various attributes.

[0291] In this way, using the retouch correction screen, it becomes possible to adjust the state of the arm retouch in finer detail than during retouch. The same screen can be used to adjust more finely than during retouch even when correcting retouches using functions other than arm retouch.

[0292] Figure 39 is a diagram showing an example of mounting the retouch correction function.

[0293] As shown in Figure 39, a program module having the function of the correction application 201A may be provided in the retouch application 1A, and the retouch correction may be performed by the retouch application 1A. In this case, the correction using the screen as shown in Figure 37 is performed as a function of the retouch application 1A. A user who has taken a photo in the photo studio can select a part of the arm to be retouched and correct the skeleton points on the retouch application 1A by himself / herself.

[0294] <<Modification Example>> Each of the above-described functions including the function of arm retouch and the function of image processing using the result of clothing determination may be realized on a server on the Internet. In this case, the user who has finished taking a photo operates the browser of the retouch PC 1 to access the server, and performs the above-described retouch on the retouch screen displayed by the browser. In this way, an information processing apparatus having each of the above-described functions may be realized by a server on the Internet.

[0295] ·Regarding the program The above-described series of processes can be executed by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware, or in a general-purpose personal computer or the like.

[0296] The program to be installed is provided by being recorded on a removable medium 111 shown in FIG. 33, which includes an optical disk (such as a CD-ROM (Compact Disc-Read Only Memory) or a DVD (Digital Versatile Disc)) or a semiconductor memory. Further, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in the ROM 102 or the recording unit 108.

[0297] The program executed by the computer may be a program in which processing is performed in time series in accordance with the order described in this specification, or a program in which processing is performed in parallel or at a necessary timing such as when a call is made.

[0298] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0299] The embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

[0300] For example, the present technology can adopt a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0301] Also, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0302] Furthermore, when a single step includes a plurality of processes, the plurality of processes included in that single step can be executed by a single device or can be executed in a shared manner by a plurality of devices.

Explanation of Signs

[0303] 1 PC for reattachment, 2 Image management server, 3 Mobile terminal, 11 Display, 121 Information processing unit, 131 Image acquisition unit, 132 Display control unit, 133 Person recognition / attribute determination unit, 134 Reattachment processing unit, 135 File management unit, 151 Face recognition unit, 152 Face feature determination unit, 153 Skeleton estimation unit, 154 Clothing determination unit, 155 Determination unit

Claims

1. A program that causes a computer to determine the type of clothing worn by a person shown in a captured image to be retouched, and switch the image processing performed on the person as the subject according to the type of clothing worn by the person as the subject. The program causes the above processing to be executed.

2. According to the type of clothing worn by the person as the subject, switch whether to perform the image processing according to the user's retouch operation or not. The program according to claim 1.

3. When an arm retouch operation is performed to thin the arm of the person as the subject as the user's retouch operation, when the clothing of the person as the subject is Western-style clothing, perform the image processing of the arm retouch, and when the clothing of the person as the subject is Japanese-style clothing, do not perform the image processing of the arm retouch. The program according to claim 2.

4. According to the type of clothing worn by the person as the subject, switch the content of the image processing according to the user's retouch operation. The program according to claim 1.

5. When an arm retouch operation is performed to thin the arm of the person as the subject as the user's retouch operation, when the clothing of the person as the subject is Western-style clothing, perform the image processing of the arm retouch, and when the clothing of the person as the subject is Japanese-style clothing, perform the image processing of thinning other parts together with the arm. The program according to claim 4.

6. When an operation is performed to deform a predetermined part of the person as the subject as the user's retouch operation, switch the part to be deformed according to the type of clothing worn by the person as the subject. The program according to claim 1.

7. Switch the parts to be thinned between the case where the clothing of the person as the subject is Western-style clothing and the case where it is Japanese-style clothing. The program according to claim 6.

8. Switch the parts to be lengthened between the case where the clothing of the person as the subject is Western-style clothing and the case where it is Japanese-style clothing. The program according to claim 6.

9. Determine the gender of the person as the subject, and when a Japanese-style male person and a Japanese-style female person are shown, perform reduction processing as the image processing on the female person or perform enlargement processing on the male person. The program according to claim 1.

10. Detect the exposed range of the skin of the person as the subject, and switch the image processing according to the width of the exposed range of the skin together with the type of clothing. The program according to claim 1.

11. Reflect the content of the image processing for the person shown in the retouched captured image onto a person wearing the same type of clothing as the clothing of the original person shown in the captured image without retouching. The program according to claim 1.

12. Determine the gender of the person of the subject, Select the person who is the source of the content of the image processing and the person who is the destination based on the determination result of the gender together with the type of clothing. The program according to claim 11.

13. Determine the shooting scene of the captured image based on the type of clothing, Reflect the content of the image processing for the person shown in the retouched captured image onto a person shown in the captured image without retouching in the same shooting scene as the retouched captured image. The program according to claim 1.

14. An information processing apparatus, Determine the type of clothing of the person shown in the captured image to be retouched, Switch the image processing performed on the person of the subject according to the type of clothing of the person of the subject. An information processing method.

15. A clothing determination unit that determines the type of clothing of the person shown in the captured image to be retouched, and A retouch processing unit that switches the image processing performed on the person of the subject according to the type of clothing of the person of the subject An information processing apparatus comprising the same.

Images