Program, information processing method, and information processing apparatus

Abstract

To perform image processing appropriately for visually slimming the arms of a person appearing in a captured image.SOLUTION: An information processing apparatus is configured to: select a part of an arm to be subjected to arm retouching for visually slimming arms of a person appearing in a captured image through image processing, in accordance with how the arm of a subject person appears; and perform image processing for arm retouching on the selected part of the arm, in accordance with a level selected by a user. This technology is applicable to a computer installed in a photo studio.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] This technology particularly relates to a program, an information processing method, and an information processing apparatus that are capable of appropriately performing image processing for thinning the arms of a person shown in a photographed image.

Background Art

[0002] As a service for commemorative photography of events such as coming-of-age ceremonies and weddings, there is a service that allows users to retouch images taken by a cameraman at a photography studio on their own (Patent Document 1). After taking the photos, the user will perform retouching using a PC or the like prepared in another space of the photography studio.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In wedding commemorative photography, the bride often wears a dress with exposed arms or a dress with see-through lace where the arms can be seen, and there is a particular need for women to show their arms beautifully.

[0005] This technology has been made in view of such a situation, and aims to be able to appropriately perform image processing for thinning the arms of a person shown in a photographed image.

Means for Solving the Problems

[0006] The program according to one aspect of the present technology causes a computer to execute a process of selecting, according to the way the arm of the subject person appears in the photographed image, a part of the arm that is the target of arm retouching to thin the arm of the person appearing in the photographed image by image processing, and performing the image processing of the arm retouching according to the level selected by the user on the selected part of the arm.

[0007] In one aspect of the present technology, a part of the arm that is the target of arm retouching to thin the arm of the person appearing in the photographed image by image processing is selected according to the way the arm of the subject person appears in the photographed image, and the image processing of the arm retouching according to the level selected by the user is performed on the selected part of the arm.

Advantages of the Invention

[0008] According to the present technology, it is possible to appropriately perform image processing for thinning the arm of the person appearing in the photographed image.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Embodiments for Carrying Out the Invention

[0010] <<Flow of Work in a Photography Studio>> FIG. 1 is a diagram showing the flow of work in a photography studio.

[0011] This technology is used, for example, in a photography studio for wedding commemorative photography. This technology is also applicable when taking commemorative photos of various events such as school entrance / graduation commemorative photos, coming-of-age ceremony commemorative photos, and Shichi-Go-San commemorative photos.

[0012] As shown on the left side of FIG. 1, photography in a photography studio is carried out by a cameraman. In the example of FIG. 1, the bride and groom standing in front of the background screen are being photographed as the subjects. The groom is in a tuxedo and the bride is in a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photography by the cameraman is arranged. Photography of the bride and groom etc. may also be carried out outside the photography studio.

[0013] Shooting by a cameraman is carried out by changing the composition and shooting a large number of still images (photos) such as 50 or 100 images. The composition of the still image is determined by the poses of the bridegroom and bride, the positional relationship between the bridegroom, bride and the cameraman, and camera parameters such as the focal length. The captured image, which is a still image captured by the cameraman, is transferred to a PC (not shown) connected to the camera and saved.

[0014] After the shooting is completed, the bridegroom and bride move to the retouch space prepared in the shooting studio and use the retouch PC1 shown on the right side of FIG. 1 to check the captured images taken by the cameraman.

[0015] Also, as users of the retouch PC1, the bridegroom and bride select, for example, the number of captured images to be finally printed and adjust the appearance of each captured image by retouching themselves. Retouching is performed on a part of the captured images selected from the captured images taken by the cameraman. For example, a predetermined number of captured images such as 5 or 10 are selected as the images to be retouched according to the plan applied by the user. Retouching (processing) includes various image processes for adjusting the appearance of the captured images. A screen used for selecting the number of images to be retouched may be displayed in front of the image selection screen. Any number can be selected as the image to be retouched.

[0016] One or more retouch PCs 1 are prepared in the retouch space. The captured images saved in the storage PC are transferred to the retouch PC1 and displayed on the display 11 of the retouch PC1. An application for retouching the captured images taken by the cameraman, i.e., the retouch app 1A, is installed on the retouch PC1.

[0017] In this way, in the photography studio where the retouching PC 1 applying this technology is installed, the retouching of the photographed images taken by the cameraman is performed by the users themselves (the bridegroom and the bride). The users can use the functions of the retouching app 1A to retouch the photographed images so that they look as they like. The retouching app 1A may be installed on a terminal different from a PC, such as a tablet terminal, and used for retouching.

[0018] Figure 2 is a diagram showing an example of providing photographed images.

[0019] The photographed images retouched by the retouching app 1A are printed by the operating company of the photography studio and handed over to the users at a later date in the form of, for example, an album as indicated by arrow #1.

[0020] In addition, the retouched photographed images are uploaded from the retouching PC 1 to the image management server 2 and managed by the image management server 2 as indicated by arrow #2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the operating company of the photography studio or by a business operator different from the operating company of the photography studio.

[0021] The users can operate their own mobile terminal 3 such as a smartphone and operate the image management app 3A to view or download the photographed images in which they are the subjects, as indicated by arrow #3. The image management app 3A is an application for viewing the photographed images managed by the image management server 2. The image management app 3A is provided, for example, by the business operator managing the image management server 2.

[0022] The provision of the photographed images taken in the photography studio is carried out in the form of an album and, as appropriate, in the form of data. The provision of the photographed images in data form may be possible at a separate charge from the charge for the album.

[0023] The provision of the photographed images in the data is performed such that not only the photographed images retouched by the user but also the photographed images not selected by the user as retouch targets are provided. For the photographed images not selected by the user as retouch targets, as will be described later, retouching is automatically performed by the retouch app 1A without depending on the user's operation.

[0024] The user can also receive, in the form of data, the photographed images not selected as retouch targets. Instead of using the image management app 3A, the user may access a predetermined site using a browser to receive the photographed images managed in the image management server 2. For example, by registering as a member of the image providing service provided by the administrator of the image management server 2, the user can receive the photographed images managed in the image management server 2 using the image management app 3A or using a browser.

[0025] Instead of the mobile terminal 3, other terminals such as a tablet terminal or a PC may be used to receive the photographed images. When the provision of the photographed images is performed in data, together with the photographed images which are still images, data of moving images capturing the state of shooting in the shooting studio may be provided.

[0026] <<Regarding the retouch app>> <Screen display> · Retouch screen FIG. 3 is a diagram showing an example of the retouch screen.

[0027] Above the center of the retouch screen, which is a screen used for retouching photographed images, a display area 31 is formed. The display area 31 is an area where one photographed image to be retouched is enlarged and displayed. In the thumbnail area 32 formed on the left side of the display area 31, a thumbnail image of the photographed image selected by the user is displayed. In the example of FIG. 3, thumbnail images of five photographed images P1 to P5 are displayed. For example, the photographed image P1 selected using the thumbnail image is enlarged and displayed in the display area 31.

[0028] Below the retouch screen, a horizontally long retouch selection area 33 is formed. Using the buttons displayed in the retouch selection area 33, the content of retouching is selected. Each time the content of retouching is selected using the buttons displayed in the retouch selection area 33, the selected content is reflected in the display of the display area 31. The user will select the content of retouching to be applied to the photographed image to be retouched using the display of the retouch selection area 33 while looking at the display of the display area 31.

[0029] FIG. 4 is a diagram showing an enlarged view of the display of the retouch selection area 33.

[0030] The retouch selection area 33 is formed from an area 33-1 which is a display area for information used for adjusting the whole face and body, an area 33-2 which is a display area for information used for adjusting the eyes and makeup, and an area 33-3 which is a display area for information used for adjusting brightness.

[0031] In the area 33-1 formed on the left side of the retouch selection area 33, adjustment buttons 51 used for adjusting the small face feeling, adjustment buttons 52 used for adjusting the shape of the face, adjustment buttons 53 used for adjusting the skin texture, adjustment buttons 54 used for adjusting the thinness of the arms, and adjustment buttons 55 used for adjusting the style (styling up) are displayed.

[0032] The adjustment button 51 has five levels (degrees of adjustment) for a small face effect. "0" for the small face effect means keeping the original face size without changing it. Also, "1" for the small face effect means reducing the face size by one level based on the parameters corresponding to "1". By selecting any level from "1" to "4", the user can adjust the size of their own face shown in the captured image displayed in the display area 31.

[0033] The adjustment button 52 has "round", "egg-shaped", and "slim" options for the face shape. By selecting any shape, the user can adjust the shape of their own face shown in the captured image displayed in the display area 31. It is also possible to keep the face shape unchanged by selecting "no retouch".

[0034] The adjustment button 53 has five levels for the skin texture. "0" for the skin texture means keeping the original skin texture without changing it. Also, "1" for the skin texture means increasing the skin texture level by one level based on the parameters corresponding to "1". By selecting any level from "1" to "4", the user can adjust the smoothness degree of their own skin texture shown in the captured image displayed in the display area 31.

[0035] The adjustment button 54 has three levels for the thinness of the arms. "0" for the arm thinness means no processing, that is, keeping the original thinness without changing it. Also, "weak" for the arm thinness means performing processing based on the parameters corresponding to "weak" and making the arm one level thinner. "Strong" for the arm thinness means performing processing based on the parameters corresponding to "strong" and making the arm even one level thinner than in the "weak" case. By selecting any level, the user can adjust the thinness of their own arms shown in the captured image displayed in the display area 31. Details of the arm retouch, which is a function for adjusting the arm thinness, will be described later.

[0036] As styling, three levels are provided. "0" for styling means no processing, that is, keeping the original style without changing the style. Also, "weak" for styling means performing processing such as making the legs longer based on the parameters corresponding to "weak", and improving the balance of the whole body by one level. By selecting any level, the user can adjust their own style shown in the captured image displayed in the display area 31.

[0037] In the area 33-2 formed at the center of the retouch selection area 33, adjustment buttons 56 used for adjusting the eye size, adjustment buttons 57 used for adjusting the eye shape, adjustment buttons 58 used for adjusting the makeup density, adjustment buttons 59 used for adjusting the nasolabial fold, and adjustment buttons 60 used for adjusting beard removal are displayed.

[0038] As for the eye size, five levels are provided. "0" for the eye size means no processing, that is, keeping the original size without changing the eye size. Also, "1" for the eye size means increasing the eye size by one level based on the parameters corresponding to "1". By selecting any level from "1" to "4", the user can adjust their own eye size shown in the captured image displayed in the display area 31.

[0039] As for the eye shape, "round eyes", "downward-slanting eyes", and "cat eyes" are provided. By selecting any shape, the user can adjust their own eye shape shown in the captured image displayed in the display area 31. By selecting "no processing", it is also possible not to change the eye shape.

[0040] As the makeup intensity, five levels are provided. "0" for the makeup intensity indicates keeping the original intensity without changing the makeup intensity. Also, "1" for the makeup intensity means increasing the makeup intensity by one level based on the parameter corresponding to "1". By selecting any level from "1" to "4", the user can adjust the makeup intensity of themselves shown in the captured image displayed in the display area 31.

[0041] As for the nasolabial folds, it is possible to select on / off. By selecting on or off, the user can choose to emphasize the highlighted part of their nasolabial folds shown in the captured image displayed in the display area 31 or keep it in the original state.

[0042] As for beard removal, three levels are provided. "0" for beard removal indicates keeping the original beard without changing it. Also, "weak" for beard removal means performing processing based on the parameter corresponding to "weak" and thinning the beard by one level. By selecting any level, the user can adjust the beard intensity of themselves shown in the captured image displayed in the display area 31. The beard removal function is effective when the person to be retouched is the groom.

[0043] In the area 33-3 formed on the right side of the retouch selection area 33, adjustment buttons 61 used for adjusting the skin brightness, adjustment buttons 62 used for adjusting the overall brightness, and adjustment buttons 63 used for adjusting the brightness of the background only are displayed.

[0044] As for the skin brightness, three levels are provided. "0" for the skin brightness indicates keeping the original brightness without changing the skin brightness. Also, "1" for the skin brightness means increasing the skin brightness level by one level based on the parameter corresponding to "1". By selecting any level, the user can adjust the skin brightness of themselves shown in the captured image displayed in the display area 31.

[0045] As the overall brightness, three levels are provided. "0" for the overall brightness indicates that the overall brightness remains unchanged and is the original brightness. Also, "1" for the overall brightness indicates that the level of the overall brightness is increased by one level based on the parameter corresponding to "1". By selecting any level, the user can adjust the overall brightness shown in the captured image displayed in the display area 31.

[0046] As the background brightness, three levels are provided. "0" for the background brightness indicates that the background brightness remains unchanged and is the original brightness. Also, "1" for the background brightness indicates that the level of the background brightness is increased by one level based on the parameter corresponding to "1". By selecting any level, the user can adjust the background brightness shown in the captured image displayed in the display area 31.

[0047] Returning to the description of FIG. 3, a batch retouch button 34 used for batch retouch is displayed on the left side of the display area 31. Batch retouch is a function for collectively selecting combinations of various retouch contents as described above.

[0048] In the example of FIG. 3, any level can be selected from three types of batch retouch levels with different combinations of retouch content levels. By using the batch retouch function, the user can select each retouch content in one operation without using the display in the retouch selection area 33.

[0049] A pre - processing button 35 is displayed on the right side of the display area 31. The pre - processing button 35 is a button operated when switching the display in the display area 31 to the state before retouch. While the pre - processing button 35 is pressed, the captured image in the state before retouch is displayed in the display area 31. By using the pre - processing button 35, the user can compare the state after retouch with the state before retouch and check whether the degree of retouch is too strong.

[0050] Below the processing button 35, a person selection button 36 used to select a person to be retouched is displayed. By using the person selection button 36, the user can select to retouch only the person shown on the left side of the captured image, only the person shown on the right side, or both of them. Face recognition or the like is performed on the captured image, and the person to be the subject of the retouch image processing is selected according to the user's operation.

[0051] The user can select the content of the retouch to be applied to the captured image displayed in the display area 31 by using such various functions. Thus, the retouch that can be performed using the retouch app 1A includes at least adjustment of the whole face and body, adjustment of the eyes and makeup, and adjustment of brightness.

[0052] The captured image to be retouched is switched using the thumbnail image, and the retouch of each of the captured images P1 to P5 is sequentially performed. When the retouch of each of the captured images P1 to P5 is completed and the end button displayed at the upper right end of the screen is operated, the retouch using the retouch screen is completed.

[0053] <<Overview of Arm Retouch>> FIG. 5 is a diagram showing an overview of arm retouch.

[0054] As shown by the arrow #11 in FIG. 5, skeleton estimation is performed on the person shown in each of the captured images selected as the retouch target. In FIG. 5, skeleton estimation of the bride's person shown in the captured image P5 is being performed. Skeleton estimation is a process of estimating the positions of the skeleton points of the person shown in the captured image. For example, skeleton estimation is performed using an inference model that takes the captured image as input and outputs information on the positions of the skeleton. The retouch app 1A is provided with an inference model for skeleton estimation. The skeleton estimation using the inference model may be performed on a server on the network.

[0055] In the example in the center of Fig. 5, the positions of the shoulders, elbows, and wrists of the bride's left and right arms are estimated. The small circular dots D1L, D2L, and D3L indicate the positions of the left shoulder, left elbow, and left wrist, respectively. Also, the dots D1R, D2R, and D3R indicate the positions of the right shoulder, right elbow, and right wrist, respectively. Note that the dots indicating the skeleton points are shown for the convenience of explanation and are not information actually displayed on the captured image. When the arm retouch function is selected, the dots indicating the skeleton points may be displayed on the captured image.

[0056] When the level of the arm thickness is selected, image processing is performed to thin the arm as shown at the tip of arrow #12. The captured image P5 shown on the right side of Fig. 5 is an image after the image processing of arm retouch. Compared with the image before the image processing, the bride's both arms are slightly thinner. Thus, the image processing of arm retouch by the retouch app 1A is performed based on the information of the skeleton points which is the result of skeleton estimation.

[0057] Fig. 6 is a diagram showing an example of the image processing of arm retouch. In Fig. 6, the vicinity of the bride's right arm in the captured image P5 shown in Fig. 5 is enlarged and shown.

[0058] As shown on the left side of Fig. 6, the area between dot D1R and dot D2R is recognized as the upper arm, and the area between dot D2R and dot D3R is recognized as the forearm. As shown by the small dots, the outer edges of the upper arm and the outer edge of the forearm are detected, and as shown by the arrows on the right side of Fig. 6, image processing is performed to move each position of the edge in the direction of the center of the arm.

[0059] On the right side of Fig. 6, image processing is performed to slightly move the outer edge of the upper arm in the direction of the center of the arm (slightly to the right) represented by the straight line connecting dot D1R and dot D2R. Also, image processing is performed to slightly move the outer edge of the forearm in the direction of the center of the arm (slightly to the upper right) represented by the straight line connecting dot D2R and dot D3R.

[0060] Similar image processing is performed on the left arm of the bride. That is, image processing is performed to move the outer edge of the upper arm part in the central direction of the arm represented by the straight line connecting dot D1L and dot D2L. Also, image processing is performed to move the outer edge of the forearm part in the central direction of the arm represented by the straight line connecting dot D2L and dot D3L.

[0061] The amount of deformation of the outer edge of the arm is determined according to the level selected by the user using the adjustment button 54 and the like. The amount of deformation is a retouch parameter that defines the amount of movement of the edge position.

[0062] In this way, the image processing for arm retouch is performed by deforming only the outer sides of the upper arm part and the forearm part, and no image processing is performed on the inner side of the arm closer to the body. Instead of deforming both the outer and inner sides of the arm, the image processing for arm retouch is performed by deforming only the outer side as a part of the arm.

[0063] When the subject person is facing the camera frontally and the left arm shown on the right side in the captured image is the object of image processing, the edge part of the left arm closer to the right end of the image is the outer side of the arm. Also, when the right arm shown on the left side in the captured image is the object of image processing, the edge part of the right arm closer to the left end of the image is the outer side of the arm.

[0064] Conversely, when the subject person is facing the camera with their back and the right arm shown on the right side in the captured image is the object of image processing, the edge part of the right arm closer to the right end of the image is the outer side of the arm. Also, when the left arm shown on the left side in the captured image is the object of image processing, the edge part of the left arm closer to the left end of the image is the outer side of the arm.

[0065] This makes it possible to prevent the body from being deformed as if dragged by the deformation of the arm, resulting in an unnatural appearance. For example, when the outer edge of the upper arm of the right arm is moved to the right, in the portion shown in color in FIG. 7 between the outer line of the upper arm before image processing and the outer line of the upper arm after image processing, image processing is performed to interpolate with the pixels on the outer side of the arm (background pixels in the example of FIG. 7). If the same image processing is performed on the inner side of the arm, the processing is to interpolate the portion between the inner line of the upper arm before image processing and the inner line of the upper arm after image processing with the pixels where the body is shown or the pixels near the body, and the body may be deformed as the arm becomes thinner. By deforming only the outer side of the arm, it is possible to prevent such an unnatural appearance.

[0066] In wedding commemorative photography, the bride often wears a dress with the arms exposed or a dress through which the arms can be seen through the lace, and there is a particular need for women to show their arms beautifully. By enabling the arms to be processed to appear thinner, it becomes possible to meet such needs.

[0067] FIG. 8 is a diagram showing an example of arm retouching when the bride's clothing is a kimono.

[0068] As shown in FIG. 8, it is possible to utilize the arm retouching function even when the bride wearing a haori is shown. Even when the arms are not visible while wearing a haori, an inference model capable of estimating the skeletal points is prepared in the retouch app 1A.

[0069] Also in the example of FIG. 8, the positions of the shoulders indicated by the dots D1L and D1R of the bride's left and right arms, the positions of the elbows indicated by the dots D2L and D2R, and the positions of the wrists indicated by the dots D3L and D3R are estimated, and the arms are made thinner by deforming the outer sides of the left upper arm and the left forearm, and the outer sides of the right upper arm and the right forearm, respectively. The outer edge of the clothing covering the upper arm and the outer edge of the clothing covering the forearm are detected, and image processing is performed so as to move each position of the edge in the direction of the center of the arm.

[0070] When wearing an uchikake, the body can appear larger depending on the direction of the body, but by using the arm retouching function, it is possible to prevent this appearance. Details of arm retouching will be described later.

[0071] <Reflect all numbers> FIG. 9 is a diagram showing an example of application of retouching parameters.

[0072] As shown in Fig. 9, the retouching contents applied to each of the captured images P1 to P5 by the user's operation are reflected in all unselected images. An unselected image is a captured image that has not been selected as a target for retouching. A captured image selected as a target for retouching on the image selection screen becomes a selected image. For example, when 5 captured images are selected from 100 captured images, the remaining 95 captured images become unselected images.

[0073] That is, the number of photographed images selected as retouching targets according to the plan (selected images) are retouched according to the content directly selected by the user using the retouching screen, whereas the unselected images not selected as retouching targets are retouched so that the retouching content selected using the selected images is automatically reflected. For example, the retouching content of the selected image is reflected on all unselected images.

[0074] In this way, the retouching application 1A is provided with a full reflection function that reflects the retouching content of a selected image to unselected images after the user has retouched the selected image.

[0075] When photographing an event such as a wedding, many shots are usually taken, such as hundreds of shots. It is a large burden for the user to manually retouch all the images taken by the photographer. By applying the retouching content of the selected image to all unselected images, the user can receive all the images taken in a state adjusted to the user's preference.

[0076] Which retouch content of which captured image among the retouched captured images P1 to P5 is reflected in which unselected image is selected according to the respective features of the selected image and the unselected image, such as whether the size of the person's face is shown to be about the same size. For example, in a certain unselected image, the retouch content of the selected image in which the face of the same person as the person shown in the unselected image is shown to be about the same size is reflected.

[0077] In this case, features such as the number of people and the size of the face shown in each captured image (selected image and unselected image) are recognized. For each unselected image, the retouch content of the selected image having the same features is reflected.

[0078] The retouch app 1A is also equipped with a function to recognize such features of the captured images. Not only the size but also other face states such as the orientation of the face may be recognized. Also, other types of features such as the gender and age of the person, the type of clothing the person is wearing, and the shooting scene are recognized.

[0079] <<Processing of Retouch App 1A>> <Overall Processing> With reference to the flowchart of FIG. 10, the processing of the retouch app 1A when providing a service to a certain set of users will be described. The processing shown in FIG. 10 starts when the shooting by the cameraman is completed and the user starts the retouch work. At the start of the retouch work, the folder in which the captured images are saved is specified by the staff of the shooting studio or the like.

[0080] In step S1, the retouch app 1A starts importing the captured images in the specified folder. Attribute determination processing is sequentially performed on the captured images imported from the folder. Details of the captured image import processing will be described later with reference to the flowchart of FIG. 11.

[0081] When the import and attribute determination of all captured images are completed, in step S2, the retouch app 1A displays an image selection screen.

[0082] In step S3, the retouch app 1A accepts the user's operation on the image selection screen and selects the captured image to be retouched. A predetermined number of captured images, such as 5 or 10, are selected as the retouch targets.

[0083] When the selection of the captured image to be retouched is completed, in step S4, the retouch app 1A displays a retouch screen.

[0084] In step S5, the retouch app 1A accepts the user's operation on the retouch screen and performs retouching.

[0085] In step S6, the retouch app 1A determines whether the retouching of all captured images is completed. If it is determined in step S6 that the retouching of all captured images is not completed, the process returns to step S4 and the above-described process is repeated.

[0086] If it is determined in step S6 that the retouching of all captured images is completed, in step S7, the retouch app 1A performs full reflection. When the reflection of the retouch content for all unselected images is completed, the process in FIG. 10 ends.

[0087] <Details of the captured image import process> With reference to the flowchart in FIG. 11, the details of the captured image import process will be described.

[0088] In step S11, the retouch app 1A displays a function on / off setting screen. The function on / off setting screen is a screen used for setting the validity / invalidity of the wrist retouch function. The setting using the function on / off setting screen is performed, for example, by the staff of the photo studio.

[0089] FIG. 12 is a diagram showing an example of a function on / off setting screen.

[0090] In the example of FIG. 12, a message is displayed asking whether to use the wrist touch function, and below it, buttons to be operated when using the wrist touch function and buttons to be operated when not using it are displayed. The staff of the photography studio will set whether to use the wrist touch function according to the user's plan or the like.

[0091] When the plan used by the user is a plan to take commemorative wedding photos, since the user is wearing clothing such as a dress with a lot of arm exposure, the wrist touch function becomes important. On the other hand, depending on the plan used by the user, such as a plan for commemorative photos of an adult ceremony taken with clothing with little arm exposure, the wrist touch function may not be important. By enabling the selection of whether to use the wrist touch function, it is possible to shorten the time required for attribute determination in the case of not using the wrist touch function.

[0092] When using the wrist touch function, skeletal estimation of the people shown in each captured image is performed. When not using the wrist touch function, by omitting the skeletal estimation process as described later, the time required for attribute determination can be shortened by the amount of time required for the skeletal estimation process.

[0093] The function on / off setting screen of FIG. 12 is displayed when the staff of the photography studio sets to select whether to use the wrist touch function every time a captured image is imported. For example, it is possible to set from a predetermined menu screen whether to always use the wrist touch function, always not use it, or select whether to use it every time a captured image is imported. When set to always use the wrist touch function, or when set to always not use the wrist touch function, the function on / off setting screen is not displayed.

[0094] For a studio where many users utilize wedding commemorative photo plans, or a studio where many users utilize coming-of-age ceremony commemorative photo plans, the customer base varies for each photography studio. By enabling settings according to user trends to determine whether function on / off setting screens are necessary, it becomes possible to improve the convenience for the staff of the photography studio.

[0095] Returning to the description of FIG. 11, in step S12, the retouch app 1A determines whether to use the function of arm retouch. If it is selected to use the function of arm retouch using the function on / off setting screen, or if it is set to always use the function of arm retouch using the menu screen, it is determined that the function of arm retouch is to be used.

[0096] If it is determined in step S12 to use the function of arm retouch, the process proceeds to step S13. The attribute determination processes in steps S13 to S16 are performed for each captured image imported from the folder.

[0097] In step S13, the retouch app 1A performs face recognition and face feature determination. Face recognition and face feature determination are processes of recognizing the face of a person shown in the captured image and determining the features of the recognized face for each face. Each process of face recognition and face feature determination is performed, for example, by image processing or using an inference model generated by machine learning. Note that the determination of attributes using the inference model is synonymous with the inference of attributes.

[0098] The retouch app 1A is provided with an inference model for face recognition that takes the captured image as input and outputs information on the position of the face shown in the captured image. Also, the retouch app 1A is provided with an inference model for face feature determination that takes as input a face image that is an image of the face recognized by face recognition and outputs information on the features of the face. The features of the face include information such as the positions of each part such as eyes, nose, and mouth, the contour points of the face, the orientation of the face, and the size of the face. For each person whose face is recognized, attribute determination after face feature determination is performed.

[0099] In step S14, the retouch application 1A performs skeleton estimation. Skeleton estimation is a process of estimating the positions of the skeleton points of a person shown in a captured image. The skeleton estimation is performed using, for example, an inference model generated by machine learning. The retouch application 1A is provided with an inference model for skeleton estimation that takes as input a person image, which is an image of a person shown in the captured image, and outputs information on the positions of the skeleton points. The skeleton estimation may be performed by image processing.

[0100] FIG. 13 is a diagram showing an example of skeleton points.

[0101] As shown by the dots in FIG. 13, for example, the positions of the skeleton points of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles are estimated. Based on the positions of the skeleton points, attributes such as gender and body orientation are also estimated.

[0102] In step S15 of FIG. 11, the retouch application 1A performs clothing determination. Clothing determination is a process of estimating the type of clothing worn by a person shown in a captured image. The clothing determination is performed using, for example, an inference model generated by machine learning. The retouch application 1A is provided with an inference model for clothing determination that takes as input a person image and outputs information on the type of clothing. The clothing determination may be performed by image processing.

[0103] In step S16, the retouch application 1A determines various attributes of the person shown in the captured image, such as gender, age, and the exposed range of the skin. The gender and age are determined based on, for example, the features of the face. The retouch application 1A is provided with an inference model that takes as input a face image or face features and outputs information on gender and age. An inference model for gender determination and an inference model for age determination may be provided separately. The determination of gender and age may be performed by image processing.

[0104] The exposed area of the skin is determined, for example, by detecting the skin color area through image processing. The skin color area is detected as the exposed area of the skin. A inference model that takes a person image as input and outputs information on the skin area may be prepared in the retouch app 1A. As other attributes, the type of object overlapping the person in the captured image, the shooting scene of the captured image, etc. may be determined.

[0105] In this way, in the retouch app 1A, various attributes of the captured image and various attributes of the person shown in the captured image are determined using machine learning. Based on these attributes determined at the time of importing the captured image, image processing using each retouch function is performed.

[0106] In step S17, the retouch app 1A saves the results of various attribute determinations such as the determination results of face recognition / face features, the results of skeleton estimation, the results of clothing determination, gender, age, and the determination results of the exposed area of the skin. The results of the attribute determinations are recorded and saved in the management file of each captured image.

[0107] FIG. 14 is a diagram showing an example of a management file.

[0108] As shown in FIG. 14, the management file includes information on the image size, selection flag, and retouched flag. The selection flag is a flag indicating whether the captured image is selected as the selected image. The retouched flag is a flag indicating whether the captured image has been retouched.

[0109] In addition, a face ID is recorded in the management file, and in association with each face ID, retouch parameters, gender (gender determined based on facial features), age, face size, contour points, face orientation, skeleton point information, gender (gender determined based on skeleton points), body orientation, clothing type, and information on the exposed area of the skin are recorded.

[0110] The management file records not only retouch parameters indicating the content of retouching for individual persons but also retouch parameters indicating the content of retouching for the entire image (parameters such as the brightness of the image), information on the shooting scene, and the like. The gender of a person may be determined by combining the determination result based on facial features and the determination result based on skeleton points, or may be determined based on only facial features or only skeleton points.

[0111] In step S18 of FIG. 11, the retouch application 1A determines whether the processing of all captured images has been completed. If it is determined in step S18 that the processing of all captured images has not been completed, the process returns to step S13, and the above-described processing is repeatedly performed.

[0112] On the other hand, if it is determined in step S12 that the arm retouch function is not used, the process proceeds to step S19. The processing after step S19 is the same as the processing after step S13, except that skeleton estimation is omitted. Redundant explanations will be omitted as appropriate.

[0113] That is, the retouch application 1A performs face recognition and face feature determination in step S19 and clothing determination in step S20. Also, the retouch application 1A determines other attributes such as the gender, age, and exposure range of the person shown in the captured image in step S21.

[0114] In step S22, the retouch application 1A saves the results of various attribute determinations such as the determination result of face recognition and face features, the result of clothing determination, the determination results of gender, age, and the skin exposure range.

[0115] In step S23, the retouch application 1A determines whether the processing of all captured images has been completed. If it is determined in step S23 that the processing of all captured images has not been completed, the process returns to step S19, and the above-described processing is repeatedly performed.

[0116] When it is determined in step S18 or step S23 that the processing of all captured images has been completed, the processing in FIG. 11 ends.

[0117] As described above, when the function of arm retouch is not used, skeleton estimation is omitted in the attribute determination at the time of capturing an image. By omitting the skeleton estimation process when the function of arm retouch is not used, it is possible to shorten the time required for attribute determination.

[0118] <<Details of Arm Retouch>> Here, the details of arm retouch will be described in the following order. Regarding the control of the image processing of arm retouch below, it is possible to use a combination of one or more. · Switching of a part to be the target of arm retouch · Method of deformation by arm retouch · Example of using the result of gender determination for arm retouch · Example of using the detection result of the exposed range of skin for arm retouch · Example of using the estimated result of age for arm retouch · Other examples of arm retouch

[0119] <Switching of a part to be the target of arm retouch> · Example 1 of the retouch target part FIG. 15 is a diagram showing an example of a part to be the target of arm retouch. Hereinafter, when a line of an arbitrary part of a person is shown in a thick line, it indicates that image processing for thinning is performed on that part.

[0120] As shown in FIG. 15A, when the subject person is photographed in a body orientation where the left half of the body is in the foreground and the right half of the body is in the background, the outer side of the upper arm and the outer side of the forearm of the left arm indicated by the thick line L1 are the retouch targets.

[0121] On the other hand, as shown in FIG. 15B, when the subject person is photographed in a body orientation where the right half of the body is in the foreground and the left half of the body is in the background, the outer side of the upper arm and the outer side of the forearm of the right arm indicated by the thick line L2 are the retouch targets.

[0122] Also, as described with reference to FIG. 5, when facing forward, the outer sides of both arms become the retouch targets as described above. The orientation of the body of the subject person is specified based on the information included in the management file. Thus, a part that becomes the target of the arm retouch image processing switches according to the appearance of the arms represented by the result of the skeleton estimation.

[0123] That is, as shown in FIG. 15, when one arm appears in front of the other arm, the image processing for arm retouch is performed on the one arm that appears in front. For example, with reference to the front, an orientation that becomes the threshold value of the body orientation is set. When the orientation of the subject's body faces outside the threshold orientation, the image processing for arm retouch is performed only on the one arm that appears in front, and when the orientation of the subject's body faces more forward than the threshold orientation, the image processing for arm retouch is performed on both arms.

[0124] As shown in FIG. 15, when only one arm appears and the other arm does not appear (is not visible), the image processing for arm retouch is performed only on the one visible arm. Even if the arm is in front, if it is hidden by another person's body or the like and not visible, the arm retouch for the one arm that appears in front is not performed.

[0125] ·Example 2 of the retouch target part FIG. 16 is a diagram showing another example of the part that becomes the target of arm retouch.

[0126] As shown in FIG. 16A, when the subject person is facing forward, the outer side of the left arm indicated by the thick line L11 and the outer side of the right arm indicated by the thick line L12 become the retouch targets.

[0127] On the other hand, as shown in FIG. 16B, when the subject person is facing forward and raising the left arm from the shoulder while lowering the right arm, for the right arm, the outer side of the right arm indicated by the thick line L12 becomes the retouch target as in the case of FIG. 16A. For the left arm, instead of the front side of the left arm indicated by the thick line L11, the back side of the left arm indicated by the thick line L21 that appears on the outside becomes the retouch target.

[0128] Even when the left arm is raised from the shoulder as shown in FIG. 16B, if the front side of the left arm (the portion indicated by the thick line L11) is deformed in the same manner as when it is lowered, the process involves interpolating the portion between the edge of the upper arm before image processing and the edge of the upper arm after image processing with pixels near the face, and the face may be deformed as the arm becomes thinner. When the arm is raised, by deforming only the back side portion away from the face, it is possible to prevent such unnatural appearances.

[0129] In this way, a part of the arm touch-up target switches depending on the angle of the arm. For example, with a horizontal angle as a reference (0 degrees), a threshold angle of the arm is set. When the angle of the subject's arm is smaller than the threshold angle, arm touch-up image processing is performed on the front side of the arm, and when the angle of the subject's arm is larger than the threshold angle, arm touch-up image processing is performed on the back side of the arm.

[0130] Even when the amount of deformation used for arm touch-up is switched as shown in FIG. 17 between the case of targeting the front side of the arm and the case of targeting the back side of the arm. Among the three levels prepared for the adjustment button 54 (FIG. 4), for example, when the "weak" level arm touch-up operation as shown in the balloon is performed, as shown on the right side of FIG. 17, arm touch-up image processing is performed using the amount of deformation corresponding to the "weak" level ("Arm touch-up: ON").

[0131] In the example of FIG. 17, even for the same upper arm portion, the amount of deformation used for arm touch-up image processing targeting the upper arm portion of the raised left arm is larger (more) than the amount of deformation used for arm touch-up image processing targeting the upper arm portion of the lowered right arm. On the right side of FIG. 17, the length of the arrow indicates the magnitude of the amount of deformation. The same applies to other figures. In this way, arm touch-up image processing is performed using different amounts of deformation according to the angle of the arm.

[0132] <Method of Deformation by Arm Touch-up> · Example of Deformation 1 FIG. 18 is a diagram showing an example of arm deformation.

[0133] As shown on the left side of FIG. 18, when the subject person is photographed in a body orientation where the left half of the body is forward and the right half of the body is rearward, the outer side of the upper arm of the left arm indicated by the thick line L31 and the outer side of the forearm of the left arm indicated by the thick line L32 are the retouch targets. In FIG. 18, unlike the case of FIG. 15, the portions that are the retouch targets are shown separately for the upper arm and the forearm.

[0134] The amount of deformation used in the arm retouch image processing may be switched as shown in FIG. 18 between the case where the outer side of the upper arm is the target and the case where the outer side of the forearm is the target. For example, when an arm retouch operation at the "weak" level as shown in the balloon is performed, as shown on the right side of FIG. 18, the arm retouch is performed using the amount of deformation corresponding to the "weak" level.

[0135] In the example of FIG. 18, the amount of deformation used in the arm retouch image processing for the upper arm of the left arm is larger than the amount of deformation used in the arm retouch image processing for the forearm of the left arm. Thus, the arm retouch image processing is performed using different amounts of deformation according to the part of the arm.

[0136] When image processing is performed using the same amount of deformation for each of the upper arm and the forearm, the balance between the upper arm and the forearm may deteriorate, but by changing the amount of deformation used for each image processing, it becomes possible to improve the balance of each part after the image processing. The amount of deformation used in the arm retouch image processing for the forearm may be made larger than the amount of deformation used in the arm retouch image processing for the upper arm.

[0137] · Example of Deformation 2 FIG. 19 is a diagram showing another example of arm deformation.

[0138] The bride's arm shown on the left side of Fig. 19A appears slightly thicker than the bride's arm shown on the left side of Fig. 19B. The outer side of the upper arm of the left arm indicated by the thick line L31 and the outer side of the forearm of the left arm indicated by the thick line L32 are the retouch targets.

[0139] For example, when an arm retouch operation at the "weak" level as shown in the balloon is performed, as shown on the right side of Fig. 19, arm retouch is performed using the deformation amount corresponding to the "weak" level. In the example of Fig. 19, image processing for arm retouch is performed using different deformation amounts according to the thickness of the arm, together with the level selected by the user. The thickness of the arm is calculated based on the distance at a predetermined position of the arm.

[0140] The deformation amount used for the image processing of arm retouch for the bride's left arm shown in Fig. 19A is larger than the deformation amount used for the image processing of arm retouch for the bride's left arm shown in Fig. 19B. Thus, even when the same level is selected as the retouch level using the adjustment button 54, image processing for arm retouch is performed using a larger deformation amount as the thickness of the arm before deformation is thicker. Conversely, image processing for arm retouch is performed using a smaller deformation amount as the thickness of the arm before deformation is thinner. This makes it possible to prevent the arm after image processing from becoming too thin.

[0141] In the example of Fig. 19, the deformation amount is changed according to the thickness of the arm, but the shape of the arm may be determined and the deformation amount may be changed according to the shape as well. For example, the type of the shape of the arm such as round or linear is determined, and the deformation amount is changed according to the type of the shape of the arm. The type of the shape of the arm is determined based on the thickness of each part of the arm, etc.

[0142] ·Example of Deformation 3 Image processing may be performed not only to deform the thickness of the arm but also to extend the length of the arm. The part to be extended can be switched according to the way the arm appears, such as only the upper arm or only the forearm.

[0143] <Example of Using the Result of Gender Judgment for Arm Retouch> In wedding commemorative photos, the shooting is carried out with a composition in which the bridegroom and the bride are photographed together. When a male figure and a female figure appear in a certain photographed image, the male figure is identified as the bridegroom and the female figure is identified as the bride. When importing the photographed image, the gender of the people appearing in the photographed image is determined as described above. The image processing of arm retouching for the photographed image in which the bridegroom and the bride are photographed together is performed with priority given to the processing for the bride.

[0144] · Example 1 with priority given to the bride Figure 20 is a diagram showing an example of deformation when the bridegroom and the bride are photographed.

[0145] The photographed image shown on the left side of Figure 20 is an image in which the bridegroom is shown on the left side and the bride is shown on the right side. By using the person selection button 36 (Figure 3) prepared on the retouch screen, the user can select any one of only the bridegroom shown on the left side of the photographed image, only the bride shown on the right side, or both of them to perform arm retouching.

[0146] When the operation of arm retouching at the "weak" level is performed as shown in, for example, the speech bubble in the state of selecting "both of them", as shown on the right side of Figure 20, no image processing of arm retouching is performed on the bridegroom ("arm retouching: OFF"), and only for the bride, image processing of arm retouching is performed using the deformation amount corresponding to the "weak" level ("arm retouching: ON"). Image processing of arm retouching according to the user's operation is performed on the bride who is a woman, and no image processing according to the user's operation is performed on the bridegroom who is a man.

[0147] In this way, by making only the arm retouching for the bride effective, it becomes possible to emphasize the thinness of the bride's arm.

[0148] The process of making the arm retouching for the bride effective may be performed only when the bridegroom and the bride are selected as the retouching targets using the person selection button 36, or may also be performed when selected individually, such as only the bridegroom or only the bride.

[0149] ·Example 2 with Priority Given to the Bride Figure 21 is a diagram showing another example of deformation when the bridegroom and the bride are captured. The captured image shown in Figure 21 is the same image as the image described with reference to Figure 20.

[0150] When the operation of arm touch at the "weak" level is performed as shown in the speech bubble, for example, in the state where "both" is selected, as shown on the right side of Figure 21, image processing of arm touch is performed on the bridegroom using the deformation amount corresponding to the "weak" level, and image processing of arm touch is performed on the bride using the deformation amount corresponding to the "strong" level. Image processing of arm touch for the female bride is performed using a deformation amount larger than the deformation amount used for image processing of arm touch for the male bridegroom. When comparing before and after image processing, the deformation amount of the bride's arm becomes larger than the deformation amount of the bridegroom's arm.

[0151] In this way, by switching the deformation amount used for image processing of arm touch between men and women and applying stronger image processing of arm touch to women, it is also possible to emphasize the thinness of the bride's arm.

[0152] Note that when the operation of arm touch is performed in the state where only the male (in the example of Figure 21, "only the person on the left") is selected, it is also possible not to perform image processing of arm touch for the male bridegroom. For example, the display of the adjustment button 54 can be grayed out so that arm touch is prohibited when only the male is selected as the touch target.

[0153] <Example of Using the Detection Result of the Skin Exposure Range for Arm Touch> In wedding commemorative photos, the bride often wears a dress that exposes the neck, chest, and both arms. When a captured image shows a person with a lot of skin exposure and a person with little skin exposure, the person with a lot of skin exposure is identified as the bride. When the captured image is imported, the skin exposure range is detected as described above.

[0154] The detection result of the exposed area of the skin and the determination result of gender may be combined so that a woman with a large exposed area of the skin can be identified as a bride. In this case, a man with a small exposed area of the skin is identified as the groom, and a woman with a small exposed area of the skin is identified as the mother of the bride or the like. The image processing of the arm touch for a photographed image in which the bride and other persons are shown together is performed so as to prioritize the processing for the bride. The processing that prioritizes the arm touch for the bride is the same as in the case where the bride and the groom are shown. Duplicate explanations will be omitted as appropriate.

[0155] · Example 1 of bride priority FIG. 22 is a diagram showing a modified example in the case where the bride and the mother of the bride are shown.

[0156] The photographed image shown on the left side of FIG. 22 is an image in which the bride is shown on the left side and the mother of the bride is shown on the right side. The woman on the left side wearing a dress and having a large exposed area of the skin is identified as the bride, while the woman on the right side wearing a kimono and having a small exposed area of the skin is identified as the mother of the bride. By using the person selection button 36 (FIG. 3) prepared on the retouch screen, the user can perform an arm touch by selecting only the bride shown on the left side of the photographed image, only the mother of the bride shown on the right side, or both of them.

[0157] When an arm touch operation at the "weak" level is performed as shown in the speech bubble, for example, in the state of selecting "both of them", as shown on the right side of FIG. 22, no arm touch is performed on the mother of the bride, and only for the bride, arm touch image processing is performed using the deformation amount corresponding to the "weak" level. Arm touch image processing according to the user's operation is performed on the bride with a large exposed area of the skin, and image processing according to the user's operation is not performed on the mother with a small exposed area of the skin.

[0158] In this way, by making only the arm touch for the bride effective, it becomes possible to emphasize the thinness of the bride's arm.

[0159] · Example 2 of bride priority FIG. 23 is a diagram showing another example of deformation when the bride and the bride's mother are shown. The captured image shown in FIG. 23 is the same image as the image described with reference to FIG. 22.

[0160] When the operation of arm touch at the "weak" level is performed as shown in the speech bubble, for example, in the state where "both" is selected, as shown on the right side of FIG. 23, the arm touch is performed on the bride's mother using the deformation amount corresponding to the "weak" level, and the arm touch is performed on the bride using the deformation amount corresponding to the "strong" level. The image processing of the arm touch for the bride with a wide exposed skin area is performed using a deformation amount larger than the deformation amount used for the image processing of the arm touch of the mother with a narrow exposed skin area. When comparing before and after the image processing, the deformation amount of the bride's arm becomes larger than the deformation amount of the mother's arm.

[0161] In this way, by switching the deformation amount used for the image processing of the arm touch according to the exposed skin area and applying stronger image processing of the arm touch to the person with a wide exposed skin area, it is also possible to emphasize the thinness of the bride's arm.

[0162] <Example of using the estimated age result for arm touch> When the captured image is imported, the age of the subject person is estimated as described above. The deformation amount used for the image processing of the arm touch may be switched based on the estimated age result. For example, the image processing of the arm touch for a person belonging to the age group of children or the elderly is performed using a deformation amount smaller than the deformation amount used for the image processing of the arm touch for a person in other age groups. By performing the image processing using the deformation amount according to the age, it is possible to prevent the image after the arm touch from becoming an unnatural image.

[0163] <Other examples of arm touch> · When multiple people are lined up FIG. 24 is a diagram showing an example of a part to be the target of the arm touch.

[0164] The photographed image shown in FIG. 24 is an image in which two persons, the bridegroom and the bride, are depicted. The bridegroom is shown on the left side, and the bride is shown on the right side. In this case, the outer side of the right arm of the bridegroom indicated by the thick line L51 and the outer side of the left arm of the bride indicated by the thick line L61 are the targets for arm touch. The adjacent arms of the two persons, namely the left arm of the bridegroom and the right arm of the bride, are excluded from the targets for arm touch. Similarly, when a plurality of three or more persons are depicted, the adjacent arms are excluded from the targets for arm touch.

[0165] In this way, by excluding the adjacent arms from the targets for arm touch and setting the opposite arms (the right arm of the bridegroom and the left arm of the bride) shown on the outside as the targets for arm touch, it becomes possible to prevent the adjacent arms of each person from being deformed and resulting in an unnatural appearance.

[0166] · When an arm is shown near the face The arm shown near the face is excluded from the targets for arm touch, and only the arm shown at a position away from the face becomes the target for arm touch. For example, when a tall bridegroom and a short bride are shown side by side, and one arm of the bridegroom is shown near the face of the bride, among the left and right arms of the bridegroom, the arm closer to the face of the bride is excluded from the targets for arm touch, and only the other arm farther from the face of the bride becomes the target for arm touch.

[0167] For example, an arm shown within a range of a predetermined distance based on the area of the face recognized by face recognition is determined as an arm shown near the face. In this way, by excluding the arm shown near the face from the targets for arm touch and setting the arm shown at a distant position as the target for arm touch, it becomes possible to prevent the face from becoming unnaturally depicted due to the deformation of the arm.

[0168] · When a hand is overlapping an arm in the image The arm with overlapping hands is excluded from the target of arm retouching, and only the arm without overlapping hands becomes the target of arm retouching. For example, in the case of a composition where the bride places her hand on the groom's arm, among the left and right arms of the groom, the arm on which the bride places her hand is excluded from the target of arm retouching, and only the other arm on which the bride does not place her hand becomes the target of arm retouching.

[0169] For example, the part beyond the skeletal point of the wrist is determined as the hand part. In this case, the arm with at least a part of the hand, such as the fingertip, overlapping is excluded from the target of arm retouching. By excluding the arm with overlapping hands from the target of arm retouching in this way, it is possible to prevent the hand overlapping the arm from being deformed along with the deformation of the arm and resulting in an unnatural appearance.

[0170] · When an object overlaps and appears on the arm When there is an object of some kind overlapping on the arm, whether the arm with the overlapping object is the target of arm retouching is determined according to the type of the object. In the shooting of wedding commemorative photos, the shooting may be carried out in a composition where the bride holds a bouquet or a stuffed toy. Also, the shooting may be carried out in a composition where the marriage certificate is spread out and facing the camera.

[0171] When the object overlapping and appearing on the arm is a bouquet or a bunch of flowers, that arm becomes the target of arm retouching. On the other hand, when the object overlapping and appearing on the arm is a stuffed toy or a marriage certificate, that arm is excluded from the target of arm retouching.

[0172] For example, even in the case of deformation accompanying the deformation of the arm, there are types of objects that do not cause a sense of discomfort that are set in advance. When the object estimated by object estimation at the time of importing the captured image is an object of the preset type, the arm with the overlapping object becomes the target of arm retouching, and when it is not an object of the preset type, the arm with the overlapping object is excluded from the target of arm retouching.

[0173] In this way, by excluding the arm on which the object is overlapping from the target of arm retouching according to the type of the object, it is possible to prevent the object from being deformed along with the deformation of the arm and resulting in an unnatural appearance.

[0174] <<Details of Image Processing Using the Result of Clothing Judgment>> At the time of capturing an image, clothing judgment is performed for each person of the subject. Clothing judgment is a process of judging the type of clothing. The types of clothing include Western-style clothing and Japanese-style clothing.

[0175] Rather than large classifications such as Western-style clothing and Japanese-style clothing, the types of clothing belonging to small classifications may be judged. Western-style clothing worn by women includes wedding dresses and colored dresses. Also, Japanese-style clothing worn by women includes white kimonos, colored kimonos, and furisode. On the other hand, Western-style clothing worn by men includes tuxedos and wedding suits. Japanese-style clothing worn by men includes haori and hakama.

[0176] Not only the type of clothing but also the attributes of the clothing such as color, shape, and pattern may be judged. In this case, the judgment result of the attributes of the clothing is used for various image processes together with the judgment result of the type.

[0177] In the following, mainly the case where Western-style clothing / Japanese-style clothing is obtained as the judgment result will be described. Various image processes such as retouching and full reflection are switched according to whether the clothing type of the person of the subject is Western-style clothing or Japanese-style clothing.

[0178] <Example of Using the Result of Clothing Judgment in Image Processing of Arm Retouching> ·Switching of On / Off of Image Processing FIG. 25 is a diagram showing an example of switching of On / Off of image processing.

[0179] The photographed image shown in FIG. 25 is an image of the bride. The bride's clothing shown in FIG. 25A is determined to be Western-style clothing, and the bride's clothing shown in FIG. 25B is determined to be traditional Japanese clothing. When the wrist touch operation is performed on each photographed image, as shown at the tip of the arrow in FIG. 25A, for the image in which the type of clothing is determined to be Western-style clothing, image processing for wrist touch is performed, and the arms become thinner.

[0180] On the other hand, as shown at the tip of the arrow in FIG. 25B, for the photographed image in which the type of clothing is determined to be traditional Japanese clothing, image processing for wrist touch is not performed. In the example of FIG. 25B, a message notifying that the wrist touch function cannot be used is displayed.

[0181] In this way, when the wrist touch operation is performed, it is possible to perform the wrist touch image processing according to the user's operation when the clothing of the subject person is Western-style clothing, and not to perform the wrist touch image processing when the clothing of the subject person is traditional Japanese clothing.

[0182] When the clothing of the subject person is traditional Japanese clothing, the effect may not be noticeable even if wrist touch image processing is performed. By enabling the use of the wrist touch function when the clothing of the subject person is Western-style clothing, it is possible to make the effect of the function more noticeable.

[0183] ·Switching of Image Processing Content FIG. 26 is a diagram showing an example of switching of image processing content.

[0184] The photographed image shown in FIG. 26 is the same image as the photographed image shown in FIG. 26. When the wrist touch operation is performed on each of the photographed image in FIG. 26A in which the clothing of the subject person is determined to be Western-style clothing and the photographed image in FIG. 26B in which the clothing of the subject person is determined to be traditional Japanese clothing, as shown at the tip of the arrow in FIG. 26A, for the image in which the type of clothing is determined to be Western-style clothing, image processing for wrist touch is performed, and the arms become thinner.

[0185] On the other hand, as shown at the tip of the arrow in Fig. 26B, for an image in which the type of clothing is determined to be a kimono, style-up image processing is performed instead of arm-retouch image processing. The style-up image processing is a process of thinning the entire body. Image processing for thinning is performed not only on the arm part but also on other parts.

[0186] As described above, when the arm-retouch operation is performed, when the clothing of the subject person is Western-style clothing, arm-retouch image processing is performed according to the user's operation, and when the clothing of the subject person is kimono, it is possible to perform image processing on other parts as well using other functions.

[0187] When the clothing of the subject person is kimono, the effect of arm-retouch image processing may not be easily noticeable even if it is performed. By performing image processing for thinning not only the arm part but also other parts when the clothing of the subject person is kimono, it becomes possible to make the effect of the function more noticeable. Different amounts of change may be used depending on the part to be subjected to the image processing.

[0188] <Example of using the result of clothing determination for image processing of other retouch functions> Switching the on / off of the function or switching the content of the image processing according to the result of the clothing determination is not limited to arm-retouch, and is similarly performed when other retouch functions selectable on the retouch screen are selected.

[0189] Fig. 27 is a diagram showing another example of switching the content of the image processing.

[0190] The photographed image shown in Fig. 27A is an image in which the clothing of the subject person is determined to be Western-style clothing, and the photographed image shown in Fig. 27B is an image in which the clothing of the subject person is determined to be kimono. Image processing when the style-up retouch function is used for the photographed image shown in Fig. 27 and the style is adjusted will be described.

[0191] When a styling operation is performed on each captured image, as shown at the tip of the arrow in Fig. 27A, for an image in which the type of clothing is determined to be Western-style clothing, image processing is performed to narrow the waist. On the other hand, as shown at the tip of the arrow in Fig. 27B, for an image in which the type of clothing is determined to be Japanese-style clothing, image processing is performed to narrow the entire body. Image processing to narrow not only the waist but also other parts is performed.

[0192] In this way, even when the same styling operation is performed, image processing is performed on different parts depending on whether the clothing of the subject person is Western-style or Japanese-style.

[0193] The style adjustment by styling is performed not only by narrowing specific parts of the body but also by appropriately stretching them. Image processing to stretch different parts may be performed depending on whether the clothing of the subject person is Western-style or Japanese-style. For example, when the clothing of the subject person is Western-style, image processing is performed to stretch the part below the waist of the subject person, and when the clothing of the subject person is Japanese-style, image processing is performed to stretch the part below the neck of the subject person.

[0194] As described above, when an operation to deform a predetermined part of the subject person, such as an arm touch operation or a styling operation, is performed by the user, the part to be deformed is switched according to the type of clothing of the subject person. There may be cases where it is preferable to switch the content of the image processing according to whether the clothing of the subject person is Western-style or Japanese-style. By switching the content of the processing as described above, it becomes possible to perform image processing suitable for the type of clothing of the subject person.

[0195] <When the bridegroom and the bride are shown> It is also possible to use the result of clothing determination for image processing on a captured image in which the bridegroom and the bride are shown together. Switching the on / off of functions or switching the content of image processing according to the result of clothing determination is also performed when the captured image shows the bridegroom and the bride together.

[0196] FIG. 28 is a diagram showing an example of image processing when the bridegroom and the bride are captured.

[0197] The captured image shown on the left side of FIG. 28 is an image in which the bridegroom and the bride are captured as the subjects of the image, and their respective outfits are determined to be traditional Japanese clothing. When taking a picture with a composition in which the bridegroom and the bride wearing traditional Japanese clothing are lined up, the body of the bride may appear large due to the volume of the clothing.

[0198] When a predetermined operation such as a styling-up operation is performed on the captured image in which the bridegroom and the bride wearing traditional Japanese clothing are shown side by side, in order to prevent the bride from appearing large, as shown at the tip of the arrow, image processing is performed on the bride to slightly reduce the size. On the right side of FIG. 28, the fact that the entire body of the bride is surrounded by a thick line indicates that the size has been reduced.

[0199] This makes it possible to prevent the bride from appearing large. Instead of reducing the size of the bride, image processing may be performed to enlarge the size of the bridegroom. At least one of the image processing for reducing the size of the bride and the image processing for enlarging the size of the bridegroom is performed.

[0200] Such image processing is performed when the bridegroom and the bride are captured as the subjects of the image and their respective outfits are determined to be traditional Japanese clothing, and is not performed when the outfits of the bridegroom and the bride are determined to be Western-style clothing.

[0201] <Other examples of retouching using the result of clothing determination> ·Examples of using in combination the detection results of the exposure range Image processing for retouching may be performed by combining the detection result of the exposed area of the skin and the result of clothing determination. For example, it is determined whether the exposed area of the skin is wide or narrow based on the area set as a threshold value, and depending on the combination of whether the exposed area of the skin is wide or not and the result of clothing determination, switching of on / off of the image processing and switching of the content of the image processing are performed.

[0202] For example, when the type of the clothing of the subject person is Western-style clothing and the exposed range of the skin is wide, the person is identified as a bride, and is identified as wearing a dress with the skin of the arm part exposed. When an arm touch operation is performed on a bride wearing a dress with the skin of the arm part exposed, image processing for emphasizing the thinness of the arm is performed. That is, the image processing of the arm touch is performed using a deformation amount larger than the deformation amount used for the arm touch of a bride wearing a dress with the skin of the arm part not exposed or the deformation amount used for the arm touch of a bride wearing Japanese-style clothing.

[0203] · Example used for scene determination The result of clothing determination may be used for scene determination. Scene determination is a process for determining the type of shooting scene. The shooting scenes include a wedding ceremony scene and a pre-shooting scene. For example, when the clothing of the bride is a white kimono or a white robe, it is determined that the shooting scene is during the wedding ceremony, and when the clothing of the bride is a colored robe or a color dress, it is determined that there is a pre-shooting scene.

[0204] The parameters used for the image processing of retouching may be switched according to the result of scene determination. Also, according to the shooting scene determined based on the result of clothing determination, the shooting image that is the source of the full reflection and the shooting image that is the destination of the reflection may be selected. For example, a shooting image of the same shooting scene as the shooting scene of the shooting image that is the destination of the reflection is selected as the shooting image that is the source of the reflection.

[0205] <Full reflection using the result of clothing determination> As described with reference to FIG. 9, full reflection is a process of reflecting the content of the retouching of the retouched shooting image on an unselected image that was not selected as the retouching target at the time of image selection. For each person shown in the unselected image, it is selected which content of the retouching for which person shown in which retouched shooting image is to be reflected. The result of clothing determination is used for the selection of the person shown in the unselected image that is the destination of the reflection of the retouching content and the person shown in the retouched shooting image that is the source of the reflection.

[0206] · Case 1 FIG. 29 is a diagram showing a first example of full reflection.

[0207] The photographed images P11 to P15 shown in the upper part of FIG. 29 are retouched photographed images. Further, the photographed image P31 shown in the lower part of FIG. 29 is an unselected image selected as a processing target at the time of full reflection. Each unselected image is sequentially selected, and image processing is performed to reflect the content of retouching. The image processing for reflecting the content of retouching is performed by applying parameters that define the content of retouching (performing retouching using the same parameters). For the sake of convenience of explanation, it is assumed that there is one person shown in all the photographed images.

[0208] In the example of FIG. 29, it is determined that the clothing of the people shown in each of the photographed images P11 to P15 is Western-style clothing. It is also determined that the clothing of the person shown in the photographed image P31 is Western-style clothing.

[0209] Further, in the example of FIG. 29, the photographed image P13 is selected from among the photographed images P11 to P15 as an image in which a person satisfying the conditions is shown. For example, a condition that the person is of the same gender as the person shown in the unselected image and has a face of the size closest to the size of the face of the person shown in the unselected image is set as the condition of the person who is the source of reflection of the content of retouching. The person shown in the unselected image becomes the person who is the destination of reflection of the content of retouching. For example, for the determination of gender identity, the gender determined based on the facial features is used.

[0210] In this case, as shown by the white arrow, all the content of the retouching performed on the source person is reflected on the destination person shown in the unselected image. The content to be reflected also includes the content of arm retouching. When the arm retouching function is used in the retouching of the photographed image P13, the arms of the person in Western-style clothing at the destination also become as thin as the arms of the person in Western-style clothing at the source. Thereby, it becomes possible to make the arms of the person in Western-style clothing shown in the unselected image as thin as the arms of the source person.

[0211] Thus, when the clothing of the people shown in all the photographed images is Western-style clothing and the clothing of the people shown in the unselected image is also Western-style clothing, a person whose content of retouching is the source of reflection is selected from among the people in Western-style clothing shown in the retouched photographed images. In addition, the content of all retouches, including arm retouching, is reflected in the person who is the destination of reflection.

[0212] · Case 2 FIG. 30 is a diagram showing a second example of full reflection. Descriptions that overlap with the above-described explanations will be omitted as appropriate. The same applies hereinafter.

[0213] In the example of FIG. 30, it is determined that the clothing of the people shown in the photographed image P32, which is an unselected image, is Japanese-style clothing. Here, the photographed images in which it is determined that the clothing of the subject people is Japanese-style clothing are shown hatched. The same applies to FIGS. 31 and 32.

[0214] Also, in the example of FIG. 30, from among the photographed images P11 to P15, the photographed image P13 in which a person of the same gender as the person shown in the unselected image and having a face shown in the size closest to the size of the face of the person shown in the unselected image is selected as the source of reflection.

[0215] In this case, as shown by the white arrow, the content of the retouching performed on the source person, excluding arm retouching, is reflected in the destination person shown in the unselected image. Since the content of arm retouching is excluded from the reflected content, even if the function of arm retouching is used in the retouching of the photographed image P13, the arms of the Japanese-style clothing person who is the destination of reflection will not become thinner unlike the arms of the Western-style clothing person who is the source of reflection. If the content of arm retouching is directly reflected in the Japanese-style clothing person, the shape of the clothing such as a kimono may collapse, but it is possible to prevent such a situation.

[0216] Thus, when the clothing of the people shown in all the photographed images is Western-style clothing and the clothing of the people shown in the unselected image is Japanese-style clothing, a person whose content of retouching is the source of reflection is selected from among the people in Western-style clothing shown in the retouched photographed images. In addition, the content of the retouching, excluding arm retouching, is reflected in the person who is the destination of reflection.

[0217] · Case 3 FIG. 31 is a diagram showing a third example of full reflection.

[0218] In the example of FIG. 31, it is determined that the clothing of the people shown in each of the photographed images P11 to P15 is Japanese-style clothing. Image processing of wrist attachment has also been performed on the Japanese-style clothing people shown in the photographed images P11 and P12. On the other hand, it is determined that the clothing of the person shown in the photographed image P33, which is an unselected image, is Western-style clothing.

[0219] Also, in the example of FIG. 31, among the photographed images P11 to P15, the photographed image P13 in which a person of the same gender as the person shown in the unselected image and whose face is shown in the size closest to the size of the face of the person shown in the unselected image is selected as the reflection source.

[0220] In this case, as shown by the white arrow, all the retouching contents performed on the person of the reflection source are reflected on the person of the reflection destination shown in the unselected image. Also, the level 1 wrist retouching that has not been performed on the person shown in the photographed image P13 is reflected. Since the image processing of the wrist retouching is performed using the deformation amount corresponding to level 1, the wrists of the Western-style clothing person of the reflection destination become thinner.

[0221] The fact that the function of wrist retouching is used in the retouching of the photographed images P11 and P12 in which Japanese-style clothing people are shown is considered to indicate that the user has a strong desire to thin the wrists. Since the wrists are more prominent in Western-style clothing than in Japanese-style clothing, even if wrist retouching is not used in the retouching of the reflection source photographed image P13, the content of the wrist retouching is reflected on the photographed image P33 in which the clothing of the subject person is determined to be Western-style clothing. As a result, it becomes possible to reflect the content of the retouching in a form that meets the user's wishes.

[0222] Thus, when the clothing of the people shown in all the photographed images is traditional Japanese clothing and the clothing of the people shown in the unselected image is Western-style clothing, a person who is the source of the retouch content is selected from among the people in traditional Japanese clothing shown in the retouched photographed images. Also, when arm retouching is used for the retouching of any of the photographed images, the content of all the retouching, including the arm retouching, is reflected in the target person. The number of photographed images (the number of photographed images in which arm retouching is used) that serves as a threshold value when reflecting the arm retouching may be set.

[0223] · Case 4 FIG. 32 is a diagram showing a fourth example of full reflection.

[0224] In the example of FIG. 32, it is determined that the clothing of the person shown in the photographed image P34, which is an unselected image, is traditional Japanese clothing. The other conditions are the same as the conditions shown in FIG. 32. Among the photographed images P11 to P15, the photographed image P13 in which a person of the same gender as the person shown in the unselected image and whose face is shown in the size closest to the size of the face of the person shown in the unselected image is selected as the source.

[0225] In this case, as shown by the white arrow, the content of the retouching performed on the source person is reflected in the target person shown in the unselected image. Since the content of the arm retouching is excluded from the reflected content, the arms of the target person in traditional Japanese clothing do not become thinner.

[0226] Thus, when the clothing of the people shown in all the photographed images is traditional Japanese clothing and the clothing of the people shown in the unselected image is traditional Japanese clothing, a person who is the source of the retouch content is selected from among the people in traditional Japanese clothing shown in the retouched photographed images. Also, the content of the retouching, excluding the arm retouching, is reflected in the target person.

[0227] As described above, it is possible to use the result of clothing determination for the full reflection image processing.

[0228] When it is possible to identify a person based on facial features, information on skeleton points, etc., the identification result of the person may be used to perform full reflection. In this case, for example, the retouching content for the bride shown in the retouched photographed image is reflected in the bride shown in the unselected image, and the retouching content for the groom shown in the retouched photographed image is reflected in the groom shown in the unselected image. As a result, the same retouching content will be reflected for the same person.

[0229] <<Configuration of Retouching PC>> FIG. 33 is a block diagram showing a configuration example of the retouching PC 1.

[0230] The CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0231] An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a recording unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0232] The input unit 106 is composed of a keyboard, a mouse, etc.

[0233] The output unit 107 is composed of a display 11, etc.

[0234] The recording unit 108 is composed of a hard disk, a non-volatile memory, etc. The recording unit 108 records programs such as the retouching application 1A executed by the CPU 101 and various information such as photographed images.

[0235] The communication unit 109 is an interface for networks such as the Internet and LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a photographed image of a user who is about to start retouching, which is transmitted from the storage PC. Also, the communication unit 109 communicates with the image management server 2 via the Internet, and transmits the retouched photographed image to the image management server 2.

[0236] The drive 110 controls writing of data to the removable media 111 and reading of data from the removable media 111. The capture image from the storage PC may be imported via the removable media 111.

[0237] FIG. 34 is a block diagram showing a functional configuration example of the retouch PC 1.

[0238] As shown in FIG. 34, in the retouch PC 1, the information processing unit 121 is realized by executing the retouch application 1A. The information processing unit 121 is composed of an image acquisition unit 131, a display control unit 132, a person recognition / attribute determination unit 133, a retouch processing unit 134, and a file management unit 135.

[0239] The image acquisition unit 131 reads and acquires a photographed image from the recording unit 108 at the start of retouching of the photographed image. When the photographing is completed, all the photographed images such as several hundred images photographed with a set of users as subjects are transferred from the storage PC and recorded in the recording unit 108. The photographed image acquired by the image acquisition unit 131 is supplied to the display control unit 132 and the person recognition / attribute determination unit 133. The import of the photographed image in step S1 of FIG. 10 is a process performed by the image acquisition unit 131.

[0240] The display control unit 132 causes an image selection screen to be displayed based on the captured image supplied from the image acquisition unit 131. Further, after five, ten, or the like captured images to be retouched are selected using the image selection screen, the display control unit 132 causes a retouch screen to be displayed and controls the display of the retouch screen. While retouching is being performed using the retouch screen, information on the result of the image processing is supplied from the retouch processing unit 134. The processes in steps S2, S3, and S4 in FIG. 10 and the process in step S11 in FIG. 11 are processes performed by the display control unit 132.

[0241] The person recognition and attribute determination unit 133 recognizes the faces of the persons shown in the respective captured images captured by the image acquisition unit 131 and determines the attributes of the respective persons. The processes in steps S12 - S16 and the processes in steps S19 - S21 in FIG. 11 are processes performed by the person recognition and attribute determination unit 133.

[0242] That is, the person recognition and attribute determination unit 133 recognizes the faces of the persons shown in the captured images and determines attributes such as gender and age for each recognized face (each person). Further, the person recognition and attribute determination unit 133 determines attributes such as skeleton points, clothing type, and exposure range for each person whose face has been recognized. Details of the configuration of the person recognition and attribute determination unit 133 will be described later with reference to FIG. 35. The recognition result and the attribute determination result by the person recognition and attribute determination unit 133 are supplied to the retouch processing unit 134 and the file management unit 135 together with the captured images that are the objects of recognition and the like.

[0243] The retouch processing unit 134 performs various image processes related to retouching in response to the user's operations on the retouch screen when retouching the captured image. The processes in steps S5 - S7 in FIG. 10 are processes performed by the retouch processing unit 134. For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouch processing unit 134 performs image processing such as deforming the position of the eye contour according to the selection content by the user.

[0244] Specifically, when any one of the levels of "1", "2", "3", or "4" is selected using the adjustment button 56 (Fig. 4), the retouch processing unit 134 extracts the eye region of the person to be retouched by mask processing or the like. The retouch processing unit 134 enlarges the image of the extracted eye region according to the selected level, and performs eye enlargement processing by pasting the enlarged eye region image at the extraction source position.

[0245] Also, when an operation related to adjusting the intensity of makeup is performed, the retouch processing unit 134 performs image processing such as synthesizing an image of a cheek with a predetermined color on the cheek part or synthesizing an image of lipstick with a predetermined color on the lip part.

[0246] When an operation related to adjusting skin texture is performed, the retouch processing unit 134 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the size (small face feeling) or shape of the face is performed, the retouch processing unit 134 performs image processing such as shifting the position of the face contour according to the selection content by the user.

[0247] Specifically, when any one of the levels of "1", "2", "3", or "4" is selected using the adjustment button 51 (Fig. 4), the retouch processing unit 134 shifts each position of the face contour of the person to be retouched in the direction of the center of the face according to the selected level, and performs image processing to make the face smaller.

[0248] When an operation related to adjusting the size (small face feeling) or shape of the face is performed, the retouch processing unit 134 performs image processing such as shifting the position of the face contour according to the selection content by the user. Also, when an operation related to deforming a specific part of a person, such as arm retouch or styling up, is performed, the retouch processing unit 134 performs the image processing of those functions as described above.

[0249] The information representing the result of the image processing by the retouch processing unit 134 is supplied to the display control unit 132 and used for displaying the retouch screen. When the retouch of all the captured images is completed, the retouch parameters, which are the information indicating the content of the retouch, are supplied to the file management unit 135.

[0250] In addition, when all are reflected, the retouch processing unit 134 reflects the content of the retouch of the retouched captured image on the unselected image. As described above, which person in which captured image the retouch content is reflected on which person in which unselected image is selected based on the face size, gender, type of clothing, etc. When all reflections are completed, the retouch parameters of each unselected image are supplied to the file management unit 135.

[0251] The file management unit 135 generates a management file in which various types of information are recorded based on the information supplied from the person recognition / attribute determination unit 133 and the retouch processing unit 134. The processes in steps S17 and S22 in FIG. 11 are the processes performed by the file management unit 135.

[0252] The file management unit 135 manages by associating the files of the respective captured images with the management file. The files of the captured images and the management file are recorded in, for example, the recording unit 108. The information of the management file managed by the file management unit 135 is appropriately supplied to the display control unit 132 and used for screen display.

[0253] FIG. 35 is a block diagram showing a configuration example of the person recognition / attribute determination unit 133 in FIG. 34.

[0254] The person recognition / attribute determination unit 133 is composed of a face recognition unit 151, a face feature determination unit 152, a skeleton estimation unit 153, a clothing determination unit 154, and a determination unit 155. Each captured image captured by the image acquisition unit 131 is input to each unit.

[0255] The face recognition unit 151 recognizes the face shown in the captured image. Among the processes of steps S13 and S19 in FIG. 11, the face recognition process is the process performed by the face recognition unit 151. An inference model for face recognition that takes the captured image as input and outputs information indicating the position of the face is prepared for the face recognition unit 151. The recognition result by the face recognition unit 151 is output to each unit.

[0256] The face feature determination unit 152 determines the features of the face recognized by the face recognition unit 151. Among the processes of steps S13 and S19 in FIG. 11, the determination of the face features is the process performed by the face feature determination unit 152. An inference model for face feature determination that takes the face image as input and outputs information on the face features is prepared for the face feature determination unit 152. The determination result by the face feature determination unit 152 is output to the reattachment processing unit 134 and the file management unit 135.

[0257] The skeleton estimation unit 153 performs skeleton estimation on the person whose face has been recognized by the face recognition unit 151. The process of step S14 in FIG. 11 is the process performed by the skeleton estimation unit 153. An inference model for skeleton estimation that takes the image of the person as input and outputs information on the positions of the skeleton points is prepared for the skeleton estimation unit 153. The result of the skeleton estimation by the skeleton estimation unit 153 is output to the reattachment processing unit 134 and the file management unit 135.

[0258] The clothing determination unit 154 performs clothing determination on the person whose face has been recognized by the face recognition unit 151. The processes of steps S15 and S20 in FIG. 11 are the processes performed by the clothing determination unit 154. An inference model for clothing determination that takes the image of the person as input and outputs information indicating the type of clothing is prepared for the clothing determination unit 154. The inference model for clothing determination is generated, for example, by learning using a large number of images showing various people in clothing with the information on the type of clothing labeled as learning data. The information on the type of clothing, which is the determination result by the clothing determination unit 154, is output to the reattachment processing unit 134 and the file management unit 135.

[0259] The determination unit 155 estimates various attributes of the person whose face has been recognized by the face recognition unit 151. Attributes recorded in the management file other than the facial features, skeleton points, and clothing type are estimated by the determination unit 155. The processes in steps S16 and S21 of FIG. 11 are the processes performed by the determination unit 155. The determination unit 155 is provided with various inference models such as an inference model for gender estimation, an inference model for age estimation, and an inference model for estimating the exposed range of the skin. The determination result by the determination unit 155 is output to the retouch processing unit 134 and the file management unit 135.

[0260] In this way, in each part of the person recognition / attribute determination unit 133, face recognition and determination of various attributes are performed using an inference model generated by machine learning. Instead of performing the processing of each part of the person recognition / attribute determination unit 133 using an inference model, it may be performed by analyzing the image.

[0261] <<Retouch correction tool>> As described above, a user who uses the service of the photography studio can perform retouching by himself / herself using the functions of the retouch app 1A and adjust the appearance of the photographed image. The retouching of the photographed image may also be performed by the staff of the photography studio. In addition, the retouching of the photographed image is automatically performed as appropriate by the full reflection function of the retouch app 1A.

[0262] Retouching by manual or full reflection is performed using the determination results of various inference models. However, if there is an error in the determination using this inference model, retouching that results in an unnatural appearance may be performed. A tool for correcting such so-called retouching failures is provided.

[0263] FIG. 36 is a diagram showing the flow of retouch correction.

[0264] The PC201 is an information processing device used for retouching correction. A correction application 201A, which is a tool for retouching correction, is installed on the PC201. The correction application 201A may be installed on the retouching PC1 so that the retouching correction is performed on the retouching PC1.

[0265] As shown by the arrow in FIG. 36, the image to be corrected and the management file are imported into the correction application 201A. The image to be corrected is a captured image with failed retouching. For example, before providing the retouched captured images to the user, the staff at the photography studio or the like checks whether the retouching of each captured image is unnatural. The checking work is, for example, a work of visually checking one by one on the screen of the retouching PC1.

[0266] The captured images determined to have unnatural retouching are selected as the images to be corrected. In the example of FIG. 36, a plurality of captured images are selected as the images to be corrected. For example, when an image to be corrected is selected on the screen of the retouching application 1A, the file of the image to be corrected and the management file of the image to be corrected are transmitted to the PC201 and imported by the correction application 201A.

[0267] The retouching correction using the correction application 201A may be performed by the staff at the photography studio, or may be performed by the operator of the business that provides the retouching application 1A and the correction application 201A. In the latter case, the file of the image to be corrected selected by the staff at the photography studio is transmitted to the PC201 via a network such as the Internet and imported by the correction application 201A.

[0268] When the file of the image to be corrected and the like are imported, a retouching correction screen is displayed by the correction application 201A. The retouching correction screen is a screen used for retouching correction. The user of the retouching correction screen is the staff at the photography studio or the operator of the business that provides the retouching application 1A and the correction application 201A.

[0269] Figure 37 is a screen showing an example of a retouch correction screen.

[0270] The retouch correction screen basically has the same screen configuration as the retouch screen displayed by the retouch app 1A. Retouch correction is performed by the same operations as those during retouch. The display in Fig. 37 is the display when an operation for correcting arm retouch is performed. The correction of arm retouch will be mainly described.

[0271] A display area 251 is formed above the center of the retouch correction screen. The display area 251 is an area where one captured image (image to be corrected) to be corrected is enlarged and displayed. A thumbnail image of the image to be corrected is displayed in a thumbnail area 252 formed on the left side of the display area 251. In the example of Fig. 37, the first captured image selected using the thumbnail image is enlarged and displayed in the display area 251.

[0272] The captured image displayed in the display area 251 is an image in which the bridegroom is shown on the left side and the bride is shown on the right side. Small circular dots indicating skeletal points and images of straight lines connecting the skeletal points are displayed on the body parts of the bridegroom and the bride.

[0273] Figure 38 is a diagram showing an enlarged view of the display in the display area 251.

[0274] Dots D11R to D14R are displayed on the right arm of the bridegroom, and dots D11L to D14L are displayed on the left arm. The upper arm part of the right arm is indicated by dot D11R indicating the shoulder position of the right arm and dot D12R indicating the elbow position of the right arm, and the forearm part of the right arm is indicated by dot D13R indicating the elbow position of the right arm and dot D14R indicating the wrist position of the right arm. Similarly, the upper arm part of the left arm is indicated by dot D11L indicating the shoulder position of the left arm and dot D12L indicating the elbow position of the left arm, and the forearm part of the left arm is indicated by dot D13L indicating the elbow position of the left arm and dot D14L indicating the wrist position of the left arm.

[0275] On one side, dots D21R to D24R are displayed on the bride's right arm, and dots D21L to D24L are displayed on the left arm. The upper arm of the right arm is indicated by dot D21R showing the position of the shoulder of the right arm and dot D22R showing the position of the elbow, and the forearm of the right arm is indicated by dot D23R showing the position of the elbow of the right arm and dot D24R showing the position of the wrist. Similarly, the upper arm of the left arm is indicated by dot D21L showing the position of the shoulder of the left arm and dot D22L showing the position of the elbow, and the forearm of the left arm is indicated by dot D23L showing the position of the elbow of the left arm and dot D24L showing the position of the wrist.

[0276] The information such as these skeleton points is displayed based on the information recorded in the management file. The straight lines connecting between the dots indicating each part of the upper arm and forearm of the right arm and the upper arm and forearm of the left arm of the bridegroom and the bride are each displayed in a different color. By displaying the information of the skeleton points in different colors, the result of skeleton estimation can be easily confirmed.

[0277] Returning to the description of FIG. 37, below the thumbnail area 252, a person selection button 253 and a skeleton point selection button 254 are displayed. The person selection button 253 is a button used when selecting a person to be the correction target of the arm attachment among the people shown in the captured image to be corrected. It is possible to select only the bridegroom shown on the left side, only the bride shown on the right side, or both the bridegroom and the bride. In the example of FIG. 37, both the bridegroom and the bride are selected. The information of the skeleton points of the person selected using the person selection button 253 is displayed on the captured image.

[0278] The skeleton point selection button 254 is a button operated when selecting an added skeleton point. It is possible to add skeleton points in case of an error in skeleton estimation. Each added skeleton point can be selected using the skeleton point selection button 254.

[0279] At the lower left of the retouch correction screen, a retouch correction area 256 is formed, and at the lower right, a skeleton point correction area 257 is formed. Each time the buttons in the retouch correction area 256 and the skeleton point correction area 257 are operated, the content of the user's correction is reflected in the display of the display area 251.

[0280] In the retouch correction area 256, a level correction button 256-1, a correction part selection button 256-2, and a direction selection button 256-3 are displayed.

[0281] The level correction button 256-1 is a button used to select the level of the arm retouch. In the example of FIG. 37, it is possible to select a preferred level from among five levels. That is, when retouching using the retouch screen of FIG. 3, it is possible to select a preferred level from among three levels as the level of the arm retouch, whereas during correction, it is possible to select a preferred level from among the finer, five levels.

[0282] The correction part selection button 256-2 is a button used to select the part to correct the arm retouch. It is possible to select the part to be corrected from among seven types: the upper arm parts (both arms) of the left and right arms, the forearm parts (below the elbows) of the left and right arms, the upper arm part of the left arm, the upper arm part of the right arm, the forearm part of the left arm, the forearm part of the right arm, and the whole. That is, when retouching using the retouch screen of FIG. 3, the user cannot select the part to be the subject of the image processing of the arm retouch, whereas during correction, it is possible to select it.

[0283] The direction selection button 256-3 is a button used to select the part (direction) to correct the arm retouch. It is possible to select the correction direction from among three types: outside, inside, and both sides of the part selected using the correction part selection button 256-2. That is, when retouching using the retouch screen of FIG. 3, only the outside of the arm is the subject of the image processing of the arm retouch, whereas during correction, it is possible to select the part to be the subject of the image processing.

[0284] By using the display of the retouch correction area 256, finer adjustment of the arm thickness can be made compared to during retouch. For example, by using the correction site selection button 256-2 and the direction selection button 256-3, it becomes possible to select the site and direction for arm retouch from the outside of the upper arm, the inside of the upper arm, the outside of the forearm, and the inside of the forearm of the subject person's arm.

[0285] In the skeleton point correction area 257, a movement button 257-1, an addition button 257-2, and a deletion button 257-3 are displayed. Buttons operated when moving the display range of the display area 251, buttons operated when returning the state of the skeleton point to the state before correction, etc. are also displayed in the skeleton point correction area 257.

[0286] The movement button 257-1 is a button operated when moving a skeleton point. For example, after operating the movement button 257-1, by pressing and sliding an arbitrary dot on the display area 251, it becomes possible to move the position of the skeleton point.

[0287] The addition button 257-2 is a button operated when adding a skeleton point. For example, after operating the addition button 257-2, by pressing an arbitrary position on the display area 251, it becomes possible to add a skeleton point at an arbitrary position.

[0288] The deletion button 257-3 is a button operated when deleting a skeleton point. For example, after operating the deletion button 257-3, by pressing an arbitrary dot on the display area 251, it becomes possible to delete the skeleton point at an arbitrary position.

[0289] By using the display of the skeleton point correction area 257, it becomes possible to correct the error in skeleton estimation and accordingly correct the arm retouch.

[0290] On the right side of the display area 251, various buttons such as an on / off button 258 that is operated when switching the display of the skeleton points on and off, and a determination result display button 259 that is operated when displaying the determination result of the captured image to be corrected are displayed. For example, when the determination result display button 259 is operated, the attribute information recorded in the management file is displayed in a list on the retouch correction screen. In a state where the attribute information is displayed in a list, it is possible to check and correct the determination results of various attributes.

[0291] In this way, using the retouch correction screen, it becomes possible to adjust the state of the arm retouch more finely than at the time of retouch. The same screen can be used to adjust more finely than at the time of retouch even when correcting a retouch using functions other than the arm retouch.

[0292] FIG. 39 is a diagram showing an example of mounting the retouch correction function.

[0293] As shown in FIG. 39, a program module having the function of the correction application 201A may be provided in the retouch application 1A so that the retouch is performed by the retouch application 1A. In this case, the correction using the screen as shown in FIG. 37 is performed as a function of the retouch application 1A. A user who has taken a photo in the photo studio can select a part of the arm to be retouched and correct the skeleton points on the retouch application 1A by himself / herself.

[0294] <<Modification Example>> Each of the above-described functions including the function of the arm retouch and the function of image processing using the result of clothing determination may be realized on a server on the Internet. In this case, the user who has finished taking a photo operates the browser of the retouch PC 1 to access the server and performs the above-described retouch on the retouch screen displayed by the browser. In this way, an information processing apparatus having each of the above-described functions may be realized by a server on the Internet.

[0295] ·Regarding the program The above series of processes can be executed either by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer or the like.

[0296] The program to be installed is provided by being recorded on a removable medium 111 shown in FIG. 33, which is composed of an optical disk (such as a CD-ROM (Compact Disc - Read Only Memory), a DVD (Digital Versatile Disc)) or a semiconductor memory. Further, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in the ROM 102 or the recording unit 108.

[0297] The program executed by the computer may be a program in which processing is performed in time series in accordance with the order described in this specification, or a program in which processing is performed in parallel or at a necessary timing such as when a call is made.

[0298] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0299] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0300] For example, the present technology can adopt a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0301] In addition, each step described in the above flowchart can be executed by one device or shared and executed by a plurality of devices.

[0302] Furthermore, when a single step includes a plurality of processes, the plurality of processes included in that single step can be executed by a single device or can be executed in a distributed manner by a plurality of devices.

Explanation of Signs

[0303] 1 PC for reattachment, 2 Image management server, 3 Mobile terminal, 11 Display, 121 Information processing unit, 131 Image acquisition unit, 132 Display control unit, 133 Person recognition / attribute determination unit, 134 Reattachment processing unit, 135 File management unit, 151 Face recognition unit, 152 Face feature determination unit, 153 Skeleton estimation unit, 154 Clothing determination unit, 155 Determination unit

Claims

1. A program that causes a computer to select, according to the way the arm of the subject person appears in the captured image, a part of the arm that is a target for arm retouching to thin the arm of the person appearing in the captured image by image processing, and perform the image processing of the arm retouching according to the level selected by the user on the said part of the arm. A program for executing the process.

2. The image processing of the arm retouching is performed by deforming the outer part of the arm that is the said part. The program according to claim 1.

3. Select different parts as the said outer part according to the angle of the arm as the way of appearance. The program according to claim 2.

4. The image processing of the arm retouching is performed using different amounts of deformation for the outer part of the upper arm and the outer part of the forearm. The program according to claim 2 or 3.

5. The image processing of the arm retouching is performed using an amount of deformation according to the angle of the arm. The program according to any one of claims 1 to 3.

6. The image processing of the arm retouching is performed using an amount of deformation according to the thickness of the arm. The program according to any one of claims 1 to 3.

7. The image processing of the arm retouching is performed using an amount of deformation according to the shape of the arm. The program according to any one of claims 1 to 3.

8. The image processing of the arm retouching is performed by deforming the said part of the arm and extending the length of the arm. The program according to claim 1.

9. Determine the gender of the subject person, and the image processing of the arm retouching is performed using an amount of deformation according to the gender of the subject person. The program according to claim 1.

10. Determine the gender of the subject person, and when a male person and a female person appear as the subject person, the image processing of the arm retouching for the female arm is performed preferentially over the image processing of the arm retouching for the male arm. The program according to claim 1.

11. Perform the image processing of the arm retouching for the female arm, and do not perform the image processing of the arm retouching for the male arm. The program according to claim 10.

12. The image processing of the arm retouching for the female arm is performed using a greater amount of deformation than the amount of deformation used for the image processing of the arm retouching for the male arm. The program according to claim 10.

13. Determine the age of the subject person, The image processing of the wrist touch is performed using a deformation amount corresponding to the age of the person of the subject. The program according to claim 1.

14. Detect the exposed range of the skin of the person of the subject, The image processing of the wrist touch for the wrist of a person with a wide exposed range of skin among the persons shown in the captured image is preferentially performed over the image processing of the wrist touch for the wrist of a person with a narrow exposed range of skin. The program according to claim 1.

15. Perform the image processing of the wrist touch for the wrist of a person with a wide exposed range of skin, Do not perform the image processing of the wrist touch for the wrist of a person with a narrow exposed range of skin. The program according to claim 14.

16. The image processing of the wrist touch for the wrist of a person with a wide exposed range of skin is performed using a larger deformation amount than the deformation amount used for the image processing of the wrist touch for the wrist of a person with a narrow exposed range of skin. The program according to claim 14.

17. Without performing the image processing of the wrist touch for the adjacent wrists of two persons shown in the captured image, perform the image processing of the wrist touch for the opposite wrist. The program according to claim 1.

18. Perform the image processing of the wrist touch for one visible wrist of the person of the subject. The program according to claim 1.

19. Perform the image processing of the wrist touch for one wrist that appears in front of the wrist of the person of the subject. The program according to claim 1.

20. Without performing the image processing of the wrist touch for the wrist closer to the face of the person of the subject, perform the image processing of the wrist touch for the wrist farther from the face. The program according to claim 1.

21. Do not perform the image processing of the wrist touch for the wrist where at least a part of the hand is overlapping. The program according to claim 1.

22. When an object overlaps the wrist of the person of the subject, switch whether to perform the image processing of the wrist touch according to the type of the object. The program according to claim 1.

23. Perform skeletal estimation of the person of the subject, Select the part that is the target of the image processing of the wrist touch according to the way of depiction represented by the result of the skeletal estimation. The program according to claim 1.

24. Set whether to always use the image processing for wrist retouching, always not use it, or select whether to use it or not each time the captured image to be retouched is imported. The program according to claim 23.

25. When using the image processing for wrist retouching, perform the skeleton estimation for each of the captured images at the time of importing the captured images to be retouched. When not using the image processing for wrist retouching, do not perform the skeleton estimation at the time of importing the captured images to be retouched. The program according to claim 24.

26. Select the part that is the target of the image processing for wrist retouching according to the user's operation. The program according to claim 1.

27. The user's operation is performed so as to be selected from the outer part of the upper arm of the arm of the subject person, the inner part of the upper arm, the outer part of the forearm, and the inner part of the forearm. The program according to claim 26.

28. An information processing apparatus selects a part of the arm that is the target of wrist retouching to thin the arm of the person shown in the captured image by image processing according to the way the arm of the subject person appears, and performs the image processing for wrist retouching according to the level selected by the user on the part of the arm. An information processing method.

29. selects a part of the arm that is the target of wrist retouching to thin the arm of the person shown in the captured image by image processing according to the way the arm of the subject person appears, and performs the image processing for wrist retouching according to the level selected by the user on the part of the arm. A retouching processing unit An information processing apparatus comprising the same.

Images