Image processing device, image processing method, and program

Abstract

To appropriately select an image meeting a user's intention.SOLUTION: A program causes a computer to function as first designation means that designates a first image group, second designation means that designates a second image group, analysis means that analyzes images included in the first image group and the second image group, determination means that determines a score reference for providing scores to the images based on a result of analysis on the second image group, scoring means that executes scoring of the first image group based on the score reference and a result of analysis on the first image group, selection means that selects a third image group from the first image group based on the score, and designation means that can designate an adjustment parameter for adjusting the third image group selected by the selection means.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for selecting one or more images from among a plurality of images according to a predetermined criterion. [Background technology]

[0002] There is an automatic layout technology that automatically selects images for creating an album from among a plurality of images, automatically determines a template for the album, and automatically assigns images to the template.

[0003] Patent Document 1 describes a technique for adjusting the frequency of appearance of a main object in an automatically selected group of images in an album candidate according to a user's specification. Patent Document 1 also describes a technique for adding a score and identification information indicating that it is a specific object to image data, and selecting images from the entire group of images according to the priority of the object that is set according to the score and user specification. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-97483 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the method described in Patent Document 1, there is a risk that an image that meets the user's intention may not be appropriately selected.

[0006] An object of the present invention is to provide a technique for appropriately selecting an image that meets the user's intention. [Means for solving the problem]

[0007] A program according to one aspect of the present invention causes a computer of an image processing device to include a first designation means for designating a first image group, a first selection means for selecting the second image group from the first image group based on a user operation for designating a second image group from the first image group, an analysis means for analyzing each image included in the second image group, a determination means for determining a criterion for selecting images from the first image group based on a result of the analysis of the second image group, a second selection means for selecting a third image group from the first image group, which is the image group from which the second image group was selected, based on the criterion based on a result of the analysis of the images included in the second image group, and a selection means for selecting the third image group. A first generation means for generating a first album by automatically arranging the program on a plurality of templates; a first display means for displaying on a display unit, and a change operation for changing parameter values ​​of the image group displayed on the display unit from values ​​corresponding to the third image group selected by the second selection means to other values; After the first album is displayed, Acceptable changes Operation reception a third selection means for selecting a fourth set of images from the first set of images using the parameter of the other value corresponding to the change operation when the change operation is accepted; and a third selection means for selecting the fourth set of images from the first set of images using the parameter of the other value corresponding to the change operation when the change operation is accepted. A second generating means for generating a second album by automatically arranging the second album in a plurality of templates; The display unit functions as a second display unit that displays on the display unit. Effect of the Invention

[0008] According to the present invention, an image that meets the user's intention can be appropriately selected. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the image processing device. [Diagram 2] FIG. 2 is a software block diagram of an album creation application. [Diagram 3] FIG. 13 is a diagram showing a display screen provided by an album creating application. [Figure 4] FIG. 13 is a diagram showing a display screen provided by an album creating application. [Diagram 5]FIG. 13 is a diagram showing a display screen provided by an album creating application. [Figure 6] 13 is a flowchart showing an automatic layout process. [Figure 7] FIG. [Figure 8] 13 is a flowchart showing details of a score criterion determination process. [Figure 9] FIG. 1 is a diagram illustrating feature amounts in a feature amount space. [Figure 10] FIG. 13 is a diagram showing a result of image selection. [Figure 11] FIG. 13 is a diagram showing a group of templates used for the layout of image data. [Figure 12] FIG. 13 is a diagram showing a display screen provided by an album creating application. [Figure 13] FIG. 13 is a diagram illustrating score adjustment. [Figure 14] FIG. 2 is a software block diagram of an album creation application. [Figure 15] 13 is a flowchart showing an automatic layout process. [Figure 16] 13 is a flowchart showing details of a score criterion determination process. [Figure 17] 13 is a flowchart illustrating a second score criterion determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are essential to the solution of the present invention. Note that the same components are given the same reference numbers and descriptions are omitted.

[0011] <<First embodiment>> In this embodiment, an example of a method of generating an automatic layout by running an application (hereinafter also referred to as "app") for creating an album in an image processing device will be described. In the following description, "image" includes still images, videos, and frame images extracted from videos, unless otherwise specified. Furthermore, images here may also include still images, videos, and frame images in videos that are stored on a network, such as a service on a network and storage on a network, and can be obtained via a network.

[0012] 1 is a block diagram showing the hardware configuration of an image processing device. The image processing device 100 may be, for example, a personal computer (hereinafter, referred to as a PC) or a smartphone. In this embodiment, the image processing device will be described as a PC. The image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, a HDD 104, a display 105, a keyboard 106, a pointing device 107, and a data communication unit 108.

[0013] The CPU (Central Processing Unit / Processor) 101 comprehensively controls the image processing device 100, and for example, reads out a program stored in the ROM 102 into the RAM 103 and executes it to realize the operation of this embodiment. In FIG. 1, there is one CPU, but the system may be configured with multiple CPUs. The ROM 102 is a general-purpose ROM, and for example, stores a program executed by the CPU 101. The RAM 103 is a general-purpose RAM, and is used as a working memory for temporarily storing various information when the CPU 101 executes a program. The HDD (Hard Disk) 104 is a storage medium (storage unit) for storing image files, a database for holding processing results such as image analysis, and templates used by an album creation application.

[0014] The display 105 is a display unit that displays to the user an electronic album as a layout result of the user interface (UI) of this embodiment and image data (hereinafter also referred to as "images"). The keyboard 106 and the pointing device 107 receive instruction operations from the user. The display 105 may have a touch sensor function. The keyboard 106 is used, for example, when the user inputs the number of pages of the album to be created on the UI displayed on the display 105. The pointing device 107 is used, for example, when the user clicks a button on the UI displayed on the display 105.

[0015] A data communication unit 108 communicates with an external device via a wired or wireless network. The data communication unit 108 transmits data laid out by an automatic layout function to a printer or a server capable of communicating with the image processing device 100. A data bus 109 connects the blocks in FIG. 1 to each other so that they can communicate with each other.

[0016] 1 is merely an example, and is not limited to this example. For example, the image processing device 100 may not have the display 105, and may display the UI on an external display.

[0017] The album creation application in this embodiment is stored in the HDD 104. Then, as will be described later, the user selects the application icon displayed on the display 105 with the pointing device 107 and starts the application by clicking or double-clicking the icon.

[0018] 2 is a software block diagram of the album creation application. The album creation application has an album creation condition designation unit 201, a user image designation unit 202, an adjustment parameter designation unit 210, and an automatic layout processing unit 218. The automatic layout processing unit 218 has an image acquisition unit 203, an image conversion unit 204, an image analysis unit 205, a score criterion determination unit 206, an image score unit 207, an adjustment parameter determination unit 208, and a two-page spread number input unit 209. It further has an image selection unit 211, a two-page spread allocation unit 212, a template input unit 213, an image layout unit 214, a layout information output unit 215, an image correction condition input unit 216, and an image correction unit 217.

[0019] The album creation application described above includes program modules corresponding to the components shown in Fig. 2. The CPU 101 executes each program module, causing the CPU 101 to function as each component shown in Fig. 2. Hereinafter, each component shown in Fig. 2 will be described assuming that each component executes various processes. Fig. 2 also shows a software block diagram relating in particular to an automatic layout processing unit 218 that executes an automatic layout function.

[0020] The album creation condition designation unit 201 designates the album creation conditions to the automatic layout processing unit 218 in response to a UI operation by the pointing device 107. In this embodiment, as the album creation conditions, it is possible to designate an album candidate image group (also called a first image group) including candidate images to be used in the album, the number of pages, the type of template, and whether or not image correction is performed on the album. It is also possible to designate the commercial material for creating the album. The album candidate image group may be designated by, for example, accompanying information of individual images such as the shooting date and time, or attribute information, or may be designated based on the structure of a file system including images such as a device and a directory. In addition, two arbitrary images may be designated, and all images taken between the dates and times when the respective image data were taken may be the target image group. In this specification, a "spread" corresponds to one display window in a display, and corresponds to a pair of adjacent pages (i.e., two pages) that a user can view at once when opening a book in a printed matter. In addition, two pages in a spread may be printed on different sheets of paper and bound together so that they are adjacent to each other, or printed on a single sheet of paper and folded in the middle to form a spread. The user image designation unit 202 allows the user to designate a group of user images (also called a second group of images) that represent the user's hobbies or preferences and that the user wishes to use in the album. One or more images may be selected as the user image group. In addition to the above designation method, a group of images used in an album created in the past may be stored in a cloud accessible via the HDD 104 or the data communication unit 108, and the user may be allowed to designate from the group of images. The image designated here may or may not be included in the image designated by the album creation condition designation unit 201. In other words, the user image group may or may not be included in the album candidate image group.

[0021] The image acquisition unit 203 acquires the image group designated by the album creation condition designation unit 201 and the user image designation unit 202 from the HDD 104. The image acquisition unit 203 outputs, as meta information, image width and height information contained in the acquired image, shooting date and time information contained in the Exif information at the time of shooting, and information indicating whether the image is included in the user image group, to the image analysis unit 205. The image acquisition unit 203 also outputs the acquired image data to the image conversion unit 204. Identification information is assigned to each image, and the meta information output to the image analysis unit 205 and the image data output to the image analysis unit 205 via the image conversion unit 204 described later can be associated by the image analysis unit 205.

[0022] The images stored in the HDD 104 include still images and frame images cut out from a video. The still images and frame images are acquired from an imaging device such as a digital camera or a smart device. The imaging device may be included in the image processing device 100 or may be included in an external device. If the imaging device is an external device, the images are acquired via the data communication unit 108. The still images and cut-out images may be images acquired from a network or a server via the data communication unit 108. Examples of images acquired from a network or a server include social networking service images (hereinafter, referred to as "SNS images"). The program executed by the CPU 101 analyzes data attached to each image to determine the storage source. The SNS images may be acquired from an SNS via an application, and the acquisition destination may be managed within the application. The images are not limited to the above-mentioned images, and may be other types of images.

[0023] The image conversion unit 204 converts the image data input from the image acquisition unit 203 into pixel count and color information for use in the image analysis unit 205, and outputs the converted information to the image analysis unit 205. In this embodiment, the image is converted to a predetermined pixel count, for example, 420 pixels on the short side, and the long side is converted to a size that maintains the ratio of the original sides. Furthermore, the image is converted to a unified color space such as sRGB for color analysis. In this way, the image conversion unit 204 converts the image into an analysis image with a unified pixel count and color space. The image conversion unit 204 outputs the converted image to the image analysis unit 205. The image conversion unit 204 also outputs the image to the layout information output unit 215 and the image correction unit 217.

[0024] The image analysis unit 205 analyzes image data of the analysis image input from the image conversion unit 204 by a method described later, and acquires image feature amounts. The analysis process executes each process of estimating the degree of focus, face detection, individual recognition, and object determination, and acquires these image feature amounts. Other image feature amounts include color, brightness, resolution, data amount, and degree of blur / shake, but other image feature amounts may also be used. The image analysis unit 205 extracts and combines necessary information from the meta information input from the image acquisition unit 203 together with these image feature amounts, and outputs the combined information as feature amounts to the score criterion determination unit 206. The image analysis unit 205 also outputs the feature amounts of the analysis result to the image score unit 207. The image analysis unit 205 also outputs the shooting date and time information to the spread allocation unit 212.

[0025] The scoring criteria determination unit 206 uses the feature amounts of the user image group designated by the user image designation unit 202, among the feature amounts obtained from the image analysis unit 205, to determine a scoring criterion for calculating a score in the image scoring unit 207 by a method described below, and provides the result to the image scoring unit 207. The score here is an index showing the suitability of each image for the layout, with a higher score indicating a more suitable image for the layout. The scoring criteria is a criterion for calculating a score in the image scoring unit 207, and by determining the scoring criterion based on the feature amounts of the user image group, it becomes possible to select images according to the user's intention.

[0026] The image scoring section 207 assigns a score to each image in the album candidate image group using the scoring criteria determined by the scoring criteria determination section 206, the adjustment parameters acquired from the adjustment parameter designation section 210, and the feature amount acquired from the image analysis section 205. The scoring results are output to the image selection section 211 and the image layout section 214.

[0027] The spread number input section 209 inputs the number of spreads in the album designated by the album creation condition designation section 201 to the spread allocation section 212. The number of spreads in the album corresponds to the number of templates in which a plurality of images are arranged.

[0028] The image selection unit 211 selects images based on the scores calculated by the image score unit 207, creates a list of layout images (also called the third image group) to be used in the album, and provides it to the spread allocation unit 212 and the adjustment parameter determination unit 208.

[0029] A spread allocation section 212 allocates each image to a spread using the shooting date information for the image group selected by the image selection section 211. Here, an example of allocation in units of spreads will be described, but allocation in units of pages may also be performed.

[0030] The template input section 213 reads from the HDD 104 a plurality of templates corresponding to the template information designated by the album creation condition designation section 201 , and inputs them to the image layout section 214 .

[0031] The image layout unit 214 performs layout processing of images for each spread. Specifically, for the spread to be processed, a template suitable for the image selected by the image selection unit 211 is determined from a plurality of templates input by the template input unit 213, and the layout of each image is determined.

[0032] The layout information output unit 215 outputs layout information to be displayed on the display 105 in accordance with the layout determined by the image layout unit 214. The layout information is, for example, bitmap data in which the data of the selected image selected by the image selection unit 211 is laid out in the determined template.

[0033] The image correction condition input section 216 provides the image correction section 217 with ON / OFF information of the image correction specified by the album creation condition specification section 201. Examples of the types of correction include brightness correction, dodging correction, red-eye correction, contrast correction, etc. ON or OFF of the image correction may be specified for each type of correction, or may be specified for all types collectively.

[0034] Image correction unit 217 performs correction on the layout information held by layout information output unit 215, based on the image correction conditions received from image correction condition input unit 216. The number of pixels of the image processed by image conversion unit 204 to image correction unit 217 can be changed according to the size of the layout image determined by image layout unit 214. In this embodiment, image correction is performed on each image after the layout image is generated, but the present invention is not limited to this, and each image may be corrected before being laid out on a spread or a page.

[0035] The adjustment parameter determination unit 208 determines adjustment parameters from the group of selected images received from the image selection unit 211, and outputs them to the adjustment parameter designation unit 210. The adjustment parameters are parameters that can be designated by the user and are parameters that can adjust the selection results of the image selection unit 211.

[0036] The adjustment parameter designation unit 210 sets the adjustment parameters received from the adjustment parameter determination unit 208 as initial values, and displays a UI screen (user interface screen) described later on the display 105, thereby allowing the user to designate the adjustment parameters. The designated adjustment parameters are output to the image scoring unit 207. In this way, if there is a discrepancy between the images selected by the image selection unit 211 and the user's intention, the user can adjust the scoring criteria using the UI screen described later, and as a result, the user can reselect images based on the adjusted scoring criteria.

[0037] When the album creation application is installed in the image processing device 100, a startup icon is displayed on the top screen (desktop) of the OS (operating system) running on the image processing device 100. When the user double-clicks the startup icon displayed on the display 105 with the pointing device 107, the application program stored in the HDD 104 is loaded into the RAM 103 and executed by the CPU 101 to start up.

[0038] <Example of display screen> 3 is a diagram showing an example of an application start-up screen 301 provided by an album creation application. The application start-up screen 301 (first screen) is displayed on the display 105. The user sets album creation conditions (described later) via the application start-up screen 301, and the album creation condition designation unit 201 acquires the settings from the user via this UI screen.

[0039] Path box 302 on application start screen 301 displays the storage location (path) in HDD 104 of multiple images (e.g. multiple image files) for which an album is to be created. When folder selection button 303 is instructed by a user clicking operation with pointing device 107, a folder selection screen that is standard on the OS is displayed. On the folder selection screen, folders set in HDD 104 are displayed in a tree structure, and the user can select a folder containing images for which an album is to be created, using pointing device 107. The path of the folder in which the album candidate images selected by the user are stored is displayed in path box 302.

[0040] The template designation area 304 is an area for the user to designate template information, and the template information is displayed as an icon. In the template designation area 304, icons of multiple template information are displayed side by side, and the user can select template information by clicking with the pointing device 107.

[0041] The number of spreads box 305 accepts the setting of the number of spreads in the album from the user. The user directly inputs a number into the number of spreads box 305 via the keyboard 106, or inputs a number into the number of spreads box 305 from a list using the pointing device 107.

[0042] A check box 306 accepts ON / OFF designation of image correction from the user. When the check box is checked, image correction ON is designated, and when the check box is not checked, image correction OFF is designated. In this embodiment, all image corrections are turned ON / OFF with one button, but this is not limited to the above, and a check box may be provided for each type of image correction.

[0043] The product designation unit 308 sets the product of the album to be created. The product can set the size of the album and the type of paper for the album. The cover type and the type of binding part can also be set individually.

[0044] When the user presses the OK button 309, the album creation condition designation unit 201 outputs the contents set on the application start screen 301 to the automatic layout processing unit 218 of the album creation application.

[0045] At this time, the path entered in the path box 302 is transmitted to the image acquisition unit 203. The number of spreads entered in the number of spreads box 305 is transmitted to the number of spreads input unit 209. Template information selected in the template designation area 304 is transmitted to the template input unit 213. The ON / OFF status of image correction in the image correction check box is transmitted to the image correction condition input unit 216.

[0046] A reset button 310 on the display screen 301 is a button for resetting each setting information on the application start screen 301 .

[0047] 4 is a diagram showing an example of a user image selection screen 401 provided by an album creation application. When an OK button 309 on the application start screen 301 is pressed, the screen displayed on the display 105 transitions to a user image selection screen 401 (second screen). The user sets a user image via the user image selection screen 401, and the user image designation unit 202 acquires the settings from the user. A user image is an image that constitutes a user image group.

[0048] A path box 402 on the user image selection screen 401 displays the storage location (path) in HDD 104 of a plurality of images (e.g., a plurality of image files) that are the target of the user images. When a folder selection button 403 is instructed by a click operation by the user with the pointing device 107, a folder selection screen is displayed. On the folder selection screen, folders set in HDD 104 are displayed in a tree structure, and the user can select a folder that contains images that are the target of album creation with the pointing device 107. The folder path of the folder selected by the user is displayed in the path box 402. The same folder path as that in the path box 302 may be displayed in the path box 402.

[0049] User image designation area 404 is an area for the user to designate a user image, and multiple image files stored in a folder in HDD 104 designated by path box 402 are displayed as icons. Multiple image icons are displayed side by side in user image designation area 404, and the user can select an image by clicking it with pointing device 107. In Fig. 4, an image marked with a check mark 405 indicates that the image is the image designated by the user. If the user selects it, a check mark 405 is added, and if the user selects it again in that state, the check mark 405 disappears.

[0050] When the user presses the OK button 406, the user image designation unit 202 acquires the contents set on the user image selection screen 401. The user image designation unit 202 outputs the acquired setting contents to the automatic layout processing unit 218 of the album creation application. At that time, a list of image files with check marks 405 is transmitted to the image acquisition unit 203 as information on the user image group.

[0051] A reset button 407 on the user image selection screen 401 is a button for resetting each setting information on the user image selection screen 401 .

[0052] FIG. 5 is a diagram showing an example of an adjustment parameter designation screen provided by an album creation application. The adjustment parameter designation screen 501 (third screen) will be described with reference to FIG. 5(a). The adjustment parameter designation screen 501 is displayed on the display 105 when the user presses, for example, an image reselection button 1204 on a layout confirmation screen (FIG. 12) described later. The user designates adjustment parameters via the adjustment parameter designation screen 501, and the adjustment parameter designation unit 210 acquires the adjustment parameters designated by the user. In addition, the user can check the initial values ​​of the adjustment parameter designation screen 501 and check the appearance frequency of images that have been automatically laid out, etc.

[0053] Photo number adjustment 502 is for adjusting the number of images to be arranged in the album with a slider bar. The user can adjust the number of images to be arranged in the album by moving the slider bar left and right. Portrait photo ratio 503 is for adjusting the ratio of photos containing people to be arranged in the album with a slider bar. Personal ratio 504 is for adjusting the ratio of specific people to be arranged in the album. For people, the number of images to be arranged in the album for each person can be adjusted based on the results of personal recognition analyzed by image analysis unit 205.

[0054] When the user presses the OK button 505, the adjustment parameter designation unit 210 acquires the contents set on the adjustment parameter designation screen 501. When the user presses the cancel button 506, the adjustment parameter designation unit 210 is closed and the adjustment parameter designation process ends. Fig. 5(b) will be described later.

[0055] <Processing flow> Fig. 6 is a flowchart showing the processing of the automatic layout processing unit 218 of the album creation application. The flowchart shown in Fig. 6 is realized, for example, by the CPU 101 reading a program stored in the HDD 104 into the RAM 103 and executing it. In the explanation of Fig. 6, the explanation will be given on the assumption that each component shown in Fig. 2, which functions as a result of the CPU 101 executing the album creation application, executes the processing. The automatic layout processing will be explained with reference to Fig. 6. Note that the symbol "S" in the explanation of each processing means a step in the flowchart (the same applies hereinafter in this specification).

[0056] In S601, the image conversion unit 204 converts an image to generate an analysis image. At the time of S601, it is assumed that various settings through the UI screens of the application launch screen 301 and the user image selection screen 401 have been completed. That is, it is assumed that the album creation conditions, the album candidate image group, and the user image group have been set. Specifically, in S601, the image conversion unit 204 identifies a plurality of image files stored in a folder in the HDD 104 designated by the album creation condition designation unit 201 and the user image designation unit 202. Then, the identified plurality of image files are read from the HDD 104 to the RAM 103. Then, the image conversion unit 204 converts the image of the read image file into an analysis image having a predetermined number of pixels and color information, as described above. In this embodiment, it converts into an analysis image having a short side of 420 pixels and color information converted to sRGB.

[0057] In S602, the image analysis unit 205 executes an analysis process of the analysis image generated in S601 to obtain a feature amount. Examples of the feature amount include meta information stored in the image and image feature amounts that can be obtained by analyzing the image. In this embodiment, the analysis process includes obtaining the degree of focus, face detection, personal recognition, and object determination, but is not limited to these, and other analysis processes may be executed. Details of the process performed by the image analysis unit 205 in S602 will be described below.

[0058] The image analysis unit 205 extracts necessary meta information from the meta information received from the image acquisition unit 203. For example, the image analysis unit 205 acquires the shooting date and time as time information of the image in the image file from the Exif information attached to the image file read from the HDD 104. Note that, for example, the position information or F value of the image may be acquired as the meta information. Furthermore, information other than that attached to the image file may be acquired as the meta information. For example, schedule information linked to the shooting date and time of the image may be acquired.

[0059] The image analysis unit 205 also acquires image features from the analysis image generated in S601. For example, the image features include the degree of focus. Edge detection is performed as a method for determining the degree of focus. A Sobel filter is generally known as a method for detecting edges. The Sobel filter detects edges, and the luminance difference between the start point and end point of an edge is divided by the distance between the start point and the end point to calculate the inclination of the edge. From the result of calculating the average inclination of the edges in an image, an image with a large average inclination can be considered to be in focus more than an image with a small average inclination. If multiple thresholds with different values ​​are set for the inclination, it is possible to output an evaluation value of the amount of focus by determining which threshold or higher the inclination is. In this embodiment, two different thresholds are set in advance, and the amount of focus is determined in three stages: "○", "△", and "×". For example, the inclination of the focus to be adopted in the album is determined as "○", the inclination of the focus that is acceptable is determined as "△", and the inclination that is unacceptable is determined as ×, and each threshold is set in advance. The threshold value may be set by, for example, the creator of the album creating application, or may be configurable on a user interface. Note that, for example, the brightness, color, saturation, or resolution of an image may be acquired as the image feature amount.

[0060] The image analysis unit 205 executes face detection on the analysis image generated in S601. Here, a known method can be used for the face detection process. For example, Adaboost, which creates a strong classifier from a plurality of prepared weak classifiers, is used for the face detection process. In this embodiment, a face image of a person (object) is detected by a strong classifier created by Adaboost. The image analysis unit 205 extracts a face image and acquires the upper left coordinate value and the lower right coordinate value of the position of the detected face image. By having these two types of coordinates, the image analysis unit 205 can acquire the position and size of the face image. Note that, although a case where an object is detected using AdaBoost has been described here, object detection may also be performed using a trained model such as a neural network.

[0061] The image analysis unit 205 performs personal recognition by comparing a face image in the image to be processed based on the analysis image detected by face detection with a representative face image stored in the face dictionary database for each personal ID. The image analysis unit 205 acquires a similarity between each of the representative face images and the face image in the image to be processed. The image analysis unit 205 also identifies a representative face image whose similarity is equal to or greater than a threshold and has the highest similarity. The personal ID corresponding to the identified representative face image is set as the ID of the face image in the image to be processed. Note that, if the similarity between all of the representative face images and the face image in the image to be processed is less than the threshold, the image analysis unit 205 registers the face image in the image to be processed as a new representative face image in the face dictionary database in association with a new personal ID.

[0062] The image analysis unit 205 executes object recognition on the analysis image generated in S601. Here, a known method can be used for the object recognition process. In this embodiment, the object is recognized by a classifier created by Deep Learning. The classifier outputs a likelihood of 0 to 1 for each object, and recognizes that an object exceeding a certain threshold is present in the image. The image analysis unit 205 can acquire the type of object, such as a pet such as a dog or a cat, a flower, a food, a building, an ornament, and a landmark, by recognizing the object image. In this embodiment, the object is discriminated, but this is not limited thereto, and each type may be acquired by recognizing a facial expression, a photographing composition, or a scene such as a trip or a wedding. In addition, the likelihood itself output from the classifier before the discrimination is performed may be used. This allows the score criterion determination unit 206 to determine a score criterion with a greater degree of freedom.

[0063] FIG. 7 is a diagram showing feature amounts. The image analysis unit 205 stores the feature amounts acquired in S602 in a storage area such as the ROM 102, distinguishing them for each ID that identifies each image (analysis image) as shown in FIG. 7. For example, as shown in FIG. 7, the shooting date and time information acquired in S602, the focus discrimination result, the number of detected face images and their position information and similarity, and the type of recognized object are stored in a table format. The position information of the face image is stored distinguishing them for each individual ID acquired in S602. Furthermore, when multiple types of objects are recognized from one image, the multiple types of objects are all stored in the row corresponding to that one image in the table shown in FIG. 7.

[0064] In S603, it is determined whether the processes of S601 to S602 have been completed for all images included in the album candidate image group and the user selected image group. If it is determined that the processes have not been completed, the processes from S601 are repeated. If it is determined that the processes have been completed, the process proceeds to S604. That is, the processes of S601 to S602 are repeatedly executed for all images stored in the specified folder, and the table shown in Fig. 7 containing information on each of the images is created.

[0065] In S604, the scoring criterion determination unit 206 determines the scoring criterion based on the feature amount obtained by the analysis result of the user images of the user image group. That is, the scoring criterion used by the image scoring unit 207 is determined based on the feature amount corresponding to the image ID of the user image group. The score described here is an index indicating the appropriateness of each image to the layout. The scoring criterion is a criterion for calculating the score in the image scoring unit 207, and is, for example, a formula or score calculation algorithm formed using a certain coefficient. The scoring criterion determination unit 206 generates control information used for the scoring criterion, determines the scoring criterion based on the control information, and provides it to the image scoring unit 207. A specific example of the control information is the average value and standard deviation for each feature amount, as described later.

[0066] Fig. 8 is a flowchart showing details of the score criterion determination process in S604. Fig. 9 is a diagram showing feature amounts in a feature amount space. The score criterion determination process performed in S604 will be described below with reference to Figs. 8 and 9.

[0067] In S801, the score criterion determination unit 206 calculates the average value and standard deviation of the group of user images for each feature amount as control information. Hereinafter, the control information will be described with reference to FIG.

[0068] FIG. 9(a) shows an example of plotting the feature amounts of each image on a feature amount space. Since the feature amount space is multidimensional and it is not possible to illustrate all of the feature amounts, FIG. 9(a) will explain two feature amounts extracted from each feature amount: similarity to personal ID1 and shooting time. However, a one-dimensional feature space using one feature amount may be used. The same processing is also performed for other feature amounts. The shaded points, such as point 901, indicate points on which feature amounts corresponding to user images are plotted. The non-shaded points, such as point 902, indicate points on which feature amounts corresponding to album candidate images are plotted. Point 903, indicated by "x" in the figure, indicates an average value vector obtained from the feature amount vector of the user image group.

[0069] FIG. 9(b) plots each image on the feature value axis of similarity to individual ID1. The more the face of individual ID1 is included in the image and the higher the correlation with the person identified by individual ID1, that is, the higher the probability that the image is the same person as individual ID1, the larger the value (to the right of the axis). Point 904 indicates the average value of the user image group on this feature value axis, and width 905 indicates the standard deviation of the user image group on this feature value axis. On the feature value axis of similarity to individual ID1 shown in FIG. 9(b), each user image has similar features, and the standard deviation 905 is small, resulting in a dense distribution. In other words, it can be seen that there is a high probability that many people with individual ID1 are included in all images in the user image group.

[0070] Fig. 9(c) plots each image on the feature axis of the shooting date and time. Point 906 indicates the average value of the user image group on this feature axis, and width 907 indicates the standard deviation of the user image group on this feature axis. On the feature axis of the shooting date and time, each user image has different characteristics, and the standard deviation 907 is a large sparse distribution. In other words, it can be seen that the user image group includes images with various shooting dates and times.

[0071] In S802, it is determined whether the process of S801 has been completed for all feature items. If it is determined that the process has not been completed, the process from S801 is repeated. If it is determined that the process has been completed, the process proceeds to S803.

[0072] In S803, the scoring criterion determination unit 206 acquires the scoring criterion pre-installed in the album creation application. Then, in S804, the scoring criterion is determined using the control information calculated in S801 (in this example, the average value and standard deviation for each feature amount). More specifically, in this embodiment, the following formula (1) and formula (2) are acquired as the scoring criterion, and the control information calculated in S801 (the average value and standard deviation for each feature amount) is applied to formula (1) to determine the scoring criterion. Then, the process proceeds to S605, where the image scoring unit 207 scores the images in the album candidate image group. Below, an example of the scoring criterion determined in S604 and the subsequent scoring process in S605 will be described together. In this embodiment, in S605, the image scoring unit 207 first calculates the score for each image to be scored (referred to as the "image of interest") and for each feature amount using formula (1) using the control information calculated in S801. The images to be scored are images in the album candidate image group. Sji=(50-10×|μi−fji| / σi) / σi···(1) Here, j is the index of the image of interest, i is the index of the feature, fji is the feature of the image of interest, Sji is the score corresponding to the feature fji, and μi and σi are the average value and standard deviation of each feature of the user image group. In other words, in a feature having a small standard deviation 905 of the user image group and a dense distribution as in FIG. 9(b), the score of the feature of the image of interest close to the average value is calculated higher than that of the feature far from the average value. On the other hand, in a feature having a large standard deviation 907 of the user image group and a sparse distribution as in FIG. 9(c), the score of the feature of the image of interest close to the average value does not differ as much as in the case of FIG. 9(b) compared to that of the feature far from the average value.

[0073] Then, in S605, the image scoring section 207 calculates the score of each image of interest using the score Sji for each image of interest and for each feature amount calculated using equation (1) and equation (2). Pj = Σi(Sji) / Ni (2) Here, Pj indicates the score of each image of interest, and Ni indicates the number of feature items. That is, the score of each image of interest is calculated as the average of the scores of each feature. Thus, in S804, the score criterion determination unit 206 determines these formulas (1) and (2) as the score criteria. Thereafter, in S605, the image scoring unit 207 applies formulas (1) and (2) as described above to score each image of interest.

[0074] In addition, since it is preferable that the images used in the album are in focus, a predetermined score may be added to a target image having a focus feature value of "◯" as shown in Fig. 9. Also, a predetermined score may be added to a target image having a specific object. For example, in a case where it is desired to select many images including animals, a predetermined score may be added to a target image including an object belonging to an animal, so that the target image is preferentially selected.

[0075] According to the above score criteria, for features with smaller standard deviation σ, such as the similarity to individual ID1 in Fig. 9(b), the more similar the target image is to the user image group, the higher the score calculated by formula (1). Therefore, for similarity to individual ID1, which is a feature with a dense distribution in the user image group, the more similar the target image is to the user image group, the higher the score will be and the more likely it will be selected.

[0076] On the other hand, for a feature with a large standard deviation σ, such as the shooting date and time in Fig. 9(b), the score calculated by formula (1) is small regardless of the difference between the average value μ and the feature fji of the target image. Therefore, the shooting date and time, which is a feature with a sparse distribution in the user image group, has little effect on image selection regardless of the features of the target image.

[0077] In this way, when the scoring criteria in this embodiment are used, images having characteristics common to the user image group are selected from the album candidate image group, making it possible to select images that have a sense of unity as a layout image group.

[0078] In this embodiment, equation (2) is used as the scoring criterion for calculating the score of each image of interest, but this is not limited to this. For example, the scores of each feature calculated by equation (1) may be weighted and added using equation (3). Pj = Σi(wi × Sji) / Ni (3) Here, wi is a weighting coefficient for each feature. This allows you to change the degree of influence (contribution rate) of each feature on the score of the target image. For example, in Figure 9, by increasing the contribution rate for similarity to individual ID1 and decreasing the weighting contribution rate for the shooting date and time, it becomes possible to assign a score that places more importance on similarity to individual ID1.

[0079] Also, a score criterion may be used in which the largest score among the scores of each feature calculated by formula (1) is set as the score of the image of interest. This allows a high score to be given when one or more of the feature amounts have characteristics similar to those of the user image group. For example, in FIG. 9, even if the feature amount other than the similarity of the image of interest to the individual ID 1 has a characteristic different from that of the user image group, such as the shooting date and time, the image of interest can be given a high score.

[0080] Also, a scoring criterion may be used that calculates the score of the image of interest based on the difference between the average vector of the user images and the vector of the image of interest in a multidimensional feature space. This enables scoring that takes all features into consideration comprehensively. For example, in FIG. 9(a), an album candidate image 902 has a feature close to the average vector 903 in the multidimensional space, so a high score is assigned to the album candidate image 902. On the other hand, an album candidate image 908 has a feature different from the average vector 903 in the multidimensional space, so a high score is not assigned to the album candidate image 908. In other words, when one or more features have a feature different from the average vector 903 of the user images, control can be performed so that a high score is not assigned to the album candidate image 902.

[0081] In this embodiment, the scoring criteria are determined using the average and standard deviation of each feature of the user image group, but this is not limited to this. For example, the scoring criteria may be determined using each feature of the user image group and its standard deviation, as shown in equation (4). Sji=Σk(50-10×|fki−fji| / σi) / (σi×Nk)···(4) Here, k is the index of the user image, fki is the feature of the user image, and Nk is the number of images included in the user image group. In this way, by comparing the feature of the image of interest with the feature of each user image, the difference between the feature of the image of interest and the user image group can be evaluated more accurately.

[0082] In this embodiment, the average value and standard deviation of the feature amounts of the user image group are used as the control information, but the control information is not limited to this, and the median value of the feature amount, distribution shape (normal distribution or Poisson distribution, etc.), quartile range, quartile deviation, etc. may also be used. For example, the median value and standard deviation of the feature amounts of the user image group may be used, or the average value and quartile deviation of the feature amounts of the user image group may be used. In other words, at least one of the average value and median value of the feature amounts of the user image group and at least one of the standard deviation, quartile deviation, and distribution shape may be used.

[0083] Referring again to FIG. 6, following S604, in S605, as described above, the image scoring section 207 acquires the scoring criteria determined by the scoring criteria determination section 206. That is, as described above, the formulas shown in equations (1) to (4) are acquired as the scoring criteria. Then, the image scoring section 207 executes scoring for each image in the album candidate image group based on the acquired scoring criteria. Scoring means assigning a score to each image (scoring). The assigned score is provided to the image selection section 211, and is referred to when selecting images to be used in the layout described below.

[0084] In S606, the image scoring section 207 judges whether or not the image scoring in S605 has been completed for all images in the album candidate image group in the user-specified folder. If it is judged that the image scoring has not been completed, the process from S605 is repeated. If it is judged that the image scoring has been completed, the process proceeds to S607.

[0085] In S607, the image selection unit 211 selects images to be laid out based on the scores for each image calculated by the image score unit 207. Hereinafter, the group of selected images will be referred to as a layout image group.

[0086] In this embodiment, the user images are all selected as images to be laid out. Then, the image selection unit 211 selects images having high scores given by the image score unit 207 from the image group designated by the album creation condition designation unit 201.

[0087] By executing the above-mentioned method in the scoring criteria determination section 206 and the image scoring section 207, images having characteristics similar to those of the user image group are selected in the image selection section 211.

[0088] FIG. 10 is a diagram showing an example of an image selection result. FIG. 10(a) shows an example of an image selection result in this embodiment. FIG. 10(b) shows an example of an image selection result in a comparative example when this embodiment is not used. First, a description will be given with reference to FIG. 10(a). For the purpose of explanation, the feature amount with the highest score is written for each image among different feature amounts A to E. In FIG. 10(a), for a group of album candidate images having different features A to E, images having features A and C are specified as a group of user images in the user image specification unit 202. Then, images having features A and C are selected as a group of layout images in the image selection unit 211. In this way, the user's intention can be understood from the group of user images and reflected in the image selection. On the other hand, FIG. 10(b) shows an example of an image selection result when there is no user-selected image. In this case, images are selected based on the tendency of the feature amounts of the entire group of album candidate images, so that many images having features A and B, which have a large number of images, are selected. 10(b), even if an image is specified by the user as in the case of FIG. 10(a), if the image is automatically selected from the group of album candidate images, the image will still be selected based on the tendency of the feature amounts of the entire group of album candidate images. As a result, many images having features A and B, which have a large number of images, will be selected.

[0089] In the image selection process of S607, images with a score equal to or greater than a certain threshold are selected. This threshold may be built into the application program in advance, may be set within the application, or may be dynamically set from the image group. Dynamically setting from the image group may involve, for example, determining a threshold that selects 20% of the total number of images in the image group. Also, by setting a lower limit for the number of selected images, the number of selected images may be adjusted so that, for example, at least one image is laid out per spread.

[0090] As a method of image selection, a higher selection probability may be set for a higher score, and the image may be selected based on the probability. In this way, by selecting based on the probability, the layout image can be changed every time the automatic layout function is executed by the automatic layout processing unit 218. For example, if the user is not satisfied with the automatic layout result, the user can obtain a layout result different from the previous one by pressing an image reselection button 1204 on the layout confirmation screen (FIG. 12) described later.

[0091] Furthermore, the image selection unit 211 may select images to be laid out from the image group designated by the album creation condition designation unit 201, rather than selecting layout images from the user image group. According to this method, by designating an image with an ideal subject arrangement or composition that is not desired to be included in the album, by using the user image designation unit 202, it becomes possible to search for and select an image close to the ideal.

[0092] Also, images to be laid out may be selected from an image group that combines the image data group and user images specified by the album creation condition specification unit 201. This method creates the possibility of searching for and selecting an image that is more suitable for the layout than the images in the user image group. In this case, the image scoring unit 207 assigns scores not only to the album candidate image group but also to the user image group.

[0093] In S608, the adjustment parameter determination unit 208 determines adjustment parameters from the image group selected in S607 and sends them to the adjustment parameter designation unit 210. The adjustment parameter designation unit 210 sets the received adjustment parameters as initial values ​​and sets values ​​in the adjustment parameter designation screen 501 described above in Fig. 5. The processing of S608 will be described below with reference to Fig. 5. In this embodiment, the number of photos 502, portrait photo ratio 503, and ratio for each individual 504 will be described as examples of adjustment parameters.

[0094] The number of photos 502 is a parameter that indicates the number of photos to be arranged in the album, that is, the number of layout images, whether the number is excessive or insufficient. Specifically, the adjustment parameter determination unit 208 determines the number of photos parameter by equation (5). Number of photos parameter = number of layout images ÷ number of spreads – number of basic photos (5) Here, the basic number of photos refers to the standard number of images to be arranged on one spread, for example, four. The basic number of photos is built into the application in advance. The adjustment parameter specification unit 210 sets an initial value for number of photos 502 based on the received number of photos parameter. For example, if the number of layout images is 28 and the number of spreads is "4," the number of photos parameter becomes "3." This parameter indicates that the current number of photos (number of layout images) is slightly more than the standard number of images.

[0095] FIG. 5B is a diagram for explaining a method for associating the number of photos 502 with the number of photos parameter. In this embodiment, the slider bar has a five-step scale 507. A range 508 of the number of photos parameter corresponding to each scale 507 is determined. The adjustment parameter designation unit 210 presents the adjustment parameter to the user by setting a cursor 509 at the position of the scale corresponding to the received number of photos parameter. This allows the user to know whether the layout image group selected by the image selection unit 211 is more or less than the album. The range 508 of the photo parameter may be incorporated in the application in advance, or may be dynamically set based on the maximum number of photos that can be arranged in one spread and the basic number of photos.

[0096] In the above formula (5), the number of photos parameter is calculated as the excess or deficiency of the standard number of images arranged on one spread, but the present invention is not limited to this. For example, the number of photos parameter may be calculated as the excess or deficiency of the number of images in the layout image group relative to the album candidate image group set by the album creation condition designation unit 201. In this case, for example, the number of photos parameter is calculated by formula (6). Number of photos parameter = number of layout images ÷ number of album candidate images × 100 (6)

[0097] By using the number of photos parameter calculated by formula (6), the user can know approximately what percentage of images in the album candidate image group have been selected as the layout image group. Note that in this embodiment, the adjustment parameter designation unit 210 converts the number of photos parameter into a slider bar for the number of photos 502, but this is not limited to this. The number of photos 502 may be directly linked to the number of layout images, or a numerical value may be displayed and input directly instead of a slider bar.

[0098] The person photo ratio 503 is a parameter that indicates the ratio of photos that include people in the layout image group. Specifically, the adjustment parameter determination unit 208 determines the person photo ratio parameter from equation (7). Portrait photo ratio parameter = number of portrait photos ÷ number of layout images (7) Here, the number of person photos is the number of images in the layout image group that have one or more faces analyzed by the image analysis unit 205. As with the number of photos 502, the adjustment parameter designation unit 210 assigns the person photo ratio parameter received from the adjustment parameter determination unit 208 to the slider bar of person photo ratio 503. Note that, although the ratio of the number of person photos to the number of images in the layout image group is used as the person photo ratio parameter here, it may also be a ratio of how many person photos are used in the album candidate image group.

[0099] The personal ratio 504 is a parameter that indicates the ratio of photos in which each person appears in the layout image group. Specifically, the adjustment parameter determination unit 208 determines the personal ratio parameter by equation (8). Individual ratio parameter k = number of photos of individual ID k ÷ number of photos of a person (8) Here, k represents the index of the personal ID. The number of photos of personal ID k is the number of images in the layout image group in which personal ID k analyzed by the image analysis unit 205 is recorded. As with the number of photos 502, the adjustment parameter designation unit 210 assigns the personal ratio parameter k received from the adjustment parameter determination unit 208 to the slider bar of personal ratio 504. Note that the slider bar of personal ratio 504 may be created as many times as the number of personal IDs, or may be created only for personal IDs detected to have a number equal to or greater than a certain threshold. Furthermore, the adjustment parameter designation unit 210 may display a representative face image of the personal ID corresponding to the slider bar adjacent to the slider bar.

[0100] These adjustment parameters allow the user to be presented with the characteristics of the layout image group. In this embodiment, the number of photos, the ratio of person photos, and the ratio of individuals are described as adjustment parameters, but the present invention is not limited to these. For example, elements other than people may be used, such as the ratio of animal photos or the ratio of flower photos, using the results of object recognition, or the color of the image may be specified. In addition, the location information included in the meta information of the image may be used to display the shooting location on a map, so that the importance of the shooting location can be specified. This concludes the description of the initial adjustment parameter determination process in S608.

[0101] The initial adjustment parameters thus determined are displayed on the adjustment parameter designation screen 501 shown in Fig. 5. When the OK button 505 is pressed, the process proceeds to S609. Since the initial adjustment parameters are not changed at the time of S608, the OK button 505 and the cancel button 506 may not be displayed, and the process may proceed to S609 with the adjustment parameter designation screen 501 displayed for a predetermined time. The process may proceed to the creation of layout information in S612, which will be described later. After the creation of the layout information is completed, the adjustment parameter designation screen 501 may transition to a layout confirmation screen (Fig. 12), which will be described later, for confirming the layout.

[0102] Alternatively, the initial adjustment parameters determined in S508 may not be presented to the user at the time of determining the initial adjustment parameters in S608. Thereafter, the adjustment parameters may be presented to the user along with a layout confirmation screen (FIG. 12) displayed as a result of the process in S612, which will be described later. Also, in this example, the process is described on the assumption that the user does not change the adjustment parameters on the adjustment parameter designation screen 501 in S608, but when the initial adjustment parameters in S608 are presented, the user may designate adjustment parameters. In this case, following S608, the process proceeds to S615, which will be described later.

[0103] Returning to FIG. 6 for further explanation, in S609, the spread allocation section 212 divides and allocates the layout image group acquired in S607 into image groups for the number of spreads input from the spread number input section 209. In this embodiment, the layout images are arranged in the order of shooting time acquired in S602, and are divided at a location where the shooting time difference between adjacent images is large. This process is performed until the layout images are divided into the number of spreads input from the spread number input section 209. In other words, division is performed (number of spreads-1) times. This makes it possible to create an album in which images are arranged in the order of shooting time. The process of S609 may be performed on a page basis, rather than on a spread basis.

[0104] In S610, the image layout unit 214 determines the image layout. An example will be described below in which the template input unit 213 inputs (a) to (p) of Fig. 11 for a certain spread in accordance with the specified template information.

[0105] 11 is a diagram showing a template group used for laying out image data. Each of the multiple templates included in the template group corresponds to a double-page spread. Template 1101 is a single-sheet template. Template 1101 includes a main slot 1102, a sub-slot 1103, and a sub-slot 1104. Main slot 1102 is the main slot (a frame for laying out images) in template 1101, and is larger in size than sub-slot 1103 and sub-slot 1104.

[0106] Here, as an example, the number of slots of the input template is specified as 3. Assume that the orientations of the three selected images are arranged according to the shooting dates and times, whether they are portrait or landscape, as shown in Figure 11(q).

[0107] Here, in each group of images allocated to a spread, the image with the highest score calculated by the image score unit 207 is set as the main slot, and the other images are set as the sub-slot. Note that whether an image is for the main slot or the sub-slot may be set based on a certain feature acquired by the image analysis unit, or may be set randomly. Also, the user-selected image may be given priority as the main slot.

[0108] Here, it is assumed that image data 1105 is for the main slot, and image data 1106 and 1107 are for the sub-slot. In this embodiment, image data with an older date and time of shooting is laid out in the upper left of the template, and images with a newer date and time of shooting are laid out in the lower right. In FIG. 11(q), image data 1105 for the main slot has the most recent date and time of shooting, so the templates in FIG. 11(i) to (l) are candidates. Also, the older image data 1106 for the sub-slot is a portrait image, and the newer image data 1107 is a landscape image, so that the template in FIG. 11(j) is determined as the template most suitable for the selected three image data, and the layout is determined. In S610, it is determined which image is to be laid out in which slot of which template.

[0109] In S611, the image correction unit 217 executes image correction. When information indicating that image correction is ON is input from the image correction condition input unit 216, the image correction unit 217 executes image correction. As image correction, for example, dodging correction (brightness correction), red-eye correction, or contrast correction is executed. When information indicating that image correction is OFF is input from the image correction condition input unit 216, the image correction unit 217 does not execute image correction. Image correction can also be executed on image data whose short side is 1200 pixels and whose size has been converted to the sRGB color space, for example.

[0110] In S612, the layout information output unit 215 creates layout information. The image layout unit 214 lays out the image data, on which the image correction in S611 has been performed, for each slot of the template determined in S610. At this time, the image layout unit 214 changes the size of the image data to be laid out in accordance with the size information of the slot, and lays it out. Then, the layout information output unit 215 generates bitmap data, in which the image data has been laid out in the template, as output image data.

[0111] In S613, the image layout unit 214 judges whether the processes of S610 to S612 have been completed for all the spreads. If it is judged that the processes have not been completed, the process repeats from S610. If it is judged that the processes have been completed, the process proceeds to S614.

[0112] In S614, the automatic layout processing unit 218 determines whether or not to recreate the layout information created in the processes of S609 to S613. That is, it determines whether or not the layout process is complete. If it is determined that the layout process is not complete (that is, the layout information is to be recreated), the process transitions to S615. If it is determined that the layout process is complete, the automatic layout process of Fig. 6 ends. The determination of whether or not to recreate the layout information is made based on, for example, whether or not the user has pressed the image reselection button on the layout confirmation screen.

[0113] 12 is a diagram showing an example of a layout confirmation screen. In the layout confirmation screen 1201 (fourth screen), a thumbnail display area 1202 in which thumbnails of each spread of a photo album are displayed is provided at the top, and a display area for a photo album 1203 which is a layout image is provided at the bottom. In this manner, in the layout confirmation screen 1201, the layout information output unit 215 displays on the display 105 the bitmap data (output image data) in which images are laid out in the template, generated in S612. This allows the user to check the layout results. When the user presses the image reselection button 1204, the process transitions to S615.

[0114] In S615, the adjustment parameter designation unit 210 performs display control to display an adjustment parameter designation screen 501 on the display 105. At this time, the adjustment parameter designation screen 501 reflects and displays the adjustment parameter values ​​determined in S608. The user designates the adjustment parameters by operating the adjustment parameter designation screen 501. This allows the user to know how the layout image group can be adjusted, and can avoid a situation in which the layout image group does not change even if the adjustment parameters are changed. The adjustment parameter designation unit 210 sends the adjustment parameters at the time when the OK button 505 is pressed to the image scoring unit 207.

[0115] In S616, the image scoring section 207 again scores each image in the album candidate image group based on the scoring criteria determined in S604 and the adjustment parameters specified in S615. In S605, the scores were assigned based on the scoring criteria, but in S616, the scores are further adjusted based on the adjustment parameters. In this embodiment, the number of photos parameter, the person photo ratio parameter, and the individual ratio parameter will be described as examples.

[0116] A layout target value, which is a target value for the total number of selected images to be used in the album, is calculated using equation (9). A person photo target value, which is the number of person photos in the layout image group, is calculated using equation (10). A person ID target value, which is a target value for the number of images for each person in the person photo group, is calculated using equation (11). A person ID target value is found for each person. Layout target value = (number of photos parameter + number of basic photos) × number of spreads (9) Target value for portrait photos = number of layout images × portrait photo ratio parameter (10) Target value of personal ID = Number of person photos × Personal ID ratio parameter (11)

[0117] FIG. 13 is a diagram for explaining score adjustment. A method of score adjustment using a personal photo ratio parameter will be described with reference to FIG. 13. FIG. 13 is a schematic diagram in which images on which each personal ID is recorded are plotted on an axis representing the score P given to the image. Axis 1301 represents an image on which personal ID1 is recorded. Axis 1302 represents an image on which personal ID2 is recorded. Note that, if an image contains multiple individuals, the same image is plotted for each personal ID. For the sake of explanation, it will be explained here assuming that the portrait photo target value is 10 photos according to formula (10), the personal ID1 target value is 6 photos, and the personal ID2 target value is 4 photos according to formula (11).

[0118] Point 1303 is the score P m The point 1304 is the score P n The image scorer 207 shows m and P n In this example, all the images on axis 1301, which is the image group of person ID 1, are assigned points. n― P m The axis 1305 represents the plot of the image of person ID1 after the score is added, and the score after the addition is P' m Then, P´m =P m +(P n― P m )=P n It is as follows.

[0119] As a result, when selecting the top 10 scoring portrait photos, 6 images can be selected for personal ID 1 and 4 images for personal ID 2. Note that although the explanation here is for a case where there are two personal IDs, in the case where there are multiple personal IDs, addition can be done in the same way so that the target value for each personal ID approaches the target value calculated by equation (11).

[0120] Next, a method of score adjustment using the person photo ratio parameter will be described. Among the layout images, images with people and images without people are arranged with the score P as the axis. From equations (9) and (10), the target value for the number of non-person photos is "layout target value - person photo target value". Therefore, similar to the score adjustment using the personal photo ratio parameter described above, when images equivalent to the layout target value are selected from the top scores, scores are added so that the number of person photos and the number of non-person photos approach their respective target values.

[0121] Finally, the score of the entire album candidate image group is added so that the number of images designated by the parameter number of photos is selected. thr When the scores of the album candidate images are arranged in descending order, the score that is located at the layout target value is P target Then, the added value is P thr -P target Let us assume that.

[0122] The score criteria determined by the score criteria determination unit 206 are influenced by a variety of feature quantities in a complex manner, and therefore in many cases are not indices that can be simply presented to the user. However, by using the above-described process to specify indices important to the album as adjustment parameters in the adjustment parameter designation unit 210 and adjust the score, adjustments that reflect the user's intentions can be made.

[0123] Continuing the explanation by returning to Fig. 6, in S617, the image selection unit 211 reselects a group of layout images from the scores for each image calculated in S616, in the same manner as in S607, and repeats the process from S609.

[0124] When the automatic layout process in FIG. 6 is completed, the layout information output unit 215 outputs to the display 105 the bitmap data (output image data) in which the images are laid out in the template, generated in S612. That is, the layout confirmation screen 1201 in FIG. 12 is displayed. The above is an explanation of the automatic layout process flow in which the user reselects images by specifying adjustment parameters. The generated image data may be uploaded to a server via the Internet based on a user instruction. Printing and bookbinding processes are executed based on the uploaded image data, and an album (photo book) as a booklet is created and delivered to the user.

[0125] As described above, according to this embodiment, it is possible to appropriately select images that meet the user's intention. More specifically, in this embodiment, the initial value of the adjustment parameter is determined based on the image selection result, so that the state of the image selection result can be shown to the user. In this embodiment, the layout image group is automatically selected based on the user image of the user image group, but the user can adjust this selection result according to his or her own intention. Therefore, by the user re-adjusting the adjustment parameter, it is possible to perform adjustment that takes into account the user's intention. As a result, it is possible to appropriately select images that meet the user's intention. Note that in this embodiment, the selection result in which the layout image group is automatically selected is adjusted based on the user image of the user image group, not the overall tendency of the layout image group. Therefore, for example, even if the overall tendency of the layout image group is different from the tendency of the user image group, the number of images and the frequency of appearance of each object appear as the selection result according to the user image group. By the user further adjusting the adjustment parameter, such a selection result can be further adjusted to take into account the user's intention.

[0126] <<Second embodiment>> In the first embodiment, an example has been described in which the score is adjusted based on the adjustment parameters specified by the adjustment parameter specification unit 210. In the second embodiment, an example will be described in which the score criteria is adjusted based on the adjustment parameters specified by the adjustment parameter specification unit 210.

[0127] Fig. 14 is a software block diagram of an album creation application in the second embodiment. The components with the same numbers as those in Fig. 2 perform the same processes as those described in the first embodiment, and therefore the description thereof will be omitted here.

[0128] The adjustment parameter designation unit 1401, like the adjustment parameter designation unit 210, displays an adjustment parameter designation screen 501 on the display 105 to allow the user to designate adjustment parameters. In this embodiment, the initial values ​​of the adjustment parameters may be designated by the user before the automatic layout process, or initial values ​​previously built into the application may be used. As described in the first embodiment, the adjustment parameters may be determined according to an image from the layout image group selected first. The designated adjustment parameters are output to the score criterion determination unit 1402.

[0129] A score criterion determination unit 1402 determines a score criterion by using the feature amounts of the user image group specified by the user image designation unit 202, among the feature amounts obtained from the image analysis unit 205, and also by using the adjustment parameters specified by the adjustment parameter designation unit 1401. The score criterion determination unit 1402 outputs the determined score criterion to an image score unit 1403.

[0130] The image scoring section 1403 uses the scoring criteria acquired from the scoring criteria determination section 1402 and the feature amount acquired from the image analysis section 205 to score each image in the album candidate image group.

[0131] Fig. 15 is a flowchart showing the processing of the automatic layout processing unit 218 of the album creation application in this embodiment. Note that, in the processing given the same numbers as in Fig. 5, the same processing as the processing described in the first embodiment is performed, and therefore the description thereof will be omitted here.

[0132] When the image analysis is completed in S501 to S503, the process proceeds to S1501. In S1501, the score criterion determination unit 1402 determines the score criterion to be used in the image score unit 1403 based on the feature amount and adjustment parameters corresponding to the image ID of the user image group.

[0133] Fig. 16 is a flowchart showing the details of the scoring criterion determination process in S1501. Note that the processes in Fig. 16 denoted by the same numbers as in Fig. 8 are the same processes as in Fig. 8, and therefore their explanations are omitted here. Note that the scoring criterion determined in S804 is called the first scoring criterion. After the first scoring criterion is determined in S804, the process proceeds to S1601.

[0134] In S1601, the score criteria determination unit 1402 acquires adjustment parameters from the adjustment parameter designation unit 1401. In this embodiment, a number of photos parameter, a person photo ratio parameter, and a person ratio parameter are used as adjustment parameters. Unlike the example of the first embodiment, in this embodiment, each adjustment parameter represents a change from a standard setting. For example, if the slider bar for the number of photos 502 on the adjustment parameter designation screen 501 has five stages and the center of the slider bar is the standard setting, the number of photos adjustment parameter takes values ​​from -2 to +2. Note that the steps of the slider bar and the standard setting are set in the program in advance. Also, instead of using the slider bar, for example, ON and OFF using a check box may be assigned to the adjustment parameters 0 and 1, respectively.

[0135] In S1602, the scoring criterion determination unit 1402 determines a second scoring criterion. The first scoring criterion determined in S804 is a scoring criterion based on the feature amount of the user image selected by the user. On the other hand, the second scoring criterion is a scoring criterion determined based on the adjustment parameters acquired in S1601. In this embodiment, the scoring criteria based on the number of photos parameter, the person photo ratio parameter, and the person ratio parameter are determined as the second scoring criterion. Then, in S1603, the scoring criterion used by the image scoring unit 1403 is determined using these first and second scoring criteria. The process of determining the second scoring criterion will be described below.

[0136] 17 is a flowchart for explaining the second score criterion determination process. In S1701, the score criterion determination section 1402 acquires the feature amount of each image in the group of album candidate images analyzed by the image analysis section 205. Here, the face area is acquired. The face area refers to the area of ​​the face region detected by the image analysis section 205 in the target image. Note that since the area of ​​the face region detected by the image analysis section 205 depends on the image size, the face area may be normalized by dividing the area of ​​the face region by the image size.

[0137] In S1702, the score criterion determination unit 1402 determines the score criterion associated with the number of photos parameter. S S is determined from equations (12) and (13). R = 0.3 × ΣPj (12) S S = number of photos parameter × R (13) Here, R represents the degree of influence that the number of photos parameter has on the score, and the degree of influence can be adjusted by changing the coefficient 0.3 in formula (12). In this embodiment, it is pre-installed in the application, but it may be made specifiable. Pj represents the score of each image of interest, as described in the first embodiment. The number of photos can be adjusted by adjusting the overall score using formula (13).

[0138] In S1703, the score criterion determination unit 1402 determines the score criterion associated with the portrait photo ratio parameter.F S is determined from equation (14). F S = portrait ratio parameter × R × (face area – average face area) ÷ standard deviation of face area (14) Here, the average face area and the standard deviation of the face area refer to the average face area and the standard deviation of the face area for all images in the group of album candidate images.

[0139] In S1704, the score criterion determination unit 1402 determines the score criterion associated with the personal photo ratio parameter. Pk S is determined from equation (15). Pk S = individual photo ratio parameter × R × (facial area of ​​individual IDk - average facial area of ​​individual IDk) ÷ standard deviation of facial area of ​​individual IDk (15) Here, the average face area and standard deviation of the individual ID k refer to the average face area and standard deviation of the individual ID k for all images in the album candidate image group. Pk S is calculated for the number Np of people specified on the adjustment parameter specification screen 501. In this way, the score criteria based on the number of photos parameter, the person photo ratio parameter, and the person ratio parameter are determined as the second score criteria. Returning to FIG.

[0140] In S1603, the scoring criterion determination unit 1402 combines the first scoring criterion and the second scoring criterion to determine the scoring criterion to be used in the image scoring unit 1403. Here, the formula (15) obtained by adding the first scoring criterion and the second scoring criterion is set as the final scoring criterion. Qj=Pj+ S Sj+ F Sj+Σ[k=1..Np] Pk Sj / Np···(15) Here, Qj represents the score of each image of interest. This allows for scoring that reflects the adjustment parameters in addition to the score that reflects the user's intention based on the user-selected images. Note that, although equation (2) or (3), equation (13), equation (14), and equation (15) are added uniformly here, the scores to be emphasized may be changed by adding weights to each score. This concludes the explanation of the score criterion determination process in S1501. Return to FIG. 15.

[0141] In S1502, the image scoring section 1403 acquires the scoring criteria determined by the scoring criteria determination section 1402, and executes scoring for each image in the album candidate image group based on the scoring criteria. The assigned scores are provided to the image selection section 211. Other processing is similar to the example described in the first embodiment. Note that in this embodiment, after the adjustment parameters are specified in S515, the scoring criteria determination processing in S1501 is performed again. Note that in the first scoring criteria determination processing in S1501, the adjustment parameter designation screen 501 may or may not be displayed, as described in the first embodiment.

[0142] As described above, according to this embodiment, a score criterion is used that is determined from two perspectives: a first score criterion based on a group of user images and a second score criterion based on adjustment parameters. This makes it possible to reflect the user's manual adjustments while taking into account the user's intentions from the user images.

[0143] <<Other embodiments>> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0144] 100 Image processing device 201 Album creation condition specification section 202 User image specification section 205 Image Analysis Unit 206 Scoring Criteria Determination Department 207 Image Scorer 210 Adjustment parameter specification section 211 Image Selection Section

Claims

1. A program comprising: The computer of the image processing device a first designation means for designating a first image group; a first selection means for selecting the second image group from the first image group based on a user operation for specifying the second image group from the first image group; analysis means for analyzing each image included in the second set of images; a determining means for determining criteria for selecting images from the first set of images based on the analysis of the second set of images; a second selection means for selecting a third image group from the first image group, which is the image group from which the second image group was selected, based on the criterion based on the analysis result of the images included in the second image group; a first generation means for generating a first album by automatically arranging the third image group in a plurality of templates by the program; a first display means for displaying the first album on a display unit; a change operation receiving means for receiving a change operation for changing a parameter value of the group of images displayed on the display unit from the value corresponding to the third group of images selected by the second selection means to another value after the first album is displayed; a third selection means for selecting a fourth group of images from the first group of images using the parameter of the other value corresponding to the change operation when the change operation is accepted; a second generation means for generating a second album by automatically arranging the fourth group of images in a plurality of templates by the program when the change operation is accepted; a second display means for displaying the second album on the display unit; A program to function as a

2. The program according to claim 1, characterized in that the third selection means determines a score for each image included in the first group of images based on the parameter of the other value corresponding to the change operation, and selects the fourth group of images based on the determined scores.

3. the criterion is determined again further based on the parameter of the other value corresponding to the modification operation; 3. The program of claim 1, wherein the fourth group of images is selected based on a re-determined criterion.

4. 4. The program according to claim 1, further comprising a second determination means for determining an initial value of the parameter based on the image selected by the second selection means.

5. a screen on which the initial value determined by the second determination means is set is displayed on the display unit; 5. The program according to claim 4, wherein the change operation is accepted via the screen.

6. 6. The program according to claim 1, wherein the parameters include at least one of the number of images selected by the third selection means, the ratio of images including people among the selected images, and the ratio of images in which a specific person is shown among the selected images.

7. the criteria are score criteria for determining a score for each image in the first set of images; 7. The program of claim 1, wherein the criterion is different from the parameter.

8. The program according to any one of claims 1 to 7, characterized in that the analysis means outputs at least one image feature obtained by analyzing each image, which is the degree of focus, the result of face detection, and the result of object discrimination, as the analysis result.

9. The computer, 9. The program according to claim 1, further functioning as a print control unit that executes a process for printing the image selected by the third selection unit.

10. a first designation means for designating a first image group; a first selection means for selecting the second image group from the first image group based on a user operation for specifying the second image group from the first image group; analysis means for analyzing each image included in the second set of images; a determining means for determining criteria for selecting images from the first set of images based on the analysis of the second set of images; a second selection means for selecting a third image group from the first image group, which is the image group from which the second image group was selected, based on the criterion based on the analysis result of the images included in the second image group; a first generation means for automatically arranging the third image group in a plurality of templates to generate a first album; a first display means for displaying the first album on a display unit; a change operation receiving means for receiving a change operation for changing a parameter value of the group of images displayed on the display unit from the value corresponding to the third group of images selected by the second selection means to another value after the first album is displayed; a third selection means for selecting a fourth group of images from the first group of images using the parameter of the other value corresponding to the change operation when the change operation is accepted; a second generation means for generating a second album by automatically arranging the fourth image group in a plurality of templates when the change operation is accepted; a second display means for displaying the second album on the display unit; 13. An image processing device comprising:

11. The image processing device according to claim 10, characterized in that the third selection means determines a score for each image included in the first group of images based on the parameter of the other value corresponding to the change operation, and selects the fourth group of images based on the determined score.

12. the criterion is determined again further based on the parameter of the other value corresponding to the modification operation; 12. The image processing apparatus according to claim 10, wherein the fourth group of images is selected based on a re-determined criterion.

13. 13. The image processing apparatus according to claim 10, further comprising a second determination means for determining an initial value of the parameter based on the image selected by the second selection means.

14. a screen on which the initial value determined by the second determination means is set is displayed on the display unit; 14. The image processing apparatus according to claim 13, wherein the change operation is accepted via the screen.

15. 15. The image processing device according to claim 10, wherein the parameters include at least one of the number of images selected by the third selection means, the ratio of images including people among the selected images, and the ratio of images in which a specific person is shown among the selected images.

16. the criteria are score criteria for determining a score for each image in the first set of images; 16. The image processing apparatus according to claim 10, wherein the criterion is different from the parameter.

17. 17. The image processing device according to claim 10, wherein the analysis means outputs at least one image feature obtained by analyzing each image, the degree of focus, the result of face detection, and the result of object discrimination, as the analysis result.

18. 18. The image processing apparatus according to claim 10, further comprising a print control unit that executes a process for printing the image selected by the third selection unit.

19. A first designation step of designating a first group of images; a first selection step of selecting the second image group from the first image group based on a user operation for specifying the second image group from the first image group; an analyzing step of analyzing each image in the second set of images; determining criteria for selecting images from the first set of images based on the analysis of the second set of images; a second selection step of selecting a third image group from the first image group, which is the image group from which the second image group was selected, based on the criterion based on the analysis result of the images included in the second image group; a first generation step of automatically arranging the third image group in a plurality of templates to generate a first album; a first display step of displaying the first album on a display unit; a change operation receiving step of receiving a change operation for changing parameter values ​​of the image group displayed on the display unit from the values ​​corresponding to the third image group selected in the second selection step to other values ​​after the first album is displayed; a third selection step of selecting a fourth group of images from the first group of images using the parameter of the other value corresponding to the change operation when the change operation is accepted; a second generation step of automatically arranging the fourth image group in a plurality of templates to generate a second album when the change operation is accepted; a second display step of displaying the second album on the display unit; 13. An image processing method comprising:

Images