Image processing apparatus, image processing method, and program

Abstract

To appropriately select an image meeting a user's intention.SOLUTION: A program causes a computer to function as first designation means that designates a first image group, second designation means that designates a second image group, analysis means that analyzes images included in the first image group and the second image group, determination means that determines a reference based on a result of analysis on the second image group, and selection means that selects an image from the first image group based on the reference and a result of analysis on the first image group.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a technique for selecting one or more images from a plurality of images according to a predetermined criterion.

Background Art

[0002] There is an automatic layout technique that automatically selects images for album creation from a plurality of images, automatically determines an album template, and performs automatic assignment of images to the template.

[0003] Patent Document 1 discloses a technique for classifying a plurality of images of album candidates into a plurality of image groups so that the images with a similarity equal to or higher than a threshold value are grouped together, and extracting an image from the images included in the classified plurality of image groups. In Patent Document 1, the ratio of the images to be extracted from each image group is determined by the ratio of the number of images included in the image group after classification or by the user specifying the extraction ratio of each image group.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the method described in Patent Document 1, there is a possibility that an image that conforms to the user's intention may not be appropriately selected.

[0006] An object of the present invention is to provide a technique for appropriately selecting an image that conforms to the user's intention.

Means for Solving the Problems

[0007] A program according to one aspect of the present invention causes a computer to function as: a first specifying means for specifying a first image group; a second specifying means for specifying a second image group; an analyzing means for analyzing each image included in the first image group and the second image group; a determining means for determining criteria for each of a plurality of feature amounts based on the analysis result of the second image group a determination means for determining a scoring criterion for assigning scores to images as the reference and a scoring means for performing scoring of the first image group based on the scoring criterion and the analysis result of the first image group; a selecting means for selecting an image from the first image group based on the criteria for each of the plurality of feature amounts and the plurality of feature amounts obtained from each image as the analysis result of the first image group a selection means for selecting an image from the first image group based on the score obtained by the scoring and causing it to function as A program for the above, wherein the determination means has a plurality of scoring criteria and is configured to switch the scoring criteria for each type of analysis result, and the determination means switches the scoring criteria for each type of analysis result according to whether the analysis result of the second image group is less than a predetermined value. .

Advantages of the Invention

[0008] According to the present invention, an image that conforms to the user's intention can be appropriately selected.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention according to the claims, and not all combinations of the features described in the present embodiments are essential for the solution means of the present invention. The same reference numerals are assigned to the same components, and the description thereof will be omitted.

[0011] <<First Embodiment>> In the present embodiment, in an image processing apparatus, a method of operating an application for creating an album (hereinafter also referred to as an "app") to generate an automatic layout will be described as an example. In the following description, the "image" includes still images, moving images, and frame images extracted from moving images, unless otherwise specified. Also, the images here can include still images, moving images, and frame images in moving images that are held on a network such as a service on the network and a storage on the network and can be acquired via the network.

[0012] FIG. 1 is a block diagram showing the hardware configuration of the image processing apparatus. As the image processing apparatus 100, for example, a personal computer (hereinafter referred to as a PC) or a smartphone can be mentioned. In the present embodiment, the image processing apparatus will be described as a PC. The image processing apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, a display 105, a keyboard 106, a pointing device 107, and a data communication unit 108.

[0013] The CPU (Central Processing Unit / Processor) 101 comprehensively controls the image processing apparatus 100, and realizes the operations of the present embodiment, for example, by reading out a program stored in the ROM 102 into the RAM 103 and executing it. In FIG. 1, there is one CPU, but it may be composed of a plurality of CPUs. The ROM 102 is a general-purpose ROM, and stores, for example, a program executed by the CPU 101. The RAM 103 is a general-purpose RAM, and is used as a working memory for temporarily storing various kinds of information when the program is executed by the CPU 101. The HDD (Hard Disk) 104 is a storage medium (storage unit) for storing image files, a database for holding processing results such as image analysis, and templates used by an album creation application, etc.

[0014] The display 105 displays an electronic album to the user as the user interface (UI) of the present embodiment and the layout result of image data (hereinafter also referred to as "image"). The keyboard 106 and the pointing device 107 receive instruction operations from the user. The display 105 may have a touch sensor function. The keyboard 106 is used, for example, when the user inputs the number of spreads of the album to be created on the UI displayed on the display 105. The pointing device 107 is used, for example, when the user clicks a button on the UI displayed on the display 105.

[0015] The data communication unit 108 communicates with an external device via a network such as wired or wireless. The data communication unit 108 transmits, for example, the data laid out by the automatic layout function to a printer or a server capable of communicating with the image processing apparatus 100. The data bus 109 connects the respective blocks in FIG. 1 so as to be mutually communicable.

[0016] Note that the configuration shown in FIG. 1 is merely an example and is not limited thereto. For example, the image processing apparatus 100 may not have the display 105 and may display the UI on an external display.

[0017] The album creation application in this embodiment is stored in the HDD 104. Then, as will be described later, when the user selects the application icon displayed on the display 105 with the pointing device 107 and performs operations such as clicking or double-clicking, the application is launched.

[0018] FIG. 2 is a software block diagram of the album creation application. The album creation application includes an album creation condition specifying unit 201, a user image specifying unit 202, and an automatic layout processing unit 218. The automatic layout processing unit 218 includes an image acquisition unit 203, an image conversion unit 204, an image analysis unit 205, a scoring criterion determination unit 206, an image scoring unit 207, a photo number adjustment amount input unit 208, a page number input unit 209, and a photo number determination unit 210. Further, it includes an image selection unit 211, a page allocation unit 212, a template input unit 213, an image layout unit 214, a layout information output unit 215, an image correction condition input unit 216, and an image correction unit 217.

[0019] Each program module corresponding to each component shown in FIG. 2 is included in the above-described album creation application. Then, by the CPU 101 executing each program module, the CPU 101 functions as each component shown in FIG. 2. Hereinafter, as an explanation of each component shown in FIG. 2, each component will be described as performing various processes. Also, FIG. 2 particularly shows a software block diagram regarding the automatic layout processing unit 218 that executes the automatic layout function.

[0020] The album creation condition specifying unit 201 specifies album creation conditions to the automatic layout processing unit 218 according to the UI operation by the pointing device 107. In the present embodiment, as album creation conditions, an album candidate image group (also referred to as a first image group) including candidate images to be used in the album, the number of spreads, the type of template, and whether to perform image correction on the album can be specified. Also, the number of photo adjustment amount for adjusting the number of sheets to be arranged in the album, and the commercial material for creating the album can be specified. The specification of the album candidate image group may be specified by, for example, the attached information of individual images such as the shooting date and time, or the attribute information, or may be specified based on the structure of the file system including images such as devices and directories. Also, it is possible to specify two arbitrary images, and all images taken between the dates when the respective image data were taken may be used as the target image group. In this specification, "spread" corresponds to one display window in display, and in a printed matter, it corresponds to a pair of adjacent pages (that is, two pages) that a user can view at once when the book is opened. Note that there are two cases for the two pages in a spread: one case is where the two pages printed on different sheets are bound so that they are adjacent to form a spread, and the other case is where the one sheet printed is folded in the middle to form a spread.

[0021] The user image specifying unit 202 causes the user to specify a user image group (also referred to as a second image group) that represents the user's hobby or preference and is to be adopted in the album. As the user image group, it is possible to select one or more and not more than the number of photos determined by the photo number determination unit 210. Also, save the image group used in the albums created in the past in the cloud that can be accessed via the HDD 104 or the data communication unit 108, and in addition to the above specified method, the user may be allowed to specify from among the image groups. The images specified here may or may not be included in the images specified by the album creation condition specifying unit 201. That is, the user image group may or may not be included in the album candidate image group.

[0022] The image acquisition unit 203 acquires the image group specified by the album creation condition specifying unit 201 and the user image specifying unit 202 from the HDD 104. As meta information, the image acquisition unit 203 outputs to the image analysis unit 205 information such as the width and height information of the acquired image, the shooting date and time information included in the Exif information at the time of shooting, and information indicating whether the image is included in the user image group. Further, the image acquisition unit 203 outputs the acquired image data to the image conversion unit 204. Identification information is assigned to each image, and the meta information output to the image analysis unit 205 and the image data output to the image analysis unit 205 via the image conversion unit 204 described later can be associated with each other in the image analysis unit 205.

[0023] Examples of the images stored in the HDD 104 include still images and frame images cut out from moving images. The still images and frame images are those acquired from imaging devices such as digital cameras and smart devices. The imaging device may be provided in the image processing apparatus 100 or may be provided in an external device. When the imaging device is an external device, the image is acquired via the data communication unit 108. Further, the still images and the cut-out images may be images acquired from a network or a server via the data communication unit 108. Examples of the images acquired from a network or a server include social networking service images (hereinafter referred to as "SNS images"). The program executed by the CPU 101 analyzes the data attached to each image to determine the storage source. The SNS image may manage the acquisition destination within the application by acquiring the image from the SNS via the application. The images are not limited to the images described above and may be other types of images.

[0024] The image conversion unit 204 converts the image data input from the image acquisition unit 203 into the number of pixels and color information for use by the image analysis unit 205, and outputs it to the image analysis unit 205. In the present embodiment, the image is converted to a predetermined number of pixels, for example, 420 pixels on the short side, and the long side is converted to a size that maintains the ratio of each original side. Further, in order to perform color analysis, it is converted to be unified in a color space such as sRGB. In this way, the image conversion unit 204 converts the image into an analysis image with the number of pixels and color space unified. The image conversion unit 204 outputs the converted image to the image analysis unit 205. Also, the image conversion unit 204 outputs the image to the layout information output unit 215 and the image correction unit 217.

[0025] The image analysis unit 205 analyzes the image data of the analysis image input from the image conversion unit 204 by the method described later to obtain image feature amounts. As the analysis processing, each process of estimating the focus degree of focus, face detection, personal recognition, and object determination is executed, and these image feature amounts are obtained. Other examples of image feature amounts include hue, brightness, resolution, data amount, and degree of blur / vibration, but other image feature amounts may also be used. The image analysis unit 205 extracts and combines the necessary information from the meta information input from the image acquisition unit 203 together with these image feature amounts, and outputs them to the score criterion determination unit 206 as feature amounts. Also, the image analysis unit 205 outputs the feature amounts of the analysis result to the image scoring unit 207. Also, the image analysis unit 205 outputs the shooting date and time information to the split assignment unit 212.

[0026] The score criterion determination unit 206 determines the score criterion for calculating the score in the image scoring unit 207 by the method described later using the feature amounts of the user image group specified by the user image specifying unit 202 among the feature amounts acquired from the image analysis unit 205, and provides it to the image scoring unit 207. The score here is an index indicating the appropriateness of the layout for each image, and the higher the score, the more suitable it is for the layout. Also, the score criterion is the criterion for calculating the score in the image scoring unit 207, and by determining the score criterion based on the feature amounts of the user image group, it becomes possible to select an image according to the user's intention.

[0027] The image scoring unit 207 scores each image in the album candidate image group using the scoring criteria determined by the scoring criteria determination unit 206 and the feature amounts acquired from the image analysis unit 205. The results of the scoring are output to the image selection unit 211 and the image layout unit 214.

[0028] The photo number adjustment amount input unit 208 inputs, to the photo number determination unit 210, the adjustment amount for adjusting the number of photos to be arranged in the album, which is specified by the album creation condition specifying unit 201.

[0029] The page spread number input unit 209 inputs, to the photo number determination unit 210 and the page spread allocation unit 212, the page spread number of the album, which is specified by the album creation condition specifying unit 201. The page spread number of the album corresponds to the number of a plurality of templates on which a plurality of images are arranged.

[0030] The photo number determination unit 210 determines the total number of photos constituting the album based on the adjustment amount specified by the photo number adjustment amount input unit 208 and the page spread number specified by the page spread number input unit 209, and inputs the result to the image selection unit 211.

[0031] The image selection unit 211 selects images based on the number of photos input from the photo number determination unit 210 and the scores calculated by the image scoring unit 207, creates a list of the layout image group (also referred to as the third image group) to be used in the album, and provides the list to the page spread allocation unit 212.

[0032] The page spread allocation unit 212 allocates each image to a page spread for the image group selected by the image selection unit 211, using the shooting date information. Here, an example of allocation in page spread units is described, but allocation in page units may also be performed.

[0033] The template input unit 213 reads a plurality of templates corresponding to the template information specified by the album creation condition specifying unit 201 from the HDD 104 and inputs the templates to the image layout unit 214.

[0034] The image layout unit 214 performs layout processing of images for individual spreads. Specifically, for a spread to be processed, it determines a template suitable for the image selected by the image selection unit 211 from the plurality of templates input by the template input unit 213, and determines the layout of each image.

[0035] The layout information output unit 215 outputs layout information for display on the display 105 according to the layout determined by the image layout unit 214. The layout information is, for example, bitmap data in which the data of the selected images selected by the image selection unit 211 is laid out in the determined template.

[0036] The image correction condition input unit 216 provides the ON / OFF information of image correction specified from the album creation condition specifying unit 201 to the image correction unit 217. Examples of the types of correction include brightness correction, burning-in correction, red-eye correction, or contrast correction. The ON or OFF of image correction may be specified for each type of correction, or may be specified collectively for all types.

[0037] The image correction unit 217 performs correction on the layout information held by the layout information output unit 215 based on the image correction conditions received from the image correction condition input unit 216. Note that the number of pixels of the image to be processed by the image correction unit 217 from the image conversion unit 204 can be changed according to the size of the layout image determined by the image layout unit 214. In this embodiment, image correction is performed on each image after generating the layout image, but it is not limited to this, and correction of each image may be performed before laying out on a spread or page.

[0038] When the album creation application is installed in the image processing apparatus 100, a startup icon is displayed on the top screen (desktop) of the OS (operating system) that operates on the image processing apparatus 100. When the user double-clicks the startup icon displayed on the display 105 with the pointing device 107, the application program stored in the HDD 104 is loaded into the RAM 103 and executed by the CPU 101 to start up.

[0039] <Example of display screen> FIG. 3 is a diagram showing an example of an application startup screen 301 provided by the album creation application. The application startup screen 301 is displayed on the display 105. The user sets the album creation conditions described later via the application startup screen 301, and the album creation condition specifying unit 201 acquires the setting contents from the user through this UI screen.

[0040] The path box 302 on the application startup screen 301 displays the storage location (path) in the HDD 104 of a plurality of images (for example, a plurality of image files) that are the targets of album creation. When the folder selection button 303 is instructed by a click operation of the pointing device 107 from the user, a selection screen of a folder that is standardly installed in the OS is displayed. In the folder selection screen, the folders set in the HDD 104 are displayed in a tree structure, and the user can select a folder containing the images to be used for album creation with the pointing device 107. The path of the folder in which the album candidate image group selected by the user is stored is displayed in the path box 302.

[0041] The template specification area 304 is an area for the user to specify template information, and the template information is displayed as an icon. In the template specification area 304, icons of a plurality of template information are arranged and displayed, and the user can select the template information by clicking with the pointing device 107.

[0042] The double-page number box 305 accepts the setting of the double-page number of the album from the user. The user can directly input numbers into the double-page number box 305 via the keyboard 106, or use the pointing device 107 to input numbers into the double-page number box 305 from a list.

[0043] The checkbox 306 accepts the ON / OFF specification of image correction from the user. The checked state indicates that image correction is ON, and the unchecked state indicates that image correction is OFF. In this embodiment, it is assumed that all image corrections are turned ON / OFF with a single button, but it is not limited to this, and checkboxes may be provided for each type of image correction.

[0044] The photo number adjustment 307 is for adjusting the number of images arranged on the double pages of the album with a slider bar. The user can adjust the number of images arranged on each double page of the album by moving the slider bar left and right. The photo number adjustment 307 can be assigned appropriate numbers such as at least -5 and at most +5, for example, so that the number of images that can be arranged within the double page can be adjusted.

[0045] The product material designation section 308 sets the product material of the album to be created. The product material can set the size of the album and the type of paper used for the album. The type of cover and the type of binding part may be set individually.

[0046] When the user presses the OK button 309, the album creation condition designation section 201 outputs the content set on the application startup screen 301 to the automatic layout processing section 218 of the album creation application.

[0047] At that time, the path input to the password box 302 is transmitted to the image acquisition unit 203. The number of pages input to the number-of-pages box 305 is transmitted to the number-of-pages input unit 209. The template information selected in the template specification area 304 is transmitted to the template input unit 213. The ON / OFF of image correction of the image correction check box is transmitted to the image correction condition input unit 216.

[0048] The reset button 310 on the display screen 301 is a button for resetting each setting information on the application startup screen 301.

[0049] FIG. 4 is a diagram showing an example of a user image selection screen 401 provided by the album creation application. When the OK button 309 on the application startup screen 301 is pressed, the screen displayed on the display 105 is switched to the user image selection screen 401. The user sets a user image via the user image selection screen 401, and the user image specification unit 202 acquires the setting content from the user. The user image is an image constituting a user image group.

[0050] The password box 402 on the user image selection screen 401 displays the storage location (path) in the HDD 104 of a plurality of images (for example, a plurality of image files) targeted by the user image. When the folder selection button 403 is instructed by a click operation using the pointing device 107 from the user, a folder selection screen is displayed. On the folder selection screen, the folders set in the HDD 104 are displayed in a tree structure, and the user can select a folder containing the images to be used for album creation using the pointing device 107. The folder path of the folder selected by the user is displayed in the password box 402. The same folder path as that of the password box 302 may be displayed in the password box 402.

[0051] The user image specification area 404 is an area for the user to specify a user image, and a plurality of image files stored in the folder in the HDD 104 specified by the pass box 402 are displayed as icons. In the user image specification area 404, icons of a plurality of images are arranged and displayed, and the user can click and select them using the pointing device 107. In FIG. 4, the image marked with the check mark 405 indicates the image specified by the user. If the user selects it, a check mark 405 will be attached, and if the user selects it again in that state, the check mark 405 will disappear.

[0052] When the user presses the OK button 406, the user image specifying unit 202 acquires the content set on the user image selection screen 401. The user image specifying unit 202 outputs the acquired setting content to the automatic layout processing unit 218 of the album creation application. At that time, the list of image files with the check mark 405 is transmitted to the image acquisition unit 203 as information on the user image group.

[0053] The reset button 407 on the user image selection screen 401 is a button for resetting each setting information on the user image selection screen 401.

[0054] <Flow of processing> FIG. 5 is a flowchart showing the processing of the automatic layout processing unit 218 of the album creation application. The flowchart shown in FIG. 5 is realized, for example, by the CPU 101 reading out the program stored in the HDD 104 into the RAM 103 and executing it. In the description of FIG. 5, it is assumed that each component shown in FIG. 2 functions by the CPU 101 executing the above album creation application and executes the processing. The automatic layout processing will be described with reference to FIG. 5. Note that the symbol "S" in the description of each process means that it is a step in the flowchart (the same applies hereinafter in this specification).

[0055] In S501, the image conversion unit 204 converts an image to generate an analysis image. At the time of S501, it is assumed that various settings have been completed through the UI screens of the application startup screen 301 and the user image selection screen 401. That is, it is assumed that the album creation conditions, the album candidate image group, and the user image group have been set. In S501, specifically, the image conversion unit 204 identifies a plurality of image files stored in the folder in the HDD 104 specified by the album creation condition specifying unit 201 and the user image specifying unit 202. Then, the identified plurality of image files are read from the HDD 104 to the RAM 103. Then, the image conversion unit 204 converts the image of the read image file into an analysis image having a predetermined number of pixels and color information as described above. In the present embodiment, it is converted into an analysis image having a short side of 420 pixels and color information converted to sRGB.

[0056] In S502, the image analysis unit 205 executes an analysis process of the analysis image generated in S501 to obtain feature amounts. Examples of the feature amounts include meta information stored in the image and image feature amounts that can be obtained by analyzing the image. In the present embodiment, as the analysis process, obtaining the focus degree of focus, face detection, personal recognition, and object determination are executed, but it is not limited thereto, and other analysis processes may be executed. Hereinafter, the details of the process performed by the image analysis unit 205 in S502 will be described.

[0057] The image analysis unit 205 extracts necessary meta information from the meta information received from the image acquisition unit 203. The image analysis unit 205, for example, acquires the shooting date and time as the time information of the image in the image file from the Exif information attached to the image file read from the HDD 104. Note that, as the meta information, for example, the position information or F value of the image may be acquired. Further, as the meta information, information other than that attached to the image file may be acquired. For example, schedule information associated with the shooting date and time of the image may be acquired.

[0058] In addition, the image analysis unit 205 acquires image feature amounts from the analysis image generated in S501. Examples of the image feature amounts include the degree of focus of the image. As a method for obtaining the degree of focus of the image, edge detection is performed. The Sobel filter is generally known as a method for edge detection. By performing edge detection with the Sobel filter and dividing the luminance difference between the start point and the end point of the edge by the distance between the start point and the end point, the slope of the edge is calculated. From the result of calculating the average slope of the edges in the image, an image with a larger average slope can be regarded as being more in focus than an image with a smaller average slope. And by setting a plurality of threshold values with different values for the slope, it is possible to output an evaluation value of the amount of focus by determining whether it is above which threshold value. In the present embodiment, two different threshold values are set in advance, and the amount of focus is determined in three levels of "○", "△", and "×". For example, the slope of the focus to be adopted in the album is determined as "○", the slope of the acceptable focus is determined as "△", and the unacceptable slope is determined as "×", and each threshold value is set in advance. The setting of the threshold value may be provided by, for example, the creator of the album creation application, or may be settable on the user interface. Note that, as the image feature amount, for example, the brightness, color tone, saturation, or resolution of the image may be acquired.

[0059] The image analysis unit 205 performs face detection on the analysis image generated in S501. Here, a known method can be used for the face detection process. For example, Adaboost, which creates a strong classifier from a plurality of prepared weak classifiers, is used for the face detection process. In the present embodiment, a face image of a person (object) is detected by the strong classifier created by Adaboost. The image analysis unit 205 extracts the face image and acquires the upper left coordinate value and the lower right coordinate value of the position of the detected face image. By having these two types of coordinates, the image analysis unit 205 can acquire the position and size of the face image. Here, a case where an object is detected using AdaBoost has been described, but object detection may be performed using a learned model such as a neural network.

[0060] The image analysis unit 205 performs personal recognition by comparing the face image in the image to be processed based on the analyzed image detected by face detection with the representative face images stored in the face dictionary database for each personal ID. The image analysis unit 205 obtains the similarity between the face image in the image to be processed and each of the plurality of representative face images. Further, it identifies the representative face image with a similarity equal to or higher than the threshold value and having the highest similarity. Then, the personal ID corresponding to the identified representative face image is set as the ID of the face image in the image to be processed. Incidentally, when the similarity between the face image in the image to be processed and all of the plurality of representative face images is less than the threshold value, the image analysis unit 205 registers the face image in the image to be processed as a new representative face image in the face dictionary database in association with a new personal ID.

[0061] The image analysis unit 205 performs object recognition on the analyzed image generated in S501. Here, a known method can be used for the object recognition process. In the present embodiment, an object is recognized by a discriminator (trained model) created by DeepLearning. The discriminator outputs a likelihood of 0 to 1 for each object, and an object exceeding a certain threshold value is recognized as being in the image. By recognizing the object image, the image analysis unit 205 can obtain the types of objects such as pets like dogs or cats, flowers, food, buildings, ornaments, and landmarks. Although objects are discriminated in the present embodiment, the present invention is not limited thereto, and each type may be obtained by recognizing expressions, shooting compositions, or scenes such as trips or weddings. Also, the likelihood itself output from the discriminator before performing the discrimination may be used. Thereby, a more flexible score criterion can be determined in the score criterion determination unit 206.

[0062] FIG. 6 is a diagram showing feature amounts. The image analysis unit 205 distinguishes the feature amounts acquired in S502 for each ID that identifies each image (analysis image) as shown in FIG. 6, and stores them in a storage area such as the ROM 102. For example, as shown in FIG. 6, the shooting date and time information, the focus determination result, the number of detected face images and their position information and similarity, and the type of recognized object acquired in S502 are stored in a table format. Note that the position information of the face images is stored separately for each personal ID acquired in S502. Also, when a plurality of types of objects are recognized from one image, all the plurality of types of objects are stored in the row corresponding to that one image in the table shown in FIG. 6.

[0063] In S503, the image analysis unit 205 determines whether or not the processes of S501 to S502 have been completed for all the images included in the album candidate image group and the user selected image group. Here, if it is determined that the processes have not been completed, the processes from S501 are repeated. If it is determined that the processes have been completed, the process proceeds to S504. That is, by repeatedly executing the processes of S501 to S502 for all the images stored in the specified folder, the table shown in FIG. 6 including the information of each of the all images is created.

[0064] In S504, the score criterion determination unit 206 determines a score criterion based on the feature amounts obtained from the analysis results of the user images in the user image group. That is, based on the feature amounts corresponding to the image IDs of the user image group, the score criterion used by the image scoring unit 207 is determined. The score described here is an index indicating the appropriateness of the layout for each image. Also, the score criterion is a criterion for calculating a score in the image scoring unit 207, and is, for example, an expression or a score calculation algorithm configured using a certain coefficient. The score criterion determination unit 206 generates control information used for the score criterion, determines the score criterion based on the control information, and provides it to the image scoring unit 207. As a specific example, the control information is the average value and standard deviation for each feature amount as described later.

[0065] FIG. 7 is a diagram showing feature amounts in a feature space. FIG. 8 is a flowchart showing details of the score criterion determination process of S504. Hereinafter, the score criterion determination process performed in S504 will be described with reference to FIGS. 7 and 8.

[0066] In S801, the score criterion determination unit 206 calculates, as control information, the average value and standard deviation of the user image group for each feature amount. Hereinafter, the control information will be described with reference to FIG. 7.

[0067] FIG. 7(a) shows an example in which the feature amounts of each image are plotted in a feature space. Since the feature space is multi-dimensional and all cannot be illustrated, in FIG. 7(a), an explanation will be given by extracting two feature amounts, namely, the similarity to individual ID1 and the shooting time, from among the feature amounts. However, a one-dimensional feature space using one feature amount may be used. The same processing is performed for other feature amounts as well. The shaded point 701 as an example indicates a point where the feature amount corresponding to the user image is plotted. The unshaded point 702 as an example indicates a point where the feature amount corresponding to the album candidate image is plotted. The point 703 indicated by "×" in the figure indicates the average value vector obtained from the feature amount vector of the user image group.

[0068] FIG. 7(b) shows each image plotted on the feature amount axis of the similarity to individual ID1. The higher the face of individual ID1 is included in the image and the higher the correlation with the person identified by individual ID1, that is, the higher the probability of being the same person as individual ID1, the larger the value (right direction of the axis). The point 704 indicates the average value of the user image group on this feature amount axis, and the width 705 indicates the standard deviation of the user image group on this feature amount axis. On the feature amount axis of the similarity to individual ID1 shown in FIG. 7(b), each user image has similar features and has a dense distribution with a small standard deviation 705. That is, it can be seen that there is a high probability that many images of the person of individual ID1 are included in all the user image groups.

[0069] FIG. 7(c) plots each image on the feature quantity axis of the shooting date and time. Point 706 indicates the average value of the user image group on this feature quantity axis, and width 707 indicates the standard deviation of the user image group on this feature quantity axis. On the feature quantity axis of the shooting date and time, each user image has different features, and the distribution is sparse with a large standard deviation 707. That is, it can be seen that the user image group includes images taken at various times.

[0070] In S802, the score criterion determination unit 206 determines whether the process of S801 has ended for all feature quantity items. Here, if it is determined that the process has not ended, the process from S801 is repeated. If it is determined that the process has ended, the process proceeds to S803.

[0071] In S803, the score criterion determination unit 206 acquires the score criteria pre - incorporated in the album creation application. Then, in S804, using the control information calculated in S801 (in this example, the average value and standard deviation for each feature quantity), the score criteria are determined. More specifically, in this embodiment, the following formulas (1) and (2) are acquired as the score criteria, and by applying the control information (average value and standard deviation for each feature quantity) calculated in S801 to formula (1), the score criteria are determined. Then, the process proceeds to S505, and the process of scoring the images in the album candidate image group by the image scoring unit 207 is performed. Hereinafter, an example of the score criteria determined in S504 and the subsequent scoring process in S505 will be described together. In this embodiment, in S505, the image scoring unit 207 first calculates the score for each image (referred to as the "target image") to be scored and for each feature quantity using formula (1) that uses the control information calculated in S801. Note that the images to be scored are the images in the album candidate image group. Sji=(50 - 10×|μi―fji| / σi) / σi···(1) Here, j is the index of the target image, i is the index of the feature amount, fji is the feature amount of the target image, Sji is the score corresponding to the feature amount fji, and μi and σi respectively represent the average value and standard deviation for each feature amount of the user image group. That is, for a feature amount with a dense distribution where the standard deviation 705 of the user image group is small as shown in Fig. 7(b), the score of the feature amount of the target image close to the average value is calculated to be higher than that of the feature amount far from the average value. On the other hand, for a feature amount with a sparse distribution where the standard deviation 707 of the user image group is large as shown in Fig. 7(c), the score of the feature amount of the target image close to the average value does not show as much difference as in the case of Fig. 7(b) compared to the feature amount far from the average value.

[0072] And in S505, the image scoring unit 207 calculates the score of each target image using the score Sji for each target image and each feature amount obtained by Equation (1) and Equation (2). Pj = Σi(Sji) / Ni ···(2) Here, Pj represents the score of each target image, and Ni represents the number of items of the feature amount. That is, the score of each target image is calculated as the average of the scores of each feature amount. In this way, in S804, the score criterion determination unit 206 determines these Equation (1) and Equation (2) as the score criteria. In subsequent S505, the image scoring unit 207 applies Equation (1) and Equation (2) as described above to score each target image.

[0073] In addition, here, since it is preferable that the images used in the album are in focus, a predetermined score may be added to the target image whose focus feature amount shown in Fig. 6 is "〇". Also, a predetermined score may be added to the target image having a certain specific object. For example, in the case where it is desired to select many images including animals, by adding a predetermined score to the target image including an object belonging to an animal, control can be performed so that it is preferentially selected.

[0074] According to the above scoring criteria, in the feature quantity where the standard deviation σ becomes small, such as the similarity for the personal ID1 in Fig. 7(b), the closer the target image is to the user image group in terms of features, the higher the score calculated by formula (1). Therefore, regarding the similarity for the personal ID1, which is a feature quantity with a dense distribution in the user image group, the closer the target image is to the user image group in terms of features, the higher the score and the easier it is to be selected.

[0075] On the contrary, in the feature quantity where the standard deviation σ becomes large, such as the shooting date and time in Fig. 7(b), regardless of the difference between the average value μ and the feature quantity fji of the target image, the score calculated by formula (1) becomes small. Therefore, regarding the shooting date and time, which is a feature quantity with a sparse distribution in the user image group, no matter what features the target image has, the influence on image selection becomes small.

[0076] As described above, when using the scoring criteria in this embodiment, from the album candidate image group, an image having features common to the user image group can be selected, and it becomes possible to select an image with a sense of unity as the layout image group.

[0077] In this embodiment, formula (2) is used as the scoring criteria for calculating the score of each target image, but it is not limited thereto. For example, according to formula (3), the scores of each feature quantity calculated by formula (1) may be weighted and added. Pj = Σi(wi × Sji) / Ni ···(3) Here, wi is the weighting coefficient for each feature quantity. Thereby, the influence degree (contribution rate) of each feature quantity on the score of the target image can be changed. For example, in Fig. 7, by increasing the contribution rate for the similarity for the personal ID1 and decreasing the weight contribution rate for the shooting date and time, it becomes possible to perform scoring that emphasizes more on the similarity for the personal ID1.

[0078] Alternatively, a scoring criterion may be used such that the highest score among the scores of each feature amount calculated by formula (1) is set as the score of the target image. As a result, when one or more of the feature amounts have features similar to the user image group, a high score can be assigned. For example, in FIG. 7, even when a feature amount other than the similarity to the personal ID1 of the target image has a feature different from the user image group, such as the shooting date and time, a high score can be assigned to the target image.

[0079] Also, on the multi-dimensional feature amount space, a scoring criterion may be used to calculate the score of the target image based on the difference between the average vector of the user image group and the vector of the target image. As a result, scoring can be performed comprehensively considering all feature amounts. For example, in FIG. 7(a), since the album candidate image 702 has features close to the average value vector 703 in the multi-dimensional space, a high score is assigned. On the other hand, since the album candidate image 708 has features different from the average value vector 703 in the multi-dimensional space, a high score is not assigned. That is, when one or more feature amounts have features different from the average value vector 703 of the user image group, it can be controlled so that a high score is not assigned.

[0080] In this embodiment, the scoring criterion is determined using the average and standard deviation of each feature amount of the user image group, but it is not limited thereto. For example, as in formula (4), a scoring criterion using each feature amount of the user image group and its standard deviation may be determined. Sji=Σk(50-10×|fki―fji| / σi) / (σi×Nk)···(4) Here, k represents the index of the user image, fki represents the feature amount of the user image, and Nk represents the number of images included in the user image group. In this way, by comparing the feature amount of the target image with the feature amounts of each user image, the difference in the feature amounts between the target image and the user image group can be evaluated more accurately.

[0081] In this embodiment, the average value and standard deviation of the feature amounts of the user image group are used as control information. However, the present invention is not limited to this, and the median value of the feature amounts, the distribution shape (such as normal distribution or Poisson distribution), the interquartile range, or the interquartile deviation may be used. For example, the median value and standard deviation of the feature amounts of the user image group may be used, or the average value and interquartile deviation of the feature amounts of the user image group may be used. That is, at least one of the average value and median value of the feature amounts of the user image group and at least one of the standard deviation, interquartile deviation, and distribution shape may be used.

[0082] Again, referring to FIG. 5, subsequent to S504, in S505, as described above, the image scoring unit 207 acquires the scoring criteria determined by the scoring criteria determination unit 206. That is, as described above, the equations shown in equations (1) to (4) are acquired as the scoring criteria. Then, the image scoring unit 207 executes scoring for each image of the album candidate image group based on the acquired scoring criteria. Scoring means assigning a score to each image (scoring). The assigned scores are provided to the image selection unit 211 and are referred to when selecting an image to be used for the layout described later.

[0083] In S506, the image scoring unit 207 determines whether the image scoring in S505 has been completed for all the images of the album candidate image group in the folder designated by the user. If it is determined that the process has not been completed, the process from S505 is repeated. If it is determined that the process has been completed, the process proceeds to S507.

[0084] In S507, the photo number determination unit 210 determines the number of photos to be arranged in the album. In this embodiment, using the adjustment amount for adjusting the number of double-page photos input from the photo number adjustment amount input unit 208 and the number of double-page inputs input from the double-page number input unit 209, the number of photos to be arranged in the album is determined by equation (5). Number of photos = [Number of double-page × (Basic number of photos + Adjustment amount)] ··· (5) Here, [·] indicates the floor function that truncates the fractional part, and the basic number of photos indicates the number of images to be arranged on a double-page without adjustment. In this embodiment, the basic number of photos is set to 6 in consideration of the appearance during layout, and it is incorporated in advance into the program of the album creation application.

[0085] Also, in this embodiment, the number of photos to be laid out is determined based on the number of spreads and the adjustment amount of the number of photos. However, the present invention is not limited to this, and the number of photos to be laid out may be determined according to the number of user images specified in the user image specifying unit 202. For example, control may be performed such that the larger the number of user images, the larger the number of photos to be laid out is set.

[0086] In S508, the image selection unit 211 selects images to be laid out from the scores for each image calculated by the image scoring unit 207 and the number of photos determined by the photo number determination unit 210. Hereinafter, the selected group of images is referred to as a layout image group.

[0087] In this embodiment, the user image group is selected as the group of images to be all laid out. Then, the image selection unit 211 selects images in descending order of the scores given by the image scoring unit 207 from the group of images specified by the album creation condition specifying unit 201 by the number obtained by subtracting the number of user image groups from the total number of images to be laid out.

[0088] By executing the above-described method in the score criterion determination unit 206 and the image scoring unit 207, images having characteristics similar to the user image group are selected in the selection by the image selection unit 211.

[0089] FIG. 9 is a diagram showing an example of an image selection result. FIG. 9(a) shows an example of an image selection result in the present embodiment. FIG. 9(b) shows an example of an image selection result when the present embodiment is not used as a comparative example. First, an explanation will be given using FIG. 9(a). For the sake of explanation, each image is marked with the feature amount having the highest score for different feature amounts from A to E. In FIG. 9(a), for an album candidate image group having different features from A to E, an image having the features of A and C is designated as the user image group in the user image specifying unit 202. Then, in the image selection unit 211, an image having the features of A and C is selected as the layout image group. In this way, the intention of the user can be drawn from the user image group and reflected in the image selection. On the other hand, FIG. 9(b) shows an example of an image selection result when there is no user-selected image. In this case, in order to select an image based on the tendency of the feature amounts of the entire album candidate image group, as a result, many images having the features of A and B with a large number of image sheets are selected. Also, in the case of FIG. 9(b), even if there is a user-specified image in the same manner as FIG. 9(a), if it is automatically selected from the album candidate image group, still, an image is selected based on the tendency of the feature amounts of the entire album candidate image group. For this reason, as a result, many images having the features of A and B with a large number of image sheets are selected.

[0090] Note that as a method of image selection, a higher selection probability may be set so that the higher the score, and the selection may be made probabilistically. In this way, by selecting probabilistically, the layout image can be changed each time the automatic layout function by the automatic layout processing unit 218 is executed. For example, when the user is not satisfied with the automatic layout result, by pressing a re-selection button (not shown in the UI), the user can obtain a layout result different from the previous time.

[0091] Further, the image selection unit 211 may select the total number of images to be laid out from the image group specified by the album creation condition specifying unit 201 without selecting a layout image from the user image group. According to this method, it is possible to search for and select an image close to the ideal by specifying, in the user image specifying unit 202, an image with an ideal subject arrangement or composition that the user does not want to include in the album.

[0092] Alternatively, the total number of images to be laid out may be selected from the image group combining the image data group specified by the album creation condition specifying unit 201 and the user images. According to this method, there is a possibility of searching for and selecting an image more suitable for layout than the images in the user image group. In this case, the image scoring unit 207 also performs scoring on not only the album candidate image group but also the user image group.

[0093] Further, when the image selection unit 211 selects, as a layout image, an image whose score calculated by the image scoring unit 207 is equal to or higher than a certain threshold value, the photo number determination unit 210 does not have to determine the number of photos. In this case, the value such that the number of selected images becomes the number of spreads is the upper limit that can be set as the threshold value.

[0094] Returning to FIG. 5 again to continue the explanation. In S509, the spread allocation unit 212 divides and allocates the layout image group acquired in S508 into image groups corresponding to the number of spreads input from the spread number input unit 209. In the present embodiment, the layout images are arranged in the order of shooting time acquired in S502 and are divided at a position where the time difference between the shooting times of adjacent images is large. Such processing is performed until it is divided into the number of spreads input from the spread number input unit 209. That is, (the number of spreads - 1) times of division are performed. Thereby, an album with images arranged in the order of shooting time can be created. Note that the process of S509 may be performed in units of pages instead of units of spreads.

[0095] In S510, the image layout unit 214 determines the image layout. Hereinafter, an example will be described in which the template input unit 213 inputs (a) to (p) of FIG. 10 for a certain layout according to the specified template information.

[0096] FIG. 10 is a diagram showing a group of templates used for the layout of image data. Each of the plurality of templates included in the template group corresponds to each layout. Template 1001 is one template. Template 1001 includes a main slot 1002, a sub-slot 1003, and a sub-slot 1004. The main slot 1002 is the main slot (the frame for laying out the image) within the template 1001 and is larger in size than the sub-slot 1003 and the sub-slot 1004.

[0097] Here, the number of slots of the input template is specified as 3 as an example. Assuming that when arranging the orientations of the three selected images (vertical or horizontal) with respect to the shooting date and time, it is as shown in FIG. 10(q).

[0098] Here, in each image group assigned to the layout, the image with the largest score calculated by the image scoring unit 207 is used for the main slot, and the other images are set for the sub-slots. Note that it may be set for the main slot or the sub-slot based on a certain feature amount acquired by the image analysis unit, or it may be set randomly. Also, the user-selected image may be preferentially set in the main slot.

[0099] Here, it is assumed that the image data 1005 is for the main slot, and the image data 1006 and 1007 are for the sub - slots. In this embodiment, the image data with an older shooting date and time is laid out at the upper left of the template, and the image with a newer shooting date and time is laid out at the lower right. In FIG. 10(q), since the image data 1005 for the main slot has the newest shooting date and time, the templates in FIGS. 10(i) to (l) are candidates. Also, since the older image data 1006 for the sub - slot is a vertical image and the newer image data 1007 is a horizontal image, as a result, the template in FIG. 10(j) is determined as the most suitable template for the three selected image data, and the layout is determined. In S510, it is determined which image is to be laid out in which slot of which template.

[0100] In S511, the image correction unit 217 executes image correction. The image correction unit 217 executes image correction when information indicating that image correction is ON is input from the image correction condition input unit 216. As image correction, for example, overexposure correction (luminance correction), red - eye correction, or contrast correction is executed. The image correction unit 217 does not execute image correction when information indicating that image correction is OFF is input from the image correction condition input unit 216. Image correction can be executed, for example, on image data whose size has been converted to a short side of 1200 pixels in the sRGB color space.

[0101] In S512, the layout information output unit 215 creates layout information. The image layout unit 214 lays out the image data on which the image correction in S511 has been executed for each slot of the template determined in S510. At this time, the image layout unit 214 scales and lays out the image data to be laid out according to the size information of the slot. Then, the layout information output unit 215 generates the bitmap data in which the image data is laid out on the template as the output image data.

[0102] In S513, the image layout unit 214 determines whether the processes of S510 to S512 have been completed for all the spreads. If it is determined that they have not been completed, the process from S510 is repeated. If it is determined that they have been completed, the automatic layout process of FIG. 5 ends.

[0103] When the automatic layout process of FIG. 5 ends, the layout information output unit 215 outputs the bitmap data (output image data) in which the images are laid out on the template, which was generated in S512, to the display 105 for display. Note that the generated image data may be uploaded to a server via the Internet based on a user instruction. Based on the uploaded image data, printing and binding processes are executed, and an album (photo book) in the form of a booklet is created and delivered to the user.

[0104] The above is the description of the processing flow for performing the automatic layout process. According to the present embodiment, an image that conforms to the user's intention can be appropriately selected. In particular, in the present embodiment, by determining a scoring criterion such that an image having characteristics similar to the user image group has a high score, it is possible to select an image with a sense of unity while grasping the user's intention. Further, according to the present embodiment, when the overall tendency of the album candidate image group is different from the tendency of the user image group, it is possible to select an image that grasps the user's intention as compared with the case of selecting an image according to the overall tendency of the album candidate image group.

[0105] <<Second Embodiment>> In the second embodiment, in the score criterion determination process of S504 described in the first embodiment, a process of determining a score criterion different from the score criterion described in the first embodiment will be described. Specifically, a score criterion is used in which a high score is given to a feature amount having a sparse distribution in the user image group. Also, examples will be described in which a score criterion is used to select an image having a feature close to the user image group among the feature amounts of such a distribution, or conversely, a score criterion is used to deliberately not select it. Note that the basic process of the automatic layout process is the same as the example described in the first embodiment, and hereinafter, the description will focus on the differences.

[0106] FIG. 11 is a diagram for explaining the score criterion of this embodiment. FIG. 11(a) shows the result of plotting each image in the first embodiment on a certain feature amount axis. The shaded point taking point 1101 as an example indicates a user image. The unshaded points taking point 1102 and point 1103 as examples indicate album candidate images. Also, the characters "H" and "L" written inside point 1102 and point 1103 indicate the degree of the score calculated by the image scoring unit 207. That is, H indicates that the degree of the score is high, and L indicates that the degree of the score is low.

[0107] In the first embodiment, in S504, the score criterion determination unit 206 uses a score criterion that has a greater influence on the score for a feature amount in which the user image group is plotted together (that is, a feature amount with a small standard deviation σ) among each feature amount. As a result, as shown in FIG. 11(a), in the feature amount having a dense distribution in the user image group, an image having a feature closer to the user image group is given a higher score and is more likely to be selected.

[0108] In this embodiment, in S504, a score criterion is used in which a feature amount in which the user image group is plotted scattered (that is, a feature amount with a large standard deviation σ) has a greater influence on the score. For example, according to formula (6), the score of each feature amount may be calculated for each target image. Sji = σi × (50 - 10 × MINk(|fki―fji|) / σi) / Nk ···(6)

[0109] Here, MINk(·) is a function that obtains the minimum value of the element (·) in each user image k. That is, in Equation (6), the difference in feature amounts between the target image and the user image closest to it is used as a scoring criterion. Also, in Equation (6), the larger the standard deviation σi of the feature point i in the user image group, the higher the score is calculated.

[0110] As a result, as shown in FIG. 11(b), the user image group is plotted in a scattered manner, and an image having features closer to a certain user image is given a higher score. Therefore, it becomes possible to select an image having features close to the user image group in a feature amount that has a sparse distribution in the user image group.

[0111] Also, in S504, as shown in FIG. 11(c), a scoring criterion may be used such that a higher score is given to a feature amount in which the user image group is plotted in a scattered manner and an image having features different from each user image. For example, the score may be calculated by Equation (7). Sji = σi × Σk(50 - 10 × σi / MINk(|fki ― fji|)) / Nk ···(7)

[0112] Thereby, from the album candidate image group, an image having features different from the user image group is selected, and together with the user image group, it becomes possible to select images having various variations as the layout image group.

[0113] Also, in the present embodiment, in the score criterion determination unit 206, principal component analysis may be performed on the user image group to obtain a mapping function to the first axis, all images may be mapped onto the first axis, and a score criterion based on the difference in feature amounts between the user image and the target image on the first axis may be used.

[0114] The above is the description of the processing content of the score criterion determination process in this embodiment. According to this embodiment, it is also possible to determine a score criterion such that an image having characteristics different from those of the user image group has a high score, and it becomes possible to select an image with rich variations while taking into account the user's intention.

[0115] <<Third Embodiment>> In the third embodiment, an example of making the score criterion determination process of S504 in the first embodiment and the second embodiment different will be described. Specifically, an example of performing a process of automatically switching between the score criterion described in the first embodiment and the score criterion described in the second embodiment will be described.

[0116] In the first embodiment and the second embodiment, as shown in FIG. 11, an example of using the distribution of the user image group in the feature space and the difference in features between the user image and the target image as the score criterion has been described.

[0117] In the first embodiment, an example of enabling the selection of an image with a sense of unity by using a score criterion in which features where the user image group is arranged together have a great influence on score calculation has been described.

[0118] In the second embodiment, an example of enabling the selection of an image with rich variations by using a score criterion in which features where the user image group is arranged dispersedly have a great influence on score calculation has been described.

[0119] In the third embodiment, based on the tendency of the features in the feature space, the score criteria described in the first embodiment and the second embodiment are automatically switched and used.

[0120] FIG. 12 is a flowchart showing the details of the score criterion determination process of S504 in this embodiment. In this embodiment, according to the flow shown in FIG. 12, in the score criterion determination unit 206, the score criterion is automatically switched based on the value of the standard deviation obtained for each feature.

[0121] In S1201, as in the first embodiment, for each feature amount, the average and standard deviation of the feature amounts of the user image group are calculated.

[0122] In S1202, based on the value of the standard deviation calculated in S1201, the score criterion to be acquired is switched. That is, in S1202, the score criterion determination unit 206 determines whether the standard deviation calculated in S1201 is less than a predetermined value. When the standard deviation is less than the predetermined value (for example, less than 3), the process proceeds to S1203, and the first score criterion is determined as the score criterion. For example, Equation (1) is determined as the score criterion. On the other hand, when the standard deviation is greater than or equal to the predetermined value (for example, 3 or more), the process proceeds to S1204, and a second score criterion different from the first score criterion is determined as the score criterion. For example, Equation (6) is specified as the score criterion.

[0123] Thereby, for example, in FIG. 7(a), regarding the similarity to the personal ID1, Equation (1) can be used as the score criterion, and regarding the shooting date and time, Equation (6) can be used as the score criterion. In this case, a high score can be given to and selected for an image in which the person with the personal ID1 appears and the shooting date and time are scattered. That is, images can be selected with different tendencies for each feature amount, and it becomes possible to select images that better reflect the user's intention.

[0124] In the present embodiment, in S1202, an example in which the score criterion is switched between the first score criterion and the second score criterion based on the standard deviation calculated in S1201 has been described. However, the score criteria to be switched to are not limited to two, and the score criteria may be switched among three or more. Further, in the present embodiment, in S1202, an example in which the score criterion is switched based on the standard deviation calculated in S1201 has been described, but it is not limited to this. The score criterion to be acquired may be switched according to the type of feature amount. Specifically, by referring to a table in which the types of feature amounts and the score criteria to be used are associated in advance in the album creation application, a score criterion uniquely determined for each feature amount may be used. That is, the score criterion may be switched for each type of feature amount.

[0125] In S1205, for all feature item, it is determined whether the processes of S1201 to S1204 have been completed. Here, if it is determined that they have not been completed, the process from S1201 is repeated. If it is determined that they have been completed, the score criterion determination process in FIG. 12 ends.

[0126] The above is the explanation regarding the processing flow for implementing the score criterion determination process in this embodiment. According to this embodiment, by using an appropriate score criterion according to the user image group and each feature amount, it becomes possible to select an image that better captures the user's intention.

[0127] <<Other Embodiments>> In addition, in the third embodiment, an example of performing a process of automatically switching between the score criterion described in the first embodiment and the score criterion described in the second embodiment has been described. However, the configuration may be such that the user can select the first score criterion or the second score criterion. For example, a UI screen (not shown) provided with a checkbox indicating whether to select a mode emphasizing variations may be presented to the user. Then, if the checkbox is checked, the process described in the second embodiment or the third embodiment may be performed, and if it is not checked, the process described in the first embodiment may be performed. Also, instead of a checkbox, the UI screen may be configured so that the user can select a mode.

[0128] In the above-described embodiment, an example was explained in which a scoring criterion is determined based on the analysis result of the user image group, and scoring of the album candidate image group is performed based on the criterion and the analysis result of the album candidate image group. Then, an example was explained in which a layout image group is selected from the album candidate image group based on the score. However, it is only necessary that a criterion can be determined based on the analysis result of the user image group, and the layout image group can be selected based on the criterion and the analysis result of the album candidate image group. That is, the criterion obtained based on the analysis result of the user image group is a criterion that reflects the intention (selection criterion) of the user to select an image. Therefore, if the layout image group is selected from the album candidate image group based on this selection criterion, it is possible to select an image that takes into account the user's intention as compared with the case of selecting an image from the overall tendency of the album candidate image group.

[0129] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0130] 100 Image processing apparatus 201 Album creation condition specifying unit 202 User image specifying unit 205 Image analysis unit 206 Scoring criterion determination unit 207 Image scoring unit 211 Image selection unit

Claims

1. A computer, a first specifying means for specifying a first image group, a second specifying means for specifying a second image group, an analyzing means for analyzing each image included in the first image group and the second image group, a determining means for determining a criterion for each of a plurality of feature amounts based on the analysis result of the second image group, the determining means being a determining means for determining a scoring criterion for assigning a score to an image as the criterion, a scoring means for performing scoring of the first image group based on the scoring criterion and the analysis result of the first image group; a selecting means for selecting an image from the first image group based on the criteria for each of the plurality of feature amounts and the plurality of feature amounts obtained from each image as the analysis result of the first image group, the selecting means being a selecting means for selecting an image from the first image group based on the score obtained by the scoring, A program for causing the computer to function as, the determining means has a plurality of scoring criteria and is configured to switch the scoring criteria for each type of analysis result, the determining means is characterized in that it switches the scoring criteria for each type of the analysis result according to whether or not the analysis result of the second image group is less than a predetermined value.

2. The program according to claim 1, wherein the determining means determines the scoring criterion using at least one of an average value and a median value of the analysis result of the second image group.

3. The program according to claim 1 or 2, wherein the determining means determines the scoring criterion using at least one of a standard deviation, a quartile deviation, and a distribution shape of the analysis result of the second image group.

4. The determination means determines the score criterion using the difference between the analysis result of the second image group and the analysis result of the image scored by the scoring means, according to any one of claims 1 to 3.

5. The determination means determines the score criterion uniquely determined for each type of analysis result, according to any one of claims 1 to 4.

6. The scoring means assigns, as the score of each image, the average value of the scores obtained for each type of analysis result, The selection means selects an image from the images with high scores, according to any one of claims 1 to 5.

7. The scoring means assigns, as the score of each image, the highest score among the scores obtained for each type of analysis result, The selection means selects an image from the images with high scores, according to any one of claims 1 to 5.

8. The analysis means outputs, as the analysis result, the meta information attached to each image, according to any one of claims 1 to 7.

9. The meta information includes information on the shooting date and time, according to claim 8.

10. The analysis means outputs, as the analysis result, at least one of the image features including the focus degree of the focus, the result of face detection, and the result of object discrimination obtained by analyzing each image, according to any one of claims 1 to 9.

11. The selection means selects at least one image from the second image group, according to any one of claims 1 to 10.

12. The program according to any one of claims 1 to 11, wherein the second image group is included in the first image group.

13. causing the computer to further function as creation means for executing a layout using the image selected by the selection means and creating image data, the program according to any one of claims 1 to 12.

14. first specifying means for specifying a first image group; second specifying means for specifying a second image group; analysis means for analyzing each image included in the first image group and the second image group; determination means for determining a criterion for each of a plurality of feature amounts based on the analysis result of the second image group, the determination means for determining a score criterion for assigning a score to an image as the criterion; scoring means for performing scoring of the first image group based on the score criterion and the analysis result of the first image group; selection means for selecting an image from the first image group based on the criteria for each of the plurality of feature amounts and the plurality of feature amounts obtained from each image as the analysis result of the first image group, the selection means for selecting an image from the first image group based on the score obtained by the scoring; comprising the determination means has a plurality of score criteria and is configured to switch the score criteria for each type of analysis result; The determination means switches the score criteria for each type of the analysis result according to whether or not the analysis result of the second image group is less than a predetermined value, an image processing apparatus.

15. a first specifying step of specifying a first image group; a second specifying step of specifying a second image group; an analysis step of analyzing each image included in the first image group and the second image group; A determination step of determining criteria for each of a plurality of feature amounts based on the analysis result of the second image group, the determination step including determining a score criterion for assigning a score to an image as the criterion; A scoring step of performing scoring of the first image group based on the score criterion and the analysis result of the first image group; A selection step of selecting an image from the first image group based on the criteria for each of the plurality of feature amounts and the plurality of feature amounts obtained from each image as the analysis result of the first image group, the selection step including selecting an image from the first image group based on the score obtained by the scoring; comprising; In the determination step, the determination step is configured to have a plurality of score criteria and switch the score criteria for each type of analysis result; In the determination step, the score criteria for each type of the analysis result are switched according to whether or not the analysis result of the second image group is less than a predetermined value. An image processing method characterized by this.

Images