Image processing device, image processing method, and recording medium

Abstract

The present invention provides an image processing device, an image processing method, and a recording medium, which can reduce the workload of a user when automatically creating a photo album. An image processing method performs the following processing: obtaining a plurality of candidate images (52); obtaining user selection information related to a user-selected image selected by a user from the plurality of candidate images (54); assigning attributes to the user-selected image based on the analysis results of the user-selected image (58); estimating a user selection criterion (60) as a criterion for selecting a user-selected image based on a user selection ratio represented by the ratio of the number of selected images having the same attributes to the total number of user-selected images; and selecting an automatically selected image from unselected images that were not selected by the user among the candidate images based on the user selection criterion (62).

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a recording medium. Background Art

[0002] There is known a service for creating an original album using images owned by a user. For example, a service is provided in which a user sends a plurality of images owned by the user to a service provider via the Internet and creates an album with a layout desired by the user using the sent plurality of images.

[0003] Patent Document 1 describes a photo album creation system that creates a photo album using user-specified images. Patent Document 2 describes an electronic photo album creation device that obtains attribute information of user-specified images and extracts images based on the attributes to create a photo album.

[0004] Patent document 3 describes a photo album device that sets important images from images displayed on a display screen according to user specifications, and creates an electronic photo album according to the user's intention in a manner that includes more important images than images other than the important images.

[0005] Creating a photo album requires a labor-intensive process, such as selecting images from a large number of images and neatly laying out the selected images. This labor-intensive process can be a burden for beginners creating photo albums. With this in mind, software that automates image selection and layout is being developed as a photo album ordering software and a photo album creation assistance device.

[0006] Patent Document 4 describes a device for assisting with creating image materials for photo albums. The device described in this document acquires a second image material surrounding a first image material selected by a user, the second image material having image characteristics that satisfy predetermined selection criteria relative to the first image material, and then creates a layout page with the second image material arranged around the first image material.

[0007] Patent Document 5 describes an image search device that calculates a feature value representing an image's characteristics, scores the images based on the feature value, and displays a predetermined number of images with the highest scores as search results. In the device described in this document, images selected by the user are considered favorites, and the weighting coefficient of the feature value is increased so that the scores increase. This results in a layout of images favored by the user.

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2011-049870

[0009] Patent Document 2: Japanese Patent Application Laid-Open No. 2014-063246

[0010] Patent Document 3: Japanese Patent Application Laid-Open No. 2014-174787

[0011] Patent Document 4: Japanese Patent Application Laid-Open No. 2016-081319

[0012] Patent Document 5: Japanese Patent Application Laid-Open No. 2014-191701

[0013] However, the automatic selection of images and automatic layout of images do not meet the user's requirements with 100% accuracy, and thus new workload for the user may be incurred, such as reselection of images and correction of layouts.

[0014] When automatically creating an album, images used in the album are automatically selected from the user's existing images. However, during this automatic image selection, it is possible that an image the user intended to select is not selected, or an image the user did not intend to select is selected. This requires the user to reselect an image from their existing images.

[0015] Furthermore, when creating an album of only the images that the user really wants to select, it is necessary to manually select only the images that the user really wants from a large number of images that the user has. Such a selection operation may become a large burden for the user. Summary of the Invention

[0016] The present invention has been made in view of such circumstances, and its object is to provide an image processing device, an image processing method, and a program that can reduce the user's workload when automatically creating a photo album.

[0017] In order to achieve the above-mentioned object, the following invention modes are provided.

[0018] The image processing device involved in the present invention has one or more processors, and the processors perform the following processing: acquiring multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired multiple candidate images; assigning attributes to the user-selected image based on the analysis results of the user-selected image; inferring a user selection criterion as a criterion for selecting a user-selected image based on a user selection ratio represented by the ratio of the number of user-selected images having the same attributes to the total number of user-selected images; and selecting an automatically selected image from the unselected images not selected by the user among the candidate images based on the user selection criterion.

[0019] The image processing device according to the present invention infers the user's selection criteria based on the attributes of the user-selected image selected from candidate images, and automatically selects unselected images from the candidate images that the user has not selected based on the inferred user selection criteria. This allows images that reflect the user's preferences to be selected, reducing the user's workload in selecting images.

[0020] The image processing method involved in the present invention performs the following processing: acquiring multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the multiple candidate images acquired; assigning attributes to the user-selected image based on the analysis results of the user-selected image; inferring a user selection criterion as a criterion for selecting a user-selected image based on a user selection ratio represented by the ratio of the number of user-selected images having the same attributes to the total number of user-selected images; and selecting an automatically selected image from unselected images not selected by the user among the candidate images based on the user selection criterion.

[0021] The recording medium involved in the present invention is a non-volatile recording medium that can be read by a computer, and a program is recorded in the recording medium, which causes the computer to implement the following processing: a process of acquiring multiple candidate images; a process of acquiring user selection information related to a user-selected image selected by a user from the multiple candidate images acquired; a process of assigning attributes to the user-selected image based on the analysis results of the user-selected image; a process of inferring a user selection criterion as a criterion for selecting a user-selected image based on a user selection ratio represented by the ratio of the number of user-selected images having the same attributes to the total number of user-selected images; and a process of selecting an automatically selected image from unselected images that have not been selected by the user among the candidate images based on the user selection criterion.

[0022] Effects of the Invention

[0023] According to the present invention, the user's selection criteria are inferred based on the attributes of the user-selected image selected from candidate images. Unselected images not selected by the user from the candidate images are automatically selected based on the inferred user selection criteria. This allows images reflecting the user's preferences to be selected, reducing the user's image selection workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a functional block diagram of the image processing device according to the first embodiment.

[0025] Figure 2 yes Figure 1 The functional block diagram of the processor is shown.

[0026] Figure 3 This is a flowchart showing the steps of the image processing method according to the first embodiment.

[0027] Figure 4 This is a schematic diagram of a selection screen showing an example of a selection screen.

[0028] Figure 5 Schematic diagram showing an example of analysis results of a selected image.

[0029] Figure 6 Schematic diagram showing an example of analysis results of an unselected image.

[0030] Figure 7 This is an illustration of how to adjust the score.

[0031] Figure 8 This is an explanatory diagram of adjustment scores showing another example of adjustment scores.

[0032] Figure 9 This is a schematic diagram of a selection screen applicable to the image processing apparatus according to the second embodiment.

[0033] Figure 10 Schematic diagram showing an example of analysis results of a selected image.

[0034] Figure 11 Schematic diagram showing an example of analysis results of an unselected image.

[0035] Figure 12 This is an illustration of how to adjust the score.

[0036] Figure 13 This diagram illustrates another example of adjusting scores.

[0037] Figure 14 This is a schematic diagram of a selection screen applicable to the image processing apparatus according to the third embodiment.

[0038] Figure 15 This diagram is an explanatory diagram of the user's preferences and the like estimated from displayed unselected images.

[0039] Figure 16 1 is an explanatory diagram of a reference score showing an example of a reference score.

[0040] Figure 17 This is an illustration of how to adjust the score.

[0041] Explanation of symbols

[0042] 10 - Image processing device, 12 - Image processing unit, 14 - Display unit, 16 - Input unit, 18 - Storage unit, 20 - Processor, 22 - Memory, 24 - Image memory, 26 - Information memory, 28 - Program memory, 30 - Communication interface, 32 - Input / output interface, 40 - Display driver, 42 - Input driver, 44 - Storage driver, 50 - Specification information acquisition unit, 52 - Image acquisition unit, 54 - User selection information acquisition unit, 56 - Analysis unit, 58 - Attribute information assignment unit, 60 - Reference estimation unit, 62 - Automatic selection unit, 64 - Layout unit, 66 - Order information transmission unit, 100 - Selection screen, 102-candidate image, 102A-user selected image, 104-selection symbol, 106-OK button, 108-scroll bar, 200-selection screen, 202-candidate image, 202A-user selected image, 202B-unselected image, 204-negative selection symbol, 206-OK button, 208-scroll bar, 300-selection screen, 300A-non-display area, 302-candidate image, 302A-user selected image, 302B-displayed unselected image, 302C-non-display candidate image, 304-selection symbol, 306-OK button, 308-scroll bar, S10 to S24-various steps of the image processing method. DETAILED DESCRIPTION

[0043] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same reference numerals are given to the same components, and repeated descriptions are omitted as appropriate.

[0044] [First embodiment]

[0045] [Configuration Example of Image Processing Device]

[0046] Figure 1 This is a functional block diagram of an image processing device according to the first embodiment. The image processing device 10 shown in this figure executes automatic album creation software, automatically selects images to be used in the album, automatically lays out the selected images, and places orders with album manufacturers.

[0047] In this specification, the term "production" may also be referred to as "generation" or "manufacturing." Furthermore, the term "device" may also be referred to as "system."

[0048] The image processing device 10 can be applied to mobile terminals such as computers and smartphones. In this specification, the term "image" is sometimes used to mean image data, which is an electrical signal representing an image. In addition, an image may also be referred to as a photograph.

[0049] The image processing device 10 includes an image processing unit 12, a display unit 14, an input unit 16, and a storage unit 18. The image processing unit 12 includes a processor 20 and a memory 22. The memory 22 includes an image memory 24, an information memory 26, and a program memory 28.

[0050] The processor 20 reads and executes the automatic album creation software stored in the program memory 28 and performs predetermined processing on the images read from the image memory 24. The processor 20 stores the acquired various information in the information memory 26.

[0051] The image processing unit 12 includes a communication interface 30 and an input / output interface 32 .

[0052] The communication interface 30 includes a communication port for connecting to a network. For example, the image processing unit 12 can communicate data with an external device via the network using the communication interface 30. The communication interface 30 is applicable to both wireless communication and wired communication.

[0053] The input / output interface 32 is preferably a port for connecting an electric signal line such as a USB cable, a slot for inserting a device such as a memory card, etc. USB is an abbreviation of Universal Serial Bus.

[0054] The image processing unit 12 includes a display driver 40 , an input driver 42 , and a storage driver 44 .

[0055] The display driver 40 controls the display of the display unit 14. The display driver 40 transmits an image signal representing an image displayed on the display unit 14 to the display unit 14. The display unit 14 displays an image corresponding to the image signal transmitted from the display driver 40.

[0056] The input driver 42 acquires a signal indicating information input from the input unit 16. The input driver 42 transmits the signal transmitted from the input unit 16 to the processor 20. The processor 20 controls the image processing unit 12 according to the signal transmitted from the input unit 16.

[0057] A keyboard, a mouse, etc. can be applied to the input unit 16. A touch panel can be applied as the input unit 16. In one embodiment in which a touch panel is applied, the input unit 16 and the display unit 14 are configured as an integrated unit.

[0058] The storage driver 44 controls writing of data to the storage unit 18 and reading of data from the storage unit 18. That is, the processor 20 can acquire images owned by the user stored in the storage unit 18 via the storage driver 44.

[0059] When a mobile terminal device is used as the image processing device 10 , a system in which the image processing unit 12 , the display unit 14 , the input unit 16 , and the storage unit 18 are housed in the same housing can be used.

[0060] Figure 2 yes Figure 1 The processing units of the processor 20 shown in this figure correspond to various functions of the processor 20.

[0061] The processor 20 includes a specification information acquisition unit 50. The specification information acquisition unit 50 acquires specification information related to the specifications of the album, such as the size and number of pages of the album. The specification information can include, for example, information related to the number of images used in the album.

[0062] The information related to the number of images used in the album may indicate, for example, the minimum number of images required to create the album or the maximum number of images that can be used to create the album. Furthermore, the information related to the number of images used in the album may be set as the number of images or as the number of pages.

[0063] That is, information about the number of images used in the album can be specified by the number of pages and the number of images used in each page. Furthermore, the specification information can include, for example, information about the size of the images used in the album. The specification information can be input by the user using the input unit 16.

[0064] The processor 20 includes an image acquisition unit 52. The image acquisition unit 52 acquires candidate images, which are multiple images owned by the user and are candidates for images used in the album. The image acquisition unit 52 can acquire candidate images in folder units. The image acquisition unit 52 can acquire candidate images from an external device connected to the network via the communication interface 30, or from an external device connected via the input / output interface 32. In addition, the candidate images are represented using symbol 102 in Figure 4 Middle picture.

[0065] The processor 20 includes a user selection information acquisition unit 54. The user selection information acquisition unit 54 acquires user selection information for a user selection image manually selected by the user from the candidate images. The user selection information acquisition unit 54 stores the user selection information in Figure 1 The information memory 26 is shown. The user selection information can be applied to the identification information of the user selected image such as the file name of the user selected image.

[0066] Regarding user image selection, an upper limit on the number of candidate images that can be selected can be set. This upper limit can be determined based on information related to the number of images used in the album. The upper limit can be set arbitrarily within a range from 0% to 100% of the total number of candidate images. The upper limit can be a predetermined fixed value or a variable value set appropriately by the user.

[0067] Furthermore, regarding the user's image selection, a lower limit value for the number of candidate images that need to be selected can be set. For example, the lower limit value can be set to 2, and the user needs to select two or more candidate images.

[0068] The processor 20 includes an analysis unit 56. The analysis unit 56 analyzes the user-selected image based on predetermined attributes such as the type of subject, focus level indicating the degree of blur and shake of the image, brightness level, color, and the presence or absence of people. The analysis unit 56 stores the analysis results of the user-selected image in the information memory 26. The user-selected image is indicated by the symbol 102A. Figure 4 Middle picture.

[0069] The processor 20 includes an attribute information assigning unit 58. The attribute information assigning unit 58 assigns attribute information of each type of attribute suitable for analyzing the user selected image to the user selected image. The attribute information assigning unit 58 stores the attribute information of each user selected image and each type of attribute in the information memory 26. Figure 5 etc. are shown in the figure.

[0070] The processor 20 includes a reference estimation unit 60. The reference estimation unit 60 estimates a user selection reference corresponding to the user's preferences and intentions based on the attribute information. Specifically, the reference estimation unit 60 can estimate the user selection reference corresponding to the user's preferences for selecting an image based on the subject determination result. Furthermore, the reference estimation unit 60 can estimate the user selection reference for selecting an image that includes a specific person as attribute information in the user-selected image.

[0071] The reference estimation unit 60 calculates the user selection ratio by dividing the number of user-selected images assigned the same attribute information by the total number of user-selected images, and estimates the user selection reference based on the user selection ratio. The reference estimation unit 60 stores the estimated user selection reference in the information memory 26.

[0072] The reference estimation unit 60 can estimate the user selection reference based on the comparison result between the user selection ratio and a predetermined threshold value. The predetermined threshold value can be a fixed value set in advance or a variable value that can be arbitrarily set by the user.

[0073] Furthermore, the predetermined threshold value may be different for each attribute. For example, the threshold value may be set lower for items such as the type of subject and the presence of a person, which are easy to compare, while the threshold value may be set lower for items such as focus, brightness, and color, which are difficult to compare.

[0074] The reference estimation unit 60 calculates an evaluation value for each user-selected image and each piece of attribute information, calculates a reference score for each user-selected image based on the evaluation value, calculates a correction value based on the user selection ratio, and uses the correction value to adjust the reference score to calculate an adjusted score. The adjusted score functions as a numerical indicator corresponding to the user's preferences, etc., for each user-selected image.

[0075] The processor 20 includes an automatic selection unit 62. The automatic selection unit 62 applies the user selection criterion estimated by the usage criterion estimation unit 60 and selects an automatically selected image from unselected images of the candidate images as an image to be used in the album.

[0076] The automatic selection unit 62 can select a predetermined number of automatically selected images from the unselected images in descending order of the adjustment scores calculated using the reference estimation unit 60. At least one of an upper limit and a lower limit can be set for the number of automatically selected images to be selected. For example, at least one of the upper limit and the lower limit can be set based on information regarding the number of images included in the album's specifications.

[0077] In addition, the unselected image uses the symbol 102B in Figure 4 The automatic selection unit 62 stores the automatically selected image information including identification information such as the file name of the automatically selected image in the information memory 26 .

[0078] The processor 20 includes a layout unit 64. The layout unit 64 creates an automatic layout for the album using user-selected images and automatically selected images according to the album specifications acquired using the specification information acquisition unit 50. The layout unit 64 displays the automatically laid-out album on the display unit 14.

[0079] The layout unit 64 can receive the automatic layout edit and perform manual editing of the automatic layout. An example of manual editing is a method in which the user manually inputs the edit content using the input unit 16 or the like for the automatic layout displayed on the display unit 14 .

[0080] The processor 20 includes an order information transmission unit 66. This unit transmits order information for the photo album to the photo album manufacturer via the communication interface 30. The order information includes the album's specifications, the images used in the album, and the user-approved album layout. The order information may also include information such as a delivery deadline and delivery method.

[0081] Furthermore, the image processing apparatus 10 may not include the layout unit 64 and the order information transmission unit 66 , and the functions of the layout unit 64 and the order information transmission unit 66 may be implemented in an external device connected to the image processing apparatus 10 .

[0082] [Steps of image processing method]

[0083] Figure 3 1 is a flowchart showing the steps of the image processing method according to the first embodiment. In the specification information acquisition step S10, Figure 2 The specification information acquisition unit 50 shown in FIG. acquires the specification information of the album. The specification information acquisition unit 50 stores the specification information in Figure 1 The information memory 26 is shown. After the specification information acquisition step S10, the process proceeds to the candidate image acquisition step S12.

[0084] In addition, Figure 1 If the photo album creation software is not installed in the image processing apparatus 10 shown, a preparation step of downloading the photo album creation software from a download site and installing the software is performed before the specification information acquisition step S10 .

[0085] Album specification information includes, for example, the size and number of pages of the album. Examples of album sizes include A4, A5, and A5 Square. Examples of page numbers include 16 pages, 24 pages, 32 pages, 40 pages, and 48 pages. The number of images required to create the album is determined based on the size and number of pages specified. For example, if 24 pages are specified, the minimum required number of images is 100.

[0086] In the candidate image acquisition step S12 , the image acquisition unit 52 acquires candidate images owned by the user.

[0087] The image acquisition unit 52 stores the acquired candidate images in the image memory 24. After the candidate image acquisition step S12, the process proceeds to the user selection information acquisition step S14.

[0088] Candidate images can be categorized and stored in multiple folders based on attributes, etc. In the candidate image acquisition step S12, the image acquisition unit 52 can select one or more folders from the multiple folders and acquire the candidate images contained in the selected folders. Alternatively, the image acquisition unit 52 can simultaneously acquire multiple images stored on the memory card as candidate images. Furthermore, in the candidate image acquisition step S12, the image acquisition unit 52 can search for images associated with the user and automatically acquire the searched images as candidate images.

[0089] In the user selection information acquisition step S14, the user selection information acquisition unit 54 acquires user selection information for the user selection image. After the user selection information acquisition step S14, the process proceeds to the user selection image analysis step S16.

[0090] In the user-selected image analysis step S16, the analysis unit 56 analyzes the user-selected image. Based on the analysis results, the reference estimation unit 60 estimates a user-selected reference corresponding to the user's preferences, etc. The reference estimation unit 60 stores the estimated user-selected reference in the information memory 26. After the user-selected image analysis step S16, the process proceeds to the automatic selection step S18.

[0091] In the automatic selection step S18, the automatic selection unit 62 selects an automatically selected image from the unselected candidate images using the user selection criteria estimated in the user-selected image analysis step S16. The automatic selection unit 62 stores the automatically selected image information of the automatically selected image in the information memory 26. After the automatic selection step S18, the process proceeds to the layout step S20.

[0092] In the layout step S20 , the layout unit 64 uses the user-selected images and the automatically selected images to create an automatic layout of the album based on the specification information such as the size and number of pages acquired in the specification information acquisition step S10 .

[0093] In the layout step S20, the layout unit 64 can display the automatic layout using the display unit 14. The layout unit 64 stores layout information related to the automatic layout in the information memory 26. After the layout step S20, the process proceeds to the editing step S22.

[0094] In the editing step S22, the layout unit 64 receives the user's manual edits to the automatic layout and edits the automatic layout. The layout unit 64 can display the edited layout using the display unit 14. The layout unit 64 stores layout edit information for the edited layout in the information memory 26. The editing step S22 can be omitted. After the editing step S22, the process proceeds to the order information sending step S24.

[0095] In the order information transmission step S24 , the order information transmission unit 66 transmits the order information of the photo album including the specification information of the photo album, images used in the photo album, layout information of the photo album, etc. to the photo album manufacturer.

[0096] After the order information transmission step S24, a predetermined end process is performed, and the processor 20 ends the image processing method. The photo album producer who receives the order information produces the photo album based on the order information.

[0097] In addition, the layout step S20, the editing step S22 and the order information sending step S24 may not be used. Figure 1 The present invention is implemented by the image processing apparatus 10 shown in the figure, and is implemented using an external device connected to the image processing apparatus 10.

[0098] [Selection screen configuration example]

[0099] Figure 4 1 is a schematic diagram of a selection screen showing an example of a selection screen. The selection screen 100 is displayed using the display unit 14. The selection screen 100 displays at least a portion of a candidate image 102. Figure 4 Thumbnail images are used as candidate images 102 displayed on the selection screen 100. The selection screen 100 may also display accompanying information such as the file names of the candidate images 102, the total number of candidate images, and the number of images selected by the user.

[0100] The user selects two or more user-selected images 102A from the plurality of candidate images 102 displayed on the selection screen 100 . Figure 4 The user selection image 102A shown is displayed with a selection symbol 104 superimposed thereon, indicating the user's selection.

[0101] The confirmation button 106 displayed on the selection screen 100 is a button operated by the user to confirm the user's selection. By operating the scroll bar 108 on the selection screen 100, the user can sequentially display all the candidate images 102 on the selection screen 100.

[0102] Figure 4 The symbol 102B shown indicates an unselected image that has not been selected by the user among the candidate images 102. The unselected image 102B is not displayed superimposed with the selection symbol 104 attached to the user-selected image 102A. Figure 4 The user selection image 102A shown is an example of a first user selection image. The user selection information of the user selection image 102A is an example of first user selection information.

[0103] [Specific Example of Analysis Processing in Selected Image Analysis Step]

[0104] Figure 4 The analysis of the user selected image 102A shown is applied to one or more attributes such as the determination result of the subject, blur and shake degree, brightness level, color, face recognition of people, and number of people, and attribute information is assigned to each attribute.

[0105] Figure 5 : is a schematic diagram showing an example of analysis results of a selected image. Figure 5 An arbitrary character string is described in the file name shown. Figure 5 In the example, as the types of attributes, the determination result of the subject, the degree of blurring and shaking, the degree of brightness, the color, and the person are given. Figure 5 The displayed blur and shake levels indicate the blur and shake levels, in other words, the focus level.

[0106] The subject determination result can classify the subject into types such as people, scenery, and objects. Figure 5 In the example of the object type, dishes, landscapes and people are given. The subject can be determined by applying image labeling technology.

[0107] The degree of blurring and jitter can be represented by a blurring and jitter evaluation value represented by a numerical value ranging from 0 to 100. A relatively large blurring and jitter evaluation value indicates a good image with a relatively small degree of blurring and jitter.

[0108] Figure 5 In the analysis results shown, the blur and jitter evaluation values ​​for four of the five user-selected images 102A are 80 or higher. This indicates that the user selection criterion for images with blur and jitter evaluation values ​​of 80 or higher can be inferred. In other words, if the user selection ratio is 80% or higher, the blur and jitter evaluation value can be inferred as the user selection criterion.

[0109] The brightness level can be represented by a brightness evaluation value expressed using a numerical value ranging from 0 to 100. A relatively large brightness evaluation value indicates a relatively bright and good image. The brightness level can be represented by the value in the Munsell color system.

[0110] Figure 5 In the analysis results shown, three of the five user-selected images 102A have brightness evaluation values ​​of 80 or higher. This indicates that the user selection criterion is that the user selected an image with a brightness evaluation value of 80 or higher. In other words, if the user selection ratio is 60% or higher, the brightness evaluation value can be inferred as the user selection criterion.

[0111] Color can be represented by a color evaluation value using a numerical value ranging from 0 to 100. A relatively large color evaluation value indicates a good image with relatively vivid chromaticity. Brightness can be represented by chroma in the Munsell color system.

[0112] Figure 5 In the analysis results shown, three of the five user-selected images 102A have color evaluation values ​​of 80 or higher. This indicates that the user selection criterion is that the user selected an image with a color evaluation value of 80 or higher. In other words, if the user selection ratio is 60% or higher, the color evaluation value can be inferred as the user selection criterion.

[0113] Character indicates whether to include a specific character. Figure 5The user-selected image 102A indicated by the file name 0002.xxx includes the person A and the person B. The user-selected image 102A indicated by the file name 0004.xxx includes the person A.

[0114] exist Figure 5 In the analysis results shown, of the five user-selected images 102A, three with file names 0001.xxx, 0003.xxx, and 0005.xxx are assigned dish attribute information. Based on this analysis result, it is possible to infer the user selection criteria for selecting images containing dishes. In other words, if the user selection ratio is 60% or higher, the subject determination result can be inferred as the user selection criteria.

[0115] Furthermore, among the five user-selected images 102A, two user-selected images 102A having file names 0002.xxx and 0004.xxx have person A assigned as attribute information. Based on this analysis result, the user's selection criteria for selecting an image including person A can be estimated.

[0116] In other words, if the user selection ratio is 40% or greater, the person information included in the person identification result can be inferred as the user selection criterion. Furthermore, the user selection criterion can also be inferred based on the ratio of user-selected persons in user-selected images 102A relative to the total number of user-selected images 102A containing persons.

[0117] For example, in Figure 5 Among the five user-selected images 102A shown, the user-selected image 102A including the person A is the user-selected image 102A having the file name 0002.xxx and the file name 0004.xxx.

[0118] Since person A is assigned to both user-selected images 102A having file names 0002.xxx and 0004.xxx, the user selection rate for person A is 100%. For example, if the threshold for estimating the user selection criterion is 80% or higher, it can be estimated that the user selection criterion is to select an image containing person A.

[0119] That is, in the analysis and processing of the user selection image 102A, attribute information of one or more types of attributes is assigned to each user selection image 102A, and a user selection criterion corresponding to the user's preferences, etc. is inferred based on the user selection ratio obtained by dividing the number of user selection images 102A having the same attribute information by the total number of user selection images 102A.

[0120] exist Figure 5While the example shown illustrates a method for estimating the user selection criterion using attribute information whose user selection ratio exceeds a predetermined threshold, the threshold for the user selection ratio can be arbitrarily determined based on factors such as the number of images selected by the user. The threshold for the user selection ratio can be a predetermined fixed value or a value manually set by the user.

[0121] in addition, Figure 5 The types of attributes shown are examples, and the types of attributes applicable to the selected image are not limited to Figure 5 For example, the number of attribute types can be more than one. For example, using Figure 2 The attributes analyzed and assigned to the non-selected images by the analysis unit 56 shown above may be at least one of the type of subject, focus level indicating the degree of blur and shake of the image, brightness, color, and the presence or absence of a person.

[0122] And, based on Figure 5 The user selection ratio of the user selection image 102A shown is an example of the first user selection ratio. The user selection criterion based on the first user selection ratio is an example of the first user selection criterion.

[0123] [Specific Example of Selection Processing in Automatic Selection Step]

[0124] exist Figure 4 In the automatic selection step S18 shown, unselected image 102B among the candidate images is analyzed, and an automatically selected image is selected from the unselected image 102B using the user selection criteria. The analysis of unselected image 102B can be performed using the same process as that for analyzing user-selected image 102A in the selected image analysis step.

[0125] Figure 6 : is a schematic diagram showing an example of analysis results of an unselected image. Figure 6 In the analysis results of the non-selected image 102B shown, Figure 5 The user shown selects the same attribute type as the analysis result of image 102A, and assigns attribute information to unselected image 102B.

[0126] The non-selected images 102B use the attribute information assigned to each non-selected image 102B, and a reference score is calculated for each non-selected image 102B. Figure 6 The reference score shown is calculated by setting the score of each character to 100 and adding the numerical value obtained by fractionalizing the number of characters to the sum of the blur and blur evaluation value, the brightness evaluation value, and the color evaluation value.

[0127] Next, an adjustment score is calculated based on the base score adjusted according to the user selection criteria. If the user selection criteria are estimated to select images containing dishes, an adjustment value is assigned a positive weight to unselected images 102B containing dishes to calculate the adjustment score. A predetermined number of automatically selected images are selected in descending order of adjustment scores.

[0128] Figure 7 : is an explanatory diagram of the adjustment score. Since the unselected image 102B with the file name 0011.xxx has the analysis result that the subject includes a dish, the positive adjustment value +300 is applied to Figure 6 The adjusted scores are calculated using the baseline scores shown.

[0129] Furthermore, when it is estimated that the selection criteria for selecting images containing people are to be selected, the non-selected images 102B containing people in the subject are given positive adjustment values ​​as positive weights, and the adjusted values ​​are calculated. Figure 6 Adjusted scores for the benchmark scores shown.

[0130] Figure 8 10. This is an explanatory diagram showing another example of adjustment scores. Since the unselected image 102B with the file name 0010.xxx and the unselected image 102B with the file name 0013.xxx both obtain the analysis result that the subject includes person A, the positive adjustment value +300 is applied to Figure 6 Adjusted scores are calculated based on the baseline scores shown.

[0131] In this manner, the adjustment value is used as a weight corresponding to the user selection criterion to calculate an adjustment score obtained by adjusting the criterion score calculated based on the analysis result of the non-selected image 102B, and the automatically selected images are selected in descending order of the adjustment score.

[0132] In this embodiment, the adjustment values ​​adjusted based on the subject determination result and the user determination criteria based on multiple attributes such as a person are set to be the same, but the present invention is not limited to this. Different adjustment values ​​may be set for each attribute.

[0133] For example, compared with the adjustment value adjusted using the user's determination criteria based on the subject determination result or person, the adjustment value can be lowered by using the user's determination criteria based on attributes such as blurring, brightness, and color.

[0134] [Specific example of automatic layout]

[0135] When laying out the plurality of images selected from the candidate images 102 as images to be used in the album on each page of the album, the user selected image 102A may be laid out with priority over the automatically selected images.

[0136] For example, when laying out the user-selected image 102A and the automatically selected image on each page of the album, the user-selected image 102A may be laid out larger than the automatically selected image, or may be laid out on a page with a smaller page number than the automatically selected image.

[0137] [Effects of the First Embodiment]

[0138] According to the image processing device and the image processing method according to the first embodiment, the following effects can be obtained. [1]

[0140] The user-selected image 102A selected by the user is analyzed and a user selection criterion is estimated. Unselected images 102B are automatically selected based on the estimated user selection criterion. Thus, the automatic layout of the album can be implemented using the user-selected image and the automatically selected image. [2]

[0142] A benchmark score based on the attribute information is calculated for each unselected image 102B. The benchmark score is adjusted using an adjustment value representing a weight based on the user's selected benchmark, and the adjusted score is calculated for each unselected image 102B. This allows the automatically selected images used in the album to be selected in descending order of adjustment score.

[0143] [Second embodiment]

[0144] [Overall Structure of Image Processing Device]

[0145] Next, an image processing apparatus according to the second embodiment will be described. In the following description, the differences from the image processing apparatus 10 according to the first embodiment will be mainly described. Figure 2 The overall configuration of an image processing device according to the second embodiment will be described.

[0146] In the image processing device involved in the second embodiment, the user manually selects user-selected images that he does not want to include in the album, and based on the analysis results of the user-selected images, the user selection criteria reflecting the user's preferences such as not wanting to include in the album are inferred, and the images that he does not want to include in the album are excluded from the automatically selected images.

[0147] In other words, when automatically selected images are selected from unselected images, unselected images that are presumed not to be included in the album are excluded, and automatically selected images to be used in the album are selected.

[0148] The user selection information acquisition unit 54 acquires user selection information of user selected images manually selected by the user from the perspective of images not to be included in the album. The analysis unit 56 analyzes the user selected images, and the attribute information assignment unit 58 assigns attribute information of each attribute type to each user image.

[0149] The reference estimation unit 60 estimates the user selection reference for images that the user does not want to include in the album based on the ratio of user-selected images having the same attribute information to the total number of user-selected images. In other words, the reference estimation unit 60 estimates the user selection reference as a negative weight.

[0150] The automatic selection unit 62 selectively excludes images that are estimated not to be included in the album by the user from the unselected images based on the user selection criteria, and selects the unselected images excluding the excluded images as automatically selected images.

[0151] The layout unit 64 uses the user selected images and the automatically selected images to implement the automatic layout of the album. Figure 9 The user selects an image using symbol 202A. Figure 9 The unselected image is shown in the figure using symbol 202B. Figure 9 Middle picture.

[0152] [Steps of image processing method]

[0153] refer to Figure 3 The image processing method according to the second embodiment will be described. In the user selection information acquisition step S14 , the user selection information acquisition unit 54 acquires user selection information of user selected images manually selected by the user, with the user not wanting to include them in the album.

[0154] In the user selected image analysis step S16 , the user selected images are analyzed, and a user selection criterion for images that the user does not want to include in the album is estimated based on the ratio of user selected images having the same attribute information to the total number of user selected images.

[0155] In the automatic selection step S18 , images that are estimated not to be included in the album by the user are selectively excluded from the unselected images based on the user selection criteria, and the unselected images excluding the excluded images are selected as automatically selected images.

[0156] In the layout step S20 , the automatic layout of the album is performed using the user-selected images and the automatically selected images.

[0157] [Selection screen configuration example]

[0158] Figure 92 is a schematic diagram of a selection screen applicable to the image processing device according to Embodiment 2. In the selection screen 200, a user selection image 202A is displayed superimposed with a minus selection symbol 204 indicating the user's selection.

[0159] When the user operates the OK button 206 , the user's selection of the image not to be included in the album is confirmed. Even in the selection screen 200 , the user can operate the scroll bar 208 to sequentially display all candidate images 202 on the selection screen 200 .

[0160] in addition, Figure 9 The user selected image 202A shown is an example of a second user selected image selected by the user from the candidate images as an image not used in the album. Figure 9 The user selection information corresponding to the user selection image 202A shown is an example of the second user selection information. Figure 9 The user selection ratio of the user selection image 202A shown is an example of the second user selection ratio. Figure 9 The user selection criterion estimated by the user selection image 202A shown is an example of the second user selection criterion.

[0161] [Specific Example of Analysis Processing in Selected Image Analysis Step]

[0162] Figure 10 1 is a schematic diagram showing an example of analysis results of a selected image. The analysis results of the selected image in the second embodiment are similar to those in the first embodiment, and subject determination, blurring and blurring degree, brightness degree, color, facial recognition of people, and number of people are used as attributes.

[0163] exist Figure 10 In the analysis results shown, three of the five user-selected images 202A were selected, each containing a landscape. This suggests that the user's selection criteria, which is a preference for not adding images containing landscapes to an album, can be inferred. In other words, if the user selection ratio is 60% or higher, the subject determination result can be inferred to be the user's selection criteria.

[0164] Furthermore, among the five user-selected images 202A, two user-selected images 202A in which the subject includes the person C are selected. From this, it can be inferred that the user's selection criterion is that the user does not want to add images in which the subject includes the person C to the album.

[0165] In other words, if the user selection ratio is 40% or greater, the person identification result can be inferred as the user selection criterion. Furthermore, similarly to the first embodiment, the person identification result can also be used to infer the user selection criterion based on the ratio of user-selected persons in user-selected images 202A relative to the total number of user-selected images 202A containing persons.

[0166] Furthermore, not limited to the subject determination result, the user selection criterion may be estimated based on attribute information such as blurring, brightness, and color, when the user selection ratio relative to unselected images is greater than a predetermined threshold.

[0167] [Specific Example of Selection Processing in Automatic Selection Step]

[0168] Figure 11 This is a schematic diagram showing an example of analysis results for unselected images. In the analysis of unselected images, similar to the first embodiment, attribute information is assigned to unselected images 202B for the same attribute types as those for user-selected image 202A. Furthermore, similar to the first embodiment, a benchmark score is calculated for each unselected image 202B using this attribute information. The benchmark scores for unselected images 202B are omitted from illustration.

[0169] In the automatic selection of unselected images 202B, a negative adjustment value is calculated as a negative weight based on the user's selection criteria for not adding the images to the album. The negative adjustment value is used to adjust the criteria score and calculate the adjustment score. A predetermined number of automatically selected images can be selected in descending order of adjustment scores.

[0170] Figure 12 2 is an explanatory diagram of the adjustment score. When the user's selection criteria are estimated as not wanting to add images containing dishes to the album, the unselected image 202B with the file name 0011.xxx is assigned an adjustment value of -300 as a negative weight to calculate the adjustment score.

[0171] Figure 13 This diagram illustrates another example of an adjustment score. When the user's selection criteria are determined to not include images containing person A in the album, the unselected images 202B with file names 0010.xxx and 202B with file names 0013.xxx are assigned an adjustment value of -300 as a negative weight to calculate the adjustment score.

[0172] Specifically, based on the user's selection criteria for not adding an image to the album, a negative weight is applied to the benchmark score of each unselected image 202B to calculate an adjustment score. A predetermined number of automatically selected images can be selected from the unselected images 202B in descending order of the adjustment scores calculated for each unselected image 202B.

[0173] [Operation and Effect of the Second Embodiment]

[0174] According to the image processing device and the image processing method according to the second embodiment, the following effects can be obtained. [1]

[0176] User-selected image 202A, manually selected as an image the user does not want to include in the album, is analyzed to infer a user selection criterion for the image the user does not want to include in the album. Images inferred based on the inferred user selection criterion are excluded from unselected images 202B, and automatically selected images are selected from unselected images 202B. This allows automatic layout of the album to be implemented using both the user-selected and automatically selected images. [2]

[0178] The unselected images 202B are assigned a baseline score based on their attribute information. The baseline score is adjusted using a negative weight based on the user selection criteria for images that the user does not want to include in the album, and an adjusted score is calculated. This allows the images used in the album to be automatically selected in descending order of adjusted scores.

[0179] [Third embodiment]

[0180] [Overall Structure of Image Processing Device]

[0181] Next, an image processing apparatus according to a third embodiment will be described. In the following description, the differences from the image processing apparatus 10 according to the first embodiment will be mainly described. Figure 2 The overall configuration of an image processing device according to the second embodiment will be described.

[0182] In the image processing device involved in the third embodiment, when the user-selected image is selected, the displayed unselected image information as the information of the displayed unselected image displayed in the selection screen among the unselected images set by the user as unselected is maintained, and the displayed unselected image information is used to infer the user's displayed unselected basis, thereby enhancing the inference of the user's selection basis.

[0183] Figure 2 The user selection information acquisition unit 54 acquires user selection information and acquires information about displayed unselected images. Examples of the information about displayed unselected images include the file names of the displayed unselected images, their positional relationship with the user-selected image on the selection screen, and the duration of their display on the selection screen.

[0184] The reference estimation unit 60 estimates a user display non-selection reference indicating the user's intention to deselect the displayed non-selected image based on the displayed non-selected image information. Using the estimated user display non-selection reference, the estimation of the user selection reference is enhanced by considering the user's intention to deselect the displayed non-selected image.

[0185] [Steps of image processing method]

[0186] exist Figure 3 In the user selection information acquisition step S14 shown, user selection information is acquired, and information about displayed unselected images is also acquired. In the user selection image analysis step S16, non-display selection information related to the displayed unselected images is acquired. Based on the analysis results of the displayed unselected images, a user display unselection criterion indicating the user's intention to unselect the displayed unselected images is inferred. Using the inferred user display unselection criterion, the inference of the user selection criterion is enhanced by taking into account the user's intention to unselect the displayed unselected images.

[0187] [Selection screen configuration example]

[0188] Figure 14 This is a schematic diagram of a selection screen applicable to the image processing device according to the third embodiment. Non-display area 300A is an area where non-display candidate images 302C, which are not displayed on selection screen 300, are arranged. This area is displayed on selection screen 300 when scroll bar 308 is operated. When non-display candidate images 302C are displayed on selection screen 300, they can be selected as user-selected images 302A. Reference numeral 302 denotes a candidate image. Reference numeral 302B denotes the display of an unselected image. Reference numeral 304 denotes a selection symbol. Reference numeral 306 denotes an OK button.

[0189] [User preferences estimated by displaying unselected images, etc.]

[0190] Figure 15 This diagram is an explanatory diagram of the user's preferences and the like estimated from displayed unselected images. Figure 15 , as the non-selected image information, the positional relationship with the user selected image 302A in the selection screen 300 and the similarity with the user selected image 302A during the display period on the selection screen 300 are exemplified.

[0191] The positional relationship with the user selected image 302A in the selection screen 300 can be expressed as a distance between the two using a numerical value such as the number of pixels. The similarity with the user selected image 302A can be expressed as a numerical value such as a ratio.

[0192] and, Figure 153 shows the result of inferring the user's intention to deselect displayed unselected image 302B. For example, if displayed unselected image 302B is displayed on selection screen 300 for a relatively short period of time, it can be inferred that the user barely visually recognized it, and therefore had no intention of not selecting it, because the displayed unselected image 302B passed through the user's visual recognition range while scrolling selection screen 300 while searching for user-selected image 302A.

[0193] On the other hand, unselected image 302B displayed on selection screen 300 for a relatively long time, located relatively close to user-selected image 302A on selection screen 300, and having a relatively high degree of similarity to user-selected image 302A can be inferred to be an image that the user visually recognizes but has not selected. In other words, it can be inferred to be an image that the user does not intend to select.

[0194] Regarding the unselected image 302B displayed on the selection screen 300 for a relatively long period of time, having a relatively close positional relationship with the user-selected image 302A in the selection screen 300, and having a relatively high similarity, it can be inferred that although it is visually recognized by the user, it is similar to the user-selected image 302A, and therefore the selection is retained.

[0195] like Figure 15 As shown, a negative adjustment value for adjusting the base score can be assigned to the result of estimating the user's intention not to select in displayed unselected image 302B, and an adjustment score is calculated by adjusting the base score calculated for each displayed unselected image 302B using the adjustment value. The negative adjustment value based on the result of estimating the user's intention not to select is a negative weight relative to the base score.

[0196] When the weight based on the user selection criterion is negative, the weight based on the user display non-selection criterion becomes positive. In other words, when the weight based on the user selection criterion is either positive or negative, the weight based on the user display non-selection criterion becomes the other of the positive or negative weights.

[0197] In addition, in this embodiment, for displaying the unselected image 302B, the user display unselected benchmark is inferred based on at least any one of the display period, the display position relationship with the selected image, and the similarity, and the adjustment value is set based on the inferred user display unselected benchmark, but it is not limited to this.

[0198] The displayed unselected image may also be determined as the user selection image 202A (second user selection image) of the second embodiment. Similarly to the second embodiment, the ratio of the displayed unselected images 302B of each attribute to the total number of displayed unselected images 302B is determined as the user selection ratio (second user selection ratio), and the user selection criterion (second user selection criterion) not used in the album is inferred based on the ratio of the displayed unselected images 302B.

[0199] Furthermore, it is also possible to display only the images that meet the specified conditions in the unselected images 302B, for example, Figure 15 The image determined as not to be selected is determined to be the user selected image (second user selected image) of the second embodiment, and the same processing as the second embodiment is performed.

[0200] [Specific Example of Selection Processing in Automatic Selection Step]

[0201] Figure 16 302B is an explanatory diagram showing an example of a reference score. The calculation of the reference score for each displayed non-selected image 302B shown in this diagram is the same as that in the first and second embodiments. The description thereof will be omitted here.

[0202] Figure 17 This is an illustration of how to adjust the score. Figure 7 The adjusted scores shown take into account Figure 15 The adjustment values ​​shown are calculated Figure 17 The displayed non-selected image 302B having the file name 0011.xxx is assigned +300 as an adjustment value corresponding to the user's selection criteria for selecting an image including a dish as a subject.

[0203] On the other hand, the displayed non-selected image 302B having the file name 0011.xxx is assigned -300 as an adjustment value corresponding to the user's intention of non-selection. The reference score is adjusted using the adjustment value to calculate the adjustment score.

[0204] [Operation and Effect of the Third Embodiment]

[0205] According to the image processing device and the image processing method according to the third embodiment, the following effects can be obtained. [1]

[0207] The user's intention of not selecting is estimated based on the displayed unselected images 302B that the user has not selected displayed on the selection screen 300. Thus, when selecting an automatically selected image from the unselected images, the user's intention of not selecting each unselected image can be considered. [2]

[0209] Displayed unselected image 302B is assigned an adjustment value as a negative weight corresponding to the user's intention not to select an image, and a correction score is calculated by correcting the base score using the adjustment value. This allows consideration of the user's intention not to select an image for each displayed unselected image 302B when selecting a predetermined number of automatically selected images in descending order of correction scores.

[0210] [Regarding Combinations of Embodiments]

[0211] The first embodiment, the second embodiment, and the third embodiment can also be combined as appropriate. For example, the second embodiment can be combined with the first embodiment.

[0212] That is, when the user Figure 4 When manually selecting the user selected image 102A from the candidate images 102 shown, the user can also manually select Figure 9 Shown is a user-selected image 202A that the user does not want to include in the album.

[0213] By using the user selection criteria estimated from the user selection image 102A and the user selection criteria estimated from the user selection image 202A, it is possible to automatically select an image according to the user's preference or the like from among the candidate images.

[0214] When calculating adjustment scores for non-selected images, positive weights calculated based on the user selection criteria estimated based on the user selection image 102A can be applied, and negative weights calculated based on the user selection criteria estimated based on the user selection image 202A can be applied.

[0215] Furthermore, the third embodiment can be combined with the second embodiment. That is, in the third embodiment, according to the user's Figure 14 The user selection criterion is estimated from the user selected image 302A selected as an image to be used in the album among the candidate images 302 shown, and the user display non-selection criterion indicating that the user does not want to use the image in the album is estimated from the display non-selected image 302B.

[0216] In a third embodiment, a user selection criterion may be inferred based on a user-selected image selected by the user from the candidate images 302 as an image not to be used in the album, and a user-displayed non-selected criterion indicating the user's intention to use it in the album may be inferred based on the displayed non-selected image 302B.

[0217] In other words, in the second embodiment, similarly to the third embodiment, the user display non-selection reference can be estimated from the displayed non-selected image based on the positional relationship between the displayed non-selected image and the user selected image and the display period of the displayed non-selected image, and the user display non-selection reference can be applied. Figure 9The negative weight calculated based on the user selection criterion estimated based on the user selection image 202A shown, and the positive weight calculated based on the user display non-selection criterion estimated based on the user display non-selection image can be applied.

[0218] [Hardware Structure of Each Processing Unit and Control Unit]

[0219] The hardware configuration of the processing unit that executes the processing of the image processing apparatus 10 and the image processing unit 12 described in the above embodiments is composed of various processors. These processors include CPUs (Central Processing Units), PLDs (Programmable Logic Devices), and ASICs (Application Specific Integrated Circuits).

[0220] A CPU is a general-purpose processor that executes programs and functions as a variety of processing units. A PLD is a processor whose circuit structure can be modified after manufacturing. An example of a PLD is an FPGA (Field Programmable Gate Array). An ASIC is a specialized circuit with a dedicated circuit structure designed to perform specific processing.

[0221] A processing unit can be composed of one of these various processors, or can be composed of two or more processors of the same or different types. For example, a processing unit can also be composed of multiple FPGAs, etc. A processing unit can also be composed of a combination of one or more FPGAs and one or more CPUs.

[0222] Furthermore, a single processor can be used to form multiple processing units. One example of using a single processor to form multiple processing units is a method in which one processor is combined with one or more CPUs and software to function as multiple processing units. This method is exemplified by computers such as client terminals and server devices.

[0223] Another example of a configuration is the use of a processor that implements the functionality of a system consisting of multiple processing units using a single IC chip. This approach is exemplified by the system-on-chip (SoC). IC stands for Integrated Circuit, and SoC is sometimes referred to as "System on Chip."

[0224] In this manner, the various processing units are configured as hardware structures using one or more of the aforementioned various processors. More specifically, the hardware structures of the various processors are circuits formed by combining circuit elements such as semiconductor elements.

[0225] [Examples of application to procedures]

[0226] It is possible to construct a program that causes a computer to implement various functions of the image processing device and various steps of the image processing method described in this manual. Figure 2 The program shown is a program for processing corresponding to the specification information acquisition unit 50, image acquisition unit 52, user selection information acquisition unit 54, analysis unit 56, attribute information assignment unit 58, reference estimation unit 60, automatic selection unit 62, layout unit 64 and order information sending unit 66.

[0227] [Application examples to network systems]

[0228] Figure 1 The image processing device 10 shown can be configured using a system comprising multiple independent devices connected to each other. For example, the image processing device 10 can be configured using a network system comprising multiple computers connected to each other via a network. The multiple computers can be server devices and client devices connected to each other via the network for data communication. Furthermore, the above system can also be applied to cloud computing.

[0229] For example, the image processing device 10 can constitute an entire system including a terminal device having a display unit 14 and an input unit 16, an image processing server device having the image processing unit 12 excluding the memory 22, and a storage server device having the memory 22 and the storage unit 18. Furthermore, the image processing device 10 can constitute only the image processing server in the above-described system.

[0230] For example, the image processing device described above can be installed on a server device, and a user's image can be uploaded from a client device. The server device can then perform various processing operations, and the processing results can be transmitted from the server device to the client device. Furthermore, the server device is not limited to a single server device. Multiple server devices can also be used to perform various processing operations.

[0231] The embodiments of the present invention described above may be modified, added to, or deleted from the components as appropriate without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many variations are possible within the technical concept of the present invention by a person having ordinary knowledge in the field. Furthermore, the embodiments, variations, and application examples may be appropriately combined and implemented.

[0232] [Another embodiment of the present invention]

[0233] In an image processing device according to another embodiment, a processor performs the following processing: for unselected images, a benchmark score is calculated based on the evaluation value of the assigned attribute, and an adjusted score is calculated by adjusting the benchmark score using a weight corresponding to the user selection benchmark; and an automatically selected image is selected from the unselected images in descending order of the adjustment scores.

[0234] According to this aspect, it is possible to automatically select unselected images based on weights corresponding to user selection criteria.

[0235] In an image processing device involved in another embodiment, the processor performs the following processing: obtaining first user selection information related to a first user-selected image selected by a user from candidate images as an image to be used in an album; inferring a first user selection criterion corresponding to the image to be used in the album as a user selection criterion based on an analysis result of the first user-selected image; and selecting an automatically selected image as an image to be used in the album from unselected images based on the first user selection criterion.

[0236] According to this aspect, it is possible to automatically select unselected images based on the first user selection criterion estimated based on the images that the user wants to use in the album.

[0237] In the image processing device according to another aspect, the processor preferentially selects, as the automatically selected image, an unselected image having an attribute with a relatively high first user selection ratio serving as a reference for the first user selection.

[0238] According to this aspect, unselected images having attributes with a high first user selection ratio are preferentially selected.

[0239] In an image processing device involved in another embodiment, the processor performs the following processing: for unselected images, a benchmark score is calculated based on the evaluation value of the assigned attribute, and an adjusted score with the benchmark score adjusted is calculated using a positive weight corresponding to the first user selection benchmark; and automatically selected images are selected from the unselected images in descending order of the adjustment scores.

[0240] According to this embodiment, it is possible to automatically select unselected images based on weights corresponding to user selection criteria and to automatically select unselected images based on images that the user wants to use in an album.

[0241] In an image processing device involved in another embodiment, the processor performs the following processing: obtaining second user selection information related to a second user-selected image selected by the user from candidate images as an image not used in the album; inferring a second user selection criterion corresponding to the image not used in the album as a user selection criterion based on the analysis result of the second user-selected image; excluding the image not used in the album from the unselected images based on the second user selection criterion, and selecting an automatically selected image from the unselected images.

[0242] According to this aspect, it is possible to exclude images not used in the album from the unselected images based on the second user selection criterion estimated based on the images not used in the album selected by the user, and automatically select the unselected images.

[0243] In the image processing device according to another aspect, the processor preferentially excludes, as the automatically selected images, non-selected images having an attribute with a relatively high second user selection ratio serving as a reference for the second user selection.

[0244] According to this aspect, non-selected images having an attribute with a high second user selection ratio are preferentially excluded.

[0245] In an image processing device involved in another method, the processor performs the following processing: for unselected images, a baseline score is calculated based on the evaluation value of the assigned attribute, and an adjusted score is calculated by adjusting the baseline score using a negative weight corresponding to the second user selection benchmark; and automatically selected images are selected from the unselected images in descending order of the adjustment scores.

[0246] According to this aspect, based on the images not used in the album selected by the user, the target images can be excluded from the unselected images, and the unselected images can be automatically selected.

[0247] In an image processing device involved in another embodiment, the processor performs the following processing: displays at least a portion of the candidate image as a display candidate image on a display unit; obtains non-display selection information related to a display unselected image that is an image not selected by the user in the display candidate image; infers a user display unselected benchmark based on an analysis result of the display unselected image; and selects an automatically selected image from the unselected images that the user has not selected in the candidate images based on the user selection benchmark and the user display unselected benchmark.

[0248] According to this aspect, it is possible to improve the accuracy of automatic selection of unselected images reflecting the user's preferences or the like.

[0249] In an image processing device involved in another embodiment, the processor performs the following processing: with respect to an unselected image, a benchmark score is calculated based on the evaluation value of the assigned attribute, and an adjusted score that adjusts the benchmark score is calculated using weights corresponding to the user selection benchmark and the user displayed unselected benchmark; an automatically selected image is selected from the unselected images in descending order of the adjustment scores; when the weight corresponding to the user selection benchmark is either a positive weight or a negative weight, the weight corresponding to the user displayed unselected benchmark is the other of the positive weight or the negative weight.

[0250] According to this aspect, it is possible to automatically select the non-selected images according to the weight corresponding to the display non-selection criterion.

[0251] In an image processing device involved in another embodiment, the processor uses at least any one of information related to the subject of the user-selected image, information related to the focus level of the user-selected image, information related to the brightness level of the user-selected image, and information related to the color of the user-selected image as an attribute, and assigns attribute information to the user-selected image.

[0252] According to this method, as attribute information of the image, at least any one of information related to the subject of the selected image, information related to the focus level of the selected image, information related to the brightness level of the selected image, and information related to the chromaticity of the selected image can be applied.

[0253] In an image processing device involved in another embodiment, the processor performs the following processing: obtains first user selection information related to a first user-selected image selected by the user from candidate images as an image to be used in the album, and second user selection information related to a second user-selected image selected by the user from candidate images as an image not to be used in the album; based on the analysis results of the first user-selected image and the second user-selected image, infers a first user selection criterion corresponding to the image used in the album and a second user selection criterion corresponding to the image not to be used in the album as a user selection criterion; based on the second user selection criterion, excludes the image not to be used in the album from the unselected images, and based on the first user selection criterion, selects an automatically selected image as an image to be used in the album from the unselected images.

[0254] According to this method, it is possible to automatically select unselected images based on a first user selection criterion inferred from images that the user intends to use in the album. Furthermore, it is possible to automatically select unselected images by excluding images not used in the album from the unselected images based on a second user selection criterion inferred from images not used in the album selected by the user.

[0255] In an image processing device involved in another method, the processor performs the following processing: for unselected images, a baseline score is calculated based on the evaluation value of the assigned attribute, and an adjusted score is calculated by adjusting the baseline score using a positive weight corresponding to the first user selection benchmark and a negative weight corresponding to the second user selection benchmark; and an automatically selected image is selected from the unselected images in descending order of the adjustment scores.

[0256] According to this aspect, it is possible to automatically select unselected images based on weights corresponding to user selection criteria.

Claims

1. An image processing device comprising one or more processors, The processor performs the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; estimating the attribute whose user selection ratio, represented by the ratio of the number of the user-selected images having the same attribute to the total number of the user-selected images, is greater than or equal to a predetermined threshold value as a user selection criterion for selecting the user-selected image; An automatically selected image is selected from unselected images not selected by the user among the candidate images based on the attribute estimated as the user selection reference.

2. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: For the non-selected images, a reference score is calculated based on the evaluation value of the attribute assigned, and an adjusted score is calculated by adjusting the reference score using a weight corresponding to the user selection criterion; The automatically selected images are selected from the unselected images in descending order of the adjustment scores.

3. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: acquiring first user selection information related to a first user-selected image selected by a user from among the candidate images as an image to be used in the album; inferring a first user selection criterion corresponding to an image used in the album as the user selection criterion based on an analysis result of the first user selection image; An automatically selected image to be used in the album is selected from the unselected images based on the first user selection criterion.

4. The image processing apparatus according to claim 3, wherein: The processor preferentially selects, as the automatically selected image, an unselected image having an attribute of a relatively high first user selection ratio serving as a basis for the first user selection criterion.

5. The image processing apparatus according to claim 3 or 4, wherein: The processor performs the following processing: For the non-selected image, a reference score is calculated based on the evaluation value of the attribute assigned, and an adjusted score is calculated by adjusting the reference score using a positive weight corresponding to the first user selection criterion; The automatically selected images are selected from the unselected images in descending order of the adjustment scores.

6. The image processing apparatus according to claim 1, 3 or 4, wherein: The processor performs the following processing: acquiring second user selection information related to a second user selected image selected by the user from the candidate images as an image not used in the album; inferring, based on the analysis result of the second user-selected image, a second user-selected criterion corresponding to an image not used in the album as the user-selected criterion; Based on the second user selection criterion, images not used in the album are excluded from the unselected images, and the automatically selected image is selected from the unselected images.

7. The image processing apparatus according to claim 6, wherein: The processor preferentially excludes, as the automatically selected image, non-selected images having an attribute with a relatively high second user selection ratio serving as a basis for the second user selection criterion.

8. The image processing apparatus according to claim 6, wherein: The processor performs the following processing: For the non-selected image, a reference score is calculated based on the evaluation value of the attribute assigned, and an adjusted score is calculated by adjusting the reference score using a negative weight corresponding to the second user selection criterion; The automatically selected images are selected from the unselected images in descending order of the adjustment scores.

9. The image processing apparatus according to any one of claims 1 to 4, wherein: The processor uses at least any one of information related to the subject of the user-selected image, information related to the focus level of the user-selected image, information related to the brightness level of the user-selected image, and information related to the color of the user-selected image as an attribute, and assigns attribute information to the user-selected image.

10. An image processing device comprising one or more processors, The processor performs the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; inferring a user selection criterion serving as a criterion for selecting the user selected image based on a user selection ratio represented by a ratio of the number of the user selected images having the same attribute to the total number of the user selected images; selecting an automatically selected image from among the candidate images an unselected image that is not selected by the user according to the user selection criterion; displaying at least a portion of the candidate images as a display candidate image on a display unit; acquiring non-display selection information related to a display non-selected image that is an image not selected by a user among the display candidate images; inferring a user display non-selection criterion based on the analysis result of the display non-selected image; An automatically selected image is selected from the non-selected images not selected by the user among the candidate images based on the user selection criterion and the user display non-selection criterion. The image processing apparatus according to claim 10 , wherein: The processor performs the following processing: For the non-selected image, a reference score is calculated based on the evaluation value of the attribute assigned, and an adjusted score is calculated by adjusting the reference score using a weight corresponding to the user selection criterion and a weight corresponding to the user display non-selection criterion; selecting the automatically selected image from the unselected images in descending order of the adjustment scores; as well as When the weight corresponding to the user selection criterion is either a positive weight or a negative weight, the weight corresponding to the user display non-selection criterion is the other of the positive weight and the negative weight.

12. An image processing method, comprising: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; estimating the attribute whose user selection ratio, represented by the ratio of the number of the user-selected images having the same attribute to the total number of the user-selected images, is greater than or equal to a predetermined threshold value as a user selection criterion for selecting the user-selected image; An automatically selected image is selected from unselected images not selected by the user among the candidate images based on the attribute estimated as the user selection reference.

13. An image processing method, comprising: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; inferring a user selection criterion serving as a criterion for selecting the user selected image based on a user selection ratio represented by a ratio of the number of the user selected images having the same attribute to the total number of the user selected images; selecting an automatically selected image from among the candidate images an unselected image that is not selected by the user according to the user selection criterion; displaying at least a portion of the candidate images as a display candidate image on a display unit; acquiring non-display selection information related to a display non-selected image that is an image not selected by a user among the display candidate images; inferring a user display non-selection criterion based on the analysis result of the display non-selected image; An automatically selected image is selected from the non-selected images not selected by the user among the candidate images based on the user selection criterion and the user display non-selection criterion.

14. A recording medium which is a non-volatile recording medium readable by a computer, wherein a program is recorded on the recording medium, the program causing the computer to implement the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; estimating the attribute whose user selection ratio, represented by the ratio of the number of the user-selected images having the same attribute to the total number of the user-selected images, is greater than or equal to a predetermined threshold value as a user selection criterion for selecting the user-selected image; and An automatically selected image is selected from unselected images not selected by the user among the candidate images based on the attribute estimated as the user selection reference.

15. A recording medium which is a non-volatile recording medium readable by a computer, wherein a program is recorded on the recording medium, the program causing the computer to implement the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; inferring a user selection criterion serving as a criterion for selecting the user selected image based on a user selection ratio represented by a ratio of the number of the user selected images having the same attribute to the total number of the user selected images; selecting an automatically selected image from among the candidate images an unselected image that is not selected by the user according to the user selection criterion; displaying at least a portion of the candidate images as a display candidate image on a display unit; acquiring non-display selection information related to a display non-selected image that is an image not selected by a user among the display candidate images; inferring a user display non-selection criterion based on the analysis result of the display non-selected image; An automatically selected image is selected from the non-selected images not selected by the user among the candidate images based on the user selection criterion and the user display non-selection criterion.

16. A computer program product comprising a program for causing a computer to implement the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; estimating the attribute whose user selection ratio, represented by the ratio of the number of the user-selected images having the same attribute to the total number of the user-selected images, is greater than or equal to a predetermined threshold value as a user selection criterion for selecting the user-selected image; and An automatically selected image is selected from unselected images not selected by the user among the candidate images based on the attribute estimated as the user selection reference.

17. A computer program product comprising a program for causing a computer to implement the following processing: Acquire multiple candidate images; acquiring user selection information related to a user-selected image selected by a user from the acquired plurality of candidate images; assigning attributes to the user-selected image according to the analysis result of the user-selected image; inferring a user selection criterion serving as a criterion for selecting the user selected image based on a user selection ratio represented by a ratio of the number of the user selected images having the same attribute to the total number of the user selected images; selecting an automatically selected image from among the candidate images an unselected image that is not selected by the user according to the user selection criterion; displaying at least a portion of the candidate images as a display candidate image on a display unit; acquiring non-display selection information related to a display non-selected image that is an image not selected by a user among the display candidate images; inferring a user display non-selection criterion based on the analysis result of the display non-selected image; as well as An automatically selected image is selected from the non-selected images not selected by the user among the candidate images based on the user selection criterion and the user display non-selection criterion.

Images