Image processing apparatus, image processing method, program, and recording medium

Abstract

To provide an image processing apparatus that can appropriately select an image for product creation from an image group provided from a user for every period, a method, a program, and a recording medium.SOLUTION: In an image processing apparatus of the present invention, a processor acquires a first image group photographed in a first period, acquires a second image group photographed in a second period different from the first period, specifies characteristics related to items having to do with the photographing or the details of the image, for a first image selected from the first image group and a second image included in the second image group, specifies the tendency of the characteristics of the selected first image for the items, and on the basis of the characteristics of the second image included in the second image group, selects the second image having the characteristics falling outside the tendency for the items from the second image group.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium on which the program is recorded for selecting an image for creating a commercial material from a group of images.

Background Art

[0002] When using a service that provides commercial materials such as photo books, a user, who is the user of the service, provides a plurality of images (image group) used as materials for the commercial materials. That is, the commercial material is created using an image selected from the image group provided by the user.

[0003] In the above service, it is conceivable to create a commercial material every predetermined period (for example, every month or year). In this case, the process of selecting a material image from the image group provided by the user and the process of creating a commercial material using the selected image are repeatedly executed every certain period. Thereby, the user can obtain an image group every period.

[0004] By the way, when selecting an image for creating a commercial material, that is, a material image, from the image group provided by the user, for example, it is desirable to select an image important to the user or an image that conforms to the user's preference. As a technique for selecting a material image from the image group provided every period, the technique described in Patent Document 1 can be cited.

[0005] The image processing apparatus described in Patent Document 1 (hereinafter, the conventional image processing apparatus) sets a criterion for image selection based on a first image selected from a first image group and a second image selected from a second image group. Then, when selecting an image from the input target image group, the conventional image processing apparatus selects an image based on the above criterion. With such a configuration, the conventional image processing apparatus can select a preferable image for the user from the target image group while maintaining the commonality with the first image and the second image selected in the past.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-46901 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Conventional image processing devices typically use a set of criteria for selecting source images, which involves selecting images that share a common theme (feature) with multiple first images and multiple second images. According to this criterion, for example, if both the first and second images contain a child, the image containing the child will be selected from the group of images as the image that shares a common theme with the first and second images.

[0008] On the other hand, when selecting stock images from a set of images provided for each period, if only images with a common theme are selected, the images included in each product will lack variety. As a result, even if products are provided for each period, each product will become monotonous, meaning it will lack variety.

[0009] The present invention aims to provide an image processing device, method, program, and recording medium that can appropriately select images for creating commercial materials from a group of images provided by a user at regular intervals. [Means for solving the problem]

[0010] To achieve the above objective, the present invention provides an image processing apparatus equipped with a processor, wherein the processor performs: a first acquisition process for acquiring a first group of images taken in a first period; a second acquisition process for acquiring a second group of images taken in a second period different from the first period; a feature identification process for identifying features related to two or more items concerning the first image used for creating the product and the second image included in the second group of images from the first group of images; a trend identification process for identifying the trend of the features of the first image used for creating the product with respect to two or more items; and a selection process for selecting a second image for creating the product from the second group of images based on the features of the second image included in the second group of images, wherein the selection process selects a second image for creating the product that has features that deviate from the trend for some of the two or more items.

[0011] Furthermore, in the trend identification process, the processor may obtain a first trend value that quantifies the trend, and in the selection process, the processor may select a second image for creating the product based on the first trend value. Furthermore, in the selection process, the processor may select a second image for product creation such that the difference between the second trend value, which quantifies the trend of features of the second image for product creation, and the first trend value is greater than or equal to a set value.

[0012] Furthermore, two or more items may include priority items set according to the trend and non-priority items other than the priority items. In this case, during the selection process, the processor may select a second image for creating the product that has characteristics consistent with the trend for the priority items and characteristics that deviate from the trend for the non-priority items. Furthermore, in the trend identification process, the processor may identify trends by dividing the first image used to create the product into multiple categories based on the characteristics of the first image used to create the product for each of two or more items, and by calculating the first ratio of the first image belonging to each category. Alternatively, the processor may divide the second image included in the second image group into multiple categories based on the characteristics of the second image included in the second image group for each of two or more items, and calculate the second ratio of the second image belonging to each category. Furthermore, in the selection process, the processor may select, for non-priority items, a second image belonging to a category where the second ratio is greater than the first ratio as the second image for creating the product.

[0013] Furthermore, in the feature identification process, the processor may further identify features of two or more items for the first image included in the first image group. In this case, in the trend identification process, the processor may identify, for two or more items, the trend of the features of the first image used for creating the product, and other trends that are trends of the features of the first image included in the first image group. Furthermore, in the selection process, the processor may select a second image for creating the product based on the trend and other trends.

[0014] Furthermore, the processor may perform a first creation process for each of several periods, in which it creates a first product for a single period using a set of images taken during that period. Furthermore, the first creation process may include a layout determination process that determines the image layout in the first product based on the user's layout editing operations. In this case, the processor may, in the first creation process for creating the first product for the second period, propose an image layout for the first product for the first period to the user, and if the user adopts the proposed layout, create the first product for the second period using the proposed layout. Furthermore, the first creation process may include a material selection process for selecting material images to be used in creating the first product for the period described above from a group of images taken during that period. In this case, after executing the first creation process to create the first product for the period described above, the processor may execute a re-creation process to recreate the first product for the period described above if the re-creation conditions are met. The re-creation process may include a re-selection process for selecting material images to be used in recreating the first product for the period described above from a group of images taken during that period described above.

[0015] Furthermore, the recreation process may be performed in a period later than the one period mentioned above. In this case, during the re-selection process, the processor may select source images from the image group taken in the one period mentioned above, based on the trend of the characteristics of the image group taken in two or more periods, including the one period mentioned above and a period later than the one period mentioned above. Furthermore, the processor may accept a user's save request for the first product. In this case, after executing the first creation process to create the first product for the aforementioned period, if the processor receives a save request for the first product for the aforementioned period, it is preferable not to execute the recreation process to recreate the first product for the aforementioned period. On the other hand, if the processor has not received a save request for the first product for the aforementioned period, it is preferable to execute the recreation process to recreate the first product for the aforementioned period. Furthermore, the processor may accept user orders for the first product. In this case, after executing the first creation process to create the first product for the aforementioned period, if the processor has received an order for the first product for the aforementioned period, it is preferable not to execute the recreation process to recreate the first product for the aforementioned period. On the other hand, if the processor has not received an order for the first product for the aforementioned period, it is preferable to execute the recreation process to recreate the first product for the aforementioned period.

[0016] Furthermore, the processor may perform a second creation process to create second products for multiple periods using the source images used for each of the first products created for each period. In this case, the second product may be composed of multiple second product components. In the second creation process, the processor may create each of the multiple second product components using the source images used to create the first product for the period corresponding to each second product component among the multiple periods. The processor may also accept user orders for the first product. In this case, during the second creation process, the processor may set the layout of the source images for each of the multiple second product components according to whether or not there are orders for the first product for each second product component and the period corresponding to it.

[0017] Furthermore, in order to solve the aforementioned problems, the present invention provides an image processing method by a processor, wherein the processor performs: a first acquisition process to acquire a first group of images taken in a first period; a second acquisition process to acquire a second group of images taken in a second period different from the first period; a feature identification process to identify features relating to two or more items related to the shooting of each of the first image used for creating the product and the second image included in the second group of images from the first group of images; a trend identification process to identify the trend of the features of the first image used for creating the product for two or more items; and a selection process to select a second image for creating the product from the second group of images based on the features of the second image included in the second group of images, wherein the selection process selects a second image for creating the product that has features that deviate from the trend for some of the two or more items.

[0018] Furthermore, according to the present invention, a program for causing a computer to execute each of the processes included in the above-described image processing method can be realized. Moreover, according to the present invention, a computer-readable recording medium can be realized, on which a program for causing a computer to execute each of the processes included in the image processing method of the present invention is recorded. [Effects of the Invention]

[0019] According to the present invention, an image for creating a commercial product can be appropriately selected from an image group provided by a user for each period. Thereby, it is possible to avoid the commercial products created for each period from becoming monotonous.

Brief Description of Drawings

[0020] [Figure 1] It is a diagram showing an example of a first commercial product. [Figure 2] It is a diagram showing an example of a second commercial product. [Figure 3] It is a diagram showing a commercial product providing system including an image processing apparatus according to an embodiment of the present invention. [Figure 4] It is a diagram showing an example of a layout editing screen. [Figure 5] It is a diagram showing an example of a pattern of an image layout. [Figure 6] It is an explanatory diagram of the functions of an image processing apparatus according to an embodiment of the present invention. [Figure 7] It is a diagram showing a basic processing flow of an image processing apparatus according to an embodiment of the present invention. <00所0102>It is a diagram showing the flow of the first creation process. [Figure 9] It is a diagram showing the flow of the material selection process. [Figure 10] It is an explanatory diagram of an image included in an image group for a certain month. [Figure 11] It is a diagram showing the procedure of the material selection process for the month after the first month. [Figure 12] It is an explanatory diagram of each of an image group for a certain month, a material image for a certain month, and an image group for the next month. [Figure 13] It is a diagram showing a first tendency value and a second tendency value. [Figure 14] It is a diagram showing the flow of the re-creation process. [Figure 15] It is a diagram showing the flow of the second creation process. [Figure 16A] It is a diagram showing the image size on the page corresponding to the order month in the second commercial product. [Figure 16B] This diagram shows the image size on the page corresponding to the month in which no orders were placed for the second product. [Modes for carrying out the invention]

[0021] Specific embodiments of the present invention (hereinafter referred to as "these embodiments") will be described with reference to the drawings. However, the embodiments described below are merely examples given to facilitate understanding of the present invention and do not limit it. Furthermore, the present invention may be modified or improved from the following embodiments without departing from its spirit. The present invention also includes equivalents thereof.

[0022] In this specification, "image" refers to image data unless otherwise specified. Examples of image data include lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, and lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format.

[0023] In this specification, the concept of "device" includes both a single device that performs a specific function on its own, and multiple devices that exist independently and in a distributed manner while cooperating (linking) to perform a specific function.

[0024] <<Regarding the product provision service of this embodiment>> Before describing the image processing apparatus and image processing method according to this embodiment, we will first describe the commercial product provision service that is implemented using them.

[0025] The product provision service is a service that creates products using multiple images provided by the service user and provides those products to the user. The products are, for example, photo books or albums created using images provided by the user. The type of product is not particularly limited and may include products other than those mentioned above, such as calendars or collage prints created by combining images provided by the user with decorative images. Furthermore, the method of providing the products is not particularly limited and may include providing the user with data for displaying the products on a display, or providing the products to the user in the form of printed materials.

[0026] Creating merchandise typically requires multiple images. Therefore, users provide images taken with photographic equipment such as digital cameras, either on an ad-hoc basis or periodically. In other words, when users obtain merchandise through a merchandise provision service, they provide a group of images containing multiple images.

[0027] To briefly explain the procedure for creating a product, first, one or more images are selected from the image set provided by the user to be used as images for creating the product (hereinafter also referred to as source images). Next, the product is created using the selected source images. In this process, the layout of the source images in the product is determined. The image layout in the product refers to the number of images used in the product, the placement of the images, and the size of the images.

[0028] Products created using source images are provided to users in response to user orders, etc. In this embodiment, two types of products can be provided. One product is a first product that is provided regularly and repeatedly over multiple periods. The first product is an edited image created by editing one or more source images, and more specifically, it is a photobook-type product shown in Figure 1 (indicated by the symbol F1 in the figure).

[0029] In this embodiment, the first product F1 is created at regular intervals, specifically, monthly. For example, if the product provision service is used for one year, a first product F1 is created for each month, meaning that 12 months' worth of first products F1 are provided to the user over the course of a year. This allows the user to organize the images taken each month on a monthly basis and to preserve important images in the form of products. Furthermore, since the first product F1 can be provided at a lower cost than the second product F2 described later, it is a product that is easy for users to use (easy to purchase).

[0030] Here, the multiple months included in the service usage period correspond to the "multiple periods" of the present invention, and each month (i.e., one month) corresponds to "one period". For convenience, below, the first product F1 created for a given month will be referred to as "the first product for a given month".

[0031] The first product F1 for each month is created using source images selected from a group of images taken during that month. In other words, when creating the first product F1 for each month, the user provides a group of images they took during that month. For example, when creating the first product for April, the user provides a group of images taken in April. The date each image in the group was taken can be identified from accompanying image information such as Exif (Exchangeable image file format) tag information. For convenience, in the following discussion, a group of images taken by a user in a particular month will be referred to as "a group of images from a particular month."

[0032] As described above, in this embodiment, the process of creating the first product F1 is executed monthly. Specifically, the process of acquiring a group of images provided by the user, selecting source images from the group of images, and determining the layout of the source images in the first product are repeated monthly. This makes the first product F1 available on a monthly basis. During the period of using the product provision service, the user can check the first product F1 for the current month each month and order the first product they wish to purchase. The first product for the month in which it is ordered (the month of the order) is provided to the user.

[0033] Furthermore, users can submit a monthly save request for the first product of each month. The first product for which a save request has been made, specifically the source images used to create the first product to be saved and the image layout of the first product to be saved, will be saved as a data file to a designated storage location.

[0034] The other product is a second product containing 12 months' worth of images provided by users. More specifically, the second product is a composite image created using the source images used in each of the 12 months of the first product, and is a yearly album type product as shown in Figure 2 (indicated as F2 in the figure).

[0035] In this embodiment, the second product F2 is created every 12 months, or on a yearly basis, and is composed of multiple pages, as shown in Figure 2. The multiple pages included in the second product F2 correspond to multiple components of the second product. Furthermore, each page of the second product F2 is composed of the material images used to create the first product for the month corresponding to that page. For example, in the second product F2, the page corresponding to April contains the material images used to create the first product for April. As described above, by providing Product F2, which contains 12 months' worth of images used to create Product 1 for each month, users can store important images they have taken throughout the year as an annual album.

[0036] Product 2, F2, is created after a certain period (specifically 12 months), for example, in the final month of a year, or more precisely, after Product 1, which is created in the final month, has been created. Product 2, F2, created from the material images of Product 1 for 12 months, is provided to users who order Product 2, F2.

[0037] <<Configuration of the image processing device according to this embodiment>> The product provision service described above is realized by the product provision system S shown in Figure 3. The product provision system S consists of an image processing device (hereinafter referred to as the image processing device 10) according to this embodiment and a user-side device 12 that can communicate with the image processing device 10 via a network N.

[0038] (User-side equipment) The user-side device 12 consists of, for example, a PC (Personal Computer), a communication terminal, or a camera with communication capabilities used by the user to access the product provision service. The display of the user-side device 12 can display images provided to the image processing device 10, and information based on data provided by the image processing device 10. The information based on data provided by the image processing device 10 includes image information of the first product F1 or the second product F2.

[0039] Furthermore, a predetermined application program (hereinafter referred to as the "service app") is installed on the user-side device 12. The user can use the product provision service by launching the service app. Specifically, the user can perform various operations for using the service through the user-side device 12 while the service app is running. Operations for using the service include, for example, service application operations, image input operations, layout editing operations, order operations, and save request operations.

[0040] The procedures for each operation and the GUI (Graphical User Interface) such as the operation screen are not particularly limited. For example, the layout editing screen shown in Figure 4 can be used as a GUI for layout editing operations.

[0041] The layout editing screen is displayed on the user's device 12, and the editing operation screen displays the images selected as images for creating the product (i.e., source images). Through this screen, the user can perform layout editing operations on the first product. For example, the user can click on each source image on the screen and drag that source image to the desired position on the screen. In this way, the user can set and change the position or size of each source image in the first product, i.e., the image layout.

[0042] Regarding the image layout for the first product, there is an initial setting (default), and the user performs a layout editing operation if they want to change the image layout from the initial setting. In addition, several patterns of image layout may be provided in advance, as shown in Figure 5. In this case, the user may select one of the above multiple patterns when performing a layout editing operation. Figure 5 illustrates four patterns represented by #1 to #4, and the placement position and size of the material images in the first product are indicated by dashed lines in each pattern.

[0043] Furthermore, during layout editing operations, the user may select the material image to be actually used in creating the product from among the candidate material images selected by the image processing device 10.

[0044] (Image processing device) The image processing device 10 is comprised of a computer, such as a server computer. The image processing device 10 performs a series of data processing related to the commercial product provision service. The number of computers comprising the image processing device 10 may be one or two or more. The image processing device 10 is realized by a processor and a program that the processor can execute, and is comprised of, for example, a general-purpose computer. For example, as shown in Figure 3, the computer comprising the image processing device 10 includes a processor 10A, memory 10B, and a communication interface 10C, etc.

[0045] The processor 10A is composed of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a TPU (Tensor Processing Unit). Memory 10B is composed of semiconductor memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The communication interface 10C is configured, for example, by a network interface card or a communication interface board.

[0046] A printer 10D may be connected to the image processing device 10 as an output device. The printer 10D receives print data of the product created by the image processing device 10 and operates according to the print data to print (output) the product.

[0047] The computer comprising the image processing device 10 has a program installed for executing a series of processes related to the commercial product provision service (hereinafter referred to as the commercial product provision program). The commercial product provision program is a program that causes the computer to perform each step included in the image processing method of the present invention. That is, when the processor 10A reads and executes the commercial product provision program, the computer equipped with the processor 10A functions as the image processing device of the present invention.

[0048] The product provision program may be obtained by reading it from a computer-readable storage medium, or by receiving (downloading) it via a communication network such as the Internet or an intranet.

[0049] As shown in Figure 3, the product provision system S is equipped with a storage device 14 that stores a group of images acquired by the image processing device 10 from the user-side device 12. The image group is stored in the storage device 14. The image processing device 10 is configured to be able to freely read images from the image group stored in the storage device 14. In addition, the image group stored in the storage device 14 is associated with the user who provided it and the year and month in which the images included in the image group were taken.

[0050] The storage device 14 may consist of storage built into or attached to the image processing device 10. Alternatively, the storage device 14 may consist of a third computer (e.g., an external server) that can communicate with the image processing device 10.

[0051] The configuration of the image processing device 10 will now be explained again from a functional standpoint. As shown in Figure 6, the image processing device 10 includes an image acquisition unit 21, a preliminary sorting unit 22, a material selection unit 23, an editing unit 24, a first product creation unit 25, a recommendation unit 26, an order receiving unit 27, a request receiving unit 28, a re-selection unit 29, a re-creation unit 30, and a second product creation unit 31. These functional units are realized through the cooperation of hardware equipment provided by the computer constituting the image processing device 10 and software including the aforementioned product provision program.

[0052] The image acquisition unit 21 acquires a group of images provided by the user. Specifically, the user can specify which images to provide to the image processing device 10 from among the images they have taken by operating the user-side device 12, and transmit the specified images to the image processing device 10. The image acquisition unit 21 acquires the group of images provided by the user by receiving the images transmitted from the user-side device 12. The group of images acquired by the image acquisition unit 21 is stored in the storage device 14.

[0053] Furthermore, if the user-side device 12 is a camera with communication capabilities, the image acquisition unit 21 may directly receive the images transmitted from the user-side device 12 and acquire the image group. Alternatively, if the user-side device 12 is a PC or the like equipped with image editing software that captures images from the camera, the image acquisition unit 21 may acquire the image group captured by the user-side device 12. In addition, if the image group input from the user-side device 12 is temporarily stored on a server on the network, the image acquisition unit 21 may acquire the image group from that server.

[0054] Furthermore, the image acquisition unit 21 acquires the image set for each month. In other words, the image acquisition unit 21 acquires the image set for each month on a monthly basis. The timing of acquiring the image set for each month is not particularly limited; for example, the user may input the captured image as soon as they take a picture, and the image acquisition unit 21 may acquire the input image each time an image is input. Alternatively, the user may input images in batches at regular intervals (for example, every month), in which case the image acquisition unit 21 may acquire all the images (image set) for one period, input every month, at once.

[0055] Here, a certain month during the period of use of the product provision service and a different month correspond to the first period and the second period of the present invention, respectively. In this embodiment, the second period is a period after the first period, for example, the month following the month corresponding to the first period. The image acquisition unit 21 acquires a group of images for the month corresponding to the first period, and then acquires a group of images for the month corresponding to the second period. The group of images for the month corresponding to the first period corresponds to the first image group of the present invention, and the images included in the first image group correspond to the first image. Similarly, the group of images for the month corresponding to the second period corresponds to the second image group of the present invention, and the images included in the second image group correspond to the second image.

[0056] The first and second periods are relative concepts of time that change as time progresses. For example, in the relationship between April and May, April corresponds to the first period, and May corresponds to the second period. Similarly, in the relationship between April, May, and June, April and May each correspond to the first period, and June corresponds to the second period. Furthermore, the month that falls under the first period is not limited to the month immediately preceding the month that falls under the second period; it may be any month prior to that. For example, if May is designated as the second period, February or March may be designated as the first period. Also, the month that falls under the first period may be multiple months (two or more months); for example, if May is designated as the second period, March and April may be designated as the first period. Furthermore, the second period may be earlier than the first period; for example, if May is the first period, April may be the second period.

[0057] The preliminary selection unit 22 performs a preliminary selection process to remove images that are unsuitable as source images (hereinafter referred to as "images to be excluded") from the image acquisition unit 21. Whether or not an image is to be excluded is determined based on the presence or absence of a human face in the image, the facial expression, the size of the face in the image, the degree of blur and blur in the image, and the color and brightness of the subject.

[0058] The material selection department 23, the editing department 24, the first product creation department 25, and the recommendation department 26 execute the process of creating the first product (hereinafter referred to as the first creation process). The first creation process is the process of creating the first product for each month, and is executed monthly, for example, on a predetermined date in each month.

[0059] The material selection unit 23 performs a process to select images to be used in the creation of the first product, i.e., material images (hereinafter referred to as the material selection process). The material selection process is a process included in the first creation process and is executed monthly. In the material selection process of each month, the material selection unit 23 selects material images from the group of images acquired by the image acquisition unit 21 during that month, that is, from the group of images taken during that month.

[0060] The material selection process will be explained in detail using the example of creating the first product for a certain month (hereinafter referred to as Month P). In the material selection process for Month P, material images to be used to create the first product for Month P are selected from the image set for Month P, specifically from the image set for Month P after the exclusion target images selected by the preliminary selection unit 22 have been removed.

[0061] In the material selection process, the material selection unit 23 first identifies characteristics of two or more items for each image included in the image group for month P. The two or more items are two or more items (themes) related to the shooting of the image. Examples of items include the subject person, the proportion of the subject person in the image, the parts of the person visible in the image, other subjects appearing with the person, the shooting scene, the shooting location, the shooting time, and the degree of perspective in the image.

[0062] The characteristics are the specific content and nature of the image related to the item. Examples of characteristics related to the above items include the following: • Subject(s): Relationship to the person(s) (child, relative, friend, etc.), and the number of people in the image, etc. • Percentage of the subject in the image: The percentage of the image occupied by the subject (%) • Other subjects in photos with the person: type of subject, size of subject, etc. • Filming Scene: Classification of the filming scene, and the content of the event shown in the image. • Filming location: Place name, building name, and whether the filming location is indoors or outdoors, etc. • Shooting period: Time of day (morning, noon, evening, or night), and season in which the photo was taken, etc. • Perspective: Is it a foreground or a background?

[0063] The method for identifying the characteristics of an item is not particularly limited, but for example, it may be automatically identified using known image analysis techniques. Specifically, known face detection techniques may be applied to detect the faces of people in an image, and the detected faces may be compared with pre-registered face images to identify the person whose face was detected.

[0064] Alternatively, known object detection techniques may be applied to detect subjects in an image, and then an estimation engine built using machine learning may be used to estimate the type of detected subject. Or, when analyzing a set of images provided by the user, if a predetermined number of images of the same person exist, it may be inferred that the person is important to the user (family or friend, etc.). Furthermore, the relationship between the person in the image and the user may be inferred from the location, time, and scene of the image.

[0065] Alternatively, information regarding the characteristics of the target image may be input by the operator of the image processing device 10 or the user providing the image, and the characteristics of the target image may be identified based on the input information.

[0066] When the material selection unit 23 selects material images to be used to create the first product for month P from the image group for month P, it selects material images according to the characteristics of the images included in the image group for month P.

[0067] Furthermore, the material selection unit 23 selects material images from the image group for Q month to be used in creating the first product for a month after P month, for example, Q month which is the month following P month. At this time, the material selection unit 23 selects the material images to be used in creating the first product for Q month according to the characteristics of the images included in the image group for Q month and the characteristics of the material images used in creating the first product for P month.

[0068] Hereinafter, the source images used to create the first product in month P will be referred to as "source images for month P," and similarly, the source images used to create the first product in month Q will be referred to as "source images for month Q." Here, months P and Q are related as the first and second periods mentioned above, with the image group for month P corresponding to the first image group mentioned above, and the image group for month Q corresponding to the second image group mentioned above.

[0069] Furthermore, in this embodiment, the material selection unit 23 can select the material image for month Q such that there is a difference in features between the material image for month P and the material image for month Q.

[0070] The editorial unit 24 receives layout editing operations from the user, and more specifically, receives data indicating the content of the layout editing operations from the user's device 12. The editorial unit 24 is configured to receive layout editing operations each time the first creation process is executed, in other words, every month. In this embodiment, there is an initial layout setting (default layout), and the user performs layout editing operations if they wish to use a layout other than the initial setting. Layout editing operations may include selecting images to be actually used in the creation of the first product from among the material images selected by the material selection unit 23.

[0071] The first product creation unit 25 creates the first product for each month based on the material images selected by the material selection unit 23 and the layout editing operations received by the editing unit 24. Specifically, the first product creation unit 25 executes a process to determine the layout of the material images in the first product (hereinafter referred to as the layout determination process) based on the layout editing operations. In the layout determination process, the size and placement position of the material images in the first product are determined.

[0072] Then, once the image layout for the first product of the month is determined, the first product F1 for the month is completed. The first product creation unit 25 generates a data file for the first product upon its completion. The generated data file for the first product is transmitted to the user-side device 12. The user-side device 12 displays an image of the first product based on the received data file. This allows the user to check the first product F1 for each month.

[0073] The recommendation unit 26 recommends (suggests) a recommended layout to the user during the first creation process. The recommended layout is set to reflect the image layout of the first product created in the previous month or earlier. For example, suppose the image layout of the first product in month P is changed from the default layout based on the user's layout editing operation. In this case, the recommendation unit 26 recommends the image layout of the first product in month P as the recommended layout to the user during the first creation process to create the first product in month Q, which is the month following month P. To explain with a specific example, suppose the user selects pattern #1 in Figure 5 when creating the first product in April. In this case, the recommendation unit 26 suggests pattern #1 as the recommended layout to the user when creating the first product in May.

[0074] The order receiving unit 27 is configured to receive user orders for the first product, and more specifically, when a user places an order using the user-side device 12, it receives data indicating the order details from the user-side device 12. In this embodiment, since the first product is created monthly, the order receiving unit 27 can receive orders monthly. When the order receiving unit 27 receives an order, it executes a process (for example, printing) to provide the ordered first product to the user. Furthermore, for the ordered first product F1, the material images constituting the first product and the image layout in that first product are determined at the time of order acceptance.

[0075] The request receiving unit 28 is configured to receive user save requests for the first product. More specifically, when a user performs a save request operation on the user-side device 12, the request receiving unit 28 receives data indicating the request details from the user-side device 12. In this embodiment, since the first product is created monthly, the request receiving unit 28 can receive save requests monthly. When the request receiving unit 28 receives a save request, the data file of the first product for which the save request was made is stored in the storage device 14, linked to the user who made the save request.

[0076] The re-selection unit 29 and the re-creation unit 30 execute a re-creation process to recreate the first product for the target month after the first creation process, which creates the first product for the target month, has been executed. The target month is the month to which the re-creation process applies, that is, the month in which the already created first product is recreated. The re-creation process is executed when predetermined conditions (hereinafter referred to as re-creation conditions) are met.

[0077] In this embodiment, if no order or save request has been received for the first product for the target month, a recreation process is executed to recreate the first product for the target month. In other words, if an order or save request has been received for the first product for the target month, the recreation process to recreate the first product for the target month is not executed. That is, whether or not the recreation process is necessary is determined based on the user's intention regarding the first product, specifically whether or not an order or save request has been made. Then, the first product for which an order or save request has been made is finalized, and the unfinished first product can be recreated.

[0078] The re-creation process for the first product of a target month is executed in a month following the target month. For example, if the target month is April, the re-creation process for the first product of that month (April) will be executed in May or later. Furthermore, unless an order or save request is received for the first product of the target month (including the recreated version), the re-creation process for the first product of the target month will be repeatedly executed in each month following the target month.

[0079] The re-selection unit 29 performs a re-selection process to select source images to be used to recreate the first product for the target month. In the re-selection process, the re-selection unit 29 re-selects source images from a group of images taken during a period of two months or more including the target month and months after the target month (hereinafter referred to as the re-selection period). At this time, the re-selection unit 29 selects source images based on the characteristic trends of the group of images taken during the re-selection period.

[0080] The re-creation unit 30 recreates the first product for the target month using the material images selected by the re-selection unit 29. The procedure for the re-creation unit 30 to recreate the first product is the same as the procedure for the first product creation unit 25 to create the first product.

[0081] The second product creation unit 31 executes a second creation process to create the annual second product F2 using the material images used to create each of the 12 months of the first product. In this embodiment, the second creation process for the year is executed after the first creation process, which creates the first product for the last month of the year, is completed.

[0082] In the second creation process, the second product creation unit 31 creates each of the multiple pages included in the second product F2 using the material images used to create the first product F1 for the month corresponding to each page. At this time, the second product creation unit 31 sets the layout of the material images on each page according to whether or not there is an order for the first product F1 for the month corresponding to each page. Specifically, the second product creation unit 31 sets the image layout on each page so that the material images of the first product for the month in which an order was placed take precedence over the material images of the first product for months in which no order was placed.

[0083] Then, once the image layout for each page is determined, the second product F2 is completed. Upon completion of the second product, the second product creation unit 31 generates a data file for that second product. The generated data file is sent to the user-side device 12. The user-side device 12 displays an image of the second product based on the received data file. This allows the user to confirm the second product F2.

[0084] <<Basic Processing Flow of Image Processing Devices>> The basic processing flow of the image processing device 10 according to this embodiment, which is the product provision flow, will be explained with reference to Figure 7.

[0085] The product provision flow is executed on a one-year cycle during the period of use of the product provision service. Each step in the product provision flow is executed by the processor 10A of the computer that constitutes the image processing device 10. In other words, the product provision flow progresses as the processor 10A reads the aforementioned product provision program and executes each step.

[0086] In the product delivery flow, processor 10A performs an image acquisition process each month to acquire a set of images taken by the user each month (S101). In the image acquisition process, processor 10A acquires a set of images taken by the user each month by receiving images input from the user-side device 12. Image input from the user may be performed each time the user takes an image. Alternatively, the set of images taken in each month may be provided all at once on a predetermined day of each month.

[0087] Subsequently, on a predetermined day in each month, the processor 10A executes a first creation process to create the first product F1 for that month (S102). In the first creation process, the processor 10A executes a material selection process to select material images from the image group acquired in step S101, and a layout determination process to determine the layout of the selected material images.

[0088] Furthermore, if there is a month in the previous month or earlier that satisfies the recreation conditions (S103), the processor 10A executes the recreation process with that month as the target month (S104). In the recreation process, the processor 10A executes a re-selection process to select the source image to be used to recreate the first product of the target month from the image group of the target month.

[0089] The above series of steps S101 to S104 are repeated every month until the final month arrives. Of the multiple image acquisition processes performed each month, the image acquisition process performed in an earlier month (first period) corresponds to the first acquisition process, and the image acquisition process performed in a later month (second period) corresponds to the second acquisition process. For example, in the relationship between April and May, the image acquisition process performed in April corresponds to the first acquisition process, and the image acquisition process performed in May corresponds to the second acquisition process.

[0090] When the final month arrives (S105), processor 10A creates the first product for the final month and then executes a second creation process to create the second product F2 as an annual album (S106). Once the second creation process is completed, the product provision flow for that year is finished.

[0091] <<Regarding the first creation process>> Next, the flow of the first creation process in the product provision flow will be explained with reference to Figure 8. The image processing method of the present invention is employed in the first creation process. That is, each step in the first creation process described below corresponds to a component of the image processing method of the present invention. The first creation process flow described below is merely an example, and unnecessary steps may be deleted, new steps added, or the execution order of steps rearranged, without departing from the spirit of the present invention.

[0092] Each step in the first creation process is executed by the processor 10A of the computer that constitutes the image processing device 10. In the following explanation, the first creation process for creating the first product F1 for month P or month Q will be used as an example.

[0093] When executing the first creation process to create the first product for month P, processor 10A performs an image acquisition process to obtain a set of images for month P from the user. Processor 10A also performs a preliminary selection process on the acquired set of images for month P (S001). Through the execution of the preliminary selection process, images unsuitable for use as source images for product creation are excluded from the set of images for month P. Images to be excluded are, for example, images that are blurry or out of focus, and images in which the brightness of the subject is not sufficiently ensured, or other images that do not meet a certain level of image quality.

[0094] When processor 10A starts the first creation process to create the first product for month P, it executes a material selection process (S002). In the material selection process, processor 10A selects material images (material images for month P) to be used to create the first product for month P from the group of images for month P from which the images to be excluded have been removed. The detailed procedure of the material selection process will be explained in detail later.

[0095] Next, processor 10A executes a layout determination process to determine the layout of the material images in the first product for month P (S003). Specifically, for each of the material images for month P selected in the material selection process, processor 10A determines the placement position and size in the first product for month P. Furthermore, if a user performs a layout editing operation on the first product for month P, processor 10A accepts that operation. In that case, processor 10A determines the image layout according to the accepted layout editing operation.

[0096] After the layout determination process, the processor 10A creates the first product for month P based on the material images selected in the material selection process and the image layout determined in the layout determination process (S004). Upon completion of the first product for month P, the processor 10A generates a data file for that product and transmits it to the user-side device 12. The user-side device 12 displays an image of the first product for month P on its display. The user reviews the first product for month P and considers whether or not to place an order (S005).

[0097] When a user operates the user-side device 12 to place an order for the first product of month P, the processor 10A accepts the order (S006) and executes processing to provide the first product of month P to the user (for example, printing) (S007). In addition, upon acceptance of the order, the processor 10A confirms the first product of month P (specifically, the source images and image layout used to create the first product of month P).

[0098] Furthermore, the user considers whether or not to save the data file for the first product in month P (S008). When the user requests saving by operating the user-side device 12, the processor 10A accepts the saving request (S009) and saves the data file for the first product in month P to the storage device 14 (S010).

[0099] Once the series of steps described above are completed, the first creation process for creating the first product for month P is finished. For month Q, which is the month following month P, the first creation process is executed following the same procedure as described above.

[0100] Assume that the image layout for the first product in month P is determined by the user's layout editing operations. In this case, during the first creation process for creating the first product in month Q, processor 10A proposes the image layout for the first product in month P as a recommended layout to the user. The user decides whether or not to adopt the proposed recommended layout and performs operations on the user-side device 12 according to that decision. By proposing a recommended layout in this way, the system can support the user's layout editing operations. If the user adopts the recommended layout, processor 10A creates the first product in month Q based on the proposed recommended layout.

[0101] <<Regarding material selection process>> Next, the flow of the material selection process in the first creation process will be explained with reference to Figure 9. The material selection process is executed in the first creation process of each month, that is, it is executed monthly. In the material selection process of each month, the processor 10A executes the feature identification process (S021), the trend identification process (S022), and the selection process (S023), as shown in Figure 9.

[0102] The details of the three processes described above differ between the first month of the product provision service usage period and subsequent months. Below, we will explain the material selection process for the first month and the material selection process for subsequent months. In the following explanation, we will use the example where the first month is April and the next month is May.

[0103] [Material selection process for the first month] In the material selection process for April, the first month, material images to be used in creating the first product for April (April material images) are selected from the April image set. In the April material selection process, a feature identification process is first performed, in which processor 10A identifies features related to two or more items for the images included in the April image set. There are no particular limitations on the number and types of items, but the following explanation assumes that three items, "subject person," "shooting location," and "shooting date," are set.

[0104] To explain the characteristics identified for each item, for example, the characteristic of the item "Subject Person" is that the relationship of the person in the image (specifically, child, parent, or grandparent, etc.) is identified. The characteristic of the item "Shooting Location" is that the specific name of the shooting location (place name) is identified, such as home or park. The characteristic of the item "Shooting Time" is that the specific time, or time of day such as morning, afternoon, or evening, is identified. The method for identifying features may be either an automatic method using image analysis, or a method of manually entering information that describes the features.

[0105] In the trend identification process during the material selection process in April, processor 10A identifies trends in the characteristics of images included in the April image set for three items, based on the characteristics identified in the feature identification process. The trends indicate the extent to which images with certain characteristics are included in the image set.

[0106] To explain the trend identification process in more detail, in the trend identification process, processor 10A divides the images included in the April image group into multiple categories for each of the three items, based on the features identified in the feature identification process. A category is a classification (category) used when classifying images that have a certain feature. For example, for the item "shooting location," there are categories such as images taken at home and images taken in a park. It should be noted that a single image may belong to multiple categories for a given item. For example, an image of a child and their mother together would belong to both the "Images of children" and "Images of mothers" categories for the item "Subjects."

[0107] Processor 10A calculates the proportion of images belonging to each category for each item. The proportion is expressed as the percentage of images belonging to each category relative to the April image set. The proportion calculated for each category for each item corresponds to the trend of the image characteristics included in the April image set.

[0108] Figure 10 shows an example of the ratios calculated for the images included in the April image set. Regarding the April image set, as shown in Figure 10, for example, under the item "Subject," 40% of the images included children, 40% included mothers, and 40% included fathers. Also, under the item "Shooting Location," 80% of the images were taken at home, and 20% were taken in a park.

[0109] After performing feature identification and trend identification processes, processor 10A performs the selection process for April. In the April selection process, processor 10A selects the material images to be used to create the first product for April (April material images) from the group of April images from which the exclusion target images have been removed. At this time, processor 10A selects the April material images based on the trends identified in the trend identification process.

[0110] Specifically, processor 10A prioritizes selecting images from categories with high ratios for each item as source images. In the case of Figure 10, images from the category with the highest ratio for each item, such as "images of children taken at home in the morning," are preferentially selected as source images. Note that the images selected as source images may include images belonging to categories other than the one with the highest ratio, in proportion to the ratio of each category.

[0111] The stock images for April are selected based on the steps outlined above. These stock images are chosen to reflect the user's preferences and interests. In other words, the ratios calculated for each category are determined according to the user's shooting tendencies, and images in categories with high ratios are likely to be images that the user likes or values. Therefore, by selecting stock images using the above procedure, it is possible to select images that are important to the user as stock images.

[0112] [Material selection process for the month following the first month] In the material selection process for May, which is an example of a month following the first month, material images to be used in creating the first product for May (May's material images) are selected from the May image set. In the May material selection process, the May material images are selected taking into account the April material images. Here, in relation to April and May, April corresponds to the first period, and May corresponds to the second period. The April image set corresponds to the first image set, and the images included in that set correspond to the first images. In other words, the April material images correspond to the first images used in creating the product. The May image set corresponds to the second image set, and the images used in that set correspond to the second images. In other words, the May material images correspond to the second images used in creating the product.

[0113] The material selection process in May proceeds according to the flow shown in Figure 11. In the feature identification process during the material selection process in May, processor 10A identifies features related to the three items mentioned above for the material images from April and the images included in the May image group (S201, S202). The procedure and method for identifying image features are the same as those used in the feature identification process during the material selection process in April.

[0114] In the subsequent trend identification process, processor 10A identifies the characteristic trends of the source images for April for the three items mentioned above (S203). Specifically, in the trend identification process, processor 10A divides the source images for April into multiple categories based on their characteristics for each item. Then, processor 10A calculates the ratio of images (source images) belonging to each category for each category. This ratio corresponds to the first ratio and is expressed as the percentage (in units of %) of the number of images belonging to each category to the total number of source images for April. The first ratio calculated for each category for each item corresponds to the characteristic trends of the source images for April, that is, the characteristic trends of the first images used to create the product. For the purposes of this discussion, we will refer to the trends in the characteristics of stock images in April as "April trends."

[0115] Figure 12 shows an example of the first ratio calculated for the source images in April. Regarding the source images in April, as shown in Figure 12, for example, for the item "Subject Person," the first ratio for images with children was 90%, for images with mothers it was 30%, and for images with fathers it was 10%. For the item "Shooting Location," the first ratio for images taken at home was 90%, and for images taken in a park it was 10%. For the item "Shooting Time," the first ratio for images taken in the morning was 40%, for images taken in the afternoon it was 60%, and for images taken at night it was 0%.

[0116] Furthermore, in the trend identification process, processor 10A calculates a first trend value that quantifies the trend for April. Quantifying the trend includes, for example, deriving a vector that represents the trend. Specifically, when the number of items is C and the number of divisions included in each item is D, the first trend value is represented by a multidimensional vector having dimensions C × D. Each component of the multidimensional vector that makes up the first trend value corresponds to the first ratio of images belonging to the division corresponding to that component for the item that corresponds to that component.

[0117] After performing feature identification processing and trend identification processing, processor 10A performs selection processing for May. In the selection processing for May, processor 10A selects source images for May from the image group for May based on the features of the images included in the image group for May. In this embodiment, processor 10A selects an image (second image) that has features that deviate from the trend for April for some of the three items mentioned above as the source image for May.

[0118] To explain in more detail, processor 10A sets a priority item from among three items and sets the other items as non-priority items (S204). The priority item is the item that is given priority when selecting source images for May from among the three items, and is set, for example, according to the trends in April. Specifically, it identifies the category with the highest first ratio, or the category in the first category that exceeds the standard value, and the item corresponding to that category becomes the priority item. In the case shown in Figure 12, the category with the highest first ratio is the image with a child in it, so the item "Subject Person" becomes the priority item.

[0119] Furthermore, the number of priority items is not limited to one; multiple items may be set as priority items. Also, there are no particular restrictions on how priority items are determined; for example, the user may specify the priority items. Non-priority items correspond to "some items" in the present invention, and in the case shown in Figure 12, the items "shooting location" and "shooting time" are non-priority items.

[0120] Next, processor 10A divides the images included in the May image group into several categories based on their characteristics, and calculates the ratio of images belonging to each category (S205). This ratio corresponds to the second ratio and is expressed as the percentage (in units of %) of the number of images belonging to each category relative to the total number of images included in the May image group.

[0121] Figure 12 shows an example of the second ratio calculated for images included in the May image set. Regarding the May image set, as shown in Figure 12, for example, for the item "Subject Person," the second ratio for images with children was 90%, for images with mothers it was 35%, and for images with fathers it was 10%. For the item "Shooting Location," the second ratio for images taken at home was 30%, and for images taken in a park it was 70%. For the item "Shooting Time," the second ratio for images taken in the morning was 40%, for images taken in the afternoon it was 60%, and for images taken at night it was 0%.

[0122] Subsequently, the processor 10A selects source images for May based on the second ratio calculated for each category of images included in the May image group and the first trend value indicating the trend in April (S206). At this time, the processor 10A selects as source images for May images from the May image group that have features in line with the trend in April for priority items, and features that deviate from the trend in April for non-priority items.

[0123] Images that exhibit characteristics consistent with the April trends for priority items are determined according to the primary ratio calculated for each category of priority items. For example, images belonging to the category with the highest primary ratio for each priority item are selected. In the case shown in Figure 12, images featuring children are considered images that exhibit characteristics consistent with the April trends for priority items. Furthermore, regarding priority items, images that exhibit characteristics consistent with the trends in April are not limited to images belonging to the category with the highest primary ratio; for example, images belonging to any of the n categories (where n is a natural number greater than or equal to 2) from the highest primary ratio are also acceptable.

[0124] On the other hand, images with characteristics that deviate from the April trend for non-priority items are determined according to the first and second ratios calculated for each category for non-priority items. Specifically, images belonging to categories where the second ratio calculated for non-priority items is higher than the first ratio are considered images with characteristics that deviate from the April trend for non-priority items. In the case shown in Figure 12, for the non-priority item "shooting location," the second ratio for images taken in parks is 70%, which is higher than the first ratio of 10%. Therefore, images taken in parks are considered images with characteristics that deviate from the April trend for non-priority items.

[0125] Furthermore, among the images belonging to the category in which the calculated second ratio for non-priority items is greater than the first ratio, images belonging to the category in which the difference between the second ratio and the first ratio exceeds the threshold may be selected as source images. Furthermore, for non-priority items, if there are multiple categories where the second ratio is higher than the first ratio, it is advisable to calculate an evaluation value for each category for the images belonging to each category. The evaluation value is the sum of a value calculated according to the image quality, the number of people in the image, and whether or not a specific person is present in the image, and a value determined according to the second ratio. The processor 10A should prioritize the category to which the image with the higher evaluation value belongs, and select the image belonging to that category as the source image.

[0126] When selecting images with features that deviate from the April trend for non-priority items, processor 10A uses the first trend value, which represents the April trend. Specifically, processor 10A selects May source images such that the difference between the second trend value, which quantifies the features of the May source images, and the first trend value is greater than or equal to a set value.

[0127] As shown in Figure 13, the second trend value, like the first trend value, is represented by a multidimensional vector having a dimension obtained by multiplying the number of items C by the number of divisions D contained in each item. Furthermore, each component of the multidimensional vector that makes up the second trend value corresponds to the second ratio of images belonging to the division corresponding to that component for the item that corresponds to that component. The difference between the second trend value and the first trend value is the absolute value of the difference between the two vectors, as shown in Figure 13. The setting value set for the difference may be a predetermined value, or it may be set by the user, or it may be changed as appropriate.

[0128] According to the above procedure, it is possible to accurately select images from the May image set that have features that deviate from the trends of April. In other words, in this embodiment, the processor 10A quantifies the trends of April and the trends of the features of the images selected as source images for May, and performs calculations using the two quantified trends. Through this calculation, images that have features that deviate from the trends of April can be rationally and appropriately derived.

[0129] Following the procedure described above, processor 10A selects source images for May from the May image set. The images selected as source images for May take into account the source images for April; specifically, the priority items have characteristics that align with the trends of April, while the non-priority items have characteristics that deviate from the trends of April. As a result, the images used to create the first product in May (the current month) can be appropriately selected based on the images used to create the product in April (the previous month).

[0130] To explain in more detail, the characteristics of images related to priority items are likely to reflect the content (themes) that users prioritize when creating their first product. Based on this, by selecting images with characteristics that align with the trends of April for priority items as stock images for May, it is possible to choose stock images that match the user's preferences or interests.

[0131] On the other hand, if there is little change between the characteristics of the stock images in May and April, the first product in April and the first product in May will be products with similar characteristics. If this situation continues for the first products of subsequent months, even if a first product is created each month, the characteristics of the images included in the first product, i.e., the stock images, will become monotonous. As a result, the first products each month may lack variety.

[0132] Therefore, in this embodiment, images with characteristics that deviate from the trends of April for non-priority items are selected as source images for May. As a result, the characteristics of the images used to create the products (source images) will differ between the first product in May and the first product in April. Consequently, the first products created each month will have more variety, preventing them from becoming monotonous (uninteresting) for users. Due to these effects, it is expected that the number of users utilizing product provision services that offer a new first product each month will increase.

[0133] In the case shown in Figure 12, the priority item was "subject person," and the non-priority item was "shooting location." Images were selected that had characteristics consistent with the April trend for the priority item, and characteristics that deviated from the April trend for the non-priority item, specifically images of children taken in a park. However, priority and non-priority items may change depending on the characteristics of the image set provided by the user. For example, suppose the April image set contains many images of children eating strawberries, while the May image set contains no images of children eating strawberries, but instead many images of children eating melons. In this case, for example, the priority item could be set to "the subject and their actions," and the non-priority item to "objects that appear with the subject." Specifically, for the priority item, images that have characteristics consistent with the April trend should be images of children eating fruit. For the non-priority item, images that deviate from the April trend should be images where the fruit with the child is different from strawberries, such as melons. Under these conditions, it would be good to select May image sets from the May image set.

[0134] To elaborate on the selection process for May, the images selected as source images for May may include not only images that meet the above-mentioned conditions (hereinafter referred to as "main images"), but also images other than the main images (hereinafter referred to as "secondary images"). Main images are images that have characteristics consistent with the trends in April for priority items, and characteristics that deviate from the trends in April for non-priority items. Secondary images are, for example, images that do not have characteristics consistent with the trends in April for priority items. It is preferable that the proportion of secondary images among the source images for May be smaller than the proportion of main images.

[0135] Furthermore, when selecting source images for May, processor 10A performs feature identification processing and trend identification processing. In the feature identification processing, processor 10A identifies the features of the source images for April and the features of the images included in the May image group for the three items mentioned above. At this time, processor 10A may further identify the features of the images included in the April image group (first image group) for the three items mentioned above.

[0136] Furthermore, in the trend identification process, the processor 10A identifies the trend in the characteristics of the source images from April (i.e., the trend for April) for the three items mentioned above. At this time, the processor 10A may further identify the trend in the characteristics of the images included in the April image group. The trend in the characteristics of the images included in the April image group corresponds to "other trends" in this invention, and will be referred to as the trend of the April image group below.

[0137] The processor 10A may then select source images for May based on the trends in April and the trends in the April image set. In this case, the images selected as source images for May may include not only images that have features consistent with the April trends regarding priority items, but also images that have features consistent with the April image set regarding priority items.

[0138] To illustrate with the case shown in Figure 11, regarding the priority item "subject person," the April image group contains images of children, images of mothers, and images of fathers in equal proportions. Therefore, images of children, images of mothers, and images of fathers each correspond to images that have characteristics consistent with the trend of the April image group in terms of the priority item. In this case, processor 10A may include images of mothers or fathers along with images of children in the May source images.

[0139] As described above, by selecting images with characteristics consistent with the trends of the April image group for priority items as source images for May, it becomes possible to select source images for May (the current month) while taking into account the shooting trends of April (the previous month). This allows for understanding the images that are important to users based on the shooting trends of the previous month, and then selecting source images for the current month based on that.

[0140] The above explains the process for selecting stock images for May, and the procedure is the same for selecting stock images for June and subsequent months. For example, stock images for June are selected from the June image group based on the characteristic trends of the stock images for April and the characteristic trends of the stock images for May. In this case, the stock images for April and May correspond to the first images used in creating the product, the June image group corresponds to the second image group, and the stock images for June correspond to the second images used in creating the product.

[0141] Suppose the June image set includes, for example, an image of grandparents, and such an image was not included in either the April or May image sets. In this case, the image of grandparents has characteristics that deviate from both the characteristic trends of the April and May image sets, so it may be selected as a June image set.

[0142] <<Recreation Process>> If the processor 10A has executed the first creation process to create the first product for a given month and the recreation conditions are met, it will execute a process to recreate the first product, using that month as the target month. In other words, in this embodiment, even if the first product F1 is created, if the recreation conditions are met, the first product F1 can be recreated as needed. The flow of the recreation process will be explained below with reference to Figure 14.

[0143] The recreation process is executed when a month exists that satisfies the recreation conditions, i.e., the target month. For example, in the month following the target month or later, processor 10A executes the recreation process to recreate the first product for the target month. Whether the recreation conditions are met is determined by whether there is an order for the first product and whether there is a save request.

[0144] Specifically, if processor 10A has not received any orders or storage requests for the first product of a given month, it will designate that month as the target month and execute a recreation process to recreate the first product of the target month. In the recreation process, as shown in Figure 14, processor 10A first executes a re-selection process to select the source images for the target month (S031). In the re-selection process, processor 10A selects the source images to be used to recreate the first product of the target month from the group of images taken in the target month (group of images for the target month).

[0145] In the re-selection process, processor 10A identifies the characteristics of the images included in the image group taken during the re-selection period, which includes the target month, and identifies the trend of the characteristics of that image group. The procedure for identifying the characteristics and trends of images in the re-selection process is the same as the characteristic identification process and trend identification process in the material selection process in the first creation process. The re-selection period is a period that begins in a month prior to the target month and continues in a month after the target month. The month prior to the target month is, for example, the month in which the commercial material provision service is started (the first month), and the month after the target month is the month in which the recreation process is executed (specifically, the present time).

[0146] The stock images used to recreate the first product for the target month are selected considering images taken in months after the target month. In other words, when recreating the first product for the target month, stock images can be selected based on the user's shooting trends in the period after the target month. This allows for a more accurate understanding of user shooting trends over a longer period, and enables the recreation of the first product for the target month based on those trends.

[0147] To explain in more detail, the longer the re-selection period, that is, the closer it gets to the final month of the year, the more image sets the image processing device 10 acquires each month. And the more image sets there are, the more accurately the user's shooting tendencies can be identified based on the image sets of each month. For example, a person who is not present in the image set of the target month may frequently appear in the image set of subsequent months. Also, even if a person appears in only a few images in the image set of the target month, that person may appear frequently in the image set of subsequent months. Such a person may be important to the user, and this becomes clear after the target month. By referring to the image sets taken during the re-selection period, the above facts can be easily recognized. As a result, it is possible to select more accurately the source images for recreating the first product for the target month.

[0148] After the re-selection process is executed, the processor 10A selects source images from the image set for the target month based on the characteristic trends of the image set taken during the re-selection period. Then, the processor 10A recreates the first product for the target month using the selected source images (S032). The process after the first product is recreated in the recreation process is the same as the process after the first product is created in the first creation process. Specifically, the processor 10A generates a data file for the recreated first product for the target month and sends that data file to the user-side device 12. As a result, the recreated first product for the target month is displayed on the display of the user-side device 12.

[0149] The user looks at the display to confirm the recreated first product for the target month and consider whether to place an order (S033). If the user places an order, the processor 10A accepts the order (S034) and performs the process of providing the recreated first product for the target month to the user, specifically printing (S035). Upon accepting the order, the processor 10A confirms the recreated first product for the target month.

[0150] Furthermore, the user considers whether it is necessary to save data for the first product of the recreated target month (S036). If the user requests saving, the processor 10A accepts the saving request (S037) and saves the data file of the first product of the recreated target month to the storage device 14 (S038).

[0151] The series of steps related to the recreation process described above are repeated as long as the recreation conditions are met for the target month. On the other hand, if the recreation conditions are no longer met, that is, if processor 10A receives an order or save request for the recreated first product, the recreation process for that first product will no longer be executed.

[0152] <<Regarding the second creation process>> Next, the flow of the second creation process in the product provision flow will be explained with reference to Figure 15. Each step in the second creation process is executed by the processor 10A of the computer that constitutes the image processing device 10. Prior to the execution of the second creation process, the processor 10A creates the first product F1 for each of the 12 months, including the final month.

[0153] In the second creation process, first, the processor 10A identifies the source images used to create the first product F1 for each of the 12 months (S041). In this step S041, when the processor 10A identifies the source images of the first product F1 for the month in which the save request was made, it reads the data file of the first product F1 from the storage device 14. Then, the processor 10A identifies the source images of the first product from the read data file.

[0154] Furthermore, if the first product for the target month is recreated, processor 10A identifies the source image of the recreated first product. If the first product for the target month has been recreated multiple times, processor 10A identifies the source image of the most recent first product (i.e., the first product that was most recently recreated).

[0155] Next, the processor 10A determines on a monthly basis whether or not an order has been received from a user for the first product of each month (S042). Specifically, the processor 10A determines whether or not an order has been placed for the first product of each month by referring to information showing the user's order history.

[0156] Subsequently, processor 10A creates a second product based on the source images of the first product for each month and whether or not there are orders for the first product for each month (S043). Specifically, processor 10A creates each of the multiple pages included in the second product using the source images of the first product for the month corresponding to each page. For example, if the Xth page of the second product corresponds to May, processor 10A creates the Xth page using the source images of the first product for May.

[0157] Furthermore, processor 10A sets the image layout on each page according to whether or not an order was placed for the first product in each month. Specifically, processor 10A sets the layout of each page so that the material images for the first product in the month of the order (the month in which an order was placed for the first product) are given priority. This is because it is presumed that there is a high possibility that an important event occurred for the user in the month of the order, and therefore the images taken in the month of the order are of high importance.

[0158] There are no particular limitations on how to prioritize the handling of material images for the first product in the month of order in the second product. For example, the image size on each page of the second product may be set to one of the following: large, medium, small, or index display size (minimum size). In this case, on the page of the second product corresponding to the month of order, the material images for the first product in the month of order may be placed in large size, as shown in Figure 16A. On the other hand, as shown in Figure 16B, the material images for the first product in months without an order may be placed in index display size on the last page of the second product. In Figures 16A and 16B, the image sizes for the pages corresponding to the order month and the pages corresponding to the months for which no order was placed are shown with dashed lines.

[0159] The second product F2 is completed through the above series of steps. Subsequently, if there is an order from the user for the second product, the processor 10A accepts the order (S044) and performs processing to provide the second product F2 to the user, such as printing (S045). The second creation process ends when the series of steps described above are completed.

[0160] <<Other Embodiments>> The embodiments described above are specific examples given to provide a clear explanation of the image processing apparatus and image processing method of the present invention, and are merely examples; other embodiments are also conceivable.

[0161] In the above embodiment, the first product F1 is created on a monthly basis. However, the creation cycle of the first product is not particularly limited, and the first product may be created daily, weekly, or yearly, or it may be created on a cycle of several days, several weeks, several months, or several years. Furthermore, in the above embodiment, the second product F2 is produced on an annual basis. However, the production cycle of the second product is not particularly limited, as long as multiple first products are produced within that period. For example, the second product may be produced daily, weekly, or monthly, or it may be produced on a cycle of several days, several weeks, several months, or several years.

[0162] Furthermore, in the above embodiment, the material selection process for selecting material images for the current month is based on the trend of the characteristics of the material images from the previous month. However, it is not limited to this, and material images for the current month may be selected based on the trend of the characteristics of material images from months prior to the previous month, for example, two months or more prior to the current month.

[0163] Furthermore, in the above embodiment, the functions of the image processing apparatus of the present invention are performed by a processor provided in the server computer, but this is not limited to this. The processor provided in the image processing apparatus of the present invention may be provided in both the server computer and the user-side device 12. In other words, some of the functional parts provided in the image processing apparatus 10 described above may be provided in the user-side device 12.

[0164] The image processing apparatus of the present invention includes various types of processors. These various types of processors include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. Furthermore, various types of processors include PLDs (Programmable Logic Devices), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. Furthermore, various types of processors include dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform particular processing.

[0165] Furthermore, one processing unit of the image processing apparatus of the present invention may be configured with one of the various processors described above, or with a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of FPGA and CPU. Furthermore, the multiple functional units of the image processing apparatus of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be combined and configured by a single processor. Furthermore, as in the embodiment described above, a single processor may be configured by a combination of one or more CPUs and software, and this processor may function as multiple functional units.

[0166] Alternatively, a processor that realizes the functions of the entire system, including multiple functional parts in an image processing device, on a single IC (Integrated Circuit) chip, such as a SoC (System on Chip), may be used. Furthermore, the hardware configuration of the various processors described above may be an electrical circuit (Circuitry) combining circuit elements such as semiconductor elements. [Explanation of Symbols]

[0167] 10 Image Processing Device 10A Processor 10B memory 10C communication interface 10D printer 12. User-side equipment 14 Storage device 21 Image acquisition unit 22 Preliminary Sorting Department 23. Material Selection Department 24 Editorial Department 25. First Product Creation Department 26 Recommendation Department 27 Order Reception Department 28 Request Reception Department 29 Re-selection department 30 Re-creation section 31. Second Product Creation Department F1 First Product F2 Second Product N Network S Product Provision System

Claims

1. An image processing apparatus equipped with a processor, The aforementioned processor, A first acquisition process to acquire the first set of images taken during the first period, A second acquisition process for acquiring a second set of images taken during a second period different from the first period, A feature identification process is performed to identify features related to items concerning the capture or content of the first image selected from the first image group and the second image included in the second image group. Regarding the above items, a trend identification process is performed to identify the trend of the features of the selected first image, A selection process is performed to select a second image from the second image group based on the characteristics of the second image included in the second image group. The image processing apparatus, in the selection process described above, selects a second image having characteristics that deviate from the trend described above for the item described above.

2. The first image selected and the second image selected in the selection process are images for creating commercial materials. In the trend identification process, the processor obtains a first trend value that quantifies the trend, The image processing apparatus according to claim 1, wherein in the selection process, the processor selects the second image based on the first trend value.

3. The image processing apparatus according to claim 2, wherein in the selection process, the processor selects the second image such that the difference between a second trend value, which is a numerical representation of the trend of the features of the second image to be selected, and a first trend value is greater than or equal to a set value.

4. The aforementioned items include priority items set according to the aforementioned trend and non-priority items other than the priority items, The image processing apparatus according to any one of claims 1 to 3, wherein in the selection process, the processor selects a second image having characteristics in line with the trend for the priority items and characteristics that deviate from the trend for the non-priority items.

5. In the trend identification process described above, the processor, With respect to the aforementioned items, the selected first image is divided into several categories based on the characteristics of the selected first image, and the first ratio of the first image belonging to each category is calculated for each category to identify the trend. With respect to the above items, the second images included in the second image group are divided into the above-mentioned divisions based on the characteristics of the second images included in the second image group, and the second ratio of the second images belonging to each of the above divisions is calculated for each division. The image processing apparatus according to claim 4, wherein, in the selection process, the processor selects a second image belonging to the category in which the second ratio is greater than the first ratio for the non-priority item.

6. In the feature identification process, the processor further identifies features relating to the item for the first image included in the first image group, In the trend identification process, the processor identifies, with respect to the item, the trend of the selected first image features and other trends that are trends of the first image features included in the first image group. The image processing apparatus according to claim 4, wherein in the selection process, the processor includes the second image having characteristics consistent with the other trends for the priority item in the second image to be selected.

7. The processor repeatedly performs a first creation process, which involves creating a first product for the period corresponding to the second period, using the second image selected by the selection process from the second group of images taken during the period corresponding to the second period, by changing the period corresponding to the second period. The first creation process includes a material selection process for selecting the second image to be used in creating the first product for the period corresponding to the second period from among the second image group taken during the period corresponding to the second period. In the material selection process, the processor performs the feature identification process, the trend identification process, and the selection process. When executing the first creation process with a certain period as the second period, the material selection process is executed with a period prior to the certain period as the first period. The image processing apparatus according to any one of claims 1 to 6, wherein, after executing the first creation process with the aforementioned period as the second period, if the first creation process is executed with a period later than the aforementioned period as the second period, the material selection process is executed with a period earlier than the aforementioned period as the first period.

8. The first creation process includes a layout determination process that determines the layout of images in the first product based on the user's layout editing operations. The image processing apparatus according to claim 7, wherein the processor, in a first creation process for creating the first product for the second period, proposes an image layout for the first product for the first period to the user, and if the user adopts the proposed layout, creates the first product for the second period using the proposed layout.

9. The processor, after executing a first creation process to create the first product for the period corresponding to the second period, executes a recreation process to recreate the first product for the period corresponding to the second period if the recreation conditions are met. The image processing apparatus according to claim 7 or 8, wherein the reproduction process includes a re-selection process for selecting the second image to be used for reproducing the first product from among the second group of images taken during the period corresponding to the second period.

10. The aforementioned recreation process for the period corresponding to the second period is performed in a period later than the period corresponding to the second period. The image processing apparatus according to claim 9, wherein in the re-selection process, the processor selects the second image from the second image group captured during the second period, based on the trend of the characteristics of the image group captured during two or more periods including the period corresponding to the second period and a period later than the period corresponding to the second period.

11. The processor receives a user's save request for the first product, After the execution of the first creation process, the processor: If the storage request for the first product is received, the recreation process for recreating the first product will not be executed. The image processing apparatus according to claim 9 or 10, which, if the request to save the first product has not been received, performs the recreation process to recreate the first product.

12. The processor receives a user's order for the first product. After the execution of the first creation process, the processor: If an order for the first product is received, the reproduction process for reproducing the first product is not performed. The image processing apparatus according to claim 9 or 10, which, if the order for the first product has not been received, performs the reproduction process for reproducing the first product.

13. The processor further performs a second creation process to create a second product for multiple periods, The second product is composed of a plurality of second product component pieces assembled together. Each of the multiple second product components corresponds to one of the multiple periods mentioned above. In the second creation process, the processor, For each of the first creation processes that is repeatedly executed with a different period corresponding to the second period, the process of identifying the second image used to create the first product is performed. An image processing apparatus according to any one of claims 9 to 12, which creates a second product by repeatedly performing the process of creating a second product component corresponding to the period corresponding to the second period, using the second image used to create the first product for the period corresponding to the second period among the plurality of periods, while changing the period corresponding to the second period.

14. The processor receives a user's order for the first product. The image processing apparatus according to claim 13, wherein in the second creation process, the processor sets the layout of the second image in each of the plurality of second product components according to whether or not there is an order for the first product for each second product component and the period corresponding thereto.

15. A processor-based image processing method, The aforementioned processor, A first acquisition process to acquire the first set of images taken during the first period, A second acquisition process for acquiring a second set of images taken during a second period different from the first period, A feature identification process is performed to identify features related to items concerning the capture or content of each of the first image selected from the first image group and the second image included in the second image group. Regarding the above items, a trend identification process is performed to identify the trend of the features of the selected first image, A selection process is performed to select a second image from the second image group based on the characteristics of the second image included in the second image group. The selection process involves selecting a second image having characteristics that deviate from the aforementioned trend for the aforementioned item.

16. The first image selected and the second image selected in the selection process are images for creating commercial materials. In the trend identification process, the processor obtains a first trend value that quantifies the trend, The image processing method according to claim 15, wherein the selection process involves the processor selecting the second image based on the first trend value.

17. The image processing method according to claim 16, wherein in the selection process, the processor selects the second image such that the difference between a second trend value, which quantifies the trend of the features of the second image to be selected, and a first trend value is greater than or equal to a set value.

18. The aforementioned items include priority items set according to the aforementioned trend and non-priority items other than the priority items, The image processing method according to any one of claims 15 to 17, wherein the selection process involves the processor selecting a second image having features consistent with the trend for the priority items and features that deviate from the trend for the non-priority items.

19. In the trend identification process described above, the processor performs the following: With respect to the aforementioned items, the selected first image is divided into several categories based on the characteristics of the selected first image, and the first ratio of the first image belonging to each category is calculated for each category to identify the trend. With respect to the above items, the second images included in the second image group are divided into the above-mentioned divisions based on the characteristics of the second images included in the second image group, and the second ratio of the second images belonging to each of the above divisions is calculated for each division. The image processing method according to claim 18, wherein the selection process involves the processor selecting a second image belonging to the category in which the second ratio is greater than the first ratio for the non-priority item.

20. In the feature identification process, the processor further identifies features related to the item for the first image included in the first image group. In the trend identification process, the processor identifies, with respect to the item, the trend of the selected first image features and other trends that are trends of the first images included in the first image group. The image processing method according to any one of claims 15 to 19, wherein the selection process involves the processor selecting the second image based on the trend and the other trends.

21. A program for causing a computer to perform each of the processes included in the image processing method according to any one of claims 15 to 20.

22. A computer-readable recording medium, A recording medium on which a program for causing a computer to perform each of the processes included in the image processing method according to any one of claims 15 to 20 is recorded.

Images