Image processing apparatus, image processing method, program, and recording medium

Abstract

To provide an image processing apparatus that can appropriately select an image for product creation from an image group provided from a user for every period, a method, a program, and a recording medium.SOLUTION: In an image processing apparatus of the present invention, a processor acquires a first image group photographed in a first period, acquires a second image group photographed in a second period different from the first period, specifies characteristics related to items having to do with the photographing or the details of the image, for a first image selected from the first image group and a second image included in the second image group, specifies the tendency of the characteristics of the selected first image for the items, and on the basis of the characteristics of the second image included in the second image group, selects the second image having the characteristics falling outside the tendency for the items from the second image group.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a recording medium on which the program is recorded, for selecting an image for creating a commercial material from a group of images. [Background technology]

[0002] When using a service that provides commercial materials such as photobooks, the user provides multiple images (image group) to be used as materials for the commercial material. In other words, the commercial material is created using images selected from the image group provided by the user.

[0003] In the above-mentioned service, it is possible to create commercial materials at predetermined intervals (for example, monthly or yearly). In this case, the process of selecting stock images from a group of images provided by the user and the process of creating commercial materials using the selected images are repeatedly executed at regular intervals. This allows the user to obtain a group of images at each period.

[0004] When selecting images for creating commercial materials, i.e., stock images, from a group of images provided by a user, it is desirable to select, for example, images that are important to the user or images that match the user's preferences. Patent Document 1 describes a technique for selecting stock images from a group of images provided for each period.

[0005] The image processing device described in Patent Document 1 (hereinafter referred to as the conventional image processing device) sets criteria for image selection based on a first image selected from a first group of images and a second image selected from a second group of images. When selecting an image from an input group of target images, the conventional image processing device selects the image based on the above criteria. With this configuration, the conventional image processing device can select an image that is preferable to the user from the group of target images while maintaining commonality with the first and second images selected in the past. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-46901 Summary of the Invention [Problem to be solved by the invention]

[0007] The criteria for selecting material images set by conventional image processing devices are to select images that have a common theme (characteristic) obtained from a plurality of first images and a plurality of second images. According to this criterion, for example, if a child appears in both the first image and the second image, an image that shows a child is selected from the target image group as an image that shares a common theme with these images.

[0008] On the other hand, if images with a common theme are selected from the image group provided for each period, the images included in each product will be monotonous, meaning that there will be little variation, even if the products are provided for each period.

[0009] The present invention aims to provide an image processing device, method, program, and recording medium that can appropriately select images for creating commercial materials from a group of images provided by a user for each period of time. [Means for solving the problem]

[0010] In order to achieve the above object, the image processing device of the present invention is an image processing device equipped with a processor, which executes a first acquisition process to acquire a first group of images taken during a first period, a second acquisition process to acquire a second group of images taken during a second period different from the first period, a feature identification process to identify features for two or more items related to the shooting of a first image from the first group of images that was used to create a commercial material and a second image included in the second group of images, a trend identification process to identify trends in the features of the first image used to create the commercial material for two or more items, and a selection process to select a second image for creating the commercial material from the second group of images based on the features of the second image included in the second group of images, and is characterized in that in the selection process, a second image having features that deviate from the trend for some of the two or more items is selected as the second image for creating the commercial material.

[0011] In addition, in the trend identification process, the processor may determine a first trend value that quantifies the trend, and in the selection process, the processor may select a second image for creating a commercial material based on the first trend value. Furthermore, in the selection process, the processor may select a second image for creating a commercial product so that the difference between a second trend value, which quantifies the tendency of the characteristics of the second image for creating a commercial product, and the first trend value is greater than or equal to a set value.

[0012] The two or more items may include priority items set according to trends and non-priority items other than the priority items. In this case, in the selection process, the processor may select, as the second image for creating a commercial product, a second image that has characteristics in line with the trends for the priority items and characteristics that deviate from the trends for the non-priority items. Furthermore, in the trend identification process, the processor may identify the trend by dividing the first images used in creating the commercial material into a plurality of sections based on the characteristics of the first images used in creating the commercial material for each of two or more items and calculating a first ratio of the first images belonging to each section for each section. Also, the processor may divide the second images included in the second image group into a plurality of sections based on the characteristics of the second images included in the second image group for each of two or more items and calculating a second ratio of the second images belonging to each section for each section. Furthermore, in the selection process, the processor may select, for a non-priority item, a second image that belongs to a category in which the second ratio is higher than the first ratio, as the second image for creating a commercial material.

[0013] In the feature identification process, the processor may further identify features relating to two or more items for the first images included in the first image group. In this case, in the trend identification process, the processor may identify, for two or more items, a trend in the features of the first images used to create the commercial material and another trend that is a trend in the features of the first images included in the first image group. In the selection process, the processor may select a second image for creating the commercial material based on the trend and the other trend.

[0014] The processor may also execute, for each period, a first creation process for creating a first commercial material for one period using a group of images captured in one period out of the plurality of periods. The first creation process may include a layout determination process for determining a layout of images in the first commercial material based on a layout editing operation by a user. In this case, the processor may propose a layout of images in the first commercial material for the first period to the user in the first creation process for creating the first commercial material for the second period, and if the user adopts the proposed layout, create the first commercial material for the second period with the proposed layout. Furthermore, the first creation process may include a material selection process for selecting, from a group of images taken during the one period, material images to be used in creating the first commercial material for the one period. In this case, the processor may execute the first creation process for creating the first commercial material for the one period, and then execute a re-creation process for re-creating the first commercial material for the one period if a re-creation condition is satisfied. The re-creation process may include a re-selection process for selecting, from a group of images taken during the one period, material images to be used in re-creating the first commercial material.

[0015] The re-creation process may be executed in a period after the one period. In this case, in the re-selection process, the processor may select material images from the group of images taken in the one period based on tendencies in characteristics of the group of images taken in two or more periods including the one period and a period after the one period. The processor may also accept a user's save request for the first product. In this case, after executing the first creation process for creating the first product for the one period, if the processor accepts a save request for the first product for the one period, the processor may not execute a re-creation process for re-creating the first product for the one period. On the other hand, if the processor has not accepted a save request for the first product for the one period, the processor may execute a re-creation process for re-creating the first product for the one period. The processor may also accept a user order for the first product. In this case, after executing the first creation process for creating the first product for the one period, if the processor accepts an order for the first product for the one period, the processor may not execute a re-creation process for re-creating the first product for the one period. On the other hand, if the processor has not accepted an order for the first product for the one period, the processor may execute a re-creation process for re-creating the first product for the one period.

[0016] The processor may further execute a second creation process to create second products for multiple periods using material images used in each of the first products created for each period. In this case, the second products may be composed of a collection of multiple second product components. In the second creation process, the processor may create each of the multiple second product components using material images used in creating the first product for a period corresponding to each of the second product components among the multiple periods. The processor may also receive a user order for the first product. In this case, in the second creation process, the processor may set the layout of the material images for each of the plurality of second product constituent pieces depending on whether or not there is an order for the first product for a period corresponding to each second product constituent piece.

[0017] In addition, in order to solve the above-mentioned problems, the image processing method of the present invention is an image processing method by a processor, in which the processor executes a first acquisition process to acquire a first group of images taken during a first period, a second acquisition process to acquire a second group of images taken during a second period different from the first period, a feature identification process to identify features for two or more items related to the shooting for each of the first images used to create commercial materials and the second images included in the second image group, a trend identification process to identify trends in the features of the first images used to create commercial materials for two or more items, and a selection process to select a second image for creating commercial materials from the second image group based on the features of the second images included in the second image group, wherein the selection process is characterized in that a second image having features that deviate from the trend for some of the two or more items is selected as the second image for creating commercial materials.

[0018] According to the present invention, it is also possible to realize a program for causing a computer to execute each of the processes included in the image processing method described above. Furthermore, according to the present invention, it is also possible to realize a computer-readable recording medium on which a program for causing a computer to execute each of the processes included in the image processing method of the present invention is recorded. [Effects of the Invention]

[0019] According to the present invention, images for creating commercial materials can be appropriately selected from a group of images provided by a user for each period, thereby preventing the commercial materials created for each period from becoming monotonous. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 10 is a diagram showing an example of a first commercial product. [Figure 2] FIG. 10 is a diagram showing an example of a second product. [Figure 3] 1 is a diagram showing a commercial material providing system including an image processing device according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram illustrating an example of a layout editing screen. [Figure 5] FIG. 10 is a diagram showing an example of an image layout pattern. [Figure 6] FIG. 2 is an explanatory diagram illustrating functions of an image processing apparatus according to an embodiment of the present invention. [Figure 7] FIG. 1 is a diagram showing a basic processing flow of an image processing apparatus according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing the flow of a first creation process. [Figure 9] FIG. 10 is a diagram showing the flow of a material selection process. [Figure 10] FIG. 10 is an explanatory diagram of images included in a group of images for a certain month. [Figure 11] FIG. 10 is a diagram showing the procedure for material selection processing for the month after the first month. [Figure 12] 10A and 10B are explanatory diagrams of a group of images for a certain month, material images for a certain month, and a group of images for the following month. [Figure 13] FIG. 10 is a diagram showing a first trend value and a second trend value. [Figure 14] FIG. 10 is a diagram showing the flow of a re-creation process. [Figure 15] FIG. 10 is a diagram showing the flow of a second creation process. [Figure 16A] FIG. 10 is a diagram showing the image size on the page corresponding to the order month for the second product. [Figure 16B] FIG. 10 is a diagram showing image sizes on pages corresponding to unordered months for the second product. DETAILED DESCRIPTION OF THE INVENTION

[0021] Specific embodiments of the present invention (hereinafter, the present embodiments) will be described with reference to the drawings. However, the embodiments described below are merely examples given to facilitate understanding of the present invention and are not intended to limit the present invention. Furthermore, the present invention may be modified or improved from the following embodiments without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0022] In this specification, unless otherwise specified, "image" refers to image data. Examples of image data include lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, and lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format.

[0023] In this specification, the concept of "device" includes a single device that performs a specific function by itself, as well as multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function.

[0024] <<About the product provision service of this embodiment>> Prior to describing the image processing device and image processing method according to this embodiment, a commercial material providing service implemented using these will be described.

[0025] The product provision service is a service that creates a product using multiple images provided by a user who is a service user, and provides the product to the user. The product is, for example, a photo book or album created using images provided by the user. The type of product is not particularly limited, and may be a product other than the above types, such as a calendar or a collage print image that combines images provided by the user with decorative images. In addition, the method of providing the product is not particularly limited, and may include providing the user with data for displaying the product on a display or the like, or printing the product and providing it to the user in printed form.

[0026] Creating commercial materials usually requires multiple images. Therefore, users provide images taken with photographic equipment such as digital cameras as needed or periodically. In other words, when obtaining commercial materials using a commercial material providing service, users provide an image group containing multiple images.

[0027] To give an overview of the procedure for creating a commercial material, first, one or more images are selected from a group of images provided by a user as images for creating the commercial material (hereinafter also referred to as "stock images"). Next, a process is performed to create the commercial material using the stock images selected from the group of images. In this process, the layout of the stock images in the commercial material is determined. The image layout in the commercial material refers to the number of images used in the commercial material, the position of the images, the size of the images, etc.

[0028] A commercial product created using material images is provided to a user in response to an order or the like from the user. In this embodiment, two types of commercial products can be provided. One type of commercial product is a first commercial product that is provided periodically and repeatedly over multiple periods. The first commercial product is an edited image created by editing one or more material images, and more specifically, is a photobook-type commercial product (denoted by the symbol F1 in the figure) shown in FIG. 1.

[0029] In this embodiment, the first product F1 is created at regular intervals, specifically, monthly. For example, if the product provision service is used for one year, a first product F1 is created for each month, meaning that 12 months' worth of first products F1 are provided to the user in one year. This allows the user to organize images taken each month by month and to keep important images in the form of products. Note that the first product F1 can be provided at a lower price than the second product F2 described below, making it easier for users to use (purchase).

[0030] Here, the multiple months included in the service usage period correspond to the "multiple periods" of the present invention, and each month (i.e., one month) corresponds to "one period." Note that, for convenience, hereinafter, the first product F1 created for a certain month will be referred to as the "first product of a certain month."

[0031] The first product F1 for each month is created using material images selected from a group of images taken in that month (the current month). In other words, when creating the first product F1 for each month, the user provides a group of images taken by the user in that month. For example, when creating the first product F1 for April, the user provides a group of images taken by the user in April. The shooting date of each image included in the group of images can be identified from accompanying information of the image, such as tag information in Exchangeable Image File Format (Exif). In the following description, a group of images taken by a user in a certain month will be referred to as a "group of images in a certain month" for convenience.

[0032] As described above, in this embodiment, the process of creating the first product F1 is executed monthly. Specifically, the process of acquiring a group of images provided by the user, the process of selecting material images from the group of images, and the process of determining the layout of the material images in the first product are repeated monthly. This allows the first product F1 to be provided monthly. During the usage period of the product provision service, the user can check the first product F1 for the current month every month and can order the first product they desire. The first product for the ordered month (order month) is provided to the user.

[0033] In addition, the user can make a request to save the first commercial material for each month. The first commercial material for which a request to save has been made, specifically, the image material used to create the first commercial material to be saved and the image layout of the first commercial material to be saved, are saved as data files in a specified storage location.

[0034] The other product is a second product that contains images for 12 months provided by the user. Specifically, the second product is a composite image created using the stock images used in the first product for each of the 12 months, and is a yearly album-type product (denoted by F2 in the figure) as shown in Figure 2.

[0035] In this embodiment, the second product F2 is created every 12 months, that is, on a yearly basis, and is composed of a collection of multiple pages as shown in FIG. 2. The multiple pages included in the second product F2 correspond to multiple second product components. Furthermore, each page of the second product F2 is composed of the material images used to create the first product for the month corresponding to that page. For example, in the second product F2, the page corresponding to April contains the material images used to create the first product for April. As described above, by providing the second product F2 containing 12 months' worth of images used to create the first product for each month, the user can store important images taken over the course of a year in an annual album.

[0036] The second product F2 is created after a period of time (specifically, 12 months) has passed, for example, in the last month of a year, and more specifically, after the first product of the last month has been created. The second product F2 created from the material images of the first product for 12 months is provided to the user who ordered the second product F2.

[0037] <<Configuration of the image processing device according to this embodiment>> The above-described commercial material provision service is realized by a commercial material provision system S shown in Fig. 3. The commercial material provision system S is configured by an image processing device according to this embodiment (hereinafter referred to as image processing device 10) and a user-side device 12 that can communicate with the image processing device 10 via a network N.

[0038] (user device) The user-side device 12 is configured, for example, by a PC (Personal Computer), a communication terminal, a camera with communication functions, or the like, which is used by the user to use the product providing service. The display of the user-side device 12 can display images provided to the image processing device 10 and information based on data provided from the image processing device 10. The information based on data provided from the image processing device 10 includes image information of the first product F1 or the second product F2, etc.

[0039] In addition, a predetermined application program (hereinafter, a service application) is installed in the user-side device 12. A user can use a product provision service by activating the service utilization application. Specifically, a user can perform various operations for using the service through the user-side device 12 with the service utilization application activated. The operations for using the service include, for example, a service application operation, an image input operation, a layout editing operation, an order operation, and a save request operation.

[0040] There are no particular limitations on the procedures for each operation and the GUI (Graphical User Interface) such as the operation screen, etc. An example of a GUI for layout editing operations is the layout editing screen shown in FIG.

[0041] The layout editing screen is displayed on the display of the user device 12, and the editing operation screen displays images selected as images for creating commercial materials (i.e., material images). Through this screen, the user can perform layout editing operations on the first commercial material. For example, the user clicks on each material image on the screen and drags the material image to a desired position on the screen. This allows the user to set or change the position or size of each material image in the first commercial material, i.e., the image layout.

[0042] Note that there is a default image layout for the first commercial material, and the user performs a layout editing operation, for example, when changing the image layout from the default. Also, several image layout patterns may be prepared in advance, as shown in FIG. 5. In this case, the user may select one of the multiple patterns when performing a layout editing operation. FIG. 5 illustrates four patterns, designated #1 to #4, and for each pattern, the placement position and size of the material image in the first commercial material are indicated by dashed lines.

[0043] Furthermore, in the layout editing operation, the user may be able to select material images to be actually used in creating commercial materials from among the material image candidates selected by the image processing device 10.

[0044] (Image processing device) The image processing device 10 is configured by a computer, for example, a server computer. The image processing device 10 executes a series of data processing related to the product provision service. The number of computers configuring the image processing device 10 may be one, or two or more. The image processing device 10 is realized by a processor and a program executable by the processor, and is configured by, for example, a general-purpose computer. For example, as shown in FIG. 3, the computer configuring the image processing device 10 includes a processor 10A, a memory 10B, a communication interface 10C, etc.

[0045] The processor 10A is configured by, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a tensor processing unit (TPU). The memory 10B is configured by, for example, semiconductor memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The communication interface 10C is configured by, for example, a network interface card or a communication interface board.

[0046] A printer 10D as an output device may be connected to the image processing device 10. The printer 10D receives print data of a commercial material created by the image processing device 10, and operates in accordance with the print data to print (output) the commercial material.

[0047] A program for executing a series of processes related to the product provision service (hereinafter referred to as the product provision program) is installed in the computer constituting the image processing device 10. The product provision program is a program for causing the computer to execute each step included in the image processing method of the present invention. That is, when the processor 10A reads and executes the product provision program, the computer including the processor 10A functions as the image processing device of the present invention.

[0048] The commercial material providing program may be acquired by reading it from a computer-readable recording medium, or may be acquired by receiving (downloading) it via a communication network such as the Internet or an intranet.

[0049] As shown in Fig. 3, the commercial material providing system S is provided with a storage device 14 that stores a group of images acquired by the image processing device 10 from the user-side device 12. The group of images is accumulated in the storage device 14. The image processing device 10 is configured to be able to freely read out images from the group of images stored in the storage device 14. The group of images accumulated in the storage device 14 is also associated with the user who provided the image and the year and month when the images included in the group of images were taken.

[0050] The storage device 14 may be configured by a storage built into or externally attached to the image processing device 10. Alternatively, the storage device 14 may be configured by a third computer (for example, an external server) that can communicate with the image processing device 10.

[0051] The configuration of the image processing device 10 will be explained again from a functional perspective. As shown in Fig. 6, the image processing device 10 has an image acquisition unit 21, a preliminary selection unit 22, a material selection unit 23, an editing unit 24, a first product creation unit 25, a recommendation unit 26, an order reception unit 27, a request reception unit 28, a reselection unit 29, a re-creation unit 30, and a second product creation unit 31. These functional units are realized by cooperation between hardware devices provided in the computer constituting the image processing device 10 and software including the aforementioned product provision program.

[0052] The image acquisition unit 21 acquires a group of images provided by a user. Specifically, the user can specify, from among images taken by the user, images to be provided to the image processing device 10 by operating the user-side device 12, and transmit the specified images to the image processing device 10. The image acquisition unit 21 acquires the group of images provided by the user by receiving images transmitted from the user-side device 12. The group of images acquired by the image acquisition unit 21 is stored in the storage device 14.

[0053] If the user-side device 12 is a photographing device with a communication function, the image acquiring unit 21 may acquire the image group by directly receiving images transmitted from the user-side device 12. Alternatively, if the user-side device 12 is a PC or the like equipped with image editing software and acquires images from a photographing device, the image acquiring unit 21 may acquire the image group acquired by the user-side device 12. Furthermore, if the image group input from the user-side device 12 is temporarily stored in a server or the like on a network, the image acquiring unit 21 may acquire the image group from the server.

[0054] Furthermore, the image acquiring unit 21 acquires a group of images for each month. That is, the image acquiring unit 21 acquires a group of images for each month. The timing for acquiring the group of images for each month is not particularly limited, and for example, the user may input the captured images as needed every time the user takes an image, and the image acquiring unit 21 may acquire the input image every time an image is input. Alternatively, the user may input images collectively every certain period (for example, every month), and in that case, the image acquiring unit 21 may acquire the images (group of images) for one period input every month all at once.

[0055] Here, a certain month during the usage period of the product provision service and a different month correspond to the first period and second period of the present invention, respectively. Furthermore, in this embodiment, the second period is a period after the first period, for example, the month following the month corresponding to the first period. The image acquisition unit 21 acquires a group of images for the month corresponding to the first period, and then acquires a group of images for the month corresponding to the second period. The group of images for the month corresponding to the first period corresponds to the first group of images of the present invention, and the images included in the first group of images correspond to the first images. Furthermore, the group of images for the month corresponding to the second period corresponds to the second group of images of the present invention, and the images included in the second group of images correspond to the second images.

[0056] The first period and the second period are relative concepts of time that change over time. For example, in the relationship between April and May, April corresponds to the first period and May corresponds to the second period. Also, in the relationship between April, May, and June, April and May each correspond to the first period, and June corresponds to the second period. The month corresponding to the first period is not limited to the month immediately preceding the month corresponding to the second period, but may also be a month preceding that month. For example, if May is the second period, February or March may be the first period. Furthermore, the month corresponding to the first period may be multiple months (two or more months); for example, if May is the second period, March and April may be the first period. Furthermore, the second period may be a period prior to the first period. For example, if the first period is May, the second period may be April.

[0057] The preliminary selection unit 22 executes a preliminary selection process to remove images that are inappropriate as material images (hereinafter referred to as images to be removed) from the group of images acquired by the image acquisition unit 21. Whether an image is an image to be removed is determined based on the presence or absence of a human face in the image, the facial expression, the size of the face in the image, the degree of blur and shaking in the image, the color and brightness of the subject, etc.

[0058] The material selection unit 23, the editing unit 24, the first product creation unit 25, and the recommendation unit 26 execute a process for creating a first product (hereinafter referred to as a first creation process). The first creation process is a process for creating a first product for each month, and is executed monthly, for example, on a day corresponding to a predetermined date in each month.

[0059] The material selection unit 23 executes a process (hereinafter referred to as a material selection process) for selecting images to be used in creating the first commercial material, i.e., material images. The material selection process is a process included in the first creation process and is executed monthly. In the material selection process for each month, the material selection unit 23 selects material images from the group of images acquired by the image acquisition unit 21 in that month, i.e., the group of images taken in that month.

[0060] The material selection process will be specifically described using an example of creating a first commercial material for a certain month (hereinafter referred to as month P). In the material selection process for month P, material images to be used in creating the first commercial material for month P are selected from the group of images for month P, more precisely, from the group of images for month P excluding the images to be excluded selected by preliminary selection unit 22.

[0061] In the material selection process, the material selection unit 23 first identifies features relating to two or more items for the images included in the image group for month P. The two or more items are two or more items (themes) related to the capture of the images. Examples of the items include the subject person, the proportion of the subject person in the image, parts of the person that appear in the image, other subjects that appear with the person, the shooting scene, the shooting location, the shooting time, the perspective of the image, etc.

[0062] The features are specific contents and properties of the images related to the items. Examples of the features related to the items include the following features. Subjects: Relationships of people (children, relatives, friends, etc.) and the number of people in the image - Subject occupancy rate in the image: The percentage of the subject in the image Other subjects appearing alongside the person: type of subject, size of subject, etc. Photographed scene: The classification of the photographed scene and the details of the event captured in the image Location: place name, building name, whether the photo was taken indoors or outdoors, etc. Shooting time: the time of the photo (morning, afternoon, evening, or night) and the season in which the photo was taken Perspective: Is it a close-up or a distant view?

[0063] The method for identifying the features of the items is not particularly limited, but may be, for example, by using a known image analysis technique or automatically. Specifically, a known face detection technique may be applied to detect the face of a person in an image, and the detected face may be compared with a pre-registered face image to identify the person whose face is detected.

[0064] Alternatively, after detecting a subject in an image using a known object detection technique, an estimation engine built by machine learning may be used to estimate the type of the detected subject. Alternatively, when analyzing a group of images provided by a user, if a predetermined number of images containing the same person are found, it may be inferred that the person is someone important to the user (such as a family member or friend). Furthermore, the relationship between the person in the image and the user may be inferred from the location, time, and scene of the image.

[0065] Alternatively, information regarding the characteristics of the target image may be input by the operator of the image processing device 10 or the user who provides the image, and the characteristics of the target image may be identified based on the input information.

[0066] When selecting material images to be used to create the first commercial material for month P from the image group for month P, the material selection unit 23 selects the material images according to the characteristics of the images included in the image group for month P.

[0067] Furthermore, the material selection unit 23 selects, from the image group for month Q, material images to be used in creating the first commercial material for a month after month P, for example, month Q, which is the month following month P. At this time, the material selection unit 23 selects material images to be used in creating the first commercial material for month Q according to the characteristics of the images included in the image group for month Q and the characteristics of the material images used in creating the first commercial material for month P.

[0068] Hereinafter, the material images used to create the first commercial material in month P will be referred to as "material images for month P," and similarly, the material images used to create the first commercial material in month Q will be referred to as "material images for month Q." Here, month P and month Q are in the relationship of the first period and second period described above, and the image group for month P corresponds to the first image group described above, and the image group for month Q corresponds to the second image group described above.

[0069] Furthermore, in this embodiment, the material selecting unit 23 can select the material image for month Q so that there is a difference between the material image for month P and the material image for month Q in terms of features.

[0070] The editing unit 24 accepts layout editing operations from the user, and more specifically, receives data indicating the content of the layout editing operations from the user-side device 12. The editing unit 24 is configured to accept layout editing operations each time the first creation process is executed, in other words, every month. In this embodiment, there is an initial layout setting (default layout), and the user performs layout editing operations when they desire a layout other than the initial setting. The layout editing operations may include an operation to select an image to be actually used in creating the first commercial material from the material images selected by the material selection unit 23.

[0071] The first commercial material creation unit 25 creates a first commercial material for each month based on the material images selected by the material selection unit 23 and the layout editing operations received by the editing unit 24. Specifically, the first commercial material creation unit 25 executes a process (hereinafter referred to as a layout determination process) for determining the layout of the material images in the first commercial material based on the layout editing operations. In the layout determination process, the size, placement position, etc. of the material images of the first commercial material are determined.

[0072] Then, the image layout for the first product for the current month is determined, and the first product F1 for the current month is completed. As the first product is completed, the first product creation unit 25 generates a data file for the first product. The generated data file for the first product is transmitted to the user-side device 12. The user-side device 12 displays an image of the first product based on the received data file. This allows the user to check the first product F1 for each month.

[0073] The recommendation unit 26 recommends (suggests) a recommended layout to the user in the first creation process. The recommended layout is set to reflect the image layout of the first product created in the previous month or earlier. For example, assume that the image layout of the first product in month P is changed from the default layout based on the user's layout editing operation. In this case, the recommendation unit 26 recommends the image layout of the first product in month P to the user as the recommended layout in the first creation process for creating the first product in month Q, which is the month following month P. To give a specific example, assume that the user selected pattern #1 in Figure 5 when creating the first product in April. In this case, the recommendation unit 26 proposes pattern #1 to the user as the recommended layout when creating the first product in May.

[0074] The order receiving unit 27 is configured to receive a user's order for the first product. Specifically, when the user performs an ordering operation on the user-side device 12, the order receiving unit 27 receives data indicating the order contents from the user-side device 12. In this embodiment, the first product is produced monthly, so the order receiving unit 27 can receive orders monthly. When the order receiving unit 27 receives an order, a process (e.g., a printing process) is executed to provide the ordered first product to the user. Furthermore, for the ordered first product F1, the material images constituting the first product and the image layout of the first product are determined at the time of order reception.

[0075] The request receiving unit 28 is configured to receive a user's save request for the first product. Specifically, when the user performs a save request operation on the user-side device 12, the request receiving unit 28 receives data indicating the request content from the user-side device 12. In this embodiment, the first product is created monthly, so the request receiving unit 28 can receive save requests monthly. When the request receiving unit 28 receives a save request, the data file of the first product for which the save request has been made is associated with the user who made the save request and stored in the storage device 14.

[0076] After the first creation process for creating the first commercial material for the target month is executed, the reselection unit 29 and the re-creation unit 30 execute a re-creation process for re-creating the first commercial material for the target month. The target month is the month that is the target of the re-creation process, that is, the month in which the already-created first commercial material is re-created. The re-creation process is executed when predetermined conditions (hereinafter, re-creation conditions) are met.

[0077] In this embodiment, if an order or save request for the first product for the target month has not been received, a re-creation process is executed to recreate the first product for the target month. In other words, if an order or save request for the first product for the target month has been received, a re-creation process to recreate the first product for the target month is not executed. In other words, whether or not a re-creation process is necessary is determined based on the user's intention regarding the first product, specifically, whether or not an order or save request has been made. Then, a first product for which an order or save request has been made is confirmed, and an unconfirmed first product can be recreated.

[0078] The re-creation process for re-creating the first product for the target month is executed in a month after the target month. For example, if the target month is April, the re-creation process for re-creating the first product for that month (April) is executed in May or later. Furthermore, unless an order or save request is received for the first product for the target month (including the re-created product), the re-creation process for re-creating the first product for the target month is repeatedly executed in each month after the target month.

[0079] The reselection unit 29 executes a reselection process to select material images to be used to recreate the first commercial material for the target month. In the reselection process, the reselection unit 29 reselects material images from a group of images taken during a period of two months or more including the target month and months after the target month (hereinafter referred to as the reselection period). At this time, the reselection unit 29 selects material images based on the tendency of the characteristics of the group of images taken during the reselection period.

[0080] The re-creation unit 30 re-creates the first commercial material for the target month using the material image selected by the re-selection unit 29. The procedure by which the re-creation unit 30 re-creates the first commercial material is the same as the procedure by which the first commercial material creation unit 25 creates the first commercial material.

[0081] The second product creation unit 31 executes a second creation process to create a second product F2 for the year using the material images used to create each of the first products for the 12 months. In this embodiment, after the first creation process to create a first product for the last month of the year is completed, the second creation process for that year is executed.

[0082] In the second creation process, the second product creation unit 31 creates each of the multiple pages included in the second product F2 using the material images used to create the first product F1 for the month corresponding to each page. At this time, the second product creation unit 31 sets the layout of the material images on each page depending on whether or not an order has been placed for the first product F1 for the month corresponding to each page. Specifically, the second product creation unit 31 sets the image layout on each page so that material images of the first product for the ordered month are given priority over material images of the first product for the unordered month.

[0083] Then, the second product F2 is completed by determining the image layout of each page. With the completion of the second product, the second product creation unit 31 generates a data file for the second product. The generated data file is transmitted to the user-side device 12. The user-side device 12 displays an image of the second product based on the received data file. This allows the user to confirm the second product F2.

[0084] <<Basic processing flow of image processing device>> A product provision flow, which is a basic processing flow by the image processing device 10 according to this embodiment, will be described with reference to FIG.

[0085] The product provision flow is executed every year during the usage period of the product provision service. Each step in the product provision flow is executed by processor 10A of the computer constituting image processing device 10. That is, the product provision flow progresses as processor 10A reads the above-mentioned product provision program and executes each step.

[0086] In the product provision flow, the processor 10A executes an image acquisition process for each month to acquire a group of images taken by the user each month (S101). In the image acquisition process, the processor 10A receives images input from the user-side device 12, thereby acquiring a group of images taken by the user each month. Images may be input by the user each time the user takes an image. Alternatively, the group of images taken in that month may be provided all at once on a predetermined day of each month.

[0087] Thereafter, the processor 10A executes a first creation process to create the first product F1 for that month on a predetermined day of each month (S102). In the first creation process, the processor 10A executes a material selection process to select a material image from the image group acquired in step S101, and a layout determination process to determine a layout for the selected material image.

[0088] Furthermore, if there is a month that satisfies the re-creation condition among the months before the previous month (S103), the processor 10A executes a re-creation process with that month as the target month (S104). In the re-creation process, the processor 10A executes a re-selection process to select material images to be used to re-create the first commercial material for the target month from the image group for the target month.

[0089] The above series of steps S101 to S104 are repeatedly executed every month until the final month arrives. Of the multiple image acquisition processes executed every month, the image acquisition process executed in the earlier month (first period) corresponds to the first acquisition process, and the image acquisition process executed in the later month (second period) corresponds to the second acquisition process. For example, in the relationship between April and May, the image acquisition process executed in April corresponds to the first acquisition process, and the image acquisition process executed in May corresponds to the second acquisition process.

[0090] When the final month arrives (S105), the processor 10A creates the first product for the final month, and then executes a second creation process to create a second product F2 as an annual album (S106). Then, when the second creation process ends, the implementation of the product provision flow for that year ends.

[0091] <<About the first creation process>> Next, the flow of the first creation process in the product provision flow will be described with reference to Figure 8. The first creation process employs the image processing method of the present invention. That is, each step in the first creation process described below corresponds to a component of the image processing method of the present invention. The flow of the first creation process described below is merely an example, and unnecessary steps may be deleted, new steps may be added, or the order in which steps are executed may be changed, without departing from the spirit of the present invention.

[0092] Each step in the first creation process is executed by the processor 10A of the computer that constitutes the image processing device 10. Note that the following description will be given taking as an example the first creation process for creating the first product F1 for month P or month Q.

[0093] When executing the first creation process for creating the first commercial material for month P, processor 10A executes an image acquisition process for acquiring a group of images for month P from the user. Processor 10A also executes a preliminary selection process on the acquired group of images for month P (S001). By executing the preliminary selection process, images that are inappropriate for use as material images for creating commercial material are excluded from the group of images for month P as images to be excluded. Images to be excluded are images that do not meet a certain level of image quality, such as images that are blurred or out of focus, and images in which the brightness of the subject is not sufficiently ensured.

[0094] When processor 10A starts the first creation process for creating the first commercial material for month P, it executes a material selection process (S002). In the material selection process, processor 10A selects material images (material images for month P) to be used for creating the first commercial material for month P from the group of images for month P from which the images to be excluded have been excluded. The detailed procedure of the material selection process will be described in detail later.

[0095] Next, processor 10A executes a layout determination process to determine the layout of the material images in the first commercial material for month P (S003). Specifically, processor 10A determines the placement position, size, etc., in the first commercial material for month P for each of the material images for month P selected in the material selection process. Also, if the user performs a layout editing operation on the first commercial material for month P, processor 10A accepts the operation. In this case, processor 10A determines an image layout according to the accepted layout editing operation.

[0096] After the layout determination process, processor 10A creates the first commercial material for month P based on the material images selected in the material selection process and the image layout determined in the layout determination process (S004). Upon completion of the first commercial material for month P, processor 10A generates a data file of the first commercial material and transmits it to user-side device 12. An image of the first commercial material for month P is displayed on the display of user-side device 12. The user checks the first commercial material for month P and considers whether or not to order it (S005).

[0097] When the user operates the user-side device 12 to place an order for the first commercial material for month P, the processor 10A accepts the order (S006) and executes processing (e.g., printing processing) for providing the first commercial material for month P to the user (S007). In addition, upon accepting the order, the processor 10A finalizes the first commercial material for month P (more specifically, the material images and image layout used to create the first commercial material for month P).

[0098] The user also considers whether or not to save the data file of the first commercial material for month P (S008). When the user operates the user-side device 12 to make a save request, the processor 10A accepts the save request (S009) and saves the data file of the first commercial material for month P in the storage device 14 (S010).

[0099] When the series of steps described above is completed, the first creation process for creating the first commercial material for month P is completed. For month Q, which is the month following month P, the first creation process is executed in the same manner as above.

[0100] Assume that the image layout for the first commercial material for month P has been determined in accordance with a layout editing operation by the user. In this case, in the first creation process for creating the first commercial material for month Q, processor 10A proposes the image layout for the first commercial material for month P to the user as a recommended layout. The user decides whether to adopt the proposed recommended layout and performs an operation according to that decision on user-side device 12. By proposing the recommended layout in this manner, it is possible to support the user's layout editing operation. Then, if the user adopts the recommended layout, processor 10A creates the first commercial material for month Q based on the proposed recommended layout.

[0101] <<About the material selection process>> Next, the flow of the material selection process in the first creation process will be described with reference to Fig. 9. The material selection process is executed in the first creation process for each month, that is, executed monthly. In the material selection process for each month, the processor 10A executes a feature identification process (S021), a trend identification process (S022), and a selection process (S023), as shown in Fig. 9.

[0102] The above three processes differ between the first month (first month) of the product provision service usage period and the months after the first month. Below, we will explain the material selection process for the first month and the material selection process for the months after the first month. In the following, an example will be described in which the first month is April and the month after the first month is May.

[0103] [First month material selection process] In the material selection process for April, which is the first month, material images (April material images) to be used to create the first commercial material for April are selected from the April image group. In the April material selection process, first, a feature identification process is executed, in which processor 10A identifies features relating to two or more items for the images included in the April image group. The number and types of items are not particularly limited, but the following description will be given assuming that three items, "subject person," "location of photo," and "time of photo," are set.

[0104] To explain the features identified for each item, for example, the feature for the item "subject person" specifies the relationship of the person in the image (specifically, child, parent, grandparent, etc.). The feature for the item "location of photo" specifies the specific name of the location where the photo was taken (place name), such as home or park. The feature for the item "time of photo" specifies the specific time, or the time period, such as morning, afternoon, or night. The method for identifying the features may be a method for automatically identifying the features by image analysis, or a method for manually inputting information indicating the features.

[0105] In the trend identification process in the material selection process for April, processor 10A identifies trends in the characteristics of images included in the image group for April for three items based on the characteristics identified in the characteristic identification process. The trends indicate how many images with certain characteristics are included in the image group.

[0106] To explain the trend identification process in more detail, in the trend identification process, processor 10A divides the images included in the image group for April for each of the three items into multiple categories based on the features identified in the feature identification process. A category is a category (category) for classifying images having a certain feature. For example, for the item "location of photography," there is a category for images taken at home and a category for images taken in a park. It is possible that an image may belong to multiple categories for a certain item. For example, an image of a child and his / her mother will belong to the category "images with children" and also to the category "images with mothers" for the item "subject person."

[0107] Processor 10A calculates the ratio of images belonging to each category for each item. The ratio is expressed as the ratio (unit: %) of the number of images belonging to each category to the group of images in April. The ratio calculated for each category for each item corresponds to the tendency of the characteristics of the images included in the group of images in April.

[0108] An example of the calculated ratios for images included in the April image group is shown in Fig. 10. Regarding the April image group, for example, as shown in Fig. 10, for the item "subject person," the ratio of images featuring children is 40%, the ratio of images featuring mothers is 40%, and the ratio of images featuring fathers is 40%. Also, for the item "photography location," the ratio of images taken at home is 80%, and the ratio of images taken in parks is 20%.

[0109] After executing the feature identification process and the trend identification process, the processor 10A executes the selection process for April. In the selection process for April, the processor 10A selects material images (April material images) to be used in creating the first commercial material for April from the group of images for April from which the images to be excluded have been removed. At this time, the processor 10A selects the material images for April based on the trend identified in the trend identification process.

[0110] Specifically, the processor 10A preferentially selects images in a category with a high ratio for each item as material images. In the case of Fig. 10, an image in a category with the highest ratio for each item, for example, "an image that includes a child and was taken at home in the morning," is preferentially selected as material images. Note that the images selected as material images may include images belonging to categories other than the category with the highest ratio, in proportions according to the ratios of those categories.

[0111] The stock images for April are selected through the above procedure. These stock images are images selected to reflect the user's preferences and interests. In other words, the ratio calculated for each item is determined according to the photography tendencies of the user who is the image provider, and images in a category with a high ratio are likely to be images that the user likes or values. Therefore, by selecting stock images through the above procedure, images that are important to the user can be selected as stock images.

[0112] [Material selection process for the month after the first month] In the material selection process for May, which is an example of the month after the first month, material images to be used in creating the first commercial material for May (May material images) are selected from the image group for May. In the material selection process for May, material images for May are selected taking into account the material images for April. Here, in the relationship between April and May, April corresponds to the first period, and May corresponds to the second period. Furthermore, the image group for April corresponds to the first image group, and the images included in that image group correspond to the first images. In other words, the material images for April correspond to the first images used in creating the commercial material. Furthermore, the image group for May corresponds to the second image group, and the images used in that image group correspond to the second images. In other words, the material images for May correspond to the second images used in creating the commercial material.

[0113] The material selection process for May proceeds according to the flow shown in Fig. 11. In the feature identification process in the material selection process for May, the processor 10A identifies features related to the above-mentioned three items for the material images for April and the images included in the image group for May (S201, S202). The manner and procedure for identifying the image features are the same as those in the feature identification process in the material selection process for April.

[0114] In the subsequent trend identification process, the processor 10A identifies trends in the characteristics of the material images in April for the above three items (S203). Specifically, in the trend identification process, the processor 10A divides the material images in April for each item into multiple categories based on their characteristics. Then, the processor 10A calculates the ratio of images (material images) belonging to each category for each category. This ratio corresponds to the first ratio, and is expressed as the ratio (unit: %) of the number of images belonging to each category to the total number of material images in April. Then, the first ratio calculated for each category corresponds to the trend of the characteristics of the material images in April, i.e., the trend of the characteristics of the first images used to create commercial materials. In the following, for convenience, the trends in the characteristics of the material images in April will be referred to as the "trends in April."

[0115] FIG. 12 shows an example of the first ratios calculated for the material images in April. Regarding the material images in April, as shown in FIG. 12, for example, for the item "subject person," the first ratio of images in which children appear is 90%, the first ratio of images in which mothers appear is 30%, and the first ratio of images in which fathers appear is 10%. For the item "photography location," the first ratio of images taken at home is 90%, and the first ratio of images taken in parks is 10%. For the item "photography time," the first ratio of images taken in the morning is 40%, the first ratio of images taken in the afternoon is 60%, and the first ratio of images taken at night is 0%.

[0116] Furthermore, in the trend identification process, processor 10A obtains a first trend value that quantifies the trend for April. Quantifying the trend includes, for example, deriving a vector that indicates the trend. Specifically, when the number of items is C and the number of categories included in each item is D, the first trend value is expressed by a multidimensional vector having the dimensions C×D. Furthermore, each component of the multidimensional vector that constitutes the first trend value corresponds to a first ratio of images that belong to the category corresponding to that component for the item corresponding to that component.

[0117] After executing the feature identification process and the trend identification process, the processor 10A executes the selection process for May. In the selection process for May, the processor 10A selects material images for May from the image group for May based on the features of the images included in the image group for May. In this embodiment, the processor 10A selects, as the material images for May, images (second images) that have features that deviate from the April trends for some of the above three items.

[0118] More specifically, the processor 10A sets a priority item from among the three items, and sets items other than the priority item as non-priority items (S204). The priority item is an item that is given priority among the three items when selecting material images for May, and is set, for example, according to trends in April. More specifically, the category with the highest first ratio or the category in which the first ratio exceeds the reference value is identified, and the item corresponding to that category becomes the priority item. In the case shown in FIG. 12, the category with the highest first ratio is an image containing a child, so the item "subject person" becomes the priority item.

[0119] The priority item is not limited to one item, and multiple items may be set as priority items. Furthermore, there are no particular limitations on how the priority items are determined, and for example, the user may specify the priority items. The non-priority items correspond to "some items" in the present invention, and in the case shown in FIG. 12, the items "photography location" and "photography time" correspond to the non-priority items.

[0120] Next, processor 10A divides the images included in the May image group into multiple categories based on their characteristics for the above three items, and calculates the ratio of images belonging to each category for each category (S205). This ratio corresponds to the second ratio, and is expressed as the ratio (unit: %) of the number of images belonging to each category to the number of images included in the May image group.

[0121] FIG. 12 shows an example of the second ratios calculated for images included in the image group for May. Regarding the image group for May, as shown in FIG. 12, for example, for the item "subject person," the second ratio of images that include children is 90%, the second ratio of images that include mothers is 35%, and the second ratio of images that include fathers is 10%. For the item "photography location," the second ratio of images that were taken at home is 30%, and the second ratio of images that were taken in parks is 70%. For the item "photography time," the second ratio of images that were taken in the morning is 40%, the second ratio of images that were taken in the afternoon is 60%, and the second ratio of images that were taken at night is 0%.

[0122] Thereafter, the processor 10A selects material images for May based on the second ratio calculated for each category of images included in the image group for May and the first trend value indicating the trend for April (S206). At this time, the processor 10A selects, from the image group for May, images that have characteristics in line with the trend for April for priority items and have characteristics deviating from the trend for April for non-priority items as material images for May.

[0123] An image having characteristics that are in line with the trends in April for a priority item is determined according to the first ratio calculated for each category for the priority item, and is, for example, an image that belongs to the category with the highest first ratio for the priority item. In the case shown in Figure 12, an image containing a child is an image having characteristics that are in line with the trends in April for a priority item. Note that images with characteristics that are in line with April trends for priority items are not limited to images that belong to the category with the highest first ratio, but may be, for example, images that belong to any of the categories with the highest first ratios up to the nth highest (n is a natural number greater than or equal to 2).

[0124] On the other hand, images with characteristics that deviate from the April trend for non-priority items are determined according to the first ratio and the second ratio calculated for each category for the non-priority items. Specifically, images belonging to a category in which the second ratio calculated for the non-priority items is higher than the first ratio correspond to images with characteristics that deviate from the April trend for non-priority items. In the case shown in FIG. 12, for the non-priority item "location of photography," the second ratio of images taken in parks is 70%, which is an increase from the first ratio of 10%. Therefore, images taken in parks correspond to images with characteristics that deviate from the April trend for non-priority items.

[0125] In addition, among images belonging to a category in which the second ratio calculated for a non-priority item is higher than the first ratio, an image belonging to a category in which the difference between the second ratio and the first ratio exceeds a threshold may be selected as a material image. Furthermore, if there are multiple categories in which the second ratio is higher than the first ratio for non-priority items, an evaluation value may be calculated for each category for the images belonging to that category. The evaluation value is the sum of a value calculated depending on the image quality of the image, the number of people in the image, and whether a specific person is present in the image, and a value determined depending on the second ratio. The processor 10A may prioritize a category containing images with higher evaluation values among the multiple categories, and select images belonging to that category as material images.

[0126] The processor 10A uses the first trend value indicating the trend for April when selecting images having characteristics that deviate from the trend for April for non-priority items. Specifically, the processor 10A selects material images for May so that the difference between the first trend value and a second trend value that quantifies the characteristics of the material images for May is equal to or greater than a set value.

[0127] As shown in FIG. 13, the second trend value, like the first trend value, is expressed by a multidimensional vector having a number of dimensions obtained by multiplying the number of items C by the number of categories D included in each item. Furthermore, each component of the multidimensional vector forming the second trend value corresponds to a second ratio of images belonging to the category corresponding to that component for the item corresponding to that component. As shown in FIG. 13, the difference between the second trend value and the first trend value is the absolute value of the difference between the two vectors. Furthermore, the setting value set for the difference may be a predetermined value, or may be set by the user and changed as appropriate.

[0128] According to the above method, it is possible to accurately select images having characteristics that deviate from the April trends from among the group of images from May. That is, in this embodiment, the processor 10A quantifies the April trends and the trends of the characteristics of the images selected as material images for May, and executes a calculation process using the two quantified trends. This calculation process makes it possible to rationally and appropriately derive images having characteristics that deviate from the April trends.

[0129] According to the procedure described above, processor 10A selects material images for May from the group of images for May. Here, the images selected as material images for May take into account material images for April, and specifically, for priority items, they have characteristics that are in line with trends in April, and for non-priority items, they have characteristics that deviate from trends in April. As a result, the images used to create the first commercial material for May (the current month) can be appropriately selected based on the images used to create the commercial material for April (the previous month).

[0130] To explain in more detail, the characteristics of images related to priority items are likely to be the content (theme) that the user prioritizes when creating the first product. Based on this, if an image with characteristics that are in line with the trends in priority items in April is selected as the stock image for May, it is possible to select a stock image that matches the user's preferences or interests.

[0131] On the other hand, if there is little change between the characteristics of the stock images in May and April, the first products in April and May will tend to be similar to each other. If this situation continues with the first products of subsequent months, even if first products are created each month, the images included in the first products, i.e., the stock image characteristics, will become monotonous. As a result, the first products for each month may lack variety.

[0132] Therefore, in this embodiment, an image having characteristics that deviate from the April trends for non-priority items is selected as the material image for May. As a result, the characteristics of the image (material image) used to create a product will differ between the first product for May and the first product for April. As a result, the first products created each month will have a richer variety, preventing them from becoming monotonous (unchanging) for users. This effect is expected to increase the number of users who use the product provision service that provides first products each month.

[0133] In the case shown in Fig. 12, the priority item is "photographed person" and the non-priority item is "photography location." Images that include children and were taken in a park were selected as images that have characteristics in line with the April trends for the priority item and characteristics that deviate from the April trends for the non-priority item. However, the priority items and non-priority items may change depending on the characteristics of the image group provided by the user. For example, the stock images for April may contain many images of children eating strawberries, while the image group for May does not contain any images of children eating strawberries, but instead contains many images of children eating melons. In this case, the priority items may be set to "the subject and their behavior," and the non-priority items may be set to "objects photographed with the subject." Specifically, an image having characteristics consistent with the April trend for the priority items may be set to an image of a child eating fruit. Furthermore, an image having characteristics deviating from the April trend for the non-priority items may be set to an image in which the child is photographed with a fruit other than strawberries, such as a melon. Under these conditions, the stock images for May may be selected from the image group for May.

[0134] To further explain the selection process for May, the images selected as material images for May may include images that satisfy the above-mentioned conditions (hereinafter referred to as main images) as well as images other than main images (hereinafter referred to as sub-images). A main image is an image that has characteristics that are in line with April trends for priority items and characteristics that deviate from April trends for non-priority items. A sub-image is, for example, an image that does not have characteristics that are in line with April trends for priority items. It is preferable that the proportion of sub-images among the material images for May be smaller than the proportion of main images.

[0135] Furthermore, the processor 10A executes a feature identification process and a trend identification process when selecting material images for May. In the feature identification process, the processor 10A identifies the features of material images for April and the features of images included in the image group for May for the above three items. At this time, the processor 10A may further identify the features of images included in the image group for April (first image group) for the above three items.

[0136] Furthermore, in the trend identification process, the processor 10A identifies the trend of the characteristics of the material images in April (i.e., the trend for April) for the above three items. At this time, the processor 10A may further identify the trend of the characteristics of the images included in the image group for April. The trend of the characteristics of the images included in the image group for April corresponds to "other trends" in the present invention, and will be referred to as the trend of the image group for April hereinafter.

[0137] Then, the processor 10A may select material images for May based on the trends in April and the trends of the image group in April. In this case, the images selected as material images for May may include images having characteristics that are in line with the trends of the image group in April for the priority items, in addition to images having characteristics that are in line with the trends of the image group in April for the priority items.

[0138] 11 as a specific example, for the priority item "subject person," the image group for April includes images of children, images of mothers, and images of fathers in equal proportions. Therefore, the images of children, images of mothers, and images of fathers correspond to images having characteristics that conform to the trends of the image group for April for the priority item. In this case, processor 10A may include images of the mother or images of the father in the material images for May, along with images of children.

[0139] As described above, if images with characteristics that match the trends of the April image group in terms of priority items are selected as stock images for May, stock images for May (the current month) can be selected taking into account the shooting trends of April (the previous month). This allows images that are important to the user to be identified from the shooting trends of the previous month, and stock images for the current month can be selected based on this.

[0140] The above has explained the flow of the material selection process for selecting material images for May, but the same procedure applies to the material selection process for selecting material images for months after June. For example, the material images for June are selected from the June image group based on the trends in the characteristics of the April material images and the trends in the characteristics of the May material images. In this case, the April material images and the May material images correspond to the first images used in creating the commercial material, the June image group corresponds to the second image group, and the June material images correspond to the second images used in creating the commercial material.

[0141] For example, if the group of images for June includes an image of grandparents, but this image was not included in either the stock images for April or May, the image with the grandparents in it may be selected as the stock image for June because it has characteristics that deviate from the trends in the characteristics of both the stock images for April and the stock images for May.

[0142] <<Re-creation process>> If the re-creation conditions are met after the first creation process for creating a first product for a certain month is executed, the processor 10A executes a process for re-creating the first product for that month. In other words, in this embodiment, even if the first product F1 is created, if the re-creation conditions are met, the first product F1 can be recreated as needed. The flow of the re-creation process will be described below with reference to FIG. 14.

[0143] The re-creation process is executed when there is a month that satisfies the re-creation conditions, i.e., the target month. For example, in the month following the target month, processor 10A executes the re-creation process to re-create the first product for the target month. Whether the re-creation conditions are met is determined by whether there is an order for the first product and whether there is a save request.

[0144] Specifically, when processor 10A has not received either an order or a storage request for a first product for a certain month, processor 10A sets that month as a target month and executes a re-creation process to re-create the first product for the target month. In the re-creation process, as shown in Fig. 14, processor 10A first executes a re-selection process to select material images for the target month (S031). In the re-selection process, processor 10A selects material images to be used in re-creating the first product for the target month from a group of images taken in the target month (a group of images for the target month).

[0145] In the reselection process, processor 10A identifies the characteristics of images included in a group of images taken during a reselection period including the target month, and identifies trends in the characteristics of the group of images. The procedure for identifying image characteristics and trends in the reselection process is the same as the feature identification process and trend identification process in the material selection process in the first creation process. The reselection period is a period that begins in the month before the target month and continues to the month after the target month. The month before the target month is, for example, the month (first month) when the use of the product provision service begins, and the month after the target month is the month in which the re-creation process is executed (specifically, the current time).

[0146] The material images used to recreate the first commercial material for the target month are selected taking into consideration the group of images taken in months after the target month. In other words, when recreating the first commercial material for the target month, the material images can be selected taking into consideration the user's photography habits in the period after the target month. This makes it possible to properly understand the user's photography habits over a longer period of time and recreate the first commercial material for the target month taking into consideration those photography habits.

[0147] More specifically, the longer the reselection period, i.e., the closer to the final month of the year, the greater the number of image groups acquired by the image processing device 10 per month. Furthermore, the greater the number of image groups, the more accurately the user's photography habits can be identified based on the image groups for each month. For example, a person not appearing in the image group for a target month may frequently appear in the image group for the month following the target month. Furthermore, even if the number of images of a certain person in the image group for the target month is small, images of that person may frequently appear in the image group for the month following the target month. Such a person may be important to the user, and this fact becomes apparent after the target month. By referring to the image group captured during the reselection period, the above facts can be easily recognized. As a result, material images for recreating the first commercial material for the target month can be more accurately selected.

[0148] After executing the reselection process, the processor 10A selects material images from the group of images for the target month based on the tendency of the characteristics of the group of images taken during the reselection period. Then, the processor 10A recreates the first commercial material for the target month using the selected material images (S032). The flow after the first commercial material is recreated in the recreating process is the same as the flow after the first commercial material is created in the first creation process. Specifically, the processor 10A generates a data file of the recreated first commercial material for the target month and transmits the data file to the user-side device 12. As a result, the recreated first commercial material for the target month is displayed on the display of the user-side device 12.

[0149] The user looks at the display, checks the recreated first product for the target month, and considers whether to place an order (S033). When the user places an order, the processor 10A accepts the order (S034) and executes a process to provide the recreated first product for the target month to the user, specifically, a printing process, etc. (S035). Upon accepting the order, the processor 10A confirms the recreated first product for the target month.

[0150] The user also considers whether or not to save the data of the recreated first commercial material for the target month (S036). If the user makes a save request, the processor 10A accepts the save request (S037) and saves the recreated data file of the first commercial material for the target month in the storage device 14 (S038).

[0151] The series of steps related to the re-creation process described above is repeatedly performed while the re-creation conditions for the target month are satisfied. On the other hand, if the re-creation conditions are no longer satisfied, that is, if processor 10A receives an order or save request for the re-created first product, the re-creation process for the first product will no longer be executed.

[0152] <<About the second creation process>> Next, the flow of the second creation process in the product provision flow will be described with reference to Fig. 15. Each step in the second creation process is executed by the processor 10A of the computer that constitutes the image processing device 10. Before the second creation process is executed, the processor 10A creates a first product F1 for each month of the 12 months, including the final month.

[0153] In the second creation process, first, processor 10A identifies, for each month of the 12 months' worth of first commercial materials, the material images used to create first commercial material F1 (S041). In this step S041, when identifying the material images of first commercial material F1 for the month for which the save request was made, processor 10A reads the data file of that first commercial material F1 from storage device 14. Then, processor 10A identifies the material images of that first commercial material from the read data file.

[0154] Furthermore, if the first commercial material for the target month is recreated, the processor 10A identifies the material image of the recreated first commercial material. Note that if the first commercial material for the target month has been recreated multiple times, the processor 10A identifies the material image of the latest first commercial material (i.e., the first commercial material recreated most recently).

[0155] Next, the processor 10A determines whether or not an order for the first product has been received from the user for each month (S042). Specifically, the processor 10A determines whether or not an order for the first product has been received for each month by referring to information indicating the user's order history.

[0156] Thereafter, processor 10A creates a second product based on the material image of the first product for each month and whether or not there has been an order for the first product for each month (S043). Specifically, processor 10A creates each of the multiple pages included in the second product using the material image of the first product for the month corresponding to each page. For example, if the Xth page of the second product corresponds to May, processor 10A creates the Xth page using the material image of the first product for May.

[0157] Furthermore, processor 10A sets the image layout for each page depending on whether or not there is an order for the first product for each month. Specifically, processor 10A sets the layout for each page so that material images of the first product for the order month (the month in which the order for the first product was placed) are preferentially arranged. This is because there is a high possibility that an important event or the like occurred in the order month for the user, and it is presumed that images taken in the order month are of high importance.

[0158] The method for preferentially handling material images of the first product of the order month in the second product is not particularly limited. For example, the image size on each page of the second product may be set to one of large, medium, small, and index display size (minimum size). In this case, on the page of the second product corresponding to the order month, material images of the first product of the order month may be arranged in large size, as shown in FIG. 16A. On the other hand, material images of the first product of the unordered month may be arranged in index display size on the final page of the second product, as shown in FIG. 16B. In addition, in FIGS. 16A and 16B, the size of the images on the pages corresponding to the ordered months and the pages corresponding to the unordered months are indicated by dashed lines.

[0159] The second product F2 is completed through the above series of steps. After that, if there is an order for the second product from the user, the processor 10A accepts the order (S044) and executes a process for providing the second product F2 to the user, such as a printing process (S045). When the series of steps described above is completed, the second creation process ends.

[0160] <<Other embodiments>> The embodiment described above is a specific example given to clearly explain the image processing device and image processing method of the present invention, and is merely an example, and other embodiments are also possible.

[0161] In the above embodiment, the first product F1 is created monthly. However, the cycle of creating the first product is not particularly limited, and the first product may be created daily, weekly, or annually, or may be created every few days, weeks, months, or years. In the above embodiment, the second product F2 is produced on an annual basis. However, the production cycle of the second product is not particularly limited as long as multiple first products are produced within that period. For example, the second product may be produced daily, weekly, or monthly, or may be produced every few days, weeks, months, or years.

[0162] In the above embodiment, in the material selection process for selecting material images for the current month, material images are selected based on the tendency of the characteristics of material images for the previous month. However, this is not limited to this, and material images for the current month may be selected based on the tendency of the characteristics of material images for the month prior to the previous month, for example, two months or more prior to the current month.

[0163] In the above embodiment, the functions of the image processing device of the present invention are performed by a processor provided in the server computer, but this is not limited to this. The processor provided in the image processing device of the present invention may be provided in both the server computer and the user-side device 12. In other words, some of the functional units provided in the image processing device 10 described above may be provided in the user-side device 12.

[0164] The processor included in the image processing device of the present invention includes various types of processors, including, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include PLDs (Programmable Logic Devices), which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays). Furthermore, various processors include dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits), which are processors having circuit configurations designed specifically for performing specific processes.

[0165] Furthermore, one processing unit of the image processing device of the present invention may be configured by one of the various processors described above, or by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Furthermore, the multiple functional units of the image processing device of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be combined into one processor. Furthermore, as in the above-described embodiment, one processor may be configured by combining one or more CPUs and software, and this processor may function as multiple functional units.

[0166] Also, for example, a processor may be used that realizes the functions of the entire system including multiple functional units in the image processing device on a single IC (Integrated Circuit) chip, as typified by an SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0167] 10 Image processing device 10A processor 10B memory 10C communication interface 10D printer 12 User side equipment 14 Storage device 21 Image acquisition unit 22 Preliminary Selection Department 23 Material Selection Department 24 Editorial Department 25 1st Product Creation Department 26 Recommendation Department 27 Order Reception Department 28 Request Reception Department 29 Re-selection department 30 Recreation Section 31 Secondary Product Creation Department F1 First Product F2 Secondary Product N Network S Product Provision System

Claims

1. An image processing device including a processor, The processor: a first acquisition process for acquiring a first group of images captured in a first period; a second acquisition process for acquiring a second group of images captured during a second period different from the first period; a feature identification process for identifying features relating to items related to photography or image content for a first image selected from the first image group and a second image included in the second image group; a trend identification process for identifying a trend of the features of the selected first image for the item; a selection process for selecting a second image from the second image group based on a feature of the second image included in the second image group; In the selection process, the second image having a feature that deviates from the trend for the item is selected.

2. the selected first image and the second image selected in the selection process are images for creating commercial materials, In the trend identification process, the processor calculates a first trend value that quantifies the trend, The image processing device according to claim 1 , wherein in the selection process, the processor selects the second image based on the first tendency value.

3. 3. The image processing device according to claim 2, wherein in the selection process, the processor selects the second images so that a difference between a second trend value, which quantifies a trend of a feature of the second image to be selected, and the first trend value is equal to or greater than a set value.

4. the items include priority items set according to the tendency and non-priority items other than the priority items, The image processing device according to claim 1 , wherein in the selection process, the processor selects the second image that has characteristics that are in line with the trend for the priority items and that have characteristics that are out of line with the trend for the non-priority items.

5. In the trend identification process, the processor identifying the tendency by dividing the selected first images into a plurality of categories based on the characteristics of the selected first images and calculating a first ratio of the first images belonging to each category for each category; for the item, dividing the second images included in the second image group into the plurality of categories based on features of the second images included in the second image group, and calculating a second ratio of the second images belonging to each category for each category; The image processing device according to claim 4 , wherein in the selection process, the processor selects a second image that belongs to the category in which the second ratio is greater than the first ratio for the non-priority item.

6. In the feature identification process, the processor further identifies features related to the item for a first image included in the first image group; In the trend identification process, the processor identifies, for the item, the trend of the feature of the selected first image and another trend that is a trend of the feature of the first image included in the first image group; The image processing device according to claim 4 , wherein in the selection process, the processor includes the second images having characteristics that are in line with the other tendencies for the priority items in the second images to be selected.

7. the processor repeatedly executes a first creation process for creating a first commercial product for a period corresponding to the second period by using the second images selected by the selection process from the second image group taken during the period corresponding to the second period, while changing the period corresponding to the second period; the first creation process includes a material selection process of selecting, from the second image group taken during a period corresponding to the second period, the second images to be used in creation of the first commercial material for the period corresponding to the second period; In the material selection process, the processor executes the feature identification process, the trend identification process, and the selection process, When the first creation process is executed with a certain period as the second period, the material selection process is executed with a period prior to the certain period as the first period; 7. An image processing device according to claim 1, wherein, after the first creation process is executed with the certain period as the period corresponding to the second period, if the first creation process is executed with a period after the certain period as the period corresponding to the second period, the material selection process is executed with a period before the certain period as the period corresponding to the first period.

8. the first creation process includes a layout determination process for determining a layout of images in the first commercial product based on a layout editing operation by a user; 8. The image processing device of claim 7, wherein, in the first creation process for creating the first product for the second period, the processor proposes to the user a layout of images in the first product for the first period, and if the user adopts the proposed layout, creates the first product for the second period using the proposed layout.

9. the processor executes the first creation process to create the first commercial material for a period corresponding to the second period, and then, if a recreate condition is satisfied, executes a recreate process to recreate the first commercial material for a period corresponding to the second period; The image processing device described in claim 7 or 8, wherein the re-creation process includes a re-selection process of selecting the second image to be used in re-creating the first product from a group of second images taken during a period corresponding to the second period.

10. the re-creation process for the period corresponding to the second period is executed in a period after the period corresponding to the second period; 10. The image processing device of claim 9, wherein in the reselection process, the processor selects the second image from the second group of images taken during a period corresponding to the second period based on trends in the characteristics of the group of images taken during two or more periods including a period corresponding to the second period and a period after the period corresponding to the second period.

11. The processor accepts a save request for the first product from a user; After the first creation process is executed, the processor: When the save request for the first product is received, the re-creation process for re-creating the first product is not executed, The image processing device according to claim 9 , further comprising: a processor configured to execute the re-creation process for re-creating the first commercial product when the save request for the first commercial product has not been received.

12. The processor accepts a user order for the first product; After the first creation process is executed, the processor: When the order for the first product is accepted, the re-creation process for re-creating the first product is not executed, The image processing device according to claim 9 , further comprising: a processor configured to execute the re-creation process for re-creating the first product when the order for the first product has not been accepted.

13. The processor further executes a second creation process to create second products for a plurality of periods; The second product is configured by assembling a plurality of second product constituent pieces, each of the plurality of second product constituent pieces corresponds to one of the plurality of periods; In the second creation process, the processor a process of identifying the second image used in creating the first commercial product for each of the first creation processes repeatedly executed while changing the period corresponding to the second period; An image processing device as described in any one of claims 9 to 12, which creates the second product by repeatedly executing a process of creating a second product component corresponding to the period corresponding to the second period using the second image used to create the first product for the period corresponding to the second period among the multiple periods, while changing the period corresponding to the second period.

14. The processor accepts a user order for the first product; An image processing device as described in claim 13, wherein in the second creation process, the processor sets the layout of the second image in each of the plurality of second product components depending on whether or not there are orders for the first product for the period corresponding to each second product component.

15. 1. A method for image processing by a processor, comprising: by the processor a first acquisition process for acquiring a first group of images captured in a first period; a second acquisition process for acquiring a second group of images captured during a second period different from the first period; a feature identification process for identifying features relating to items related to photography or image content for each of the first images selected from the first image group and each of the second images included in the second image group; a trend identification process for identifying a trend of the features of the selected first image for the item; a selection process of selecting a second image from the second image group based on a feature of the second image included in the second image group; In the selection process, the second image having a feature that deviates from the trend for the item is selected.

16. the selected first image and the second image selected in the selection process are images for creating commercial materials, In the trend identification process, the processor calculates a first trend value that quantifies the trend, The image processing method according to claim 15 , wherein in the selection process, the processor selects the second image based on the first tendency value.

17. 17. The image processing method according to claim 16, wherein in the selection process, the processor selects the second images so that a difference between a second trend value, which quantifies a trend of a feature of the second image to be selected, and the first trend value is equal to or greater than a set value.

18. the items include priority items set according to the tendency and non-priority items other than the priority items, 18. An image processing method according to claim 15, wherein in the selection process, the processor selects the second image that has characteristics that are in line with the trend for the priority items and characteristics that are out of line with the trend for the non-priority items.

19. In the trend identification process, the processor: identifying the tendency by dividing the selected first images into a plurality of categories based on the characteristics of the selected first images and calculating a first ratio of the first images belonging to each category for each category; for the item, dividing the second images included in the second image group into the plurality of categories based on features of the second images included in the second image group, and calculating a second ratio of the second images belonging to each category for each category; The image processing method according to claim 18 , wherein in the selection process, the processor selects a second image that belongs to the category in which the second ratio for the non-priority item is increased more than the first ratio.

20. In the feature identification process, the processor further identifies features related to the item for a first image included in the first image group; In the trend identification process, the processor identifies, for the item, the trend of the feature of the selected first image and another trend that is a trend of the feature of the first image included in the first image group; The image processing method according to claim 15 , wherein in the selection process, the processor selects the second image based on the tendency and the other tendency.

21. A program for causing a computer to execute each process included in the image processing method according to any one of claims 15 to 20.

22. A computer-readable recording medium, 21. A recording medium having recorded thereon a program for causing a computer to execute each process included in the image processing method according to any one of claims 15 to 20.

Images