Image processing apparatus, image processing method, program, and recording medium

Abstract

To provide an image processing device, method, a program, and a recording medium which can properly select an image for commercial material creation from an image group provided in each period from a user.SOLUTION: In an image processing device according to the present invention, a processor acquires a first image group captured in a first period, acquires a second image group captured in a second period different from the first period, specifies a feature about two or more items related to imaging for a first image used in commercial material creation in the first image group and a second image included in the second image group, specifies the tendency of a feature of the first image used in commercial material creation for the two or more items, and selects the second image for commercial material creation from the second image group on the basis of a feature of the second image included in the second image group. The processor selects the second image having a feature deviating from the tendency for some items in the two or more items as the second image for commercial material creation.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium on which the program is recorded, for selecting an image for creating a commercial material from a group of images.

Background Art

[0002] When using a service that provides commercial materials such as photo books, a user, who is the person using the service, provides a plurality of images (image group) to be used as materials for the commercial materials. That is, the commercial material is created using an image selected from the image group provided by the user.

[0003] In the above service, it is conceivable to create a commercial material every predetermined period (for example, every month or year). In this case, the process of selecting a material image from the image group provided by the user and the process of creating a commercial material using the selected image are repeatedly executed every certain period. As a result, the user can obtain an image group for each period.

[0004] By the way, when selecting an image for creating a commercial material, that is, a material image, from the image group provided by the user, it is desirable to select, for example, an image important to the user or an image that conforms to the user's preference. As a technique for selecting a material image from the image group provided every period, the technique described in Patent Document 1 can be cited.

[0005] The image processing apparatus described in Patent Document 1 (hereinafter, the conventional image processing apparatus) sets a criterion for image selection based on a first image selected from a first image group and a second image selected from a second image group. Then, when selecting an image from the input target image group, the conventional image processing apparatus selects an image based on the above criterion. With such a configuration, the conventional image processing apparatus can select a preferable image for the user from the target image group while maintaining the commonality with the first image and the second image selected in the past.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2020-46901 [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] The criteria for selecting a material image set by a conventional image processing apparatus are to select an image having a theme (feature) commonly obtained from a plurality of first images and a plurality of second images. According to this criterion, for example, when a child is shown in both the first image and the second image, an image in which the child is shown is selected from the target image group as an image having a common theme with these.

[0008] On the other hand, when selecting a material image from among the image groups provided for each period, if only images having a common theme are selected, the images included in each commercial material will not look distinctive among the commercial materials. As a result, even if commercial materials are provided for each period, each commercial material will become monotonous, that is, lacking in variety.

[0009] An object of the present invention is to provide an image processing apparatus, method, program, and recording medium capable of appropriately selecting an image for creating a commercial material from an image group provided for each period from a user. [Means for Solving the Problems]

[0010] In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus including a processor. The processor executes a first acquisition process of acquiring a first image group captured in a first period, a second acquisition process of acquiring a second image group captured in a second period different from the first period, a feature identification process of identifying features regarding two or more items related to shooting for a first image used for product material creation among the first image group and a second image included in the second image group, a tendency identification process of identifying a tendency of features of the first image used for product material creation regarding the two or more items, and a selection process of selecting a second image for product material creation from among the second image group based on the features of the second image included in the second image group. In the selection process, the second image having features deviating from the tendency regarding some of the two or more items is selected as the second image for product material creation.

[0011] Further, in the tendency identification process, the processor may obtain a first tendency value obtained by quantifying the tendency, and in the selection process, the processor may select the second image for product material creation based on the first tendency value. Furthermore, in the selection process, the processor may select the second image for product material creation such that a difference between a second tendency value obtained by quantifying the tendency of the features of the second image for product material creation and the first tendency value is equal to or greater than a set value.

[0012] Also, the two or more items may include a priority item set according to the tendency and a non-priority item other than the priority item. In this case, in the selection process, the processor may select, as the second image for product material creation, a second image having features following the tendency for the priority item and having features deviating from the tendency for the non-priority item. Furthermore, in the trend identification process, for each of two or more items, the processor may identify the trend by dividing the first image used for product creation into a plurality of sections based on the features of the first image used for product creation, and calculating the first ratio of the first images belonging to each section for each section. Also, for each of two or more items, the processor may divide the second images included in the second image group into a plurality of sections based on the features of the second images included in the second image group, and calculate the second ratio of the second images belonging to each section for each section. Furthermore, in the selection process, for non-priority items, the processor may select, as the second image for product creation, the second images belonging to the section in which the second ratio has increased compared to the first ratio.

[0013] Also, in the feature identification process, the processor may further identify features regarding two or more items for the first images included in the first image group. In this case, in the trend identification process, the processor may identify, for two or more items, the trend of the features of the first image used for product creation and another trend that is the trend of the features of the first images included in the first image group. Also, in the selection process, the processor may select the second image for product creation based on the trend and the other trend.

[0014] Also, the processor may execute, for each period, a first creation process of creating a first product using an image group captured in one period among a plurality of periods. Also, the first creation process may include a layout determination process of determining the layout of the images in the first product based on the user's layout editing operation. In this case, in the first creation process of creating the first product for the second period, the processor proposes to the user the layout of the images in the first product for the first period, and if the user adopts the proposed layout, the processor may create the first product for the second period with the proposed layout. Further, the first creation process may include a material selection process of selecting material images used for creating the first commercial material for the one period from among the group of images captured during the one period. In this case, after executing the first creation process of creating the first commercial material for the one period, when the recreation condition is satisfied, the processor may execute a recreation process of recreating the first commercial material for the one period. The recreation process may include a re-selection process of selecting material images used for recreating the first commercial material from among the group of images captured during the one period.

[0015] Further, the recreation process may be executed in a period after the one period. In this case, in the re-selection process, the processor may select material images from among the group of images captured during the one period based on the tendency of the characteristics of the group of images captured during two or more periods including the one period and the period after the one period. Further, the processor may receive a user's save request for the first commercial material. In this case, after executing the first creation process of creating the first commercial material for the one period, when the processor receives a save request for the first commercial material for the one period, it is preferable that the processor does not execute a recreation process of recreating the first commercial material for the one period. On the other hand, when the save request for the first commercial material for the one period has not been received, the processor may execute a recreation process of recreating the first commercial material for the one period. Further, the processor may receive a user's order for the first commercial material. In this case, after executing the first creation process of creating the first commercial material for the one period, when the processor receives an order for the first commercial material for the one period, it is preferable that the processor does not execute a recreation process of recreating the first commercial material for the one period. On the other hand, when the order for the first commercial material for the one period has not been received, the processor may execute a recreation process of recreating the first commercial material for the one period.

[0016] Further, the processor may further execute a second creation process of creating a second commercial material for a plurality of periods using the material images used for each of the first commercial materials created for each period. In this case, the second commercial material may be configured by a collection of a plurality of second commercial material components. And in the second creation process, the processor may create each of the plurality of second commercial material components using the material images used for creating the first commercial material for the period corresponding to each second commercial material component among the plurality of periods. Further, the processor may receive a user's order for the first commercial material. In this case, in the second creation process, the processor may set the layout of the material images in each of the plurality of second commercial material components according to the presence or absence of an order for the first commercial material for the period corresponding to each second commercial material component.

[0017] Further, to solve the above problems, an image processing method of the present invention is an image processing method by a processor, and includes: a first acquisition process of acquiring a first image group captured in a first period by the processor; a second acquisition process of acquiring a second image group captured in a second period different from the first period by the processor; a feature identification process of identifying features regarding two or more items related to shooting for each of a first image used for creating a commercial material among the first image group and a second image included in the second image group; a trend identification process of identifying a trend of the features of the first image used for creating a commercial material for two or more items; and a selection process of selecting a second image for creating a commercial material from the second image group based on the features of the second image included in the second image group. In the selection process, a second image having features deviating from the trend for some of the two or more items is selected as the second image for creating a commercial material.

[0018] Further, according to the present invention, a program for causing a computer to execute each process included in the above image processing method is realizable. Furthermore, according to the present invention, a computer-readable recording medium in which a program for causing a computer to execute each process included in the image processing method of the present invention is recorded is realizable.

Effect of the Invention

[0019] According to the present invention, it is possible to appropriately select an image for creating a commercial material from a group of images provided by a user for each period. Thereby, it is possible to avoid the commercial materials created for each period from becoming monotonous.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Embodiments for Carrying Out the Invention

[0021] Specific embodiments of the present invention (hereinafter, these embodiments) will be described with reference to the drawings. However, the embodiments described below are merely examples given to facilitate the understanding of the present invention and do not limit the present invention. Further, the present invention can be changed or improved from the following embodiments without departing from its gist. Also, equivalents of the present invention are included.

[0022] In this specification, unless otherwise specified, "image" shall mean image data. Examples of image data include non-reversible compression image data such as in the JPEG (Joint Photographic Experts Group) format, and reversible compression image data such as in the GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format.

[0023] In this specification, the concept of "device" includes a single device that exhibits a specific function by itself, and also includes a plurality of devices that are distributed, exist independently of each other, and cooperate (collaborate) to exhibit a specific function.

[0024] <<Regarding the Commercial Material Provision Service of This Embodiment>> Prior to the description of the image processing apparatus and the image processing method according to this embodiment, the commercial material provision service implemented using these will be described.

[0025] The merchandise providing service is a service that creates merchandise using a plurality of images provided by a user who is a service user, and provides the merchandise to the user. The merchandise is, for example, a photo book or an album created using images provided by the user. The type of merchandise is not particularly limited, and may be merchandise other than the above types. For example, it may be a calendar or a collage print image obtained by synthesizing an image provided by the user and a decorative image. Also, the method of providing the merchandise is not particularly limited, and data for displaying the merchandise on a display or the like may be provided to the user, or the merchandise may be printed and provided to the user in the form of a printed matter.

[0026] Normally, a plurality of images are required to create the merchandise. Therefore, the user provides images taken by a photographing device such as a digital camera at any time or regularly. That is, when obtaining the merchandise using the merchandise providing service, the user provides an image group including a plurality of images.

[0027] To outline the procedure for creating the merchandise, first, a process of selecting one or more images from the image group provided by the user as images for creating the merchandise (hereinafter also referred to as material images) is performed. Next, a process of creating the merchandise using the material images selected from the image group is performed. In this process, the layout of the material images in the merchandise is determined. The image layout in the merchandise refers to the number of images used in the merchandise, the arrangement position of the images, and the size of the images.

[0028] The merchandise created using the material images is provided to the user according to an order or the like by the user. In this embodiment, two types of merchandise can be provided. One merchandise is a first merchandise that is periodically and repeatedly provided over a plurality of periods. The first merchandise is an edited image created by editing one or more material images. Specifically, it is a photo book-type merchandise shown in FIG. 1 (denoted by reference numeral F1 in the figure).

[0029] In this embodiment, the first commercial material F1 is created at regular intervals, specifically, monthly. For example, when using the commercial material providing service for one year, the first commercial material F1 for each month is created, that is, the first commercial material F1 for 12 months in one year is provided to the user. Thereby, the user can organize the images taken each month on a monthly basis, and can also leave important images in the form of commercial materials. Since the first commercial material F1 can be provided at a lower cost than the second commercial material F2 described later, it is a commercial material that is easy for the user to use (easy to purchase).

[0030] Here, the plurality of months included in the service usage period correspond to the "plurality of periods" of the present invention, and each month (that is, one month) corresponds to "one period". In the following, for convenience, the first commercial material F1 created for a certain month will be referred to as the "first commercial material of a certain month".

[0031] The first commercial material F1 for each month is created using the material images selected from the image group taken in that month (the current month). In other words, when creating the first commercial material F1 for each month, the user provides the image group taken by the user in that month. For example, when creating the first commercial material for April, the image group taken in April by the user is provided by the user. The shooting date of each image included in the image group can be specified from the attached information of the image such as the tag information of Exif (Exchangeable image file format). In the following, for convenience, the image group taken by the user in a certain month will be referred to as the "image group of a certain month".

[0032] As described above, in this embodiment, the process of creating the first commercial material F1 is executed monthly. Specifically, the process of acquiring the image group provided by the user, the process of selecting the material images from the image group, and the process of determining the layout of the material images in the first commercial material are repeated on a monthly basis. Thereby, the first commercial material F1 can be provided monthly. The user can check the first commercial material F1 of the current month every month during the usage period of the commercial material providing service, and can order the desired first commercial material. The first commercial material of the ordered month (order month) is provided to the user.

[0033] In addition, the user can make a storage request for the first commercial material each month on a monthly basis. The first commercial material for which the storage request has been made, specifically, the material image used for creating the first commercial material to be stored and the image layout in the first commercial material to be stored are stored as data files in a predetermined storage destination.

[0034] Another commercial material is the second commercial material that records images for 12 months provided by the user. Specifically described, the second commercial material is a composite image created using the material images used for each of the first commercial materials for 12 months. More specifically, it is an annual album-type commercial material (denoted as F2 in the figure) as shown in FIG. 2.

[0035] In the present embodiment, the second commercial material F2 is created every 12 months, that is, on a yearly basis, and as shown in FIG. 2, a plurality of pages are assembled and configured. The plurality of pages included in the second commercial material F2 correspond to a plurality of second commercial material constituent pieces. In addition, each page of the second commercial material F2 is configured by the material image used to create the first commercial material of the month corresponding to that page. For example, in the second commercial material F2, the material image used to create the first commercial material for April is arranged on the page corresponding to April. By providing the second commercial material F2 that records the images used for creating the first commercial material for each month as described above for 12 months, important images among the images taken by the user over the course of one year can be stored as an annual album.

[0036] The second commercial material F2 is created at the time when a plurality of periods (specifically, 12 months) have elapsed, for example, at the end of the year. More specifically, it is created after the first commercial material for the last month is created. The second commercial material F2 created from the material images of the first commercial materials for 12 months is provided to the user who ordered the second commercial material F2.

[0037] <<Regarding the configuration of the image processing apparatus according to the present embodiment>> The above-described merchandise offering service is realized by a merchandise offering system S shown in FIG. 3. The merchandise offering system S is composed of an image processing apparatus (hereinafter, image processing apparatus 10) according to the present embodiment and a user-side device 12 capable of communicating with the image processing apparatus 10 via a network N.

[0038] (User-side device) The user-side device 12 is constituted by, for example, a PC (Personal Computer) used by a user to utilize the merchandise offering service, a communication terminal, a camera with a communication function, or the like. On the display of the user-side device 12, an image provided to the image processing apparatus 10, information based on data provided from the image processing apparatus 10, and the like can be displayed. The information based on the data provided from the image processing apparatus 10 includes image information of the first merchandise F1 or the second merchandise F2.

[0039] In addition, a predetermined application program (hereinafter, service application) is installed in the user-side device 12. The user can utilize the merchandise offering service by starting the service utilization application. Specifically described, the user can perform various operations for service utilization through the user-side device 12 in a state where the service utilization application is started. Operations for service utilization include, for example, a service application operation, an image input operation, a layout editing operation, an order operation, and a save request operation.

[0040] Note that the procedure of each operation and the GUI (Graphic User Interface) such as an operation screen are not particularly limited. As the GUI for the layout editing operation, for example, a layout editing screen shown in FIG. 4 can be mentioned.

[0041] The layout editing screen is drawn on the display of the user-side device 12, and the image selected as the image for creating the commercial material (i.e., the material image) is displayed on the editing operation screen. Through this screen, the user can perform layout editing operations on the first commercial material. For example, the user clicks on each material image on the screen and drags the material image to the desired position on the screen. Thereby, the user can set and change the position or size of each material image in the first commercial material, that is, the image layout.

[0042] Note that there is an initial setting (default) for the image layout in the first commercial material, and the user performs a layout editing operation when, for example, changing the image layout from the initial setting. Also, for the image layout, several patterns may be prepared in advance as shown in FIG. 5. In this case, when performing the layout editing operation, the user may select any one of the above-mentioned plurality of patterns. FIG. 5 illustrates four patterns represented by #1 to #4, and for each pattern, the arrangement position and size of the material images in the first commercial material are shown by broken lines.

[0043] Also, in the layout editing operation, the user may be able to select the material image actually used for creating the commercial material from among the candidates of the material images selected by the image processing apparatus 10.

[0044] (Image Processing Apparatus) The image processing apparatus 10 is composed of a computer, for example, a server computer. The image processing apparatus 10 executes a series of data processing related to the commercial material providing service. Note that the number of computers constituting the image processing apparatus 10 may be one or two or more. The image processing apparatus 10 is realized by a processor and a program executable by the processor, and is composed of, for example, a general-purpose computer. For example, as shown in FIG. 3, the computer constituting the image processing apparatus 10 includes a processor 10A, a memory 10B, a communication interface 10C, and the like.

[0045] The processor 10A is constituted by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or the like. The memory 10B is constituted by, for example, semiconductor memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The communication interface 10C is constituted by, for example, a network interface card or a communication interface board.

[0046] A printer 10D as an output device may be connected to the image processing apparatus 10. The printer 10D receives print data of a commercial material created by the image processing apparatus 10 and operates according to the print data to print (output) the commercial material.

[0047] A program (hereinafter referred to as a commercial material providing program) for executing a series of processes related to a commercial material providing service is installed in the computer constituting the image processing apparatus 10. The commercial material providing program is a program for causing a computer to execute each step included in the image processing method of the present invention. That is, when the processor 10A reads and executes the commercial material providing program, the computer including the processor 10A functions as the image processing apparatus of the present invention.

[0048] The commercial material providing program may be obtained by being read from a computer-readable recording medium, or may be obtained by being received (downloaded) through a communication network such as the Internet or an intranet.

[0049] As shown in FIG. 3, the commercial material providing system S is provided with a storage device 14 that stores an image group acquired by the image processing device 10 from the user-side device 12. An image group is stored in the storage device 14. The image processing device 10 is configured to be able to freely read out images in the image group stored in the storage device 14. In addition, the user who is the provider of the image group and the year and month when the images included in the image group were taken are associated with the image group stored in the storage device 14.

[0050] The storage device 14 may be constituted by a storage built in or externally attached to the image processing device 10. Alternatively, a third computer (for example, an external server) capable of communicating with the image processing device 10 may constitute the storage device 14.

[0051] The configuration of the image processing device 10 will be described again from the functional aspect. As shown in FIG. 6, the image processing device 10 includes an image acquisition unit 21, a preliminary selection unit 22, a material selection unit 23, an editing unit 24, a first commercial material creation unit 25, a recommendation unit 26, an order reception unit 27, a request reception unit 28, a re-selection unit 29, a re-creation unit 30, and a second commercial material creation unit 31. These functional units are realized by the cooperation of the hardware devices provided by the computer constituting the image processing device 10 and the software including the aforementioned commercial material providing program.

[0052] The image acquisition unit 21 acquires an image group provided by the user. Specifically, the user can operate the user-side device 12 to specify an image to be provided to the image processing device 10 among the images taken by the user and transmit the specified image to the image processing device 10. The image acquisition unit 21 acquires the image group provided by the user by receiving the image transmitted from the user-side device 12. The image group acquired by the image acquisition unit 21 is stored in the storage device 14.

[0053] In addition, when the user-side device 12 is a photographing device having a communication function, the image acquisition unit 21 may directly receive the images transmitted from the user-side device 12 to acquire an image group. Alternatively, when the user-side device 12 is a PC or the like equipped with image editing software and captures images from a photographing device, the image acquisition unit 21 may acquire the image group captured by the user-side device 12. Further, when the image group input from the user-side device 12 is temporarily stored in a server or the like on the network, the image acquisition unit 21 may acquire the image group from the server.

[0054] In addition, the image acquisition unit 21 acquires the image group of each month in each month. That is, the image acquisition unit 21 acquires the image group of each month on a monthly basis. The timing of acquiring the image group of each month is not particularly limited. For example, each time the user takes a picture, the captured image is input at any time, and each time an image is input, the image acquisition unit 21 may acquire the input image. Alternatively, the user may input images together at regular intervals (for example, every month), and in that case, the image acquisition unit 21 may acquire the images (image group) for one period input every month all at once.

[0055] Here, a certain month during the usage period of the commercial product providing service and a month different from that month respectively correspond to the first period and the second period of the present invention. In addition, in the present embodiment, the second period is a period after the first period, for example, the month following the month corresponding to the first period. The image acquisition unit 21 acquires the image group of the month corresponding to the first period, and then acquires the image group of the month corresponding to the second period. The image group of the month corresponding to the first period corresponds to the first image group of the present invention, and the images included in the first image group correspond to the first images. Further, the image group of the month corresponding to the second period corresponds to the second image group of the present invention, and the images included in the second image group correspond to the second images.

[0056] The first period and the second period are relative concepts of time and change with the passage of time. For example, in the relationship between April and May, April corresponds to the first period and May corresponds to the second period. Also, in the relationship between April, May, and June, each of April and May corresponds to the first period and June corresponds to the second period. Note that the months corresponding to the first period are not limited to the month immediately before the month corresponding to the second period, and may be months before that month. For example, when May is the second period, February or March may be the first period. Also, the months corresponding to the first period may be multiple months (two or more months). For example, when May is the second period, March and April may be the first period. Also, the second period may be a period before the first period. For example, when May is the first period, April may be the second period.

[0057] The preliminary screening unit 22 executes a preliminary screening process for excluding images that are inappropriate as material images (hereinafter referred to as images to be excluded) from the image group acquired by the image acquisition unit 21. Whether an image corresponds to an image to be excluded is determined according to the presence or absence of a person's face in the image, the expression of the face, the size of the face in the image, the degree of blur and shake in the image, and the color and brightness of the subject, etc.

[0058] The material selection unit 23, the editing unit 24, the first commercial material creation unit 25, and the recommendation unit 26 execute a process of creating the first commercial material (hereinafter referred to as the first creation process). The first creation process is a process of creating the first commercial material for each month, and is executed monthly, for example, on a day corresponding to a predetermined date of each month.

[0059] The material selection unit 23 executes a process of selecting an image used for creating the first commercial material, that is, a material image (hereinafter referred to as a material selection process). The material selection process is a process included in the first creation process and is executed monthly. In the material selection process for each month, the material selection unit 23 selects a material image from the image group acquired by the image acquisition unit 21 in that month, that is, the image group taken in that month.

[0060] Regarding the material selection process, a case of creating the first commercial material for a certain month (hereinafter referred to as month P) will be specifically described as an example. In the material selection process for month P, from the image group of month P, strictly speaking, from the image group of month P from which the images to be excluded selected by the preliminary screening unit 22 have been removed, the material image used for creating the first commercial material for month P is selected.

[0061] In the material selection process, the material selection unit 23 first identifies features regarding two or more items for the images included in the image group of month P. The two or more items are two or more items (themes) regarding the shooting of the images. Examples of the items include the subject person, the ratio of the subject person in the image, the part of the person shown in the image, other subjects shown together with the person, the shooting scene, the shooting location, the shooting time period, the perspective of the image, and the like.

[0062] The features are the specific content and nature of the images regarding the items. Examples of the features regarding the above items include the following features. · Subject person: the relationship of the persons (child, relative, friend, etc.) and the number of persons in the image, etc. · Ratio of the subject in the image: the occupancy rate (%) of the subject in the image · Other subjects shown together with the person: the type of the subject and the size of the subject, etc. · Shooting scene: the classification of the shooting scene and the content of the event shown in the image, etc. · Shooting location: the name of the place, the name of the building, and whether the shooting location is indoor or outdoor, etc. · Shooting time period: the shooting time (morning, noon, evening or night) and the season when the image was shot, etc. · Perspective: whether it is a close-up or a long shot

[0063] The method for identifying the features regarding the items is not particularly limited. For example, known image analysis techniques may be used or they may be automatically identified. Specifically, a known face detection technique may be applied to detect the faces of the persons in the image, and the detected faces may be compared with the pre-registered face images to identify the persons whose faces are detected.

[0064] Alternatively, after detecting a subject in an image by applying a known object detection technique, the type of the detected subject and the like may be estimated using an estimation engine constructed by machine learning. Or, when analyzing an image group provided by a user, if there are a predetermined number or more images in which the same person appears, it may be inferred that the person is an important person (such as a family member or a friend) to the user. Further, the relationship between the person shown in the image and the user may be inferred from the shooting location, shooting time, shooting scene, etc. of the image.

[0065] Alternatively, information regarding the features of the target image may be input by the operator of the image processing apparatus 10 or the user who provides the image, and the features of the target image may be specified based on the input information.

[0066] When the material selection unit 23 selects a material image to be used for creating the first commercial material of month P from among the image group of month P, the material image is selected according to the features of the images included in the image group of month P.

[0067] Also, the material selection unit 23 selects a material image to be used for creating the first commercial material of month Q, which is a month after month P, for example, the next month of month P, from among the image group of month Q. At this time, the material selection unit 23 selects a material image to be used for creating the first commercial material of month Q according to the features of the images included in the image group of month Q and the features of the material image used for creating the first commercial material of month P.

[0068] Hereinafter, the material image used for creating the first commercial material of month P will be referred to as the "material image of month P", and similarly, the material image used for creating the first commercial material of month Q will be referred to as the "material image of month Q". Here, month P and month Q are in the relationship of the aforementioned first period and second period, the image group of month P corresponds to the aforementioned first image group, and the image group of month Q corresponds to the aforementioned second image group.

[0069] Furthermore, in the present embodiment, the material selection unit 23 can select a material image of month Q so that a difference occurs between the material image of month P and the material image of month Q in terms of features.

[0070] The editing department 24 receives the user's layout editing operation. Specifically, it receives data indicating the content of the layout editing operation from the user-side device 12. The editing department 24 is configured to receive the layout editing operation each time the first creation process is executed, in other words, monthly. In this embodiment, there is an initial layout setting (default layout), and the user performs a layout editing operation when they desire a layout other than the initial setting. The layout editing operation may include an operation of selecting an image actually used for creating the first commercial material from the material images selected by the material selection unit 23.

[0071] The first commercial material creation unit 25 creates the first commercial material for each month based on the material images selected by the material selection unit 23 and the layout editing operation received by the editing department 24. Specifically, the first commercial material creation unit 25 executes a process of determining the layout of the material images in the first commercial material (hereinafter referred to as the layout determination process) based on the layout editing operation. In the layout determination process, for the material images of the first commercial material, the size, arrangement position, etc. are determined.

[0072] Then, when the image layout in the first commercial material for the current month is determined, the first commercial material F1 for the current month is completed. When the first commercial material is completed, the first commercial material creation unit 25 generates a data file for the first commercial material. The generated data file for the first commercial material is transmitted toward the user-side device 12. On the user-side device 12, an image of the first commercial material is displayed based on the received data file. Thereby, the user can confirm the first commercial material F1 for each month.

[0073] In the first creation process, the recommendation unit 26 recommends (proposes) a recommended layout to the user. The recommended layout is set by reflecting the image layout of the first commercial material created before the previous month. For example, assume that the image layout of the first commercial material in month P has been changed from the initial layout based on the user's layout editing operation. In this case, in the first creation process of creating the first commercial material in month Q, which is the month following month P, the recommendation unit 26 recommends to the user the image layout of the first commercial material in month P as the recommended layout. To explain with a specific example, assume that when creating the first commercial material in April, the user selects pattern #1 in FIG. 5. In this case, when creating the first commercial material in May, the recommendation unit 26 proposes pattern #1 to the user as the recommended layout.

[0074] The order reception unit 27 is configured to receive the user's order for the first commercial material. Specifically, when the user performs an order operation on the user-side device 12, it receives data indicating the order details from the user-side device 12. In this embodiment, since the first commercial material is created monthly, the order reception unit 27 can receive orders monthly. When the order reception unit 27 receives an order, a process (such as a printing process) for providing the ordered first commercial material to the user is executed. Also, for the ordered first commercial material F1, at the time of order reception, the material images constituting the first commercial material and the image layout in that first commercial material are determined.

[0075] The request reception unit 28 is configured to receive the user's save request for the first commercial material. Specifically, when the user performs a save request operation on the user-side device 12, it receives data indicating the request details from the user-side device 12. In this embodiment, since the first commercial material is created monthly, the request reception unit 28 can receive save requests monthly. When the request reception unit 28 receives a save request, the data file of the first commercial material for which the save request was made is associated with the user who made the save request and stored in the storage device 14.

[0076] After the first creation process for creating the first commercial material for the target month is executed, the re-selection unit 29 and the re-creation unit 30 execute a re-creation process for re-creating the first commercial material for the target month. The target month is the month that is the target of the re-creation process, that is, the month in which the created first commercial material is re-made. The re-creation process is executed when a predetermined condition (hereinafter referred to as the re-creation condition) is satisfied.

[0077] In this embodiment, when an order or a save request for the first commercial material for the target month has not been received, a re-creation process for re-creating the first commercial material for the target month is executed. In other words, when an order or a save request for the first commercial material for the target month is received, the re-creation process for re-creating the first commercial material for the target month is not executed. That is, the necessity of the re-creation process is determined according to the user's intention regarding the first commercial material, specifically, the presence or absence of an order or a save request. Then, for the first commercial material for which an order or a save request has been made, it is finalized, and the unfinalized first commercial material can be re-created.

[0078] The re-creation process for re-creating the first commercial material for the target month is executed in a month after the target month. For example, when the target month is April, the re-creation process for re-creating the first commercial material for that month (April) is executed after May. Also, unless an order or a save request for the first commercial material for the target month (including the re-created one) is received, the re-creation process for re-creating the first commercial material for the target month is repeatedly executed in each month after the target month.

[0079] The re-selection unit 29 executes a re-selection process for selecting a material image used to re-create the first commercial material for the target month. In the re-selection process, the re-selection unit 29 re-selects a material image from a group of images taken during a period of two or more months (hereinafter referred to as the re-selection period) including the target month and months after the target month. At this time, the re-selection unit 29 selects a material image based on the tendency of the characteristics of the group of images taken during the re-selection period.

[0080] The re-creation unit 30 re-creates the first commercial material for the target month using the material image selected by the re-selection unit 29. The procedure for the re-creation unit 30 to re-create the first commercial material is the same as the procedure for the first commercial material creation unit 25 to create the first commercial material.

[0081] The second product creation unit 31 executes a second creation process for creating the annual second product F2 using the material images used for creating each of the first products for 12 months. In the present embodiment, after the first creation process for creating the first product in the last month of a year is completed, the second creation process for that year is executed.

[0082] In the second creation process, the second product creation unit 31 creates each of the plurality of pages included in the second product F2 using the material images used for creating the first product F1 of the month corresponding to each page. At this time, the second product creation unit 31 sets the layout of the material images on each page according to the presence or absence of an order for the first product F1 corresponding to each page. Specifically, the second product creation unit 31 sets the image layout on each page so that the material image of the first product of the ordered month takes precedence over the material images of the first products of the unordered months without an order.

[0083] Then, when the image layout of each page is determined, the second product F2 is completed. Along with the completion of the second product, the second product creation unit 31 generates a data file of the second product. The generated data file is transmitted toward the user-side device 12. In the user-side device 12, an image image of the second product is displayed based on the received data file. Thereby, the user can confirm the second product F2.

[0084] <<Basic processing flow of the image processing apparatus>> The product providing flow, which is the basic processing flow by the image processing apparatus 10 according to the present embodiment, will be described with reference to FIG. 7.

[0085] The product providing flow is executed on a one-year cycle during the usage period of the product providing service. Each step in the product providing flow is executed by the processor 10A of the computer constituting the image processing apparatus 10. That is, the product providing flow proceeds by the processor 10A reading out the above-described product providing program and executing each step.

[0086] In the commercial material providing flow, the processor 10A executes an image acquisition process for acquiring a group of images taken by the user each month on a monthly basis (S101). In the image acquisition process, the processor 10A receives an image input from the user-side device 12 to acquire a group of images taken by the user each month. The image input from the user may be performed each time the user takes an image. Alternatively, on a predetermined day of each month, a group of images taken that month may be provided collectively.

[0087] After that, on a predetermined day of each month, the processor 10A executes a first creation process for creating the first commercial material F1 of the current month (S102). In the first creation process, the processor 10A executes a material selection process for selecting a material image from the group of images acquired in step S101 and a layout determination process for determining a layout for the selected material image.

[0088] Also, if there is a month that satisfies the recreating condition among the months before the previous month (S103), the processor 10A executes a recreating process with that month as the target month (S104). In the recreating process, the processor 10A executes a re-selection process for selecting a material image used for recreating the first commercial material of the target month from the group of images of the target month.

[0089] The above series of steps S101 to S104 are repeatedly executed monthly until the final month arrives. Among the multiple image acquisition processes executed monthly, the image acquisition process executed in a month earlier (the first period) corresponds to the first acquisition process, and the image acquisition process executed in a month later (the second period) corresponds to the second acquisition process. For example, in the relationship between April and May, the image acquisition process executed in April corresponds to the first acquisition process, and the image acquisition process executed in May corresponds to the second acquisition process.

[0090] When the final month arrives (S105), after creating the first commercial material for the final month, the processor 10A executes a second creation process (S106) for creating a second commercial material F2 as an annual album. When the second creation process ends, the execution of the commercial material provision flow for that year ends.

[0091] <<Regarding the First Creation Process>> Next, during the commercial material provision flow, the flow of the first creation process will be described with reference to FIG. 8. In the first creation process, the image processing method of the present invention is adopted. That is, each step in the first creation process described below corresponds to a component of the image processing method of the present invention. Note that the flow of the first creation process described below is merely an example, and unnecessary steps may be deleted, new steps may be added, or the execution order of the steps may be changed within the scope not departing from the gist of the present invention.

[0092] Each step in the first creation process is executed by the processor 10A of the computer constituting the image processing apparatus 10. Hereinafter, the first creation process for creating the first commercial material F1 for month P or month Q will be described as an example.

[0093] In executing the first creation process for creating the first commercial material for month P, the processor 10A executes an image acquisition process for acquiring an image group for month P from the user. Further, the processor 10A executes a preliminary selection process on the acquired image group for month P (S001). By executing the preliminary selection process, images that are inappropriate for use as material images for creating commercial materials are excluded as target images to be excluded from the image group for month P. Target images to be excluded are images that do not meet a certain image quality, such as images with blur or defocus, and images where the brightness of the subject is not sufficiently ensured.

[0094] When the processor 10A starts the first creation process for creating the first commercial material for the P month, it executes a material selection process (S002). In the material selection process, the processor 10A selects a material image (P month material image) used to create the first commercial material for the P month from among the image group for the P month from which the images to be excluded have been excluded. The detailed procedures and the like of the material selection process will be described in detail later.

[0095] Next, the processor 10A executes a layout determination process to determine the layout of the material images in the first commercial material for the P month (S003). Specifically, for each of the P month material images selected in the material selection process, the processor 10A determines the arrangement position, size, etc. in the first commercial material for the P month. Also, when the user performs a layout editing operation on the first commercial material for the P month, the processor 10A accepts the operation. In that case, the processor 10A determines an image layout according to the accepted layout editing operation.

[0096] After the layout determination process, the processor 10A creates the first commercial material for the P month based on the material images selected in the material selection process and the image layout determined in the layout determination process (S004). When the first commercial material for the P month is completed, the processor 10A generates a data file for the first commercial material and transmits it to the user-side device 12. On the user-side device 12, an image of the first commercial material for the P month is displayed on the display. The user checks the first commercial material for the P month and considers whether to place an order (S005).

[0097] When the user operates the user-side device 12 to place an order for the first commercial material for the P month, the processor 10A accepts the order (S006) and executes a process (such as a printing process) for providing the first commercial material for the P month to the user (S007). Also, along with the acceptance of the order, the processor 10A finalizes the first commercial material for the P month (specifically, the material images and image layout used to create the first commercial material for the P month).

[0098] Also, the user considers whether to save the data file of the first commercial material for month P (S008). When the user operates the user-side device 12 to make a save request, the processor 10A accepts the save request (S009) and saves the data file of the first commercial material for month P in the storage device 14 (S010).

[0099] When the series of steps described above is completed, the first creation process for creating the first commercial material for month P ends. For month Q, which is the month following month P, the first creation process is executed according to the same flow as above.

[0100] Note that it is assumed that the image layout in the first commercial material for month P is determined according to the layout editing operation by the user. In this case, in the first creation process for creating the first commercial material for month Q, the processor 10A proposes to the user the image layout in the first commercial material for month P as a recommended layout. The user determines whether to adopt the proposed recommended layout and performs an operation according to the determination on the user-side device 12. By proposing the recommended layout in this way, the user's layout editing operation can be supported. When the user adopts the recommended layout, the processor 10A creates the first commercial material for month Q based on the proposed recommended layout.

[0101] <<Regarding the material selection process>> Next, the flow of the material selection process in the first creation process will be described with reference to FIG. 9. The material selection process is executed in the first creation process for each month, that is, it is executed monthly. In the material selection process for each month, as shown in FIG. 9, the processor 10A executes a feature identification process (S021), a trend identification process (S022), and a selection process (S023).

[0102] The contents of the above three processes are different between the first month (the initial month) during the commercial material providing service usage period and the months after the initial month. Hereinafter, the material selection process for the initial month and the material selection process after the initial month will be described respectively. In the following, a case where the first month is April and the month after the first month is May will be taken as an example for explanation.

[0103] [Material Selection Process for the First Month] In the material selection process for April, which is the first month, a material image (April material image) used for creating the first commercial material for April is selected from the image group of April. In the material selection process for April, first, a feature identification process is executed. In the feature identification process, the processor 10A identifies features regarding two or more items for the images included in the image group of April. Regarding the number and types of items, there are no particular limitations, but in the following, a case where three items, "subject person", "shooting location", and "shooting time", are set will be assumed for explanation.

[0104] Regarding the features identified for each item, for example, as the feature regarding the item "subject person", the relationship of the people shown in the image (specifically, children, parents, grandparents, etc.) is identified. As the feature regarding the item "shooting location", a specific name of the shooting location (location name) such as home or park is identified. As the feature regarding the item "shooting time", a specific time, or a time zone such as morning, afternoon, and night is identified. Note that the method of identifying features may be a method of automatically identifying by image analysis, or a method of manually inputting information indicating the features.

[0105] In the trend identification process in the material selection process for April, the processor 10A identifies the trends of the features of the images included in the image group of April for the three items based on the features identified in the feature identification process. The trend indicates what kind of features the images in the image group have and to what extent they are included.

[0106] To explain the trend identification process in more detail, in the trend identification process, the processor 10A divides the images included in the April image group into a plurality of categories for each of the three items based on the features identified in the feature identification process. A category is a category (category) when classifying images having a certain feature. For example, for the item "shooting location", there are categories such as images taken at home and images taken in the park. Note that there may be cases where one image belongs to multiple categories for a certain item. For example, an image in which a child and a mother are shown together belongs to the category of "image in which the child is shown" and also belongs to the category of "image in which the mother is shown" for the item "subject person".

[0107] The processor 10A calculates the ratio of the images belonging to each category for each item for each category. The ratio is expressed as the ratio of the number of images belonging to each category to the April image group (unit: %). And the ratio calculated for each category for each item corresponds to the trend of the features of the images included in the April image group.

[0108] An example of the ratio calculated for the images included in the April image group is shown in FIG. 10. Regarding the April image group, as shown in FIG. 10, for example, for the item "subject person", the ratio of images in which a child is shown is 40%, the ratio of images in which a mother is shown is 40%, and the ratio of images in which a father is shown is 40%. Also, for the item "shooting location", the ratio of images taken at home is 80%, and the ratio of images taken in the park is 20%.

[0109] After the processor 10A executes the feature identification process and the trend identification process, it executes the April selection process. In the April selection process, the processor 10A selects the material images (April material images) used for creating the first commercial material in April from the April image group excluding the images to be excluded. At this time, the processor 10A selects the April material images based on the trend identified in the trend identification process.

[0110] Specifically, the processor 10A preferentially selects, as the material image, the image of the category with the highest ratio for each item. In the case of FIG. 10, for each item, the image of the category with the highest ratio, for example, "the image in which a child is shown and was taken at home in the morning" is preferentially selected as the material image. Note that the image selected as the material image may include, at a ratio corresponding to the ratio of the category, images belonging to categories other than the category with the highest ratio.

[0111] Through the above procedures, the material images for April are selected. These material images are the images selected reflecting the user's preferences, interests, etc. That is, the ratio calculated for each item is determined according to the shooting tendency of the user who is the image provider, and the image of the category with the highest ratio is likely to correspond to the image preferred by the user or the image that the user values. Therefore, by selecting the material image according to the above procedures, it is possible to select, as the material image, the image important to the user.

[0112] [Material Selection Process for the Month after the First Month] In the material selection process for May, which is an example of the month after the first month, the material image (May material image) used for creating the first commercial material in May is selected from the image group of May. In the material selection process for May, the material image for May is selected taking into account the material image for April. Here, in the relationship between April and May, April corresponds to the first period, and May corresponds to the second period. Also, the image group for April corresponds to the first image group, and the images included in that image group correspond to the first images. That is, the material image for April corresponds to the first image used for creating the commercial material. Also, the image group for May corresponds to the second image group, and the images used in that image group correspond to the second images. That is, the material image for May corresponds to the second image for creating the commercial material.

[0113] The material selection process for May proceeds according to the flow shown in Fig. 11. In the feature identification process in the material selection process for May, the processor 10A identifies features related to the above-mentioned three items for the material images for April and the images included in the image group for May (S201, S202). The manner and procedure for identifying the image features are the same as the feature identification process in the material selection process for April.

[0114] In the subsequent trend identification process, the processor 10A identifies the trend of the characteristics of the material images in April for the above three items (S203). Specifically, in the trend identification process, the processor 10A divides the material images in April for each item into a plurality of sections based on their characteristics. The processor 10A then calculates the ratio of images (material images) belonging to each section for each section. This ratio corresponds to the first ratio, and is expressed as the ratio (unit: %) of the number of images belonging to each section to the total number of material images in April. Then, the first ratio calculated for each section for each item corresponds to the trend of the characteristics of the material images in April, that is, the trend of the characteristics of the first images used in creating the commercial material. In the following, for the sake of convenience, the trend in the characteristics of image material in April will be referred to as the "trend in April."

[0115] FIG. 12 shows an example of the first ratios calculated for the material images in April. As for the material images in April, as shown in FIG. 12, for example, for the item "subject person", the first ratio of images in which children appear is 90%, the first ratio of images in which mothers appear is 30%, and the first ratio of images in which fathers appear is 10%. For the item "photography location", the first ratio of images taken at home is 90%, and the first ratio of images taken in parks is 10%. For the item "photography time", the first ratio of images taken in the morning is 40%, the first ratio of images taken in the afternoon is 60%, and the first ratio of images taken at night is 0%.

[0116] In the trend identification process, the processor 10A obtains a first trend value obtained by quantifying the trend in April. Quantifying the trend includes, for example, deriving a vector indicating the trend. Specifically, when the number of items is C and the number of categories included in each item is D, the first trend value is represented by a multi-dimensional vector having dimensions of C×D. Further, each component of the multi-dimensional vector forming the first trend value corresponds to the first ratio of the images belonging to the category corresponding to the component for the item corresponding to the component.

[0117] After executing the feature identification process and the trend identification process, the processor 10A executes a selection process for May. In the selection process for May, the processor 10A selects a material image for May from the image group for May based on the features of the images included in the image group for May. In the present embodiment, the processor 10A selects, as the material image for May, an image (second image) having features deviating from the trend in April for some of the above three items.

[0118] More specifically, the processor 10A sets a priority item from among the three items and sets the items other than the priority item as non-priority items (S204). The priority item is an item that is prioritized when selecting the material image for May among the three items, and is set according to the trend in April, for example. More specifically, the category with the highest first ratio or the category in which the first category exceeds the reference value is specified, and the item corresponding to that category becomes the priority item. In the case shown in FIG. 12, since the category with the highest first ratio is the image in which the child appears, the item "subject person" becomes the priority item.

[0119] Note that the priority item is not limited to one item, and a plurality of items may be set as priority items. Also, the method of determining the priority item is not particularly limited, and for example, the user may specify the priority item. The non-priority items correspond to "some of the items" in the present invention. In the case shown in FIG. 12, the items "shooting location" and "shooting time" correspond to the non-priority items.

[0120] Next, the processor 10A divides the images included in the May image group into a plurality of categories based on their features for the above three items, and calculates the ratio of the images belonging to each category for each category (S205). This ratio corresponds to the second ratio and is expressed as the ratio (unit: %) of the number of images belonging to each category to the number of images included in the May image group.

[0121] An example of the second ratio calculated for the images included in the May image group is shown in FIG. 12. Regarding the May image group, as shown in FIG. 12, for example, for the item "subject person", the second ratio of the images with children is 90%, the second ratio of the images with the mother is 35%, and the second ratio of the images with the father is 10%. For the item "shooting location", the second ratio of the images taken at home is 30%, and the second ratio of the images taken in the park is 70%. For the item "shooting time", the second ratio of the images taken in the morning is 40%, the second ratio of the images taken in the afternoon is 60%, and the second ratio of the images taken at night is 0%.

[0122] Thereafter, the processor 10A selects the May material images based on the second ratio calculated for each category for the images included in the May image group and the first tendency value indicating the tendency in April (S206). At this time, the processor 10A selects, as the May material images, the images in the May image group that have the features along the April tendency for the priority items and have the features deviating from the April tendency for the non-priority items.

[0123] The images having the features along the April tendency for the priority items are determined according to the first ratio calculated for each category for the priority items. For example, the images belonging to the category with the highest first ratio for the priority items correspond. In the case shown in FIG. 12, the images with children are the images having the features along the April tendency for the priority items. Note that the images having the features along the April tendency for the priority items are not limited to the images belonging to the category with the highest first ratio. For example, they may be any of the images belonging to the categories from the one with the highest first ratio to the nth (n is a natural number of 2 or more).

[0124] For other non-priority items, images having characteristics deviating from the April trend are determined according to the first ratio and the second ratio calculated for each category of non-priority items. Specifically, an image belonging to a category in which the second ratio calculated for a non-priority item has increased compared to the first ratio corresponds to an image having characteristics deviating from the April trend for the non-priority item. In the case shown in FIG. 12, for the non-priority item "shooting location", the second ratio of the images taken in the park is 70%, which is higher than the first ratio of 10%. Therefore, the images taken in the park correspond to images having characteristics deviating from the April trend for the non-priority item.

[0125] Note that, among the images belonging to the category in which the second ratio calculated for the non-priority item has increased compared to the first ratio, images belonging to the category in which the difference between the second ratio and the first ratio exceeds the threshold value may be selected as material images. Also, when there are multiple categories in which the second ratio has increased compared to the first ratio for non-priority items, it is advisable to calculate an evaluation value for each image belonging to each category. The evaluation value is the total value of a value calculated according to the image quality of the image, the number of people in the image, and whether or not a specific person exists in the image, and a value determined according to the second ratio. The processor 10A may prioritize the category to which the image with a higher evaluation value belongs among the above multiple categories, and select the image belonging to that category as the material image.

[0126] When selecting an image having characteristics deviating from the April trend for non-priority items, the processor 10A uses the first trend value indicating the April trend. Specifically, the processor 10A selects the material images for May such that the difference between the second trend value obtained by quantifying the characteristics of the material images for May and the first trend value is equal to or greater than the set value.

[0127] As shown in FIG. 13, the second tendency value is represented by a multi-dimensional vector having the number of dimensions obtained by multiplying the number of items C by the number of divisions D included in each item, similar to the first tendency value. Further, each component of the multi-dimensional vector forming the second tendency value corresponds to the second ratio of the images belonging to the division corresponding to the component for the item corresponding to the component. As shown in FIG. 13, the difference between the second tendency value and the first tendency value is the absolute value of the difference between the two vectors. Also, the set value set for the difference may be a predetermined value, or may be set by the user and may be appropriately changed.

[0128] According to the above, an image having a feature deviating from the tendency in April can be accurately selected from the image group in May. That is, in the present embodiment, the processor 10A quantifies the tendency in April and the tendency of the features of the image selected as the material image in May, and executes arithmetic processing using the two quantified tendencies. By this arithmetic processing, an image having a feature deviating from the tendency in April can be reasonably and appropriately derived.

[0129] According to the procedure described above, the processor 10A selects a material image in May from the image group in May. Here, the image selected as the material image in May takes into account the material image in April. Specifically, it has features along the tendency in April for the priority items and features deviating from the tendency in April for the non-priority items. As a result, the image used for creating the first commercial material in May (the current month) can be appropriately selected based on the image used for creating the commercial material in April (the previous month).

[0130] More specifically, the features of the image regarding the priority items are likely to be the content (theme) that the user prioritizes in creating the first commercial material. Based on this, if an image having features along the tendency in April for the priority items is selected as the material image in May, a material image that matches the user's preference or interest can be selected.

[0131] On the one hand, if the change between the features of the material image in May and the features of the material image in April is small, the first commercial material in April and the first commercial material in May will tend to be similar to each other. If such a situation continues for the first commercial materials in subsequent months, even if the first commercial materials are created monthly, the features of the images included in the first commercial materials, that is, the features of the material images, will become monotonous. As a result, the first commercial materials created monthly may lack variations.

[0132] Therefore, in this embodiment, an image having features deviating from the April trend for non-priority items is selected as the material image for May. As a result, in terms of the features of the images (material images) for creating commercial materials, the first commercial material in May will be different from the first commercial material in April. As a result, for the first commercial materials created monthly, the variations can be enriched, and it is possible to avoid the materials becoming monotonous (unimpressive) for the user. Due to such an effect, it is expected that the number of users using the commercial material providing service that provides the first commercial material monthly will increase.

[0133] In the case shown in FIG. 12, it is assumed that the priority item is the "subject person" and the non-priority item is the "shooting location". Then, as an image having features along the April trend for the priority item and features deviating from the April trend for the non-priority item, an image in which a child is shown and was taken in a park was selected. However, the priority items and non-priority items may vary according to the characteristics of the image group provided by the user. For example, assume that the material images in April contain many images of children eating strawberries, while the image group in May does not contain images of children eating strawberries. Instead, it contains many images of children eating melons. In this case, for example, the priority item can be set as "the subject person and the action of that person", and the non-priority item can be set as "the object shown together with the subject person". Specifically, for the priority item, an image having characteristics following the trend in April can be determined as an image showing a child eating fruit. Also, for the non-priority item, an image having characteristics deviating from the trend in April can be determined as an image in which the fruit shown together with the child is a fruit different from strawberries, for example, a melon. Under such conditions, the material images in May can be selected from the image group in May.

[0134] Supplementing the selection process for May, the images selected as the material images in May may include, in addition to the images (hereinafter referred to as main images) that satisfy the above conditions, images other than the main images (hereinafter referred to as sub-images). The main image is an image having characteristics following the trend in April for the priority item and having characteristics deviating from the trend in April for the non-priority item. The sub-image corresponds to, for example, an image that does not have characteristics following the trend in April for the priority item. Note that among the material images in May, the ratio of the sub-images is preferably smaller than the ratio of the main images.

[0135] Also, when selecting the material images in May, the processor 10A executes a feature identification process and a trend identification process. In the feature identification process, the processor 10A identifies the characteristics of the material images in April and the characteristics of the images included in the image group in May for the above three items. At this time, the processor 10A may further identify the characteristics of the images included in the image group in April (the first image group) for the above three items.

[0136] Also, in the tendency identification process, the processor 10A identifies the tendency of the features of the material images in April (i.e., the tendency in April) for the above three items. At this time, the processor 10A may further identify the tendency of the features of the images included in the image group in April. The tendency of the features of the images included in the image group in April corresponds to the "other tendency" in the present invention, and hereinafter will be referred to as the tendency of the image group in April.

[0137] Then, the processor 10A may select the material images in May based on the tendency in April and the tendency of the image group in April. In this case, the images selected as the material images in May may include, in addition to the images having features along the tendency in April for the priority item, the images having features along the tendency of the image group in April for the priority item.

[0138] Taking the case of FIG. 11 as a specific example for explanation, for the priority item "subject person", in the image group in April, the images of children, the images of mothers, and the images of fathers are each included at the same ratio. Therefore, the images of children, the images of mothers, and the images of fathers each correspond to the images having features along the tendency of the image group in April for the priority item. In this case, the processor 10A may include the images of mothers or the images of fathers together with the images of children in the material images in May.

[0139] As described above, if the images having features along the tendency of the image group in April for the priority item are selected as the material images in May, the material images in May (the current month) can be selected taking into account the shooting tendency in April (the previous month). Thereby, the images important to the user can be grasped from the shooting tendency of the previous month, and based on this, the material images in the current month can be selected.

[0140] So far, the process of selecting material images for May has been described. However, the same procedure applies to the process of selecting material images for months after June. For example, the material images for June are selected from the image group for June based on the trend of the characteristics of the material images for April and the trend of the characteristics of the material images for May. In this case, the material images for April and May correspond to the first image used for product creation, the image group for June corresponds to the second image group, and the material images for June correspond to the second image for product creation.

[0141] Suppose the image group for June includes, for example, an image of grandparents, and such an image is not included in either the material images for April or the material images for May. In this case, since the image of grandparents has characteristics that deviate from both the trend of the characteristics of the material images for April and the trend of the characteristics of the material images for May, it may be selected as the material image for June.

[0142] <<Recreation Process>> After the processor 10A executes the first creation process for creating the first product for a certain month, if the recreation condition is satisfied, the processor 10A executes the process of recreating the first product with that month as the target month. That is, in this embodiment, even if the first product F1 is created, if the recreation condition is satisfied, the first product F1 can be remade as needed. Hereinafter, the flow of the recreation process will be described with reference to FIG. 14.

[0143] The recreation process is executed when there is a month that satisfies the recreation condition, that is, the target month. For example, at a time after the next month of the target month, the processor 10A executes the recreation process of recreating the first product for the target month. Whether the recreation condition is met is determined by the presence or absence of an order for the first product and the presence or absence of a storage requirement.

[0144] Specifically, when the processor 10A has not received any orders or storage requests for the first commercial material of a certain month, it sets that month as the target month and executes a recreation process to recreate the first commercial material of the target month. In the recreation process, as shown in FIG. 14, the processor 10A first executes a reselection process to select a material image of the target month (S031). In the reselection process, the processor 10A selects a material image to be used for recreating the first commercial material of the target month from among the image group taken in the target month (image group of the target month).

[0145] In the reselection process, the processor 10A identifies the features of the images included in the image group taken during the reselection period including the target month, and identifies the trend of the features of that image group. The procedure for identifying the features and trends of the images in the reselection process is the same as the feature identification process and trend identification process in the material selection process in the first creation process. Also, the reselection period starts from the month before the target month and continues until the month after the target month. The month before the target month is, for example, the start month (first month) of the commercial material providing service, and the month after the target month is the month in which the recreation process is executed (specifically, the current time).

[0146] The material image used to recreate the first commercial material of the target month is selected considering the image group taken in the months after the target month. That is, when recreating the first commercial material of the target month, the material image can be selected based on the user's shooting trend in the period after the target month. Thereby, the user's shooting trend can be appropriately grasped over a longer period, and based on that shooting trend, the first commercial material of the target month can be recreated.

[0147] Specifically, the longer the re-selection period, that is, the closer it gets to the last month of the year, the greater the number of image groups that the image processing apparatus 10 acquires monthly. And the greater the number of image groups, the more accurately the shooting tendency of the user can be specified based on the image groups of each month. For example, a person who does not appear in the image group of the target month may frequently appear in the image groups of the months after the target month. Also, even if the number of images in which a certain person appears is small in the image group of the target month, there may be cases where images in which that person appears are frequently included in the image groups after the target month. Such a person may be an important person to the user, and this fact will become clear after the target month. By referring to the image groups taken during the re-selection period, the above facts can be easily recognized. As a result, the material images for re-creating the first commercial material of the target month can be selected more accurately.

[0148] After executing the re-selection process, the processor 10A selects material images from among the image groups of the target month based on the tendency of the characteristics of the image groups taken during the re-selection period. Then, the processor 10A re-creates the first commercial material of the target month using the selected material images (S032). The flow after re-creating the first commercial material in the re-creation process is the same as the flow after creating the first commercial material in the first creation process. Specifically, the processor 10A generates a data file of the re-created first commercial material of the target month and transmits the data file to the user-side device 12. As a result, the re-created first commercial material of the target month is displayed on the display of the user-side device 12.

[0149] The user looks at the display, checks the re-created first commercial material of the target month, and considers whether to place an order (S033). If the user places an order, the processor 10A accepts the order (S034) and executes a process of providing the re-created first commercial material of the target month to the user, specifically, printing processing or the like (S035). Note that along with accepting the order, the processor 10A finalizes the re-created first commercial material of the target month.

[0150] Also, the user considers whether it is necessary to save data for the first commercial material of the recreated target month (S036). If the user makes a save request, the processor 10A accepts the save request (S037) and saves the data file of the first commercial material of the recreated target month in the storage device 14 (S038).

[0151] The series of steps related to the recreation process described so far is repeatedly executed while the recreation conditions for the target month are satisfied. On the other hand, when the recreation conditions are no longer satisfied, that is, when the processor 10A receives an order or a save request for the recreated first commercial material, the recreation process for that first commercial material is no longer executed thereafter.

[0152] <<Regarding the Second Creation Process>> Next, during the commercial material provision flow, the flow of the second creation process will be described with reference to FIG. 15. Each step during the second creation process is executed by the processor 10A of the computer constituting the image processing apparatus 10. Before the second creation process is executed, the processor 10A creates the first commercial material F1 for each of the 12 months including the final month.

[0153] In the second creation process, first, the processor 10A specifies, for each of the 12 months' worth of first commercial materials, the material image used for creating the first commercial material F1 on a monthly basis (S041). In this step S041, when the processor 10A specifies the material image of the first commercial material F1 for the month in which the save request was made, it reads the data file of the first commercial material F1 from the storage device 14. Then, the processor 10A specifies the material image of the first commercial material from the read data file.

[0154] Also, when the first commercial material of the target month is recreated, the processor 10A specifies the material image of the recreated first commercial material. When the first commercial material of the target month has been recreated multiple times, the processor 10A specifies the material image of the latest first commercial material (that is, the first commercial material recreated most recently).

[0155] Next, the processor 10A identifies monthly whether an order has been received from the user for the first commercial material of each month (S042). Specifically, the processor 10A identifies the presence or absence of an order for the first commercial material of each month by referring to the information indicating the user's order history.

[0156] Thereafter, the processor 10A creates a second commercial material based on the material image of the first commercial material of each month and the presence or absence of an order for the first commercial material of each month (S043). Specifically, the processor 10A creates each of the plurality of pages included in the second commercial material using the material image of the first commercial material of the month corresponding to each page. For example, when the Xth page of the second commercial material corresponds to May, the processor 10A creates the Xth page using the material image of the first commercial material of May.

[0157] In addition, the processor 10A sets the image layout on each page according to the presence or absence of an order for the first commercial material of each month. Specifically, the processor 10A sets the layout of each page so that the material image of the first commercial material of the order month (the month in which an order was placed for the first commercial material) is preferentially arranged. This is because for the order month, it is highly likely that there were important events etc. for the user, and it is presumed that the importance of the image taken in the order month is high.

[0158] The method of preferentially handling the material image of the first commercial material of the order month in the second commercial material is not particularly limited. For example, it is assumed that the size of the image on each page of the second commercial material is set to any one of large size, medium size, small size, and index display size (minimum size). In this case, in the second commercial material, on the page corresponding to the order month, as shown in FIG. 16A, the material image of the first commercial material of the order month may be arranged in a large size. On the other hand, as shown in FIG. 16B, the material image of the first commercial material of the non-order month may be arranged in the index display size on the last page of the second commercial material. Note that in FIGS. 16A and 16B, the size of the image on each of the page corresponding to the order month and the page corresponding to the non-order month is represented by a dashed line.

[0159] Through the above series of steps, the second commercial material F2 is completed. After that, when there is an order from a user for the second commercial material, the processor 10A accepts the order (S044) and executes a process for providing the second commercial material F2 to the user, such as a printing process or the like (S045). When the series of steps described above are completed, the second creation process ends.

[0160] <<Other Embodiments>> The embodiments described above are specific examples given for the purpose of easily explaining the image processing apparatus and the image processing method of the present invention, and are merely examples, and other embodiments are also conceivable.

[0161] In the above embodiment, the first commercial material F1 is created monthly. However, the creation cycle of the first commercial material is not particularly limited, and the first commercial material may be created daily, weekly, or annually, or may be created at a cycle of several days, several weeks, several months, or several years. Also, in the above embodiment, the second commercial material F2 is created on an annual basis. However, the creation cycle of the second commercial material is not particularly limited as long as a plurality of first commercial materials are created within that period. For example, the second commercial material may be created daily, weekly, or monthly, or may be created at a cycle of several days, several weeks, several months, or several years.

[0162] Also, in the above embodiment, in the material selection process of selecting the material image of the current month, the material image is selected based on the tendency of the characteristics of the material image of the previous month. However, it is not limited to this, and the material image of the current month may be selected based on the tendency of the characteristics of the material image of a month earlier than the previous month, for example, a month more than two months before the current month.

[0163] Also, in the above-described embodiment, the functions of the image processing apparatus of the present invention are assumed to be performed by a processor provided in a server computer, but the present invention is not limited to this. The processor included in the image processing apparatus of the present invention may be provided in both the server computer and the user-side device 12. That is, a part of the functional units provided in the above-described image processing apparatus 10 may be provided in the user-side device 12.

[0164] The processor included in the image processing apparatus of the present invention includes various processors. The various processors include, for example, a CPU which is a general-purpose processor that executes software (program) and functions as various processing units. Also, the various processors include a PLD (Programmable Logic Device) which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). Furthermore, the various processors include a dedicated electric circuit or the like which is a processor having a circuit configuration dedicatedly designed for performing specific processing, such as an ASIC (Application Specific Integrated Circuit).

[0165] Also, one processing unit included in the image processing apparatus of the present invention may be configured by one of the above-described various processors, or may be configured by a combination of two or more processors of the same type or different types, for example, a combination of a plurality of FPGAs, or a combination of an FPGA and a CPU. Also, a plurality of functional units included in the image processing apparatus of the present invention may be configured by one of the various processors, or two or more of the plurality of functional units may be collectively configured by one processor. Also, as in the above-described embodiment, a form in which one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of functional units may be adopted.

[0166] Alternatively, for example, in a form using a processor that realizes the functions of the entire system including a plurality of functional units in an image processing apparatus with one IC (Integrated Circuit) chip, such as represented by a SoC (System on Chip). Also, the hardware configuration of each of the various processors described above may be an electric circuit (Circuitry) formed by combining circuit elements such as semiconductor elements.

Explanation of Signs

[0167] 10 Image processing apparatus 10A Processor 10B Memory 10C Communication interface 10D Printer 12 User-side device 14 Storage device 21 Image acquisition unit 22 Preliminary screening unit 23 Material selection unit 24 Editing unit 25 First commercial material creation unit 26 Recommendation unit 27 Order reception unit 28 Request reception unit 29 Re-selection unit 30 Re-creation unit 31 Second commercial material creation unit F1 First commercial material F2 Second commercial material N Network S Commercial material providing system

Claims

1. An image processing apparatus including a processor, wherein the processor performs a first acquisition process of acquiring a first image group captured during a first period, a second acquisition process of acquiring a second image group captured during a second period different from the first period, a feature identification process of identifying features regarding two or more items related to imaging for a first image used for product material creation among the first image group and a second image included in the second image group, a trend identification process of identifying a trend of the features of the first image used for product material creation regarding the two or more items, and a selection process of selecting a second image for product material creation from among the second image group based on the features of the second image included in the second image group, and in the selection process, the second image having features deviating from the trend for some of the two or more items is selected as the second image for product material creation. An image processing apparatus.

2. In the trend identification process, the processor obtains a first trend value obtained by quantifying the trend, and in the selection process, the processor selects the second image for product material creation based on the first trend value. The image processing apparatus according to claim 1.

3. In the selection process, the processor selects the second image for product material creation such that a difference between a second trend value obtained by quantifying the trend of the features of the second image for product material creation and the first trend value is equal to or greater than a set value. The image processing apparatus according to claim 2.

4. The two or more items include priority items set according to the trend and non-priority items other than the priority items, and in the selection process, the processor selects, as the second image for product material creation, the second image having features along the trend for the priority items and features deviating from the trend for the non-priority items. The image processing apparatus according to any one of claims 1 to 3.

5. In the trend identification process, the processor for each of the two or more items, based on the features of the first image used for product material creation, divides the first image used for product material creation into a plurality of sections, and calculates a first ratio of the first images belonging to each of the sections for each of the sections, thereby identifying the trend. For each of the two or more items, based on the features of the second images included in the second image group, the second images included in the second image group are divided into the plurality of sections, and a second ratio of the second images belonging to each of the sections is calculated for each section. In the selection process, the processor selects, as the second image for creating the commercial product, a second image belonging to a section in which the second ratio has increased from the first ratio, for the non-priority item, according to the image processing apparatus of claim 4.

6. In the feature identification process, the processor further identifies features related to the two or more items for the first images included in the first image group. In the trend identification process, the processor identifies, for the two or more items, a trend of the features of the first images used for creating the commercial product and another trend that is a trend of the features of the first images included in the first image group. In the selection process, the processor includes, as the second image for creating the commercial product, a second image having features along the other trend for the priority item, according to the image processing apparatus of claim 4.

7. The processor repeatedly executes a first creation process of creating a first commercial product for a period corresponding to the second period, using the second image selected by the selection process from the second image group taken during the period corresponding to the second period, by changing the period corresponding to the second period. The first creation process includes a material selection process of selecting, from among the second image group taken during the period corresponding to the second period, a second image used for creating the first commercial product for the period corresponding to the second period. In the material selection process, the processor executes the feature identification process, the trend identification process, and the selection process. When executing the first creation process with a certain period as the period corresponding to the second period, the material selection process is executed with a period before the certain period as the period corresponding to the first period. After executing the first creation process with a certain period as the period corresponding to the second period, when executing the first creation process with a period after the certain period as the period corresponding to the second period, the material selection process is executed with a period before the certain period as the period corresponding to the first period, according to the image processing apparatus according to any one of claims 1 to 6.

8. The first creation process includes a layout determination process for determining the layout of the image in the first commercial material based on the user's layout editing operation. In the first creation process of creating the first commercial material for the second period, the processor proposes to the user the layout of the image in the first commercial material for the first period, and when the proposed layout is adopted by the user, the first commercial material for the second period is created with the proposed layout. The image processing apparatus according to claim 7.

9. After executing the first creation process of creating the first commercial material for the period corresponding to the second period, when the recreation condition is satisfied, the processor executes a recreation process of recreating the first commercial material for the period corresponding to the second period. The recreation process includes a reselection process of selecting the second image to be used for recreating the first commercial material from the second image group captured during the period corresponding to the second period. The image processing apparatus according to claim 7 or 8.

10. The recreation process for the period corresponding to the second period is executed in a period after the period corresponding to the second period. In the reselection process, based on the tendency of the features of the image group captured in two or more periods including the period corresponding to the second period and the period after the period corresponding to the second period, the processor selects the second image from the second image group captured during the period corresponding to the second period. The image processing apparatus according to claim 9.

11. The processor receives a save request from the user for the first commercial material. After the execution of the first creation process, the processor When the save request for the first commercial material is received, the processor does not execute the recreation process of recreating the first commercial material. When the save request for the first commercial material is not received, the processor executes the recreation process of recreating the first commercial material. The image processing apparatus according to claim 9 or 10.

12. The processor receives an order from the user for the first commercial material. After the execution of the first creation process, the processor When the order for the first commercial material is received, the processor does not execute the recreation process of recreating the first commercial material. The image processing apparatus according to claim 9 or 10, wherein when the order for the first commercial material has not been received, the recreation process for recreating the first commercial material is executed.

13. The processor further executes a second creation process for creating a second commercial material for a plurality of periods, the second commercial material is configured by a collection of a plurality of second commercial material components, each of the plurality of second commercial material components corresponds to one of the plurality of periods, in the second creation process, the processor for each of the first creation processes repeatedly executed by changing the period corresponding to the second period, a process of specifying the second image used for creating the first commercial material; using the second image used for creating the first commercial material for the period corresponding to the second period among the plurality of periods, a process of creating a second commercial material component corresponding to the period corresponding to the second period, and repeatedly executing the process while changing the period corresponding to the second period to create the second commercial material. The image processing apparatus according to any one of claims 9 to 12.

14. The processor receives a user's order for the first commercial material, in the second creation process, the processor sets the layout of the second image in each of the plurality of second commercial material components according to the presence or absence of an order for the first commercial material for the period corresponding to each second commercial material component. The image processing apparatus according to claim 13.

15. An image processing method by a processor, by the processor, a first acquisition process of acquiring a first image group taken in a first period; a second acquisition process of acquiring a second image group taken in a second period different from the first period; for each of the first image used for creating a commercial material and the second image included in the second image group among the first image group, a feature specifying process of specifying features related to two or more items regarding shooting; a tendency specifying process of specifying a tendency of features of the first image used for creating the commercial material for the two or more items; a selection process of selecting a second image for creating a commercial material from the second image group based on the features of the second image included in the second image group is executed, in the selection process, the second image having features deviating from the tendency for some of the two or more items is selected as the second image for creating the commercial material. Image processing method.

16. In the tendency identification process, the processor obtains a first tendency value obtained by quantifying the tendency. In the selection process, the processor selects a second image for creating a commercial material based on the first tendency value, according to the image processing method described in claim 15.

17. In the selection process, the processor selects a second image for creating a commercial material such that the difference between a second tendency value obtained by quantifying the tendency of the features of the second image for creating a commercial material and the first tendency value is equal to or greater than a set value, according to the image processing method described in claim 16.

18. The two or more items include priority items set according to the tendency and non-priority items other than the priority items. In the selection process, the processor selects, as the second image for creating a commercial material, the second image that has features along the tendency for the priority items and features deviating from the tendency for the non-priority items, according to the image processing method described in any one of claims 15 to 17.

19. In the tendency identification process, the processor For each of the two or more items, based on the features of the first image used for creating a commercial material, divides the first image used for creating a commercial material into a plurality of sections, and calculates a first ratio of the first image belonging to each section for each section to identify the tendency. For each of the two or more items, based on the features of the second images included in the second image group, divides the second images included in the second image group into the plurality of sections, and calculates a second ratio of the second images belonging to each section for each section. In the selection process, the processor selects, as the second image for creating a commercial material, the second image belonging to the section in which the second ratio has increased compared to the first ratio for the non-priority items, according to the image processing method described in claim 18.

20. In the feature identification process, the processor further identifies features related to the two or more items for the first images included in the first image group. In the tendency identification process, the processor identifies, for the two or more items, the tendency of the features of the first image used for creating a commercial material and another tendency that is the tendency of the features of the first images included in the first image group. In the selection process, the second image for creating the commercial material is selected by the processor based on the tendency and the other tendency. The image processing method according to any one of claims 15 to 19.

21. A program for causing a computer to execute each process included in the image processing method according to any one of claims 15 to 20.

22. A computer-readable recording medium, A recording medium on which a program for causing a computer to execute each process included in the image processing method according to any one of claims 15 to 20 is recorded.

Images