Image processing device, image processing method, program, and recording medium

Abstract

To provide an image processing device capable of seamlessly restarting layout editing even when moving from a terminal device of a user having an image group to a terminal device of a user not having the image group, an image processing method, a program, and a storage medium.SOLUTION: In an image processing device, an image processing method, a program, and a storage medium, a composite image is displayed in a first terminal device. When the composite image is being edited in the first terminal device, candidate images which are a part of images other than use images used in the composite image and are usable in the composite image when editing of the composite image in a second terminal device is restarted are selected from an image group on the basis of defined criteria. The use images and the candidate images transmitted from the first terminal device and received by a server are stored in the server.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium for automatically creating a layout of a composite image from a group of images. [Background technology]

[0002] When creating a layout for a composite image such as a photo album (photo book), hard cover album, or shuffle print, a user must select multiple images to be used in the composite image from a large number of images contained in the user's image group, and then beautifully lay out the selected multiple images, which is a time-consuming editing task. For beginners, this editing task is a heavy burden.

[0003] In light of this, applications have been provided that automatically select multiple images to be used in a composite image from a user's image collection and automatically create a layout for the composite image using the selected multiple images, such as a composite image ordering application (ordering software) that runs on a user's terminal device such as a PC (Personal Computer), tablet PC, or smartphone.

[0004] Here, Patent Documents 1 to 4 are prior art documents related to the present invention.

[0005] Patent document 1 describes an album creation system in which an album orderer terminal generates low-quality images from high-quality images and supplies them to an album creation device, which analyzes the low-quality images and selects from the low-quality images to be placed in an album, places them in the album to create an album sample, and transmits the album sample to the album orderer terminal to allow a user to view it.

[0006] Patent Document 2 describes an information processing device that receives the size and number of pages of an album from a user, accepts designation of image files, analyzes and scores them, divides multiple image files into multiple groups, selects highly scored images from each group, lays out the image data contained in each group in each area of ​​a template, displays a confirmation / editing screen for the album layout results, and, once the confirmation and editing work is completed, transmits the current album data to an external server via a network.

[0007] Patent Document 3 describes an information processing device that scores image files, creates a layout list consisting of multiple layouts, and allows a user to edit the laid-out album. Patent Document 3 also describes that, among image files not used in the layout list, image files with a scoring evaluation value equal to or greater than a predetermined value, or a predetermined number of image files with high evaluation values, are left without being deleted, and when re-editing, a layout list is created including the images that were left without being deleted.

[0008] Patent document 4 describes a photobook creation device that, if a page deletion image exists due to a layout change, stores the page deletion image, and when an edited page on which an image is already placed is displayed as an edited page, displays a list of page placement candidate images with the page deletion image highlighted. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2007-194966 A [Patent Document 2] JP 2019-067237 A [Patent Document 3] Patent No. 6521856 [Patent Document 4] JP 2019-128647 A Summary of the Invention [Problem to be solved by the invention]

[0010] Conventionally, a user uses an order application on a terminal device to create a layout of a composite image from a group of images, transmits information on the images used in the composite image and the created layout to a server, and orders a product of the composite image. In this case, images from the group of images that are not used in the composite image are stored only in the terminal device of the user who created the composite image, so there is a problem that resuming layout editing, such as adding a new image from the group of images to the composite image, can only be done on the terminal device of the user.

[0011] On the other hand, if an environment could be realized in which layout editing work could be seamlessly resumed even when moving from a terminal device of a user who has a group of images to a terminal device of a user who does not have the group of images, it would lead to improved convenience for users who have limited time for editing work, such as mothers with young children. However, in this case, another problem arises in that the user is forced to wait for a waiting time for all of the images that can be used in the composite image to be transmitted from the user's terminal device to the server.

[0012] A first object of the present invention is to provide an image processing device, an image processing method, a program, and a recording medium that enable seamless resume of layout editing even when moved from a terminal device of a user who has a group of images to a terminal device of a user who does not have the group of images. The second object of the present invention is to provide, in addition to the first object, an image processing device, an image processing method, a program, and a recording medium that can significantly reduce a user's waiting time for transmitting a group of images that can be used in a composite image from the user's terminal device to a server. [Means for solving the problem]

[0013] In order to achieve the above object, the present invention provides an image processing apparatus including a server, a first terminal device, and a second terminal device different from the first terminal device, which are connected via a network, and which creates a layout of a composite image using some images selected from an image group held by the first terminal device, The processor includes: displaying the composite image having the layout created on the first terminal device; During layout editing on the first terminal device, select from the image group, based on a predetermined criterion, a candidate image which is a part of images other than the used image used in the composite image and which can be used in the composite image when the layout editing on the second terminal device is resumed; To provide an image processing device that causes a server to store an image to be used, a candidate image, and an additional image that is newly added to a composite image from an image group other than the image to be used by editing the layout on the first terminal device, which are transmitted from a first terminal device and received by the server.

[0014] Here, the processor: creating, from the image group, an analysis image group having a resolution required for analyzing the image group and a smaller amount of data than the image group, in the first terminal device; In the server, the group of images for analysis transmitted from the first terminal device and received by the server is analyzed, an image to be used for analysis is selected from the group of images for analysis based on a result of the analysis of the group of images for analysis, a layout is created using the image to be used for analysis, and information on the layout is stored; It is preferable that the first terminal device displays a composite image whose layout has been created based on layout information transmitted from the server and received by the first terminal device.

[0015] The processor also The server calculates a score for each of the analysis images included in the group of analysis images; In the first terminal device, it is preferable to select as candidate images from a group of images other than the used images, images corresponding to analysis images whose scores transmitted from the server and received by the first terminal device are equal to or greater than a predetermined reference value, or images corresponding to a predetermined percentage of analysis images whose scores are in ascending order of the scores.

[0016] The processor also In the server, attributes of each analysis image included in the group of analysis images are analyzed; In the first terminal device, from a group of images other than the used image, it is preferable to select as candidate images an image corresponding to an analysis image whose attributes transmitted from the server and received by the first terminal device match the attributes of the additional image, and an image corresponding to an analysis image whose attributes are within a defined range from the attributes of the additional image, or to select as candidate images an image whose attributes do not match the attributes of the additional image and whose attributes are outside a defined range from the attributes of the additional image.

[0017] The processor also The server selects a candidate image for analysis from a group of images for analysis other than the image to be used for analysis, Preferably, in the first terminal device, a candidate image is selected from the group of images that corresponds to a candidate image for analysis transmitted from the server and received by the first terminal device.

[0018] In addition, when editing of the layout on the second terminal device is resumed, it is preferable that the processor displays on the second terminal device, the analysis image corresponding to the image to be used, the analysis image corresponding to the candidate image, and the analysis image corresponding to the additional image, which are transmitted from the server and received by the second terminal device, as images that can be used in the composite image while editing the layout on the second terminal device.

[0019] The processor also In the first terminal device, analyzing the group of images, selecting from the group of images an image to be used based on a result of the analysis of the group of images, creating a layout using the image to be used, and displaying the composite image with the layout created; It is preferable that the server stores layout information transmitted from the first terminal device and received by the server.

[0020] The processor also In the first terminal device, as a result of analyzing the group of images, it is preferable to calculate a score for each image included in the group of images, and select as candidate images from the group of images other than the used image those images whose scores are equal to or above a predetermined reference value, or images in a predetermined percentage from the highest scores.

[0021] The processor also In the first terminal device, it is preferable to analyze the attributes of each image included in the image group, and select as candidate images from the image group other than the used image an image whose attributes match those of the additional image and whose attributes are within a defined range from the attributes of the additional image, or select as candidate images an image whose attributes do not match those of the additional image and whose attributes are outside a defined range from the attributes of the additional image.

[0022] The processor also In the first terminal device, it is preferable to select as a candidate image from a group of images other than the used images, based on information of an edited image on which an edit operation has been performed by the user while editing the layout on the first terminal device, and an image not used in the composite image after editing the layout on the first terminal device.

[0023] The processor also In the first terminal device, it is preferable to select candidate images from a group of images other than the image in use based on the user's past order history.

[0024] Moreover, it is preferable that the processor transmits, as candidate images, a predetermined number of images or a predetermined percentage of images in the group of images from the first terminal device to the server.

[0025] Moreover, it is preferable that the processor transmits, as candidate images, a number of images according to the network status from the first terminal device to the server.

[0026] It is also preferable that the processor transmits at least some of the images included in the group consisting of the used image and the candidate image from the first terminal device to the server while the layout is being edited on the first terminal device.

[0027] Furthermore, it is preferable that after editing the layout on the first terminal device, the processor transmits the remaining images and the additional images included in the group from the first terminal device to the server in response to an instruction from the user.

[0028] Furthermore, it is preferable that after editing the layout on the first terminal device, the processor transmits the used image, the candidate image and the additional image from the first terminal device to the server in response to an instruction from the user.

[0029] In addition, in response to a notification sent from the second terminal device and received by the first terminal device, the processor preferably transmits from the first terminal device to the server as candidate images images from the group of images that correspond to information on images other than the used image, candidate image, and additional image contained in the notification.

[0030] It is also preferable that, when editing of the layout on the second terminal device is resumed, the processor causes the used image, candidate image and additional image transmitted from the server and received by the second terminal device to be displayed on the second terminal device as images that can be used in the composite image while editing of the layout on the second terminal device is being performed.

[0031] The present invention also provides an image processing method for creating a layout of a composite image using some images selected from an image group held by an image processing device including a server, a first terminal device, and a second terminal device different from the first terminal device, which are connected via a network, the method comprising: A step of displaying the composite image in which the layout has been created on the first terminal device by an image display unit of the first terminal device; a step in which a candidate image selection unit of the first terminal device selects, from the image group, a part of images other than the images used in the composite image based on a predetermined criterion while editing the layout on the first terminal device, a candidate image that can be used in the composite image when editing the layout on the second terminal device is resumed; An image processing method is provided, which includes a step in which an information storage unit of a server stores an image to be used, a candidate image, and an additional image to be newly added to a composite image from an image group other than the image to be used by editing the layout on the first terminal device, the images being transmitted from a first terminal device and received by the server.

[0032] a step of generating, from the image group, an analysis image group having a resolution required for analyzing the image group and an amount of data smaller than that of the image group, by an analysis image group generating unit of the first terminal device; An image analysis unit of the server analyzes the group of analysis images transmitted from the first terminal device and received by the server; A use image selection unit of the server selects an image to be used for analysis from the group of images to be analyzed based on a result of the analysis of the group of images to be analyzed; A step in which a layout creation unit of the server creates a layout using the analysis use image; A step in which an information storage unit of the server stores layout information; It is preferable that the method further includes a step in which an image display unit of the first terminal device displays a composite image whose layout has been created based on layout information transmitted from the server and received by the first terminal device.

[0033] Also, an image analysis unit of the first terminal device analyzes the image group; A step in which a use image selection unit of the first terminal device selects a use image from the image group based on a result of the analysis of the image group; A step in which a layout creation unit of a first terminal device creates a layout using a use image; A step of displaying the composite image with the layout created by an image display unit of the first terminal device; It is preferable that the method further includes a step of: storing, by an information storage unit of the server, the layout information transmitted from the first terminal device and received by the server.

[0034] The present invention also provides a program for causing a computer to execute each step of the image processing method described above.

[0035] The present invention also provides a computer-readable recording medium having recorded thereon a program for causing a computer to execute each step of the image processing method described above. Effect of the Invention

[0036] According to the present invention, by storing information on the images used, the images added, and the layout in a server, layout editing can be resumed seamlessly even when moving from a first terminal device having a group of images to a second terminal device having no group of images. Furthermore, by storing the candidate images in the server, even if the user moves from the first terminal device to the second terminal device and resumes editing the layout, while editing the layout on the second terminal device, the user can use not only the used images and additional images but also the candidate images to edit the layout, thereby improving the convenience of editing and expanding the scope of editing.

[0037] Furthermore, according to the present invention, by selecting only candidate images from the image group and sending them to the server, the user's waiting time for images that can be used in the composite image to be sent to the server can be significantly reduced compared to sending all images included in the image group to the server. Furthermore, by transmitting the used image and candidate image to the server while the layout is being edited on the first terminal device, i.e., in the background of the user's layout editing operation, the transmission time experienced by the user for transmitting the used image and candidate image can be significantly reduced compared to setting aside a dedicated time for transmitting the used image and candidate image to the server after the layout has been edited, and then transmitting these images to the server. [Brief description of the drawings]

[0038] [Figure 1] 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating a configuration of a server according to an embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating a configuration of a first terminal device according to an embodiment of the present invention. [Figure 4] FIG. 11 is a block diagram illustrating a configuration of a second terminal device according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a conceptual diagram illustrating an operation of an image processing device according to an embodiment. [Figure 6] FIG. 2 is a conceptual diagram illustrating a display screen of a PC application according to an embodiment. [Figure 7] FIG. 13 is a conceptual diagram illustrating an embodiment of a display screen of a smartphone application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing apparatus, an image processing method, a program, and a recording medium according to the present invention will be described in detail below based on preferred embodiments shown in the accompanying drawings.

[0040] Fig. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. The image processing device 10 shown in Fig. 1 includes a server 12, a first terminal device 14, and a second terminal device 16 different from the first terminal device 14, which are connected via a network (communication line) 18. The image processing device 10 automatically creates a layout of composite images such as a photo album (photo book) using a group of images held by the first terminal device 14, that is, some images (photos) selected from a group of images held by a user.

[0041] In this embodiment, server 12 has a function of automatically creating a layout of a composite image from information on the composite image and a group of images for analysis transmitted from first terminal device 14, and a function of storing information on images to be used, candidate images, additional images, and layouts transmitted from first terminal device 14. Server 12 is not limited to one unit, and may be multiple units, and is configured by a workstation or the like having a control device, a storage device, a communication device (transmitter and receiver), etc.

[0042] The first and second terminal devices 14, 16 are user terminal devices used by the same user, and have functions such as displaying a composite image and editing its layout. The first and second terminal devices 14, 16 are configured by desktop PCs, notebook PCs, tablet PCs, or mobile terminals such as smartphones, each having a control device, an input device, a storage device, a communication device, and a display (display device). The number of terminal devices may be three or more.

[0043] The network 18 has a function of connecting the server 12, the first terminal device 14, and the second terminal device 16 to each other by at least one of wired and wireless communication, and transmitting and receiving various data between the server 12, the first terminal device 14, and the second terminal device 16. The network 18 is configured by, for example, Wi-Fi (Wireless Fidelity) and a telephone line (carrier line). In addition, multiple terminal devices of multiple users can be connected via the network 18.

[0044] Next, the server 12, the first terminal device 14 and the second terminal device 16 will be described in detail.

[0045] Fig. 2 is a block diagram showing the configuration of a server according to an embodiment. The server 12 shown in Fig. 2 includes a receiving unit 20, an analysis image group storage unit 22, an image analysis unit 24, an image selection unit 26, a layout creation unit 28, an information storage unit 30, and a transmission unit 32.

[0046] The receiving unit 20 receives various data transmitted from the first and second terminal devices 14, 16. In this embodiment, the receiving unit 20 receives information on a composite image that the user wishes to create, a group of images for analysis, images to be used, candidate images, additional images, and information on a layout after layout editing on the first terminal device 14, transmitted from the first terminal device 14. The receiving unit 20 also receives information on a layout after layout editing on the second terminal device 16, transmitted from the second terminal device 16.

[0047] The composite image information is information such as the size (A5, A4, A5 square, etc.) and number of pages (16 pages, 24 pages, 32 pages, 40 pages, 48 ​​pages, etc.) of the composite image that the user wants to create. The minimum number of images required to create the composite image is determined by the size of the composite image, the number of pages, etc. For example, if the number of pages in a photo album is 24, the minimum number of images required is determined to be 100.

[0048] An image group for analysis is an image group for analysis created from an image group held by the first terminal device 14, which has the resolution required for analyzing the image group and has a smaller amount of data than the image group. In this embodiment, the used image is an image from the image group that corresponds to the image used for analysis used in the composite image whose layout is automatically created by the image processing device 10. A candidate image is a part of the image group other than the used image, and is an image that can be used in a composite image while editing the layout on the second terminal device 16. When resuming layout editing on the second terminal device 16 that does not have the image group, the image processing device 10 selects, as a candidate image, from the image group, an image that is considered to be likely to be used in a composite image by the user while editing the layout on the second terminal device. The added image is an image that is newly added from the image group to the composite image by editing the layout on the first terminal device 14. In other words, the composite image after editing the layout on the first terminal device 14 includes the used image and the added image.

[0049] The layout information is information about the images included in the composite image, the arrangement of the images in the layout, the size of the images, and the like.

[0050] Next, the analysis image group storage unit 22 stores the analysis image group received by the receiving unit 20 from the first terminal device 14.

[0051] Next, the image analysis unit 24 analyzes the analysis images stored in the analysis image group storage unit 22 for one or more analysis items.

[0052] The analysis items are not limited to, but may include detection of subjects (people and non-people objects) appearing in an image, detection of the main character, detection of a specific subject specified by the user, detection of a person's face, detection of the number of people, detection of a person's facial expression, and detection of a person's behavior, etc. Analysis items may include detection of scenes captured in an image (night view, ocean, beach, sky, etc.), detection of events (sports day, wedding, graduation ceremony, etc.), and detection of user preferences (preferential use of ocean images, preferential use of mountain images, etc.). The analysis items may include an analysis of image quality, such as image brightness, color (saturation), contrast, and degree of blur. The analysis items may include analysis of image attributes, such as the shooting time and shooting location contained in additional information of the image, such as Exif (Exchangeable Image File Format).

[0053] The analysis items may include detection of an image containing a subject whose similarity is equal to or greater than a reference value. The method of calculating the similarity is not particularly limited, but the similarity between images can be detected based on a plurality of feature amounts of the images, for example. The feature amount is, for example, the image resolution, the amount of data, the degree of blur, the type of the main subject, the size of the main subject relative to the image, the position of the main subject, the color tone, or a combination of a plurality of these feature amounts. Furthermore, the feature amount may be one-dimensional or multiple-dimensional, output by a trained model trained in advance by supervised learning or unsupervised learning upon receiving an input of an image. Here, although there are cases where the feature amount output from the trained model cannot be interpreted by humans, at least a group of values ​​that is uniquely output when one image is input can be used as the feature amount of the present invention. Each image included in the image group is distributed in a two-dimensional feature amount space with, for example, the first feature amount on the horizontal axis and the second feature amount on the vertical axis, and it can be said that the smaller the distance between the images in the feature amount space, the greater the similarity between the images.

[0054] The analysis items may include an analysis of the entire group of images for analysis, such as an analysis of the number of images per unit time and the distribution (density) of images. The results of the analysis of the entire group of images for analysis can be used to classify (group) the images to be placed on the double-page spreads of a photo album. For example, if the group of images is divided into spring, summer, fall, and winter groups, the number of images taken for each season may vary, such as there being more summer images. If there are more summer images than images from other seasons, the number of summer images used in the photo album can be increased in proportion to the number of summer images, and the number of double-page spreads using summer images can be increased from 4 to 5 out of the 12 double-page spreads in the photo album.

[0055] The analysis items may include calculation of a score (evaluation value) of an image. The method of calculating the score is not particularly limited, but for example, the total score of a plurality of analysis items can be used as the score for each image. Also, the scores of images can be weighted, such as by weighting the score of an image that includes the main character, or by weighting the score of an image taken on a shooting date with a large number of shots.

[0056] For example, as shown in Table 1, for each image (file name: images A, B, C, D, E, F, G, ...) included in the image group, scores for multiple analysis items (degree of blur, brightness, color, and number of people (faces)) are calculated, and the total score of the multiple analysis items is the score for each image. For example, image A has a score of 80 points for the degree of blur, a score of 90 points for brightness, a score of 85 points for color, a score of 100 points for one face, and a score of 200 points for people (faces), for a total value (score) of 455 points.

[0057] [Table 1]

[0058] Next, the use image selection unit 26 selects an image to be used for analysis, which is to be used in the composite image, from the group of images to be analyzed, based on the results of the analysis of the group of images to be analyzed.

[0059] The method of selecting images to be used for analysis is not particularly limited, but for example, the number of images to be used in the composite image can be selected from images of a specific subject, images of a specific scene, images of a specific event, or images that match the user's preferences among the group of images to be analyzed. The number of images to be used in the composite image can also be selected in descending order of image score. Furthermore, images can be selected by combining the results of multiple analysis items.

[0060] Next, the layout creation unit 28 automatically creates a layout of the composite image using information on the image used for analysis and the composite image.

[0061] For example, in the case of a photo album, the layout creation unit 28 classifies the images to be used for analysis into a number of groups corresponding to the number of pages in the photo album based on additional information about the images, etc., and for each page, from the images to be used for analysis classified into the groups corresponding to each page, selects the number of images to be used for analysis to be placed on each page based on the image scores, etc., and creates a layout using the selected number of images to be used for analysis.

[0062] Although the method of creating the layout is not particularly limited, it is possible to give priority to images with high scores in the layout based on the results of the analysis. For example, images with high scores may be placed in the center of the composite image, or the size of images with high scores may be increased.

[0063] Next, the information storage unit 30 stores various data in the server 12. In the present embodiment, the information storage unit 30 stores information on the layout automatically created by the layout creation unit 28, information on the edited layout transmitted from the first and second terminal devices 14, 16, etc. In addition, the information storage unit 30 integrates and stores the used image, candidate image, and additional image transmitted from the first terminal device 14 for each composite image.

[0064] It is preferable that the information storage unit 30 stores the used image and added image separately from the candidate image. This allows the used image and added image (images currently being used in the composite image) to be displayed separately from the candidate image (images that can be newly added to the composite image) when the layout editing is resumed on the second terminal device 16, so that the user can recognize at a glance the image used in the composite image and the image that can be newly added to the composite image.

[0065] The transmission unit 32 transmits various types of data to the first and second terminal devices 14, 16. In this embodiment, the transmission unit 32 transmits information on the automatically created layout, the results of the analysis of the group of images for analysis, and the like to the first terminal device 14. In addition, when editing of the layout is resumed on the second terminal device 16, the transmission unit 32 transmits the images to be used, the candidate images, the additional images, information on the layout, and the like to the second terminal device 16.

[0066] Fig. 3 is a block diagram showing the configuration of a first terminal device according to an embodiment. The first terminal device 14 shown in Fig. 3 includes a receiving unit 34, an instruction acquiring unit 36, an image storage unit 38, an image display unit 40, an analysis image group creating unit 42, a layout editing unit 44, a candidate image selecting unit 46, and a transmitting unit 48.

[0067] The receiving unit 34 receives (downloads) various data transmitted from the server 12. In the present embodiment, the receiving unit 34 receives, from the server 12, the results of the analysis of the group of images for analysis, information on the layout of an automatically created composite image, and the like.

[0068] Next, the instruction acquisition unit 36 ​​acquires various instructions input by the user using an input device such as a keyboard, a mouse, a touch pad, or a touch panel.

[0069] The instructions from the user are not particularly limited, but in this embodiment, include instructions to specify information about a composite image that the user wants to create, instructions to specify information about a group of images to be used in the composite image, instructions to edit the layout, instructions to end layout editing, etc.

[0070] The user can input, as information about the image group, instructions to specify the folder location of the image group stored in the first terminal device 14, instructions to read the image group stored in an external storage device such as an SD (Secure Digital) card or USB (Universal Serial Bus) memory, and instructions to specify the image group to be used in the composite image from the image group stored in the first terminal device 14 based on the image capture period, etc., to specify the image group to be used in the composite image all at once.

[0071] Next, the image storage unit 38 stores a group of images that can be used in a composite image, which are specified by instructions from the user.

[0072] Next, the image display unit 40 displays various images on the first terminal device 14. In the present embodiment, the image display unit 40 displays an input screen for the user to specify information on the composite image he or she wishes to create and information on the image group, a layout editing screen for editing the layout of the composite image, and each image included in the image group. The image display unit 40 also displays a composite image whose layout has been automatically created based on the layout information transmitted from the server 12.

[0073] Next, the analysis image group creation unit 42 creates an analysis image group from the images stored in the image storage unit 38 .

[0074] The resolution required for analysis is not particularly limited, but can be a resolution lower than the resolution of each image contained in the original image group, for example, a resolution higher than the VGA (Video Graphics Array) resolution of 640 pixels wide by 480 pixels high. Methods for reducing the amount of data include, but are not limited to, resizing the images by reducing the physical width and height of each image included in the original image group, and compressing the images.

[0075] Next, the layout editing unit 44 edits the layout of the composite image displayed on the image display unit 40 in accordance with instructions from the user while the layout is being edited on the first terminal device 14.

[0076] Next, the candidate image selection unit 46 selects, in the background while the layout is being edited on the first terminal device 14, from the group of images based on a set criterion (candidate image selection criterion), a candidate image which is a portion of images other than the images used in the composite image and which can be used in the composite image when the layout editing is resumed on the second terminal device 16. In other words, a candidate image is an image which the user is likely to want to use in the composite image when the layout editing is resumed on the second terminal device 16.

[0077] The criteria for the candidate image selection unit 46 to select the candidate images are not particularly limited, but may be, for example, the following methods (1) to (4). Furthermore, two or more of these methods (1) to (4) may be combined.

[0078] (1) Use of the results of analysis of analytical images Candidate images can be selected from a group of images other than the image to be used based on the score of each image for analysis included in the group of images for analysis. For example, from a group of images other than the images used, images corresponding to images for analysis whose scores are equal to or greater than a predetermined standard value (threshold value), or images corresponding to a predetermined percentage of images for analysis whose scores are the highest (for example, 20% of the images in the group of images), are selected as candidate images. The reference score and ratio can be set arbitrarily.

[0079] Table 2 adds the items "Laid out" and "Candidate image" to the above-mentioned Table 1. In the "Laid out" item of Table 2, the image to be used is represented by "Y", and images other than the image to be used are represented by "X". In the "Candidate image" item, the image to be used is represented by "-", the image selected as the candidate image is represented by "Y", and the image not selected as the candidate image is represented by "X". As shown in Table 2, from among images A, B, C, D, E, F, G, ..., image C with a score of 555 and image C with a score of 455 are selected as the images to be used, in order of their scores. In this case, when selecting, as candidate images, the top 20% of images (one image in the case of Table 2) from images B, D, E, F, G, ... other than images A and C to be used, image E with a score of 435, which is the next highest score after images A and C, is selected as the candidate image.

[0080] [Table 2]

[0081] (2) Editing operations by users For example, information on an editing operation image on which an editing operation has been performed by a user is stored, and from a group of images other than the used images, an image on which an editing operation has been performed by the user while editing the layout on the first terminal device 14 and which is not used in the composite image after editing the layout on the first terminal device 14 can be selected as a candidate image based on the stored information on the editing operation image.

[0082] For example, an image that was once added to the composite image from the group of images other than the images used during layout editing on the first terminal device 14, but was subsequently deleted from the composite image and is no longer included in the composite image after the layout is completed, is considered to be an image that is likely to be added to the composite image when layout editing is resumed on the second terminal device 16. Also, an image that is enlarged and displayed during layout editing on the first terminal device 14 is considered to be an image that is of interest to the user. Therefore, these images are selected as candidate images.

[0083] (3) Attributes of additional images From the group of images other than the used image, an image corresponding to the analysis image whose attribute matches the attribute of the added image, and an image corresponding to the analysis image whose attribute is within a range determined from the attribute of the added image can be selected as a candidate image. In this way, by selecting an image whose attribute is the same as or close to the added image as a candidate image, it is possible to provide the user with a candidate image that is likely to be added to the composite image when resuming editing of the layout on the second terminal device 16.

[0084] For example, images with the same shooting date as the additional image and images with shooting dates and times within a specified range from the shooting date and time of the additional image (images taken several days before or after the shooting date of the additional image, images taken several hours before or after the shooting time of the additional image, etc.) are selected as the selected images. In addition, using location information from a Global Positioning System (GPS), images photographed at the same location as the additional image and images photographed at locations within a specified range from the location of the additional image are selected as candidate images. Furthermore, based on the results of the analysis of the group of images for analysis, images that contain the same subjects (people and non-people objects) and images whose subject similarity is equal to or exceeds a reference value are selected as candidate images.

[0085] On the other hand, an image whose attribute does not match the attribute of the added image and whose attribute corresponds to an analysis image outside a range determined from the attribute of the added image can be selected as a candidate image. In this way, by deliberately selecting an image whose attribute does not match the attribute of the added image and that is far from the added image as a candidate image, a new candidate image that the user did not expect can be provided to the user as an image to be added to the composite image when editing of the layout is resumed on the second terminal device 16.

[0086] For example, an image whose shooting date is different from that of the additional image and that is taken on a shooting date outside a specified range (e.g., if images taken on the first and third days are added as additional images to a composite image from a group of images taken on the first, second, and third days, an image taken on the second day, etc.) is selected as a candidate image. In addition, by using GPS location information, images that are taken at a location different from that of the additional image and outside the specified range are selected as candidate images. Furthermore, based on the results of the analysis of the group of images for analysis, images that contain different subjects and have a subject similarity less than a reference value are selected as candidate images.

[0087] If the composite image is a photo album, the images are divided into groups of images that can be used on each page of the photo album, and a layout is created on each page using images corresponding to each group. For example, if the photo album has 12 pages, the images are divided into groups of images from January to December that correspond to the 12 pages, and a layout is created on each page using images corresponding to each group. In this case, rather than selecting, for each page, from among the images in the group other than the used images corresponding to other pages, an image whose attribute matches or does not match the attributes of the added image as a candidate image, it is preferable to select, for each page, from among the images in the group other than the used images corresponding to each page, an image whose attribute matches or does not match the attributes of the added image as a candidate image. This makes it possible to prevent suggesting images that seem out of place, such as proposing an image from a group corresponding to a January page as a candidate image for a July page.

[0088] (4) User's past order history For example, the user's past order history can be stored, and candidate images can be selected from a group of images other than the image to be used, based on the user's past order history.

[0089] For example, based on the result of the image similarity analysis, an image that includes the same subject as that in an image previously ordered and a subject with a similarity equal to or greater than a reference value is selected as a candidate image. In this way, by selecting an image with the same or similar similarity as the added image as a candidate image, it is possible to provide the user with a candidate image that is likely to be added to the composite image when resuming layout editing on the second terminal device 16.

[0090] Also, based on the similarity of the images, images that include a subject different from the subject in the previously ordered image and a subject with a similarity below a reference value are selected as candidate images. In this way, by deliberately selecting an image with a similarity far from the added image as a candidate image, it is possible to provide the user with a new candidate image that the user did not expect as an image to be added to the composite image when resuming layout editing on the second terminal device 16.

[0091] The number of candidate images is not particularly limited, but for example, a predetermined number of images or a predetermined percentage of the images (e.g., 20% of the images) of the group of images can be transmitted from the first terminal device 14 to the server 12 as candidate images. Furthermore, the first terminal device 14 may transmit to the server 12 as candidate images a number of images according to the status of the network 18. For example, the number of candidate images is determined according to the type of the network 18, such as how many images if the type of the network 18 is Wi-Fi and how many images if the type of the network 18 is a telephone line. Alternatively, the number of candidate images may be determined according to the communication speed of the network 18, such as how many images if the communication speed of the network 18 is equal to or higher than a reference value and how many images if the communication speed is lower than the reference value.

[0092] Next, the transmission unit 48 transmits (uploads) various types of data to the server 12. In the case of this embodiment, the transmission unit 48 transmits information on the composite image and a group of images for analysis, etc. to the server 12. In addition, the transmission unit 48 transmits candidate images to the server 12 while the layout is being edited on the first terminal device 14, and transmits to the server 12 images used, images added, information on the layout after layout editing on the first terminal device 14, etc. after the layout is edited on the first terminal device 14.

[0093] 4 is a block diagram showing the configuration of a second terminal device 16 according to an embodiment of the present invention. The second terminal device 16 shown in FIG. 4 includes a receiving unit 54, an instruction acquiring unit 56, an image display unit 60, a layout editing unit 64, and a transmitting unit 68.

[0094] The receiving unit 54 receives various data transmitted from the server 12. In the present embodiment, the receiving unit 54 receives from the server 12 information such as the image to be used, the candidate image, the additional image, and the layout after editing on the first terminal device 14.

[0095] Next, the instruction acquisition unit 56 acquires various instructions input by the user using the input device.

[0096] Instructions from the user are not particularly limited, but in this embodiment include an instruction to specify a composite image for resuming layout editing, an instruction to edit the layout, and an instruction to end layout editing.

[0097] Next, the image display unit 60 displays various images on the second terminal device 16. In this embodiment, the image display unit 60 displays an input screen for the user to specify a composite image for which the user wants to resume layout editing. Furthermore, when resuming layout editing on the second terminal device 16, the image display unit 60 displays a layout editing screen based on the information on the used image, added image, and layout transmitted from the server 12, and displays the used image, candidate image, added image, etc. as images that can be used in the composite image.

[0098] Next, the layout editing unit 64 edits the layout of the composite image displayed on the image display unit 60 in accordance with instructions from the user while the layout is being edited on the second terminal device 16.

[0099] Next, the transmission unit 68 transmits various data to the server 12. In this embodiment, the transmission unit 68 transmits to the server 12 information on the layout after the layout has been edited on the second terminal device 16, and the like.

[0100] Next, the operation of the image processing device 10 of the present invention will be described using an example in which the first terminal device 14 is a PC and the second terminal device 16 is a smartphone.

[0101] The user edits the layout of photo album pages on PC 14 using a PC application that realizes the functions of image processing device 10, and after finishing the layout editing work, moves from PC 14 to smartphone 16 and resumes layout editing on smartphone 16 using a smartphone application that also realizes the functions of image processing device 10.

[0102] Fig. 5 is a conceptual diagram of an embodiment showing the operation of an image processing device. The leftmost vertical line in Fig. 5 represents the flow of a user's operation from top to bottom, and similarly, from the leftmost vertical line to the right, the flow of an operation of a PC application, the flow of an operation of a smartphone application, and the flow of an operation of the server 12 are represented.

[0103] As shown in FIG. 5, the user launches a PC application on PC 14 having a group of images (step S1), and when starting to create a photo album, enters information such as the size and number of pages of the photo album the user wants to create, as well as information about the group of images to be used in the photo album, on an input screen (product order screen) displayed on image display unit 40 (step S2) (step S3).

[0104] In response to this, the instruction acquisition unit 36 ​​acquires the photo album information and image group information input by the user, and acquires the image group specified by the image group information and stores it in the image storage unit 38 (step S4). Next, the analysis image group creation unit 42 creates an analysis image group from the images stored in the image storage unit 38, and the transmission unit 48 transmits the photo album information and the analysis image group to the server 12 (step S4). Since the analysis image group has a significantly smaller amount of data than the original image group, the transmission of the analysis image group is completed in a much shorter time than the transmission of the image group.

[0105] Next, in the server 12, the receiving unit 20 receives the photo album information and the group of images to be analyzed transmitted from the PC 14 (step S5). Next, the image analysis unit 24 analyzes the group of images for analysis, the image to be used selection unit 26 selects from the group of images for analysis images to be used for analysis on each page of the photo album based on the results of the analysis of the group of images for analysis, and the layout creation unit 28 automatically creates a layout for each page of the photo album using the photo album information and the images to be used for analysis (step S5). Next, the layout information of the photo album is stored in the information storage unit 30, and the layout information and the results of the analysis of the group of images for analysis are transmitted to the PC 14 by the transmission unit 32 (step S6).

[0106] Next, in PC 14, the receiving unit 34 receives the layout information and the analysis results of the group of images for analysis transmitted from the server 12, and based on the layout information, an editing screen for the photo album layout is displayed on the image display unit 40 (step S7). In addition to the photo album page being edited, the layout editing screen displays a list of images included in the image group as images that can be used in the photo album.

[0107] Next, the user views the layout of the page being edited, and inputs an instruction to edit the layout (step S8). Since the PC 14 has the image group, the user can use all images contained in the image group to perform editing work while editing the layout on the PC 14. It is not essential that the user edits the layout on the PC 14.

[0108] The automatically created layout may not contain the images that the user wanted to include in the photo album, or may contain images that the user did not want to include in the photo album. In this case, the user browses the layout of the page being edited and performs editing work such as adding new images to the photo album from the group of images other than the images used, or deleting unnecessary images from the photo album.

[0109] In response to this, the instruction acquisition unit 36 ​​acquires an instruction to edit the layout, and in response to this instruction, the layout editing unit 44 edits the layout (step S9). Furthermore, while the layout is being edited, the candidate image selection unit 46 selects candidate images from a group of images other than the images to be used based on predetermined criteria (step S9), and the transmission unit 48 transmits the selected candidate images to the server 12 in real time in the background, also while the layout is being edited (step S10).

[0110] In this way, by selecting only candidate images from the image group and transmitting them to server 12, the user's waiting time for transmitting images that can be used in the photo album to server 12 can be significantly reduced compared to transmitting all images included in the image group to server 12. Furthermore, by transmitting candidate images to server 12 while the layout is being edited on PC 14, i.e., in the background of the user's layout editing operation, the transmission time experienced by the user for transmitting the candidate images can be significantly reduced compared to when the candidate images are transmitted to server 12 after the layout has been edited and a dedicated time is set aside for transmitting the candidate images to server 12.

[0111] Next, in the server 12, the receiving unit 20 receives the candidate image transmitted from the PC 14, and stores it in the information storage unit 30 (step S11).

[0112] When the layout editing on the PC 14 is completed, the user inputs an instruction to end the layout editing (step S12).

[0113] In response to this, the transmitting unit 48 in the PC 14 transmits information on the images used, the images added, and the layout after editing to the server 12 (step S13).

[0114] As in this embodiment, when a candidate image is selected and transmitted to the server 12 during layout editing on the PC 14, an additional image added to the photo album by the user after the layout editing on the PC 14 may have already been transmitted as a candidate image to the server 12. In this case, since this additional image has already been transmitted to the server 12 as a candidate image, it is not necessary to transmit this additional image to the server 12 again, but it is desirable to transmit information to the server 12 for distinguishing between the candidate image and the additional image.

[0115] Next, in the server 12, the receiving unit 20 receives the images to be used, the images to be added, and information on the edited layout, etc., transmitted from the PC 14, and stores them in the information storage unit 30 (step S14).

[0116] The information storage unit 30 stores, for each photo album, an integrated set of information on the used image, candidate image, additional image, and layout after editing on the PC 14. As a result, when resuming editing of the layout on the smartphone 16, by specifying the photo album in which editing is to be resumed, it is possible to specify information on the used image, candidate image, additional image, and layout after editing on the PC 14 for this photo album.

[0117] Next, on the smartphone 16, the user launches a smartphone application and, when resuming layout editing, inputs information specifying the photo album in which the user wants to resume layout editing on the input screen displayed on the image display unit 60 (step S15).

[0118] In response to this, the instruction acquisition unit 56 acquires the photo album designation information input by the user, and the transmission unit 48 transmits the photo album designation information to the server 12 (step S16).

[0119] Next, in the server 12, the receiving unit 20 receives the photo album designation information transmitted from the smartphone 16 (step S17). Next, the transmitting unit 32 transmits to the smartphone 16 information such as the images to be used, the candidate images, the additional images, and the layout of the photo album corresponding to the photo album designation information stored in the information storage unit 30 (step S18).

[0120] Next, in the smartphone 16, the receiving unit 54 receives the used images, candidate images, added images, layout information, etc. of the photo album transmitted from the server 12, and based on the layout information, an editing screen for the layout of the photo album is displayed on the image display unit 60 (step S19). In addition to the page being edited, the used images, added images, and candidate images are displayed as images that can be used in the photo album on the layout editing screen.

[0121] Next, the user views the layout of the page being edited, and inputs an instruction to edit the layout while editing the layout on the smartphone 16 (step S20). Although the smartphone 16 does not have a set of images, the user can use candidate images as well as the used image and the added image to perform editing work while editing the layout on the smartphone 16. It is not essential that the user edits the layout on the smartphone 16.

[0122] In response to this, the instruction acquisition unit 56 acquires an instruction to edit the layout, and in response to this instruction, the layout editing unit 64 edits the layout (step S21).

[0123] When the layout editing on the smartphone 16 is finished, the user inputs an instruction to end the layout editing (step S20).

[0124] In response to this, the transmitting unit 68 of the smartphone 16 transmits information on the edited layout and the like to the server 12 (step S21).

[0125] Next, in the server 12, the receiving unit 20 receives the layout information transmitted from the smartphone 16, and stores it in the information storage unit 30 (step S22).

[0126] In this way, by storing information on the images used, the additional images, and the layout in the server 12, layout editing can be resumed seamlessly even if the user moves from a PC 14 that has a group of images to a smartphone 16 that does not have the group of images. Furthermore, by storing the candidate images in the server 12, even if the user moves from the PC 14 to the smartphone 16 to resume editing the layout, while editing the layout on the smartphone 16, the user can edit the layout using not only the used image and the additional image but also the candidate images, thereby improving the convenience of editing and widening the scope of editing.

[0127] Fig. 6 is a conceptual diagram showing an embodiment of a display screen of a PC application, Fig. 6 shows a layout editing screen for editing the layout of each page of a photo album.

[0128] The center of the layout editing screen displays the page of the photo album being edited; in the illustrated example, the double-page spread 70 being edited, and below that, the page numbers of the double-page spread 70 being edited ("Page 2" and "Page 3") are displayed.

[0129] An edit menu 72 is displayed on the right side of the layout editing screen, that is, to the right of the double-page spread 70 being edited. From top to bottom, the edit menu 72 displays buttons such as "Preview," "Help," "Photo Edit," etc.

[0130] A list 74 of thumbnail images of the pages of the photo album is displayed on the left side of the layout editing screen, that is, on the left side of the double-page spread 70 being edited, and a scroll bar is provided on the right side of this page list 74. In the illustrated example, the page list 74 displays a plurality of pages included in the photo album in ascending order of page number from top to bottom, and the page number is displayed below the thumbnail image of each page.

[0131] A list 76 of thumbnail images of the images included in the image group is displayed at the bottom of the layout editing screen, that is, below the double-page spread 70 being edited. In the illustrated example, this image list 76 displays at least some of the images included in the image group in chronological order from left to right in descending order of the image capture time. In addition, a "magnifying glass" button for enlarging the image is displayed on each image, and images with a score of a certain value or more are marked with a star mark indicating that they are recommended images. By marking images with a star mark, the user can recognize at a glance that an image marked with a star mark is a recommended image.

[0132] Next, an operation will be described in which the user edits the layout of the double-spread page 70 being edited on the PC 14.

[0133] A user can edit the layout on the PC using input devices such as a keyboard and a mouse.

[0134] As shown in FIG. 6, a list 74 of photo album pages is displayed on the left side of the layout editing screen. In this state, the user can view at least some of the thumbnail images of the pages of the photo album as a page list 74. In addition, the user can scroll up and down to display and view thumbnail images of the remaining pages that are not displayed among all the thumbnail images of the pages.

[0135] When the user clicks with a mouse on a thumbnail image of a page whose layout he / she wishes to edit from the page list 74, the thumbnail image of the clicked page is selected from among the thumbnail images of the pages displayed in the page list 74, and the page corresponding to the thumbnail image of the clicked page is displayed in the center of the layout editing screen as the two-page spread 70 being edited. In this state, the user can view the double-page spread 70 being edited.

[0136] Furthermore, the user can display the page corresponding to the thumbnail image of another page by clicking the thumbnail image of the other page from the page list 74. Furthermore, the user can display the previous page of the double-page spread 70 being edited by clicking the "<" button to the left of the page number, and can display the next page of the double-page spread 70 being edited by clicking the ">" button to the right of the page number.

[0137] Next, when the user clicks the "Photo Edit" button in the edit menu 72 on the layout edit screen, a list of images 76 is displayed at the bottom of the layout edit screen. In this state, the user can view at least some of the thumbnail images of the images included in the image group as an image list 76. Furthermore, the user can click the "View all photos" button to display the remaining images in the image group that are not displayed in the image list 76. Furthermore, when the user clicks with the mouse on the "magnifying glass" button of one of the images displayed in the image list 76, the clicked image is enlarged and displayed on the image display unit 40. By enlarging the image, the user can view the enlarged image and check its details.

[0138] Next, when the user clicks an image from the image list 76 that the user wants to add to the double-page spread 70 being edited, the clicked image becomes selected. The user can select one or more images from the image list 76 all at once. Next, when the user drags and drops a selected image from the image list 76 onto the page being edited with the mouse, the image corresponding to the selected image is automatically laid out on the double-page spread 70 being edited.

[0139] In this way, the user can select and add an image corresponding to the image that the user wishes to add to the double-page spread 70 being edited from the image list 76. Editing the page layout is not limited to adding images to the page being edited, but various other edits can be performed, such as rotating the image frame, rotating and cropping an image, deleting a photo from the page being edited, and changing the position of photos on the page.

[0140] Fig. 7 is a conceptual diagram of an embodiment showing a display screen of a smartphone application. The left side of Fig. 7 shows a layout editing screen for editing the layout of a page, and the right side shows an image display screen that displays a list of images that can be used on the page being edited.

[0141] The area above the center of the layout editing screen displays the page of the photo album being edited; in this example, double-page spread 80; below that, the page number of the double-page spread 80 being edited ("Page 2 | Page 3") is displayed.

[0142] A main menu 82 is displayed at the bottom of the layout editing screen. In the main menu 82, buttons such as "Photo Edit," "Layout," "Background Color," etc. are displayed from left to right. In the illustrated example, a "Photo Edit" submenu 84 is displayed at the top of the main menu 82. In the "Photo Edit" submenu 84, buttons such as "Frame Rotation," "Photo Rotation / Cut," "Remove," "Swap," and "Add" are displayed from upper left to lower right.

[0143] The area extending from the top to about 1 / 3 of the image display screen displays a double-page spread 80 being edited, and the area below that, extending from the bottom to about 2 / 3 of the image display screen, displays a list 86 of thumbnail images of the images included in the image group. Also, at the bottom of the image display screen, a button 88 for "Add selected photos and layout" is displayed over the image list 86 in gray, indicating that it is disabled.

[0144] In the illustrated example, in the image list 86, at least some of the analysis images included in the analysis image group are displayed in chronological order from the upper left to the lower right in descending order of the image capture time. Among the analysis image group, the used image and the added image are displayed as "in use" and are displayed in blue-purple to indicate that they are used in the double-page spread 80 being edited. The candidate images are displayed surrounded by a red frame and are displayed in color to indicate that they can be used in the double-page spread 80 being edited. Among the analysis image group, unused images other than the used image, the added image, and the candidate images are displayed in gray to indicate that they cannot be used in the double-page spread 80 being edited. In addition, each image is displayed with a "magnifying glass" button for enlarging the image, and images with a score of a certain value or more are marked with a star mark indicating that they are recommended images.

[0145] This allows the user to recognize at a glance that the images displayed in bluish purple are the images used and added images used in the double-page spread 80 being edited, that the images displayed in color are candidate images that can be used in the double-page spread 80 being edited, and that the images displayed in gray are unused images that cannot be used in the double-page spread 80 being edited. Also, the user can recognize at a glance that the images marked with a star are recommended images.

[0146] The colors for distinguishing between the used image and the additional image, the candidate image and the unused image are not particularly limited. Also, the frame surrounding the candidate image may or may not be displayed. Furthermore, the unused image may or may not be displayed.

[0147] Next, an operation will be described when the user edits the layout of the double-page spread 80 being edited on the smartphone 16.

[0148] The smartphone's image display unit 60 is composed of a touch panel, and by touching the touch panel with a finger, the user can use the touch panel not only as the image display unit 60 but also as an instruction acquisition unit 56 to perform layout editing work.

[0149] As shown on the left side of FIG. 7, a double-page spread 80 being edited is displayed on the layout editing screen. In this state, the user can view the double-page spread 80 being edited. In addition, the user can display the previous page of the double-page spread 80 being edited by tapping the "<" button to the left of the page number with his / her finger, and can display the next page of the double-page spread 80 being edited by tapping the ">" button to the right of the page number.

[0150] Next, when the user taps the “Photo Edit” button in the main menu 82 on the layout editing screen, a “Photo Edit” submenu 84 is displayed above the main menu 82 .

[0151] Next, when the user taps the "Add" button in the submenu 84, the layout editing screen is switched to an image display screen. The image display screen displays the page being edited, in the illustrated example, a two-page spread 80, and a list of images 86. In this state, the user can view at least a part of the analysis images included in the group of images for analysis. Furthermore, the user can swipe the area of ​​the image list 86 up or down with a finger to display the remaining images in the group of images for analysis that are not displayed in the image list 86. Furthermore, when the user taps the "magnifying glass" button of one of the images displayed in the image list 86, the tapped image is enlarged and displayed on the image display unit 60. By enlarging the image, the user can view the enlarged image and check its details.

[0152] Next, when the user taps a candidate image that the user wants to add to the double-page spread 80 being edited from the image list 86, the tapped candidate image becomes selected. The user can select one or more candidate images at once from among the candidate images. Furthermore, when one or more candidate images are selected, a button 88 for "Add selected photos and layout" becomes active. Next, when the user taps the enabled "Add selected photos and layout" button 88, the selected candidate images are automatically laid out on the double-page spread 80 being edited. Also, the selected candidate image becomes an added image, the red border disappears, and it is displayed in blue-purple.

[0153] In this way, the user can select from the candidate images a candidate image that the user wishes to add to the double-page spread 80 that the user is editing, and add it. Editing the page layout is not limited to adding an image to the page that is being edited, but various other edits can be performed, such as rotating an image frame, rotating or cropping a photo, deleting a photo from the page that is being edited, and changing the position of a photo on the page.

[0154] In this embodiment, the layout of the composite image is created by the server 12, but the present invention is not limited to this, and the layout of the composite image may be created by the first terminal device 14. In this case, the image analysis unit of the first terminal device 14 analyzes the image group, the use image selection unit selects the use image from the image group based on the result of the image group analysis, the layout creation unit creates the layout of the composite image using the use image and composite image information, and the image display unit displays the composite image whose layout has been automatically created. Also, the information storage unit of the server 12 stores the layout information transmitted from the first terminal device 14. Since the image group contains more information than the image group for analysis, by analyzing the image group, a layout can be created based on more accurate analysis results than by analyzing the image group for analysis.

[0155] When creating a layout on the first terminal device 14, as in the case of creating a layout on the server 12, the image analysis unit of the first terminal device 14 calculates a score for each image included in the image group as a result of analyzing the image group, and the candidate image selection unit can select as candidate images from the image group other than the used image those images whose scores are equal to or above a predetermined standard value, or images in a predetermined percentage of the highest scores.

[0156] In addition, the image analysis unit of the first terminal device 14 analyzes the attributes of each image included in the image group, and the candidate image selection unit selects as candidate images from the image group other than the used image an image whose attribute matches the attribute of the added image and an image whose attribute is within a defined range from the attributes of the added image, or selects as candidate images an image whose attribute does not match the attribute of the added image and whose attribute is outside a defined range from the attributes of the added image.

[0157] In the present embodiment, the first terminal device 14 selects candidate images from a group of images, but this is not limited thereto. For example, a candidate image selection unit of server 12 may select a candidate image for analysis from a group of images for analysis other than the image used for analysis, and a candidate image selection unit of first terminal device 14 may select a candidate image from the group of images that corresponds to the candidate image for analysis transmitted from server 12 and received by first terminal device 14.

[0158] The timing of transmitting the used image and the candidate images from the first terminal device 14 to the server 12 is not particularly limited, so long as it is after the automatic generation of the layout. For example, while the layout is being edited on the first terminal device 14, at least some of the images included in the group consisting of the used image and the candidate image may be transmitted from the first terminal device 14 to the server 12 in real time in the background. If it is not possible to transmit all of the images included in the group during layout editing, then after the layout is edited on the first terminal device 14, the remaining images included in the group, the additional images, and information on the edited layout are transmitted from the first terminal device 14 to the server 12 in response to an instruction from the user. In this way, by transmitting not only candidate images but also the image to be used to the server 12 while editing the layout on the first terminal device 14, the transmission time experienced by the user for transmitting the image to be used can be significantly reduced compared to setting aside a dedicated time for transmitting the image to be used to the server 12 after editing the layout, and then transmitting the image to be used to the server 12 after setting aside a dedicated time for transmitting the image to be used to the server 12 after editing the layout.

[0159] Alternatively, after the layout has been edited on the first terminal device 14, rather than during the layout editing, information on the images to be used, the candidate images, the additional images and the edited layout may be transmitted from the first terminal device 14 to the server 12 in response to instructions from the user.

[0160] Furthermore, in response to a notification sent from the second terminal device 16 and received by the first terminal device 14, images from the group of images corresponding to information on images other than the used image, candidate image, and additional image contained in the notification may be sent from the first terminal device 14 to the server 12 as candidate images. When the first terminal device 14 is a smartphone and the second terminal device 16 is a PC, a notification including information on an image that the user wants to use in the composite image is sent from the PC to the smartphone. For example, when the user clicks on an image displayed in gray, indicating that it cannot be used in the composite image, during layout editing on the PC, a notification including information on the clicked image is sent to the smartphone. A user usually carries a smartphone with him or her and has the power turned on. When a notification is sent from the PC to the smartphone, an unused image corresponding to the image information included in the notification is sent from the smartphone to the server 12 as a candidate image from the image group. When editing the layout on the smartphone, this unused image is sent from the server 12 and received by the smartphone. This allows the user to use unused images in the composite image even if they have not been selected as candidate images.

[0161] In the image processing device 10, the user can input instructions specifying images that the user wants to include in the composite image and images that the user does not want to include in the composite image. When an instruction specifying an image that the user wants to include in the composite image is input, the layout creation unit 28 creates a layout using the image corresponding to this instruction in the composite image. On the other hand, when an instruction specifying an image that the user does not want to include in the composite image is input, the layout creation unit 28 creates a layout without using the image corresponding to this instruction in the composite image. This makes it possible to automatically create a composite image from a group of images that contains images that the user wants to include and does not contain images that the user does not want to include. This eliminates the need for the user to perform editing tasks such as adding images that the user wanted to include to the composite image or deleting images that the user did not want to include from the composite image.

[0162] When the user resumes the layout editing on the second terminal device 16, it is not essential that the user inputs information specifying the composite image for which the user wishes to resume the layout editing. For example, based on login information (such as account name and password) when the user logs in to the server 12, when the user logs in to the server 12 from the second terminal device 16 using the same login information as when the user logged in to the server 12 from the first terminal device 14, layout information may be automatically acquired from the server 12. Alternatively, after logging in to the server 12 from the second terminal device 16 using the same login information as when the user logged in to the server 12 from the first terminal device 14, the layout information may be acquired from the server 12 when the user taps a "resume editing button" displayed on the input screen. In addition, when logging in to the server 12 from the first terminal device 14, the MAC (Media Access Control) address information of the first terminal device 14 may be obtained, and when the same login information is used to log in to the server 12 from the second terminal device 16, layout information may be automatically obtained from the server 12.

[0163] Instead of the image to be used, the candidate image, and the additional image stored in the information storage unit 30, the analysis image corresponding to the image to be used, the analysis image corresponding to the candidate image, and the analysis image corresponding to the additional image stored in the analysis image group storage unit 22 may be transmitted from the server 12 and received by the second terminal device 16, and the analysis image group may be displayed on the second terminal device 16 as images that can be used in the composite image, instead of the original image group. In this way, by sending the analysis image when resuming layout editing on the second terminal device 16, the user's waiting time for the transmission of a group of images that can be used in the composite image can be significantly reduced compared to sending the original images.

[0164] In the above embodiment, an example has been described in which the first terminal device 14 is a PC and the second terminal device 16 is a smartphone, but this is not limited to this, and the same applies when the first terminal device 14 is a smartphone and the second terminal device 16 is a PC, when both the first and second terminal devices 14, 16 are PCs or smartphones, or when each of the first and second terminal devices 14, 16 is a terminal device other than a PC or a smartphone.

[0165] In the device of the present invention, the hardware configuration of the processing unit that executes various processes, such as the analysis image group memory unit 22, image analysis unit 24, image to be used selection unit 26, layout creation unit 28, instruction acquisition units 36, 56, analysis image group creation unit 42, layout editing units 44, 64 and candidate image selection unit 46, may be dedicated hardware or various processors or computers that execute programs.

[0166] The receiving units 20, 34, 54 and the transmitting units 32, 48, 68 are communication devices (receiving circuits and transmitting circuits) that receive and transmit various types of data, such as a network interface circuit, a modem, and a router. The information storage unit 30 and the image storage unit 38 are configured by, for example, a memory device such as a semiconductor memory, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or the like. The image display units 40 and 60 are image display devices such as a liquid crystal display or an organic EL (Electronic Luminescent) display.

[0167] Various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for performing specific processing.

[0168] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Also, multiple processing units may be configured with one of the various processors, or two or more of the multiple processing units may be combined into one processor. The processor may be located anywhere, for example, in the server 12, in any of the user's terminal devices including the first and second terminal devices 14, 16, or in each of them. In addition, if there are multiple processors, they may be linked together.

[0169] For example, as typified by computers such as servers and terminal devices, one processor is configured by combining one or more CPUs and software, and this processor functions as multiple processing units.Also, as typified by systems on chips (SoCs), there is a form in which a processor is used to realize the functions of an entire system including multiple processing units in a single IC (Integrated Circuit) chip.

[0170] Furthermore, the hardware configuration of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0171] Furthermore, the method of the present invention can be implemented, for example, by a program for causing a computer to execute each of the steps. Also, a computer-readable recording medium on which this program is recorded can be provided.

[0172] Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and various improvements and modifications may be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0173] 10 Image processing device 12 Server 14 First terminal device 16 Second terminal device 18 Network (communication lines) 20,34,54 Receiver 22 Image group storage for analysis 24 Image Analysis Unit 26 Image selection section 28 Layout Creation Department 30 Information storage section 32,48,68 Transmitter 36,56 Instruction acquisition part 38 Image storage section 40,60 Image display section 42 Analysis image group creation section 44,64 Layout Editorial Department 46 Candidate image selection section 70, 80 Editing page spread 72 Edit Menu A list of 74 thumbnail images for photo album pages List of thumbnail images of 76 and 86 images 82 Main Menu 84 Submenu 88 Buttons

Claims

1. An image processing device comprising: a server; a first terminal device; and a second terminal device different from the first terminal device, which are connected via a network; and which creates a composite image using some images selected from a group of images held by the first terminal device, a processor, the processor comprising: displaying the composite image on the first terminal device; selecting, from the image group, a candidate image that is a part of images other than a used image used in the composite image based on a predetermined criterion while editing the composite image on the first terminal device, and that can be used in the composite image when editing of the composite image is resumed on the second terminal device; The image processing device causes the server to store the used image and the candidate images transmitted from the first terminal device and received by the server.

2. The processor, The image processing device according to claim 1, further comprising: storing in the server an additional image that is newly added to the composite image from the group of images other than the used image by editing the composite image on the first terminal device, the additional image being transmitted from the first terminal device and received by the server.

3. The processor, creating, from the group of images, a group of images for analysis that has a resolution required for analyzing the group of images and has a smaller amount of data than the group of images, in the first terminal device; In the server, the group of images for analysis transmitted from the first terminal device and received by the server is analyzed, an image to be used for analysis is selected from the group of images for analysis based on a result of the analysis of the group of images for analysis, the composite image is created using the image to be used for analysis, and information on the layout of the composite image is stored; The image processing apparatus according to claim 2 , wherein the first terminal device displays the composite image based on information on a layout of the composite image transmitted from the server and received by the first terminal device.

4. The processor, calculating a score for each of the analysis images included in the group of analysis images in the server; 4. The image processing device according to claim 3, wherein the first terminal device selects as the candidate image from the group of images other than the used image an image corresponding to an analysis image whose score transmitted from the server and received by the first terminal device is equal to or greater than a predetermined reference value, or an image corresponding to a predetermined percentage of analysis images whose scores are in ascending order of the scores.

5. The processor, In the server, an attribute of each analysis image included in the group of analysis images is analyzed; 5. The image processing device according to claim 3, wherein the first terminal device selects as the candidate images from the group of images other than the used image an image corresponding to an analysis image whose attribute transmitted from the server and received by the first terminal device matches an attribute of the additional image, and an image corresponding to an analysis image whose attribute is within a defined range from the attributes of the additional image, or selects as the candidate images an image whose attribute does not match the attributes of the additional image and whose attribute is outside a defined range from the attributes of the additional image.

6. The processor, In the server, a candidate image for analysis is selected from the group of images for analysis other than the image used for analysis; The image processing apparatus according to claim 3 , wherein the first terminal device selects, from the image group, the candidate image that corresponds to the candidate image for analysis transmitted from the server and received by the first terminal device.

7. 7. The image processing device according to claim 3, wherein, when editing of the composite image is resumed on the second terminal device, the processor displays, on the second terminal device, an analysis image corresponding to the image to be used, an analysis image corresponding to the candidate image, and an analysis image corresponding to the additional image, which are transmitted from the server and received by the second terminal device, as images that can be used in the composite image while editing the composite image on the second terminal device.

8. The processor, In the first terminal device, analyzing the image group, selecting the use image from the image group based on a result of the analysis of the image group, creating the composite image using the use image, and displaying the composite image; The image processing apparatus according to claim 2 , wherein the server stores information on the layout of the composite image transmitted from the first terminal device and received by the server.

9. The processor, 9. The image processing device according to claim 8, wherein, in the first terminal device, as a result of analyzing the group of images, a score of each image included in the group of images is calculated, and from the group of images other than the used images, images whose score is equal to or greater than a predetermined standard value, or images whose score is a predetermined percentage from the highest, are selected as the candidate images.

10. The processor, 10. The image processing device according to claim 8 or 9, wherein the first terminal device analyzes attributes of each image included in the image group, and selects as the candidate images from the image group other than the used image an image whose attribute matches an attribute of the additional image and an image whose attribute is within a range determined from the attributes of the additional image, or selects as the candidate images an image whose attribute does not match an attribute of the additional image and whose attribute is outside a range determined from the attributes of the additional image.

11. The processor, An image processing device as described in any one of claims 3 to 5 and 8 to 10, wherein, in the first terminal device, based on information of an editing operation image on which an editing operation has been performed by the user from the group of images other than the used image, an image on which an editing operation has been performed by the user while the composite image is being edited on the first terminal device and which is not used in the composite image after the composite image is edited on the first terminal device is selected as the candidate image.

12. The processor, 12. The image processing apparatus according to claim 3, wherein the first terminal device selects the candidate image from the group of images other than the used image based on a past order history of a user.

13. 13. The image processing device according to claim 2, wherein the processor transmits a predetermined number of images or a predetermined percentage of the image group from the first terminal device to the server as the candidate images.

14. The image processing device according to claim 2 , wherein the processor transmits, as the candidate images, a number of images according to a state of the network from the first terminal device to the server.

15. 15. The image processing device according to claim 2, wherein the processor transmits at least a portion of the images included in a group consisting of the used image and the candidate images from the first terminal device to the server while the composite image is being edited on the first terminal device.

16. The image processing device according to claim 15 , wherein the processor transmits the remaining images included in the group and the additional image from the first terminal device to the server in response to an instruction from a user after editing the composite image on the first terminal device.

17. 15. The image processing device according to claim 2, wherein the processor transmits the used image, the candidate image, and the additional image from the first terminal device to the server in response to instructions from a user after editing the composite image on the first terminal device.

18. 18. An image processing device according to any one of claims 2 to 17, wherein the processor transmits from the first terminal device to the server, in response to a notification transmitted from the second terminal device and received by the first terminal device, an image from the group of images corresponding to information of an image other than the used image, the candidate image, and the additional image contained in the notification, as the candidate image.

19. An image processing device as described in any one of claims 2 to 6 and claims 8 to 18, wherein the processor, when resuming editing of the composite image on the second terminal device, displays the used image, the candidate image and the additional image transmitted from the server and received by the second terminal device as images that can be used in the composite image while editing the composite image on the second terminal device.

20. 1. An image processing method for creating a composite image using some images selected from an image group held by an image processing device including a server, a first terminal device, and a second terminal device different from the first terminal device, the image processing method comprising the steps of: an image display unit of the first terminal device displaying the composite image on the first terminal device; a step in which a candidate image selection unit of the first terminal device selects, from the image group, a candidate image that is a part of images other than a used image used in the composite image based on a predetermined criterion while editing the composite image on the first terminal device, and that is usable in the composite image when editing of the composite image is resumed on the second terminal device; an information storage unit of the server storing the used image and the candidate images transmitted from the first terminal device and received by the server.

21. a step of generating, from the image group, an analysis image group having a resolution required for analyzing the image group and an amount of data smaller than that of the image group, by an analysis image group generating unit of the first terminal device; an image analysis unit of the server analyzing the group of analysis images transmitted from the first terminal device and received by the server; a step of a use image selection unit of the server selecting an image to be used for analysis from the group of images to be analyzed based on a result of the analysis of the group of images to be analyzed; a step of a layout creation unit of the server creating the composite image using the analysis use image; The information storage unit of the server stores information on the layout of the composite image; The image processing method according to claim 20, further comprising a step of: displaying the composite image by the image display unit of the first terminal device based on layout information of the composite image transmitted from the server and received by the first terminal device.

22. An image analysis unit of the first terminal device analyzes the image group; a use image selection unit of the first terminal device selecting the use image from the image group based on a result of the analysis of the image group; A step in which a layout creation unit of the first terminal device creates the composite image using the use image; a step of displaying the composite image by the image display unit of the first terminal device; 21. The image processing method according to claim 20, further comprising a step of: storing, by an information storage unit of the server, information on the layout of the composite image transmitted from the first terminal device and received by the server.

23. 23. A program for causing a computer to execute each step of the image processing method according to any one of claims 20 to 22.

24. 23. A computer-readable recording medium having recorded thereon a program for causing a computer to execute each step of the image processing method according to any one of claims 20 to 22.

Images