Image editing device, image editing method, program, and recording medium

Abstract

Provided are an image editing device, an image editing method, a program, and a recording medium that make it possible to perform editing within an appropriate range when editing any image in a composite image generated using a plurality of images.　An image editing device according to one embodiment of the present invention comprises a processor. The processor acquires a composite image including a plurality of images, and first editing information for a first image among the plurality of images in the composite image, calculates a degree of unsuitability of the first editing information on the basis of the relationship between the first image and a second image other than the first image among the plurality of images in the composite image in which the first image has been edited on the basis of the first editing information, and generates second editing information in which a degree of editing is more restricted than that in the first editing information when the degree of unsuitability satisfies a restriction condition.

Description

Image editing device, image editing method, program, and recording medium

[0001] One embodiment of the present invention relates to an image editing device, an image editing method, a program, and a recording medium for editing a composite image generated using a plurality of images.

[0002] A composite image generated using multiple images (material images) available to a user may be edited based on a user's editing operation. Through image editing processing, the composite image is modified into an image in which the material images are arranged in a layout preferred by the user. For example, by editing the composite image so that the size of an image specified by the user among the material images included in the composite image is increased, the subject in the specified image can be made to stand out more.

[0003] When editing of a composite image is performed, the edited image may overlap other images in the composite image, as shown in Patent Document 1. Furthermore, according to the technology described in Patent Document 1, when a first image is edited and placed in front of a second image in the composite image, the arrangement order of the selected images (whether to be positioned closer to the front or further back) can be changed according to the detected instruction.

[0004] Japanese Patent Application Laid-Open No. 2024-079018

[0005] As with the technology described in Patent Document 1, by changing the arrangement order of two or more overlapping images, a desired image can be placed closer to the front in the edited composite image to make it stand out. However, if the size of the image placed at the forefront is enlarged too much through editing, other images placed further back will be hidden by the enlarged image and will be difficult to see in the composite image. In this way, if editing is performed on one of the multiple images included in the composite image without any restrictions, it may affect the other images, resulting in a decrease in the quality and value of the composite image.

[0006] One embodiment of the present invention has been made in consideration of the above circumstances, and aims to provide an image editing device, an image editing method, a program, and a recording medium that allow editing to be performed within an appropriate range when editing one of the images in a composite image generated using multiple images.

[0007] The above object is achieved by an image editing device described in [1] below: [1] An image editing device including a processor, the processor acquiring a composite image including a plurality of images and first editing information for a first image of the plurality of images in the composite image, calculating a degree of incompatibility of the first editing information based on a relationship between the first image and a second image other than the first image of the plurality of images in the composite image obtained by editing the first image based on the first editing information, and generating second editing information having a degree of editing that is more restricted than the first editing information if the degree of incompatibility satisfies a restriction condition.

[0008] Furthermore, one embodiment of the present invention can provide an image editing device described in any of [2] to

[19] below. [2] The image editing device described in [1], in which, when the degree of incompatibility satisfies a restriction condition, the processor generates second editing information such that an amount of change in the first image due to image processing related to the second editing information is smaller than an amount of change in the first image due to image processing related to the first editing information. [3] The image editing device described in [1] or [2], in which the processor generates a composite image in which the first image is edited based on the second editing information. [4] The image editing device described in [3], in which the processor causes a display to display the composite image in which the first image is edited based on the second editing information. [5] The image editing device described in [4], in which, when the degree of incompatibility does not satisfy the restriction condition, the processor causes the display to display the composite image in which the first image is edited based on the first editing information, and, when the degree of incompatibility satisfies the restriction condition, the processor does not display the composite image in which the first image is edited based on the first editing information, but causes the display to display the composite image in which the first image is edited based on the second editing information. [6] The image editing device according to any one of [1] to [5], wherein the processor calculates, as the incompatibility of the first editing information, a degree of overlap between the second image and the first image in a composite image in which the first image is edited based on the first editing information. [7] The image editing device according to any one of [1] to [5], wherein the processor calculates, as the incompatibility of the first editing information, a degree of overlap between the first image and a second image adjacent to the first image in a composite image in which the first image is edited based on the first editing information. [8] The image editing device according to any one of [1] to [5], wherein the processor identifies a target area in the second image in which a specific subject is located, and calculates, as the incompatibility of the first editing information, a degree of overlap between the target area in the second image and the first image in a composite image in which the first image is edited based on the first editing information. [9] An image editing device according to any one of [1] to [5], wherein the processor identifies a target area in the second image where a specific subject is located, and calculates the degree of overlap between the target area in the second image adjacent to the first image and the first image in a composite image in which the first image is edited based on the first editing information, as the degree of incompatibility of the first editing information.

[10] The image editing device described in any one of [6] to [9], wherein the degree of overlap is the area of ​​the region in the second image where the first image is overlapped, or the ratio of the area of ​​the region to the area of ​​the second image in the composite image.

[11] The image editing device described in [8] or [9], wherein the processor acquires first editing information by accepting a user's editing operation performed through a screen displaying the composite image, and displays an object indicating a target region in the second image in the composite image displayed on the screen.

[12] The image editing device described in any one of [1] to [5], wherein the processor calculates a visibility for each predetermined region constituting the second image, and calculates a degree of incompatibility of the first editing information based on the visibility.

[13] The image editing device described in

[12] , wherein the processor calculates, as the degree of incompatibility of the first editing information, a total visibility obtained by adding up the visibility of predetermined regions overlapping with the first image in the second image adjacent to the first image in the composite image obtained by editing the first image based on the first editing information.

[14] The image editing device described in

[13] , wherein the processor displays a converted image obtained by converting the composite image on a display device based on the visibility of each predetermined area constituting the second image in the composite image.

[15] The image editing device described in

[14] , wherein the color of each portion of the converted image in an area corresponding to the second image in the composite image is determined based on the visibility of the predetermined area corresponding to each portion.

[16] The image editing device described in any of [1] to

[15] , wherein the first editing information and the second editing information are information for changing at least one of the size and position of the first image in the composite image, and the processor changes the size of the second image in the composite image in association with editing the first image.

[17] The image editing device described in

[16] , wherein the processor, when reducing the size of the second image in the composite image in association with editing the first image based on the first editing information, calculates the degree of reduction of the second image as the degree of incompatibility of the first editing information.

[18] The image editing device according to any one of [1] to

[17] , wherein, when a composite image includes a plurality of second images, the processor calculates a degree of incompatibility of the first editing information for each second image based on a relationship between the first image and each second image in the composite image, determines a ranking for each degree of incompatibility calculated for each second image according to the degree of incompatibility, and generates second editing information if the degree of incompatibility selected based on the ranking satisfies a restriction condition.

[19] The image editing device according to any one of [1] to

[18] , wherein the processor processes the second image in the composite image in conjunction with editing the first image, and causes a display to display a composite image in which the first image has been edited based on the first editing information or the second editing information and the second image has been processed in conjunction with the editing of the first image.

[0009] The above-mentioned object is also achieved by an image editing method described in any one of

[20] to

[23] below.

[20] The image editing method includes the steps of: a processor acquiring a composite image including a plurality of images and first editing information for a first image among the plurality of images in the composite image; a processor calculating a degree of incompatibility of the first editing information based on a relationship between the first image and a second image among the plurality of images other than the first image in the composite image obtained by editing the first image based on the first editing information; and, if the degree of incompatibility satisfies a restriction condition, the processor generating second editing information having a more restricted degree of editing than the first editing information.

[21] The image editing method described in

[20] , further including a processor generating a composite image obtained by editing the first image based on the second editing information.

[22] The image editing method described in

[20] or

[21] , wherein, in the step of calculating the degree of incompatibility, the processor calculates the degree of overlap between the second image and the first image in the composite image obtained by editing the first image based on the first editing information as the degree of incompatibility of the first editing information.

[23] The image editing method according to any one of

[20] to

[22] , wherein the first editing information and the second editing information are information for changing at least one of the size and position of the first image in the composite image, and the image editing method further includes a step in which the processor changes the size of the second image in the composite image in conjunction with editing the first image.

[0010] A program according to one embodiment of the present invention is a program for causing a computer to execute each step included in the image editing method according to any one of

[20] to

[23] . A recording medium according to one embodiment of the present invention is a processor-readable recording medium on which a program for causing a computer to execute each step included in the image editing method according to any one of

[20] to

[23] is recorded.

[0011] According to the present invention, when editing a first image of a composite image created using multiple images, if the incompatibility of the editing information satisfies a restriction condition, the editing of the first image is restricted. This makes it possible to edit the first image within an appropriate range, thereby ensuring the quality and value of the composite image resulting from the edited first image.

[0012] FIG. 1 is a diagram showing an example of a composite image. FIG. 2 is a diagram showing an image editing device and related devices according to an embodiment of the present invention. FIG. 3 is a diagram showing functions of an image editing device according to an embodiment of the present invention. FIG. 4 is an explanatory diagram of editing of a composite image. FIG. 5 is an explanatory diagram of adjusting a second image in a composite image. FIG. 6 is a diagram showing an example in which editing of a first image is restricted. FIG. 7 is a diagram showing a preview screen of a composite image. FIG. 8 is a diagram showing the procedure of an image editing method according to an embodiment of the present invention. FIG. 9 is a diagram showing the flow of editing-related processing. FIG. 10 is a diagram relating to a modified example in which the degree of overlap between a first image and a second image is used as the degree of incompatibility. FIG. 11 is an explanatory diagram of a converted image, showing a region of the converted image that corresponds to the second image in the composite image. FIG. 12 is a diagram showing an example in which the size of the second image is adjusted while editing the first image.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The present invention will be described in detail below with reference to the accompanying drawings. For convenience of explanation, the following description may be given in terms of a GUI (Graphical User Interface).

[0014] Furthermore, the basic data processing technologies (communication / transmission technologies, data acquisition technologies, data recording technologies, data processing / analysis technologies, machine learning technologies, image processing technologies, image display technologies, visualization technologies, etc.) required to realize the present invention are well-known technologies, and therefore a description thereof will be omitted. It should be noted that the devices and apparatuses used to apply the above-mentioned well-known technologies may be appropriately selected from those available at the time of implementing the present invention.

[0015] In this specification, the concept of "device" includes not only a single device that performs a specific function, but also a combination of multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function. In addition, in this specification, the concept of "system" includes a system that is composed of multiple devices connected in a state where they can communicate with each other, and a system that is composed of a single device.

[0016] In this specification, a "user" is a person who uses the functions of the image editing device of the present invention for the purpose of enjoying the benefits of implementing the present invention. Specifically, a user of the present invention is a person who receives a composite image provided by the image editing device of the present invention, and who, when the provided composite image is edited, uses the edited composite image.

[0017] In addition, in this specification, the term "person" refers to an entity that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, as well as computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources.

[0018] Additionally, in this specification, machine learning algorithms may include neural networks, convolutional neural networks, recurrent neural networks, attention, transformers, variational autoencoders, generative adversarial networks, deep learning neural networks, Boltzmann machines, matrix factorization, factorization machines, m-way factorization machines, field-aware factorization machines, field-aware neural factorization machines, support vector machines, Bayesian networks, decision trees, and random forests, as well as other machine learning algorithms.

[0019] In addition, in this specification, unless otherwise specified, "image" refers to digital image data (hereinafter simply referred to as image data). Image data includes, for example, image data compressed in a lossy format such as JPEG (Joint Photographic Experts Group) format, and image data compressed in a lossless format such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics). An image is made up of multiple pixels, and each pixel corresponds to a "predetermined area."

[0020] <<First Embodiment of the Present Invention>> (Outline of First Embodiment) An outline of a first embodiment of the present invention will be described below. In the first embodiment, by implementing the present invention, a commercial material can be created and provided to a user. The commercial material is, for example, a photo book or album including multiple pages, or a collage print image consisting of a single page. In the first embodiment, a case in which a photo book is provided as an example of a commercial material will be described.

[0021] As shown in FIG. 1 , each page of the photo book is composed of a composite image Pg generated using two or more images (hereinafter referred to as element images). The composite image Pg for each page is generated by arranging two or more element images according to a preset layout. The layout refers to the number of element images used to generate the composite image, as well as the position and size of each element image in the composite image. Note that the state in which the layout is preset may also include a situation in which the layout is set immediately before the composite image Pg is generated. Note that the layout of the composite image may be different for each page or may be consistent across pages.

[0022] In the first embodiment, a user operates a terminal owned by the user (hereinafter referred to as the user terminal 12) to request the generation of a photo book as a commercial product. Then, upon receiving the request for generating a photo book from the user, the image editing device 10 of the present invention generates the photo book, specifically, generates composite images Pg that form each page of the photo book. The generated composite images Pg, or the generated and edited composite images Pg, are output by the image editing device 10 and provided to the user who requested the generation of the photo book. The method for providing the photo book is not particularly limited, and data for displaying each page of the photo book may be transmitted to the user terminal 12, or each page of the photo book may be printed and provided in the form of a booklet.

[0023] (Device Configuration in First Embodiment) Next, the device configuration in the first embodiment will be described with reference to Fig. 2. In the first embodiment, as shown in Fig. 2, an image editing device 10 and a user terminal 12 are used to generate and use a photo book. The image editing device 10 and the user terminal 12 are connected to each other in a state where they can communicate with each other via an external communication network N such as the Internet or a mobile communication line. Note that Fig. 2 shows only one user terminal 12, but in reality, there are a number of user terminals 12 corresponding to the number of users who will use the photo book.

[0024] 2, the user terminal 12 is configured by a smartphone, tablet terminal, PC (Personal Computer), wearable terminal, other information communication terminal, camera with communication function, etc. The user terminal 12 is not limited to a device owned by the user, but may be a device that is not owned by the user but can be used by logging in by entering account information, such as a terminal installed in a store, etc.

[0025] In addition, an application program for generating a photobook (hereinafter referred to as a photobook app) is installed on the user terminal 12. A user can launch the photobook app on the user terminal 12 and request the generation of a photobook or use the generated photobook through a GUI provided by the photobook app. Specifically, a user can perform operations related to the generation and use of a photobook through the user terminal 12 in which the photobook app is running. User operations related to the generation and use of a photobook include, for example, an operation to request the generation of a photobook (i.e., ordering a photobook), an image selection operation, an editing operation, and an operation to confirm image editing.

[0026] The user terminal 12 also includes a display 12a, which is used as a display device for displaying the generated photo book. The display 12a included in the user terminal 12 may be a display (e.g., a touch panel display) mounted on the main body of the user terminal 12, or may be a display connected to the user terminal 12 by wire or wirelessly. The display connected to the user terminal 12 may be, for example, a stationary display or a head-mounted display that can be worn by the user.

[0027] The image editing device 10 is composed of one or more computers, and specifically may be configured as a personal computer, a workstation, a server, or the like. When the image editing device 10 is configured as a server, the server may be a server for a cloud service, specifically a server for an ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input into the user terminal 12, the server performs various processes and calculations based on the input information, and the calculation results are output on the user terminal 12. This allows the user terminal 12 to use the functions of the server.

[0028] 2, the computer constituting the image editing device 10 has a processor 10a, a memory 10b, a communication interface 10c, and a storage 10d. An operating system (OS) program and an information processing program for image editing are installed in the computer constituting the image editing device 10. These programs correspond to the "program" and program product of the present invention, and are programs for controlling the computer constituting the image editing device 10.

[0029] That is, the processor 10a executes the program, causing the computer to function as the image editing device 10. Specifically, in order to provide a photo book in response to a user's request, the processor 10a executes a series of processes described below in accordance with the program.

[0030] The above program (program product) may be obtained by reading it from a computer-readable recording medium (non-transitory medium), or may be obtained (downloaded) via a network such as the Internet or an intranet.

[0031] Furthermore, the image editing device 10 communicates with the user terminal 12 via the communication interface 10c while executing processes related to providing a photo book. Information necessary for generating a photo book and various image processes, etc., is stored in the storage 10d. The storage 10d may be built into the computer that constitutes the image editing device 10, or may be externally attached to the computer, or may be a NAS (Network Attached Storage), etc. The storage 10d may also be configured as a database server, online storage, etc., connected to the image editing device 10 so as to be able to communicate with it.

[0032] (Functions of the Image Editing Device) Next, the functions of the image editing device 10 according to the first embodiment will be described. As shown in FIG. 3, the image editing device 10 has an image acquisition function, an operation reception function, an image synthesis function, an editing function, an adjustment function, an editing restriction function, and an output function. These functions are realized by cooperation between the hardware devices (i.e., the processor 10a, the memory 10b, the communication interface 10c, and the storage 10d) of the computer constituting the image editing device 10 and a program stored in the computer. In addition, some of the above functions (e.g., the image synthesis function, the adjustment function, and the editing restriction function) may be realized by artificial intelligence (AI). Each function will be described below.

[0033] [Image Acquisition Function] The image acquisition function is a function for acquiring images necessary for generating the composite image Pg that constitutes each page of the photo book, i.e., material images. The acquired material images are two or more images that constitute an image group. In the first embodiment, the material images are provided by, for example, a user, and more specifically, are images taken by the user using a digital camera or a camera provided on the user terminal 12, i.e., digital photographic images. The image editing device 10 can acquire (more specifically, receive) images taken by the user via a network through communication with the user terminal 12. However, this is not limited to this, and images selected by the user from existing images present on the network may also be acquired (more specifically, downloaded) via the network.

[0034] [Operation Acceptance Function] The operation acceptance function accepts operations performed by a user through the user terminal 12 when generating a photo book, and more specifically, receives data indicating the operation content from the user terminal 12. Specifically, when a user requests (orders) the generation of a photo book, the image editing device 10 accepts the request. Furthermore, when a user performs a selection operation or a designation operation related to the generation of a photo book, the image editing device 10 accepts the selection or designation. Furthermore, when a user performs an editing operation on a composite image generated by the image editing device 10, the image editing device 10 accepts the editing operation.

[0035] [Image Composition Function] The image composition function is a function for acquiring composite images that form each page of a photo book based on user operations. More specifically, the image editing device 10 generates a composite image by combining two or more element images according to a preset layout. Here, the layout may be a predetermined layout, or may be a layout selected by the user from among multiple candidate layouts. Alternatively, a layout set or adjusted by the user may be used. Alternatively, the system may learn layouts previously used by the user and use a layout determined based on the learning results (more specifically, using an estimation model constructed by machine learning).

[0036] The material images used in the composite image may be selected from a plurality of images (i.e., a group of images) acquired by the image acquisition function described above. The material images may be selected based on a user selection operation. Alternatively, the image editing device 10 may analyze each of the acquired images and automatically select images suitable for generating the composite image as material images based on the analysis results.

[0037] The image editing device 10 generates a number of composite images Pg corresponding to the number of pages in the photo book. As described above, each composite image Pg includes two or more element images (see FIG. 1). The generated composite images Pg are displayed on the display 12a of the user terminal 12 by the output function described below, and are confirmed by the user.

[0038] In the initial composite image generated by the image editing device 10 (i.e., the composite image before editing), two or more material images are arranged at the size and position set for each material image, as shown in Fig. 1. Also, in the initial composite image, as shown in Fig. 1, a margin is provided between adjacent images, in other words, the material images are arranged so that adjacent images do not overlap each other.

[0039] [Image Editing Function] The image editing function edits the composite image generated by the image composition function described above in accordance with a user's editing operation, as shown in FIG. 4 . Specifically, the user checks the composite image displayed on the user terminal 12 and determines whether editing is necessary. If editing is necessary, the user designates, via the user terminal 12, a material image (hereinafter, the first image) to be edited in the composite image, and performs an editing operation to change at least one of the size and placement position of the first image in the composite image. In the example shown in FIG. 4 , the enlarged material image in the lower left of the composite image Pg corresponds to the first image. The editing operation is performed using a commonly known procedure. For example, a pinch operation is performed to enlarge or reduce the size of the first image. Furthermore, to change (move) the placement position of the first image, the user clicks on the first image in the composite image on the screen and drags it to the desired placement position.

[0040] The resizing editing operation includes a cropping operation that extracts (crops) an image of a specified area in the first image, an outpainting operation that uses AI to add a new image to the outside of the first image to expand the first image, and an operation that adds a frame image to the outer edge of the first image. The cropped image, the image expanded beyond the original image by outpainting, and the image with a frame image added may be enlarged or reduced. In this case, the resizing may be performed automatically by the image editing device 10.

[0041] The image editing device 10 accepts the user's editing operation by receiving information indicating the content of the user's editing operation (hereinafter referred to as first editing information) from the user terminal 12 through communication with the user terminal 12. That is, by accepting the editing operation, the image editing device 10 acquires the first editing information for the first image in the composite image. The first editing information is information for changing at least one of the size and position of the first image in the composite image.

[0042] The image editing device 10 then edits the first image in the composite image based on the acquired first editing information. In other words, the image editing device 10 generates a composite image in which at least one of the size and the layout position of the first image is changed based on the first editing information (hereinafter also referred to as an edited composite image). Note that the edited composite image may include a composite image in which the first image is edited based on the first editing information, and a composite image in which the first image is edited based on second editing information (described later).

[0043] The edited composite image, like the unedited composite image, is displayed on the display 12a of the user terminal 12 by the output function described below and can be confirmed by the user. Then, when the editing is confirmed, the composite image at that time (more specifically, the edited composite image) forms the corresponding page of the photo book and can be used as one page of the photo book.

[0044] [Adjustment Function] The adjustment function is a function that automatically adjusts the size of images other than the first image (hereinafter referred to as second images) among the multiple images included in the composite image in association with editing of the first image using the image editing function described above. Specifically, in a composite image in which the first image is edited, the first image may overlap the second image, as shown in Figure 5, and in particular, the first image may overlap the second image adjacent to the first image. Note that in Figure 5, for convenience of illustration, each element image in the composite image is represented by a simple rectangle.

[0045] Here, the overlap of the first image with the second image means that the area of ​​the second image where the first image overlaps is no longer displayed (strictly speaking, it is replaced with a part of the first image). More specifically, the second image is placed on a display layer below the display layer of the first image, so that the area of ​​the second image located directly below the first image is cut off.

[0046] When the first image and the second image overlap in the composite image, the image editing device 10 processes the second image, specifically, reduces the size of the second image to a size that does not overlap with the first image, as shown in Fig. 5. At this time, the size of the second image may be reduced while maintaining the aspect ratio of the second image at a constant value, or the size of the second image may be reduced while changing the aspect ratio.

[0047] Furthermore, when reducing the size of the second image, one of the four vertices of the second image may be used as a reference, and the size of the second image may be reduced so that the remaining vertices approach the reference vertex, as shown in Figure 5. In this case, the reference vertex in the second image may be determined depending on the editing content of the first image. Specifically, when an editing operation is performed to enlarge the size of the first image or change the position of the first image, the vertex (the vertex marked with a black dot in Figure 5) that is farthest in the direction of the operation, more specifically, in the direction of hand movement during the editing operation, may be used as the reference.

[0048] The reduction degree when reducing the size of the second image, and the changed aspect ratio when reducing the size of the second image while changing the aspect ratio, may be set according to the positional relationship between the changed first image and the second image. In this case, the reduction degree and the changed aspect ratio may be set according to a predetermined procedure. For example, the reduction degree and the changed aspect ratio may be associated with the positional relationship between the first image and the second image, and the correspondence may be stored as table data. Then, based on the table data, the reduction degree or the changed aspect ratio corresponding to the positional relationship between the first image and the second image at that time may be set. Alternatively, these values ​​may be set using a machine learning model that outputs the reduction degree and the changed aspect ratio by inputting the positional relationship between the first image and the second image.

[0049] Furthermore, if the reduction rate of the second image, when resized to avoid overlap with the first image, exceeds its upper limit, the resized second image may not be included in the composite image and may be replaced with an unused image. Alternatively, the second image may be removed from a page (hereinafter, the initial page) of the photo book that contains a second image reduced at a reduction rate exceeding the upper limit and moved to a page other than the initial page (specifically, the next page or the previous page, etc.). The minimum size to which the second image is reduced, in other words, the upper limit of the reduction rate, may be determined based on the size of the subject included in the second image, particularly the specific subject described below.

[0050] In addition, the image editing device 10 may adjust the size of the second image according to the characteristics of the second image, for example, the clarity of the second image, or the facial expression of a person if the subject of the second image is a person.

[0051] Furthermore, the adjustment function of the image editing device 10 may include a function for adjusting the configuration of the photo book in addition to the function for adjusting the size of the second image. Specifically, the number of pages in the photo book, the layout of each page, the number of material images included on each page, the content of the material images, etc. may be adjusted according to the group of images acquired by the image acquisition function. Furthermore, for example, if a user has previously requested the generation of a photo book, the configuration of the photo book may be adjusted based on the user's past request history.

[0052] [Editing Restriction Function] The editing restriction function is a function that restricts editing of a first image when editing based on the first editing information is inappropriate. Specifically, the image editing device 10 analyzes the acquired first editing information and calculates the incompatibility of the first editing information. If the calculated incompatibility satisfies a predetermined condition (hereinafter, the restriction condition), the image editing device 10 generates second editing information that is more restricted in editing than the first editing information. Like the first editing information, the second editing information is information for changing at least one of the size and position of the first image in the composite image.

[0053] The degree of editing is an index indicating the extent to which the first image is edited. When the size of the first image is enlarged or reduced, the degree of enlargement or reduction corresponds to the degree of editing. When the position of the first image in the composite image is changed, the amount of movement of the position corresponds to the degree of editing. Restricting the degree of editing means restricting image processing related to the first editing information (specifically, changing at least one of the size and position of the first image). In other words, the second editing information is generated so that the amount of change in the first image caused by image processing related to the second editing information is smaller than the amount of change in the first image caused by image processing related to the first editing information.

[0054] When the second editing information is generated, the image editing device 10 generates a composite image Pg in which the first image is edited based on the second editing information. Furthermore, the image editing device 10 processes the second image (particularly, the second image adjacent to the first image) using the adjustment function described above for the composite image Pg in which the first image is edited based on the second editing information, specifically adjusting the size of the second image to match the editing of the first image. Details regarding the degree of incompatibility, the restriction conditions, and the second editing information will be described later.

[0055] Furthermore, when an operation to extract an area smaller than a predetermined size is performed as an editing operation to extract (crop) a part of the first image, the image editing device 10 may limit the editing and change the extraction range to a predetermined size. Here, the predetermined size may be set based on the visibility (specifically, saliency) of the first image, which will be described later.

[0056] [Output Function] The output image function is a function that outputs a photo book, more specifically, a composite image Pg that constitutes each page of the photo book, to the user, and specifically, a function that generates display data for the composite image and transmits the display data to the user terminal 12. The output (displayed) composite image may include a composite image Pg before editing and a composite image Pg after editing. Furthermore, the edited composite image Pg may include a composite image Pg before the editing is finalized and a composite image Pg after the editing is finalized.

[0057] Furthermore, the composite image Pg before the edits are confirmed is displayed as a preview when the edits are confirmed. The composite image Pg before the edits are confirmed may include a composite image Pg in which only the first image in the composite image has been edited (see FIG. 6 ), and a composite image Pg in which the first image has been edited and the second image has been processed using the adjustment function described above, i.e., a composite image Pg in which the size of the second image has been adjusted in conjunction with the editing of the first image (see FIG. 5 ).

[0058] Furthermore, when the editing restriction function described above restricts editing of the first image based on the first editing information and generates second editing information, a composite image Pg in which the first image has been edited based on the second editing information is output (previewed) as the edited composite image. In this case, the composite image in which the first image has been edited based on the first editing information is not output (previewed).

[0059] On the other hand, if editing of the first image based on the first editing information is not restricted, a composite image Pg is generated in which the first image is edited based on the first editing information, and the composite image Pg is output (preview displayed). In this case, since the second editing information is not generated, the composite image in which the first image is edited based on the second editing information is not output (preview displayed).

[0060] (Regarding the Incompatibility Degree and the Restriction Conditions) Next, the incompatibility degree and the restriction conditions will be described in detail. The incompatibility degree is an index that indicates the influence that the edited first image has on the second image in the composite image when the first image in the composite image is edited based on the first editing information. Specifically, the incompatibility degree is calculated based on the relationship between the first image and the second image in the composite image, more specifically, the positional relationship between the first image and the second image edited based on the first editing information.

[0061] In the first embodiment, the degree of overlap between the second image and the first image in a composite image in which the first image is edited based on the first editing information is calculated as the incompatibility of the first editing information. The degree of overlap is the area of ​​the region in the second image where the first image is overlapped (hereinafter referred to as the overlap region), or the ratio of the area of ​​the overlap region to the area of ​​the second image in the composite image. The areas of the second image and the overlap region may be calculated as actual area values, or the number of pixels included in each of the second image and the overlap region may be calculated.

[0062] In the first embodiment, the degree of overlap with the first image of a second image in the composite image is calculated as the degree of incompatibility. Furthermore, if a plurality of second images are present in the composite image, the degree of overlap with the first image of each of the plurality of second images is calculated as the degree of incompatibility. However, this is not limited to this, and the degree of incompatibility, i.e., the degree of overlap with the first image, may be calculated for a second image adjacent to the first image among the plurality of second images. This is because a second image adjacent to the first image is most susceptible to the effects of editing of the first image and is therefore most likely to overlap with the first image.

[0063] Furthermore, if the first image edited based on the first editing information is placed below the second image (more specifically, on a display layer below the display layer of the second image), the first image may be cut off by the second image. In this case, the degree of incompatibility, i.e., the degree of overlap with the second image, may be calculated for the edited first image.

[0064] The restriction condition is a condition related to the degree of incompatibility that is set in advance, and in the first embodiment, the degree of incompatibility exceeds a threshold. If the degree of incompatibility satisfies the restriction condition, editing of the first image based on the first editing information is restricted, and instead, second editing information is generated, and editing of the first image based on the second editing information is performed.

[0065] Note that setting the restriction condition in advance also includes a case where the restriction condition is set immediately before an editing operation is performed on the first image (in other words, immediately before the first editing information is acquired). The content of the restriction condition is not particularly limited as long as it is a condition that is appropriate for determining whether the content of the editing based on the first editing information is inappropriate. For example, the restriction condition may be that the incompatibility degree is outside a predetermined numerical range.

[0066] (Regarding Generation of Second Editing Information) The second editing information is generated by the image editing device 10 when the degree of incompatibility of the first editing information satisfies a restriction condition. In the first embodiment, the second editing information is generated so that the amount of change in the first image Ps1 before and after editing when the first image Ps1 is edited based on the second editing information is smaller than the amount of change when the first image Ps1 is edited based on the first editing information. To explain in more detail, the second editing information is generated so that the degree of incompatibility does not satisfy the restriction condition, that is, so that the degree of overlap between the first image and the second image edited based on the second editing information is equal to or less than a threshold.

[0067] The method for generating the second editing information will be specifically described using the editing case shown in Fig. 6. In this case, the size of the first image Ps1 is changed based on the first editing information, specifically, enlarged to the position indicated by the dashed line in the figure, and the degree of editing, i.e., the degree of enlargement, is assumed to be Ra.

[0068] The restriction condition is satisfied when the degree of overlap between the first image Ps1 enlarged at the enlargement level Ra and the second image Ps2 (more specifically, the second image Ps2 adjacent to the first image Ps1) is equal to or greater than a threshold value. In this case, the size of the first image Ps1 is enlarged in the same direction as the first editing information, and second editing information is generated so that the enlargement level becomes Rb. Here, the enlargement level Rb based on the second editing information is calculated using the following relational expression (1): Rb = Ra × α (1) In the above expression (1), α is a coefficient greater than or equal to 0 and less than 1, and is set so that the degree of overlap between the first image Ps1 and the second image Ps2 edited based on the second editing information is equal to or less than a threshold value. In the case shown in FIG. 6 , α is set so that the degree of overlap is equal to the threshold value.

[0069] Note that α may be 0, in which case editing of the first image Ps1 (changing the layout position or size) is postponed. The coefficient α may also be set according to the degree of incompatibility, i.e., the degree of overlap between the second image Ps2 and the first image Ps1.

[0070] On the other hand, if the degree of overlap between the first image Ps1 and the second image Ps2 edited based on the first editing information is equal to or less than the threshold, the restriction condition is not satisfied. In this case, the first image Ps1 is edited based on the first editing information, and in the case shown in Figure 6, the size of the first image Ps1 is enlarged at the original enlargement level Ra (to the position indicated by the dashed line in the figure). In this case, the second editing information is not generated.

[0071] When the first image Ps1 is edited based on the first editing information or the second editing information, the image editing device 10 displays a preview of the composite image Pg at the stage where the first image Ps1 has been edited on the user terminal 12 of the user (more specifically, the user who performed the editing operation), as shown in Fig. 7. The user can check the size or position of the first image Ps1 after the change in the composite image Pg by looking at the preview-displayed composite image Pg.

[0072] In particular, by displaying the composite image Pg in which the first image Ps1 has been edited based on the second editing information, the user can understand that editing of the first image Ps1 based on the first editing information has been restricted. As a result, even if an editing operation that excessively changes the size or position of the first image Ps1 is performed, the editing according to the editing operation is restricted, and a composite image Pg in which the first image Ps1 has been edited to an appropriate extent is generated.

[0073] In other words, if the incompatibility of the first editing information satisfies the restriction condition, the composite image Pg in which the first image is edited based on the first editing information is not previewed. Instead, the composite image Pg in which the first image is edited based on the second editing information is previewed. Taking the case illustrated in FIG. 6 as an example, even if the user performs an editing operation to enlarge the first image Ps1 to the dashed line position in the figure and the editing operation is accepted, the enlargement of the first image Ps1 is interrupted when the degree of overlap between the second image Ps2 and the first image Ps1 increases to a threshold value. As a result, the display 12a of the user terminal 12 previews the composite image Pg in which the first image Ps1 is enlarged until the degree of overlap between the second image Ps2 and the first image Ps1 reaches the threshold value (see FIG. 7 ).

[0074] On the other hand, if the incompatibility of the first editing information does not satisfy the restriction condition, a preview of the composite image Pg in which the first image Ps1 has been edited based on the first editing information is displayed. In this case, the user can see the preview of the composite image Pg and know that the editing operation on the first image Ps1 has been performed within an appropriate range.

[0075] When previewing a composite image Pg in which the first image Ps1 has been edited based on the second editing information, suggested information regarding image editing may also be displayed. The suggested information may, for example, be to exclude the second image Ps2 adjacent to the first image Ps1 from the composite image Pg, or to add an unused second image Ps2 to the composite image Pg as a new element image. The manner in which the suggested information is displayed is not particularly limited, and may, for example, be displayed superimposed on the composite image Pg in the form of a text box or window. In this case, the text box or window may be displayed translucently.

[0076] Furthermore, when editing the first image Ps1 based on the second editing information, similarly to when editing the first image Ps1 based on the first editing information, the second image Ps2 is processed in conjunction with the editing of the first image Ps1, specifically, the size of the second image Ps2 is adjusted. At this time, for example, the second image Ps2, particularly the second image Ps2 adjacent to the first image Ps1, may be reduced in size to a size that does not overlap with the first image Ps1.

[0077] (Image Editing Flow in First Embodiment) Next, as an example of the operation of the image editing device 10 according to the first embodiment, a processing flow (hereinafter referred to as image editing flow) for editing composite images Pg that constitute each page of a photo book will be described.

[0078] The image editing flow uses the image editing method of the present invention. In other words, each step (process) in the image editing flow described below corresponds to a component of the image editing method of the present invention. However, the processing flow described below is merely an example, and some steps in the processing flow may be deleted, new steps may be added to the processing flow, or the execution order of two steps in the processing flow may be reversed, as long as it does not deviate from the spirit of the present invention.

[0079] The image editing flow starts when a user requests (orders) the creation of a photo book and the image editing device 10 accepts the request, and proceeds as shown in Fig. 8. At each step in the image editing flow, the processor 10a of the computer constituting the image editing device 10 primarily reads the image editing program and executes the data processing corresponding to that step.

[0080] Specifically, the processor 10a acquires a group of two or more elemental images to generate a composite image for each page of the photo book (S001). Next, the processor 10a generates a composite image for each page of the photo book based on a user's operation (S002). That is, the processor 10a acquires a composite image Pg by combining two or more elemental images using a preset layout. The elemental images used in the composite image are selected by the user from the group of images acquired in step S001, for example. After generating the composite image Pg, the processor 10a displays a preview of the generated composite image Pg on the display 12a of the user terminal 12 (S003).

[0081] Thereafter, it is assumed that the user performs an editing operation on the composite image displayed on the user terminal 12 (S004). In this case, the processor 10a acquires first editing information indicating the content of the editing operation through communication with the user terminal 12 (S005).

[0082] Next, the processor 10a analyzes the first editing information acquired in step S005 (S006) and determines whether editing restriction is necessary based on the analysis result (S007). If it is determined that editing restriction is necessary, the processor 10a executes a process to restrict editing based on the first editing information and its associated process (S008). Note that the series of processes executed in steps S006 to S008 will be referred to as editing-related processes hereinafter.

[0083] On the other hand, if it is determined that editing restrictions are unnecessary, the processor 10a edits the first image Ps1 in the composite image based on the first editing information (S009). In other words, the processor 10a changes at least one of the size and position of the first image in the composite image Pg based on the first editing information. In addition, in conjunction with editing the first image Ps1, the processor 10a processes the second image Ps2 other than the first image in the composite image Pg, specifically adjusting the size of the second image Ps2 (S010).

[0084] When the user performs an editing operation on the composite image Pg (Yes in S004), an edited composite image is generated by the above procedure. The edited composite image is previewed on the display 12a of the user terminal 12, just like the unedited composite image (S011). Then, when the user performs a confirmation operation on the edited composite image displayed on the user terminal 12 (S012), the processor 10a confirms the edited composite image Pg displayed at that time as the corresponding page in the photo book (S013).

[0085] The series of steps S002 to S013 following the generation of the composite image are repeated a number of times corresponding to the number of pages included in the photo book (S014). Then, once the composite images that make up each page in the photo book have been determined, the processor 10a provides the photo book to the user (S015). Specifically, the processor 10a makes the composite images Pg that make up each page of the photo book displayable on the display 12a of the user terminal 12. When the above steps are completed, the image editing flow ends.

[0086] Next, the editing-related process will be described with reference to Fig. 9. In the editing-related process, first, the processor 10a identifies the content of the editing operation performed by the user from the first editing information acquired in step S005 (S021). In this step S021, the first image Ps1 to be edited in the composite image Pg and the editing content for the first image Ps1 are identified.

[0087] Then, the processor 10a calculates the incompatibility of the first editing information based on the relationship between the first image Ps1 and the second image Ps2 in the composite image Pg when the first image Ps1 is edited with the specified content (S022). Specifically, the processor 10a calculates the degree of overlap between the first image Ps1 and the second image Ps2 as the incompatibility. Furthermore, if the composite image Pg includes multiple second images Ps2, in step S022, the processor 10a calculates the incompatibility based on the first editing information for each second image Ps2 based on the relationship between the first image Ps1 and each second image Ps2 in the composite image Pg. The following describes a case in which the incompatibility (i.e., the degree of overlap with the first image Ps1) is calculated for each of the multiple second images Ps2 included in the composite image Pg. In addition, since the first image Ps1 may be placed below the second image Ps2 in the edited composite image Pg and may be cut off, in step S022, the degree of incompatibility, i.e., the degree of overlap with the second image Ps2, is also calculated for the first image Ps1.

[0088] The processor 10a then determines whether there are any second images whose incompatibility (i.e., the degree of overlap with the first image Ps1) increases as a result of editing the first image Ps1 (S023). Here, it is assumed that there are multiple second images Ps2 whose incompatibility increases as a result of editing the first image Ps1. In this case, the processor 10a determines a ranking for each of the calculated incompatibility degrees for each second image Ps2 (S024). In step S024, the processor 10a assigns a higher ranking to a degree of incompatibility that is greater.

[0089] The processor 10a then selects a degree of incompatibility to be determined from the degrees of incompatibility calculated for each second image Ps2 based on the ranking set in step S024, and determines whether the selected degree of incompatibility satisfies the restriction condition (S025). In the first embodiment, the processor 10a determines whether the degree of incompatibility with the highest ranking, i.e., the largest degree of incompatibility among the degrees of incompatibility calculated for each second image Ps2, satisfies the restriction condition.

[0090] If it is determined that the incompatibility of the object to be judged does not satisfy the restriction condition (No in S025), and if there is no second image whose incompatibility increases due to editing of the first image Ps1 (No in S023), the editing-related processing ends and the process proceeds to step S009.

[0091] On the other hand, if it is determined that the degree of incompatibility to be determined (i.e., the maximum degree of incompatibility) satisfies the restriction condition (Yes in S025), the processor 10a restricts the editing of the first image Ps1 based on the first editing information. Instead, the processor 10a generates second editing information that has a higher degree of editing than the first editing information in the manner described above (S026).

[0092] The processor 10a then edits the first image Ps1 based on the second editing information (S027). That is, in step S027, a composite image Pg is generated in which the first image Ps1 has been edited based on the second editing information. In the composite image Pg in which the first image Ps1 has been edited based on the second editing information, the degree of overlap (degree of incompatibility) between the first image Ps1 and the second image Ps2 does not satisfy the restriction condition.

[0093] The processor 10a then causes the display 12a of the user terminal 12 to preview the composite image Pg, in which the first image Ps1 has been edited based on the second editing information (S028). The user views the previewed composite image Pg and considers whether to confirm the editing of the first image Ps1 based on the second editing information. If the user confirms the editing, the user performs a confirmation operation on the user terminal 12 (S029). On the other hand, if the user does not confirm the editing of the first image Ps1 based on the second editing information, the process returns to step S004, and the processor 10a accepts the user's editing operation again; in other words, it reacquires the first editing information.

[0094] When the editing of the first image Ps1 is confirmed, the processor 10a adjusts the size of the second image Ps2 in the composite image Pg, particularly the second image Ps2 adjacent to the first image Ps1, based on the editing of the first image Ps1 based on the second editing information (S030). In step S030, the size of the second image to be adjusted is adjusted so that it does not overlap with the first image Ps1 (more specifically, the first image Ps1 after editing). Furthermore, if there are multiple second images Ps2 adjacent to the first image Ps1, the size of the second image Ps2 adjacent to the first image Ps1 that overlaps with the first image Ps1 before the adjustment is adjusted so that it does not overlap with the first image Ps1.

[0095] The processor 10a then displays a preview of the composite image Pg, in which the size of the second image Ps2 to be adjusted has been adjusted as described above (in other words, the composite image Pg in which the second image Ps2 has been processed in conjunction with the editing of the first image Ps1), on the display 12a of the user terminal 12 (S031). The user views the previewed composite image Pg and considers whether to confirm the size adjustment of the second image Ps2. If the user confirms the size adjustment, the user performs a confirmation operation on the user terminal 12 (S032). Then, when the image editing device 10 accepts the operation to confirm the size adjustment of the second image Ps2, the editing-related processing ends. On the other hand, if the user does not confirm the size adjustment of the second image Ps2, the process returns to step S004, where the processor 10a accepts the user's editing operation again; in other words, it reacquires the first editing information.

[0096] <<Effectiveness of One Embodiment of the Present Invention>> According to the first embodiment, when a first image of a composite image generated using multiple images is edited based on first editing information, the incompatibility of the first editing information is calculated. If the incompatibility satisfies a restriction condition, editing of the first image based on the first editing information is restricted. Instead, second editing information with a more restricted degree of editing than the first editing information is generated, and the first image is edited based on the second editing information. This prevents inappropriate editing of the first image (excessive changes to size or position). As a result, the first image can be edited within an appropriate range, ensuring the quality and value of the composite image in which the first image has been edited.

[0097] Furthermore, the degree of incompatibility is calculated based on the relationship between the first image and the second image in the composite image, i.e., in the first embodiment, the degree of overlap between the first image and the second image adjacent to the first image. This makes it possible to calculate the degree of incompatibility of the first editing information from the perspective of the effect that editing of the first image has on the second image (specifically, the degree to which the first image cuts off the second image). As a result, it is possible to obtain a composite image in which the first image has been edited within an appropriate range while avoiding a situation in which editing of the first image makes the second image difficult to see.

[0098] <<Other Embodiments>> Specific embodiments of the present invention have been described above, but the above-described embodiments are merely examples given to facilitate understanding of the present invention and are not intended to limit the present invention. That is, the present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof. Furthermore, embodiments of the present invention may include a combination of the above-described embodiments with one or more of the following modifications.

[0099] (Variant example in which the degree of overlap between the first image and the second image is used as the degree of incompatibility) In the first embodiment, the degree of overlap between the second image Ps2 and the first image Ps1 in a composite image in which the first image Ps1 is edited based on the first editing information, particularly the degree of overlap between the second image Ps2 adjacent to the first image Ps1 and the first image Ps1, was calculated as the degree of incompatibility of the first editing information.

[0100] Alternatively, the degree of overlap between the target area in the second image Ps2 and the first image Ps1 may be calculated as the degree of incompatibility. Specifically, the degree to which the target area in the second image Ps2 is cut off by editing the first image based on the first editing information may be determined as the degree of incompatibility with the first editing information.

[0101] The target area is the area in the second image Ps2 where a specific subject is located, and the specific subject is, for example, a main subject. Known image analysis technology, subject detection technology, and the like can be used to determine the specific subject and target area in the second image Ps2. Specifically, if a person's face is included in the second image Ps2, the person may be defined as the specific subject, and the area in which the person's face is captured may be defined as the target area. Furthermore, when selecting material images to be used in the composite image Pg from a group of images, a subject that appears more than a predetermined number of times in the group of images may be defined as the specific subject, and the area in which the subject is captured may be defined as the target area. Furthermore, a subject that is in focus in the second image Ps2, a subject that occupies the largest area in the second image Ps2, a subject closest to the center of the second image Ps2, or a subject with a specific facial expression such as a smile may be defined as the specific subject, and the area in which the subject is captured may be defined as the target area. Furthermore, a subject specified by the user using a bounding box or the like in the second image Ps2 may be defined as the specific subject, and the area in which the subject is captured may be defined as the target area. The method for determining the specific subject and target area is not limited to the above method, and the specific subject and target area may be determined based on visibility (more specifically, saliency) described below.

[0102] In a composite image in which the first image Ps1 is edited based on the first editing information, the degree of overlap between the second image Ps2, particularly the target area in the second image Ps2 adjacent to the first image Ps1, and the first image Ps1 may be calculated as the incompatibility of the first editing information. Taking the composite image Pg shown in Figure 10 as an example, suppose a user performs an editing operation to enlarge the first image Ps1 of the composite image Pg to the size indicated by the dashed line in the figure. In this case, first editing information indicating the editing content of enlarging the first image Ps1 to the size indicated by the dashed line is obtained.

[0103] Then, by analyzing the first editing information, the size of the enlarged first image Ps1 and its position in the composite image Pg are identified. Based on the identification result, it is determined whether the enlarged first image Ps1 overlaps with the adjacent second image Ps2. Furthermore, if the enlarged first image Ps1 overlaps with the second image Ps2, the processor 10a determines whether the first image Ps1 overlaps with the target area At in the second image Ps2 (the hatched area in the case shown in FIG. 10 ).

[0104] If the enlarged first image Ps1 overlaps with the target area At of the second image Ps2 adjacent to the first image Ps1, the processor 10a calculates the degree of overlap between the target area At and the first image Ps1 as the degree of incompatibility. The degree of overlap is the area of ​​the area of ​​the target area At of the second image where the first image overlaps (the overlap area), or the ratio of the area of ​​the overlap area to the area of ​​the second image in the composite image. The areas of the second image and the overlap area may be calculated as actual area values, or the number of pixels contained in each of the second image and the overlap area may be calculated. Note that in the case shown in FIG. 10, the target area At is rectangular. However, the shape of the target area At is not particularly limited as long as it is set to surround a specific subject, and may be, for example, a circle, an ellipse, a square other than a rectangle, a polygon other than a square, or an irregular shape.

[0105] Furthermore, while an editing operation is being performed on the first image Ps1 of the composite image Pg, the composite image Pg is displayed on the display 12a of the user terminal 12. At this time, a rectangular frame object F indicating the target area At is displayed on the second image Ps2 of the composite image Pg, as shown in FIG. 10 . This rectangular frame object F is an object superimposed on the composite image Pg and defines the boundary position (outer edge) of the target area At in the second image Ps2. The display of the rectangular frame object F allows the user to perform editing operations on the first image Ps1 while recognizing the target area At in the second image Ps2. Note that the display manner of the rectangular frame object F, i.e., the display color, line thickness, display time, etc. of the rectangular frame object F, is not particularly limited and may be determined arbitrarily.

[0106] When the rectangular frame object F is displayed, the degree of incompatibility may be determined based on the positional relationship between the edited first image Ps1 and the rectangular frame object F. Specifically, if the edited first image Ps1 approaches the rectangular frame object F, the degree of incompatibility may be set to 1, for example, and in other cases the degree of incompatibility may be set to 0. Note that the degree of incompatibility is not limited to the binary values ​​of 0 and 1. For example, the degree of incompatibility may be set to a value corresponding to the degree of overlap between the first image Ps1 and the target region At of the second image Ps2 adjacent to the edited first image Ps1, or may be a normalized (scaled) value, i.e., a value in the range of 0 to 1.

[0107] In the above case, the restriction condition is that the degree of incompatibility is equal to or greater than a threshold, for example, 1. If the degree of incompatibility satisfies the restriction condition, editing of the first image based on the first editing information is restricted, and instead, second editing information is generated, and editing of the first image based on the second editing information is performed.

[0108] (Regarding Other Incompatibility Degrees) The incompatibility degree of the first editing information may be determined using an index value other than the degree of overlap between the second image Ps2 and the first image Ps1. Specifically, for example, the size of the second image Ps2 (particularly the second image Ps2 adjacent to the first image Ps1) in the composite image may be reduced as a result of editing the first image Ps1 based on the first editing information. In this case, the processor 10a of the image editing device 10 may calculate the degree of reduction of the second image Ps2 as the degree of incompatibility of the first editing information. The degree of reduction is the ratio of the size of the second image Ps2 before editing the first image Ps1 to the size of the second image Ps2 after editing the first image Ps1. The smaller the size of the second image Ps2 after reduction, the greater the degree of reduction (i.e., the degree of incompatibility). If the degree of reduction is equal to or greater than a reference value, the degree of incompatibility of the first editing information satisfies the restriction condition. Here, the reference value of the reduction degree may be determined according to the size of the subject in the second image Ps2, in particular the size of the specific subject. For example, the reference value may be set to be stricter as the size of the specific subject becomes smaller, specifically, a smaller value.

[0109] Furthermore, when the second image Ps2 is reduced in size as a result of editing the first image Ps1, the area of ​​the reduced second image Ps2 may be calculated as the degree of incompatibility of the first editing information. In this case, if the area of ​​the reduced second image Ps2 is smaller than a reference value, the degree of incompatibility of the first editing information satisfies the restriction condition.

[0110] Furthermore, there may be cases where the first image Ps1 edited based on the first editing information overlaps with the second image Ps2, resulting in a portion of the second image Ps2 being cut off. In this case, the long and short sides of the portion of the second image Ps2 that is visible without overlapping with the first image Ps1 may be calculated, and the ratio (long side / short side) may be scaled (normalized) to calculate the incompatibility of the first editing information. In this case, if this value falls outside a predetermined numerical range, the incompatibility of the first editing information satisfies the restriction condition.

[0111] Furthermore, for a second image Ps2 adjacent to the first image Ps1 in a composite image Pg obtained by editing the first image Ps1 based on the first editing information, the visibility may be calculated for each pixel constituting the second image. The processor 10a of the image editing device 10 may then calculate the incompatibility of the first editing information based on the visibility calculated for each pixel. Visibility is an index indicating the degree (possibility) to which each part of an image can be recognized by a human, and specifically includes the degree of attention, saliency, and ROI (Region of Interest). Visibility can be calculated using a known calculation method that applies image processing technology.

[0112] Here, a configuration for calculating the degree of incompatibility based on saliency, which is an example of visibility, is defined as a second embodiment of the present invention, and a method for calculating the degree of incompatibility in the second embodiment will be described below. Note that, in the following, differences between the second embodiment and the first embodiment will be described, while descriptions of the commonalities between the first embodiment and the second embodiment will be largely omitted.

[0113] In the second embodiment, when the processor 10a of the image editing device 10 acquires the first editing information, it calculates the saliency for each of the plurality of pixels constituting the second image Ps2 of the composite image Pg. The saliency of each pixel in the image is a feature expressed as a numerical value and can be calculated using a known calculation method from the feature (e.g., luminance or brightness) of each pixel. Furthermore, if the number of pixels constituting the second image Ps2 is m (m is a natural number), m saliencies are calculated, and each saliency is scaled (normalized) so that the sum of the m saliencies is 1.

[0114] The processor 10a then identifies pixels of the second image Ps2 adjacent to the first image Ps1 that overlap with the first image Ps1 (hereinafter, "overlapping pixels") in the composite image Pg obtained by editing the first image Ps1 based on the first editing information. The processor 10a then calculates the total saliency (corresponding to the total visibility) obtained by adding up the saliency of all the overlapping pixels as the incompatibility of the first editing information.

[0115] As described above, in the second embodiment, the total saliency, which is the sum of the saliency of pixels in the second image Ps2 that are hidden by the first image Ps1 (i.e., overlapping pixels), is used as the incompatibility of the first editing information. If the total saliency is equal to or greater than a reference value, the incompatibility of the first editing information satisfies the restriction condition. Here, the reference value of the total saliency may be determined according to the size of the subject in the second image Ps2, particularly the size of the specific subject. For example, the reference value may be set stricter as the size of the specific subject increases, specifically, a smaller value may be set.

[0116] In the second embodiment, the processor 10a may display the converted image Pt on the display 12a of the user terminal 12, either in place of the composite image Pg or together with the composite image Pg, while the user is performing editing operations on the first image Ps1. The converted image Pt is obtained by converting the composite image Pg based on the saliency of each pixel constituting the second image Ps2 in the composite image Pg. For example, the converted image Pt is a heat map image colored according to the saliency of each pixel of the second image Ps2, as shown in FIG. 11 . That is, the color of each portion of the converted image Pt in the region corresponding to the second image Ps2 in the composite image Pg is determined according to the saliency of the pixel corresponding to that portion in the second image Ps2. FIG. 11 illustrates an extracted region of the converted image Pt corresponding to the second image Ps2 in the composite image Pg.

[0117] While an editing operation is being performed on the first image Ps1, the converted image Pt is displayed on the display 12a of the user terminal 12, allowing the user to recognize prominent areas in the second image Ps2, i.e., areas with high saliency, during the editing operation. This makes it easier to perform editing operations so that the first image Ps1 does not overlap prominent areas in the second image Ps2.

[0118] (Variations Related to Display of Composite Image) In the above-described embodiment, when the first image is edited based on the first editing information or the second editing information, a preview of the composite image at that stage, i.e., a composite image in which the first image has been edited based on the editing information, is displayed on the display 12a of the user terminal 12. Also, in the above-described embodiment, the second image in the composite image is processed in conjunction with the editing of the first image, and more specifically, the size of the second image is adjusted, and a preview of a composite image including the processed second image, i.e., a composite image in which the second image has been processed (size adjusted) in conjunction with the editing of the first image, is displayed. However, it is not necessary to preview the composite image at each stage. It is sufficient to display at least a composite image at a stage in which editing of the first image and editing of the second image are completed, i.e., a composite image in which the first image has been edited based on the first editing information or the second editing information and the second image has been processed in conjunction with the editing of the first image.

[0119] (Variations Related to Size Adjustment of Second Image) In the first and second embodiments, after the editing of the first image Ps1 (changing at least one of the position or size) is finalized, the second image Ps2 is processed, specifically, the size of the second image Ps2 is adjusted. However, this is not limited to this. As shown in FIG. 12 , during the editing of the first image Ps1, the size of the second image Ps2 may be adjusted in conjunction with a change in at least one of the position and size of the first image Ps1. In this case, the processor 10a of the image editing device 10 may calculate the degree of incompatibility of the first editing information, for example, the degree of reduction of the second image Ps2, while editing the first image based on the user's editing operation, i.e., the first editing information.

[0120] Then, when the first image Ps1 is edited until the reduction degree as the degree of incompatibility satisfies the restriction condition, editing of the first image Ps1 based on the first editing information may be restricted (suspended) at that point. As a result, even if further editing operations (strictly speaking, editing operations that increase the degree of incompatibility) are performed after the editing of the first image Ps1 is restricted, editing of the first image Ps1 based on further editing operations is restricted. Furthermore, second editing information is generated based on the editing operations performed at the time the editing is restricted, and the first image Ps1 is edited based on the second editing information. Furthermore, the size of the second image Ps2 (particularly, the second image Ps2 adjacent to the first image) in the composite image Pg is adjusted in accordance with the editing of the first image based on the second editing information. Note that when the first image Ps1 is edited, the second image Ps2 does not necessarily need to be modified (specifically, size adjusted) accordingly.

[0121] (Modification of Images Used in Composite Image) In the above-described embodiment, the images (material images) used in the composite image are photographed images captured by a device such as a digital camera. However, the material images may be images other than photographed images. For example, they may be scanned images read using a scanner, or captured images such as screenshots, or the composite image may be generated using an illustration image created using a drawing tool or the like. Alternatively, images generated by a generation AI such as an image generation AI may be used as material images.

[0122] (Regarding the Computer Constituting the Image Editing Apparatus) In the above-described embodiment, each process performed by the image editing apparatus 10 is executed by a computer. The computer may execute these processes using a processor, a program, or a combination thereof. The computer may be a general-purpose computer, a computer for a specific application, a system such as a workstation, or any other hardware element capable of executing a program.

[0123] Furthermore, some of the functions of the image editing device 10 may be performed by the user terminal 12. In this case, the image editing device 10 is configured by a computer (e.g., a server) that can communicate with the user terminal 12 and the user terminal 12 working together. Alternatively, the user terminal 12 (more specifically, the user terminal 12 on which an image editing program is installed) may independently configure the image editing device 10.

[0124] (Regarding the Processor of the Image Editing Apparatus) The processor of the image editing apparatus according to an embodiment of the present invention may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor may be configured with hardware such as a programmable logic device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field Programmable Gate Array), a dedicated circuit for executing specific processing such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit).

[0125] Furthermore, the processor has each unit or each means that executes various processes in one embodiment of the present invention. Furthermore, the type of hardware may be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may exist in devices that are physically separate from each other, or may exist in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is configured by an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0126] Furthermore, one embodiment of the present invention may be realized by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode are configured by a program. A program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform the respective function. The program may also be program code and multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storages). The program may also be stored in multiple non-transitory computer-readable media that reside in physically separate devices. Program code or code segments may represent procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending or receiving information, data, arguments, parameters, or memory contents.

[0127] REFERENCE SIGNS LIST 10 Image editing device 10a Processor 10b Memory 10c Communication interface 10d Storage 12 User terminal 12a Display At Target area F Rectangular frame object N External communication network Pg Composite image Ps1 First image Ps2 Second image Pt Transformed image

Claims

1. An image editing device comprising a processor, the processor: acquires a composite image including a plurality of images and first editing information for a first image of the plurality of images in the composite image; calculates a degree of incompatibility of the first editing information based on a relationship between the first image and a second image other than the first image of the plurality of images in the composite image in which the first image has been edited based on the first editing information; and, if the degree of incompatibility satisfies a restriction condition, generates second editing information having a more restricted degree of editing than the first editing information.

2. The image editing device of claim 1, wherein when the degree of incompatibility satisfies the restriction condition, the processor generates the second editing information so that the amount of change in the first image due to image processing related to the second editing information is smaller than the amount of change in the first image due to image processing related to the first editing information.

3. The image editing device according to claim 1, wherein the processor generates the composite image in which the first image is edited based on the second editing information.

4. The image editing device according to claim 3, wherein the processor causes a display to display the composite image in which the first image has been edited based on the second editing information.

5. The image editing device of claim 4, wherein the processor, if the degree of incompatibility does not satisfy the restriction condition, causes the display to display the composite image in which the first image has been edited based on the first editing information, and, if the degree of incompatibility satisfies the restriction condition, does not display the composite image in which the first image has been edited based on the first editing information, but causes the display to display the composite image in which the first image has been edited based on the second editing information.

6. The image editing device of claim 1, wherein the processor calculates the degree of overlap between the second image and the first image in the composite image in which the first image is edited based on the first editing information as the degree of incompatibility of the first editing information.

7. The image editing device of claim 1, wherein the processor calculates the degree of overlap between the second image adjacent to the first image and the first image in the composite image in which the first image is edited based on the first editing information as the incompatibility of the first editing information.

8. The image editing device of claim 1, wherein the processor identifies a target area in the second image in which a specific subject is located, and calculates the degree of overlap between the target area in the second image and the first image in the composite image in which the first image is edited based on the first editing information as the degree of incompatibility of the first editing information.

9. The image editing device of claim 1, wherein the processor identifies a target area in the second image in which a specific subject is located, and calculates the degree of overlap between the target area in the second image adjacent to the first image and the first image in the composite image in which the first image is edited based on the first editing information, as the degree of incompatibility of the first editing information.

10. An image editing device described in any one of claims 6 to 9, wherein the degree of overlap is the area of ​​the region in the second image where the first image is overlapped, or the ratio of the area of ​​the region to the area of ​​the second image in the composite image.

11. An image editing device as described in claim 8 or 9, wherein the processor acquires the first editing information by accepting a user's editing operation performed through a screen on which the composite image is displayed, and displays an object indicating the target area in the second image in the composite image displayed on the screen.

12. The image editing device according to claim 1, wherein the processor calculates the visibility of each predetermined area constituting the second image, and calculates the degree of incompatibility of the first editing information based on the visibility.

13. The image editing device of claim 12, wherein the processor calculates the total visibility of the second image adjacent to the first image that overlaps with the first image in the composite image in which the first image is edited based on the first editing information as the incompatibility of the first editing information.

14. The image editing device according to claim 13, wherein the processor causes a display to display a converted image obtained by converting the composite image based on the visibility of each of the predetermined areas constituting the second image in the composite image.

15. An image editing device as described in claim 14, wherein the color of each portion of the converted image in the area corresponding to the second image in the composite image is determined according to the visibility of the specified area corresponding to each portion.

16. The image editing device of claim 1, wherein the first editing information and the second editing information are information for changing at least one of the size and position of the first image in the composite image, and the processor changes the size of the second image in the composite image in conjunction with editing the first image.

17. The image editing device of claim 16, wherein when the size of the second image in the composite image is reduced in association with editing of the first image based on the first editing information, the processor calculates the degree of reduction of the second image as the degree of incompatibility of the first editing information.

18. The image editing device of claim 1, wherein, when the composite image includes a plurality of the second images, the processor: calculates the incompatibility of the first editing information for each of the second images based on the relationship between the first image and each of the second images in the composite image; determines a ranking for each of the incompatibility calculated for each of the second images according to the incompatibility; and generates the second editing information if the incompatibility selected based on the ranking satisfies the restriction condition.

19. The image editing device of claim 1, wherein the processor processes the second image in the composite image in conjunction with the editing of the first image, and causes a display to display the composite image in which the first image has been edited based on the first editing information or the second editing information and the second image has been processed in conjunction with the editing of the first image.

20. An image editing method comprising the steps of: a processor acquiring a composite image including a plurality of images and first editing information for a first image among the plurality of images in the composite image; a processor calculating a degree of incompatibility of the first editing information based on a relationship between the first image and a second image other than the first image among the plurality of images in the composite image in which the first image has been edited based on the first editing information; and a processor generating second editing information having a more restricted degree of editing than the first editing information if the degree of incompatibility satisfies a restriction condition.

21. The image editing method of claim 20, further comprising the step of: a processor generating the composite image in which the first image is edited based on the second editing information.

22. An image editing method as described in claim 20, wherein in the step of calculating the degree of incompatibility, the processor calculates the degree of overlap between the second image and the first image in the composite image in which the first image is edited based on the first editing information as the degree of incompatibility of the first editing information.

23. The image editing method of claim 20, wherein the first editing information and the second editing information are information for changing at least one of the size and position of the first image in the composite image, and the processor further comprises a step of changing the size of the second image in the composite image in conjunction with editing the first image.

24. A program for causing a computer to execute each step included in the image editing method according to any one of claims 20 to 23.

25. A computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the image editing method according to any one of claims 20 to 23.

Images