Information processing device, information processing method, and program

Abstract

To automatically complete a missing portion of an image without ground truth data.SOLUTION: An image completion processing unit sets an ID indicating content set in an image input unit. The image input unit determines whether a content type corresponding to the specified ID is "image" or not. The image completion processing unit determines whether a missing portion exists in the input image, and identifies the missing portion. The image completion processing unit sets a completion region in a region adjacent to the missing portion of the input image and outside the input image. The image completion processing unit requests an image generation unit of a server to generate a completed image based on information set in the image input unit and completion region information. The image generation unit generates the completed image with a generative AI technology using a generative model, transmits the generated completed image to the image completion processing unit of a client PC, and terminates processing.SELECTED DRAWING: Figure 8

Description

Technical Field

[0002] , ,

[0004] , ,

[0006] , , , , , , ,

[0005] , , , , , ,

[0003]

[0001] The present invention relates to an information processing technology for complementing missing parts of an input image.

Background Art

[0002] In recent years, when producing posters or flyers, services are provided to support production by editing the layout from a large number of prepared templates so that anyone can easily obtain a product of a certain quality.

[0003] When trying to use an image the user has in the selected template, part of the object on the corresponding image may be missing. If the missing part is an essential part, the user has to prepare another image suitable for the intended purpose.

[0004] By the way, recently, it has been becoming possible to generate necessary content by generative AI technology. In generative AI technology, when a user inputs an image or text as an input prompt to a generative model, characters, images, videos, etc. with a high probability of conforming to the "context" expressed by the input prompt can be generated. By using this technology, the user can easily obtain an image (complemented image) with the missing parts complemented. However, it is necessary for the user to specify which part of the image to complement.

[0005] Patent Document 1 discloses a technology for automatically correcting image data for forming a manuscript image based on the read image of the manuscript and correct data. In Patent Document 1, the read image of the manuscript is compared with correct data in which a plurality of lines with different thicknesses and directions are drawn, missing parts such as broken lines and changes in line width on the read image are detected, and the image data for forming the manuscript image is automatically corrected.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2008-250046 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, the technology described in Patent Document 1 has drawbacks, such as the need to prepare correct data in advance and load it each time, and the fact that the correction target is limited to line images. [Means for solving the problem]

[0008] The present invention relates to an information processing apparatus comprising: acquisition means for acquiring an input image; extraction means for extracting objects from the input image; identification means for identifying missing portions of the objects extracted by the extraction means based on the contour features of the objects; and generation means for generating a complementary image in which the missing portions identified in the input image have been filled in. identification The means involves the extraction means extracting multiple objects from the input image, and if the multiple objects are parts of the same object, identifying the region between the multiple objects that constitute the same object as the missing portion. The generating means is The method is characterized by generating a complementary image by filling in the missing portion identified in the input image. [Effects of the Invention]

[0009] According to the present invention, missing portions of an image can be automatically filled in even without correct data. [Brief explanation of the drawing]

[0010] [Figure 1] An example of a system configuration for a layout data editing device targeting an image output device in this embodiment. [Figure 2] An example of the hardware configuration of the image output device in this embodiment. [Figure 3]An example of the hardware configuration of the client PC and server in this embodiment. [Figure 4] An example of a functional block for a printing system targeting the image output device in this embodiment. [Figure 5] An example of image generation processing using a generative model. [Figure 6] An example of a layout data database in this embodiment. [Figure 7] This figure shows an example of the layout data editing screen on a client PC in this embodiment. [Figure 8] An example of a processing flow for generating complementary images when editing layout data in Embodiment 1. [Figure 9] An example of image completion processing using a generative model. [Figure 10] An example of a processing flow for generating complementary images when content is cropped in Embodiment 2. [Figure 11] An example of content placement. [Figure 12] An example of a processing flow for determining whether or not to generate a complementary image based on the placement position of the content in Embodiment 3. [Figure 13] An example of a processing flow for generating multiple complementary images in Embodiment 4. [Figure 14] An example of a processing flow for generating a complementary image using a text prompt in Embodiment 5. [Figure 15] An example of a processing flow for filling in the missing parts of the interpolated image generated in Embodiment 6. [Figure 16] An example of a processing flow for receiving a user's request to generate a complementary image in Embodiment 7. [Figure 17] An example where the extracted object is truncated. [Figure 18] An example of a processing flow in Embodiment 8 where, when the same object is extracted as multiple objects during content cropping, a complementary image is generated with the area between the same objects as the complementary region. [Figure 19]An example of a processing flow for receiving a complementary area for generating a complementary image in Embodiment 9. [Figure 20] An example of a dialog for having the user set a complementary area for generating a complementary image. [Figure 21] An example of content overlap. [Figure 22] An example of a processing flow for determining whether to generate a complementary image based on the overlapping state of content in Embodiment 10. [Figure 23] [[ID=`12]]An example of a processing flow for generating a plurality of complementary images with different complementary areas in Embodiment 11.

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention according to the claims, and not all combinations of the features described in the present embodiments are essential for the solution means of the present invention.

[0012] <Embodiment 1> First, the information processing system according to the present embodiment will be described. The information processing system according to the present embodiment is a printing system that involves layout data editing for an image output device. In the printing system, editing of layout data and transmission of a print job to the image output device are performed on a PC connected from the outside. When generating a print job, an editing operation of print settings is performed on the screen of the PC as necessary.

[0013] Figure 1 shows an example of the system configuration in the network environment of this system. As shown in Figure 1, the client PC 102 can connect to the server 104 and image output devices (100 to 101) via the network 103. The client PC 102 performs editing work on layout data such as posters and flyers, and requests some editing, data processing, and rendering processing related to the layout data from the server 104. Furthermore, the client PC 102 generates a print job with print settings added to the edited layout data and sends the print job to the image output devices 100 and 101. In this embodiment, there are two image output devices, but it is not limited to this, and there may be one or three or more. Similarly, there is one client PC and one server, but there may be two or more.

[0014] As an example of printing, this paper describes a system in which a print job is sent from a printing application installed on a PC to an image output device 100 via a printer driver. For example, a client PC 102 has a printing application and a printer driver installed. The printing application obtains device information of the associated image output device 100 from the printer driver, as well as print parameters such as paper type, paper size, and print quality, and can edit the print settings from the obtained parameters. Based on the above print settings and the layout data image rendered on the server 104, a print job is configured, and the print job is sent to the image output device via the print driver's spooler to execute the printing process. The image output device executes printing based on the print settings of the received print job. The image output device also maintains device information such as configuration information regarding the ink and paper it handles, as well as status information such as idle status and print errors. Furthermore, if printing cannot be executed normally due to problems with the image output device such as insufficient paper or empty ink, or due to an error in the print settings, a warning message is displayed on the main unit panel to inform the user of the reason why printing could not be performed normally.

[0015] Figure 2 shows an example of the hardware configuration of the image output device 100. Note that the image output device 101 has a similar configuration, so its description is omitted. The image output device 100 is controlled by the CPU 200. The CPU 200 operates based on control programs stored in the program ROM 201, or control programs stored in the external memory 208. The CPU 200 outputs image signals as output information to the printing unit (printer engine) 207, which is connected to the printing unit I / F 205 via the system bus 203. The CPU 200 can communicate with the client PC 102 via the input unit 204, and can notify the client PC 102 of information within the image output device 100. The CPU 200 can also receive output data to be output to the printing unit 207 via the input unit 204. The RAM 202 functions as the main memory of the CPU 200 and as a work area, and its memory capacity can be expanded by optional RAM connected to an expansion port (not shown). RAM 202 is used for output information expansion area, environmental data storage area, non-volatile memory, etc. External memory 208 such as hard disk drive (HDD) and IC card is accessed by memory controller 206. External memory 208 can be connected as an option and stores font data, emulation programs, form data, ink usage, paper feed type and size information, and main unit status information. The operation unit 209 is equipped with a panel that can display various information.

[0016] Figure 3 is an example block diagram showing the configuration of the client PC 102 and server 104 computer in Figure 1. The computer's internal components 307 include a CPU 300, ROM 301, RAM 302, keyboard controller 304, display controller 305, and disk controller 306. The CPU 300 reads various programs such as control programs, system programs, and application programs from external memory 310 to RAM 302 via the disk controller 306. The CPU 300 executes the various programs read into RAM 302 to perform various data processing and control the display of the display 309. The CPU 300 may also read control programs, etc., from ROM 301. The CPU 300 may also be a dedicated circuit such as an ASIC. The CPU 300 and the dedicated circuit are examples of hardware circuits and hardware processors. The disk controller 306 controls access to external memory 310 such as HD, CD-ROM, DVD-ROM, and USB. RAM 302 is configured to allow expansion of its capacity with optional RAM (not shown), and is primarily used as a work area for CPU 300. Keyboard controller 304 controls key input from keyboard 308 and pointing devices (not shown). Display controller 305 controls the display of display 309. In the embodiments of the present invention, unless otherwise specified, CPU 300 controls each part connected to the main bus 303 via the main bus 303. Of course, in server 104, components that are not necessarily essential, such as the display 309, do not need to be included in the configuration.

[0017] Figure 4 shows an example of the functional blocks of this system, relating to the image output device groups 100 and 101 and the client PC 102 and server 104 described in Figures 1 to 3. First, the functional blocks within the client PC 102 and server 104 will be explained.

[0018] The layout data editing department 401 adds and deletes content such as text and images to be placed on posters and flyers, and adjusts the layout of each content item. If processing such as cropping or filling is required for the content, the layout data editing department 401 requests processing from the data content editing department 409 on server 104. The layout data is either saved as a cache in the layout data DB 400 on client PC 102, or saved in the layout data DB 410 on server 104 for each client PC 102 (or for each user account if user accounts exist).

[0019] The print job transmission unit 405 generates a print job and transmits the generated print job to the image output device 100. When the print job transmission unit 405 generates a print job, it requests the preview image generation unit 413 or the print image generation unit 414 of the server 104 to perform a preview of the layout data or generate a print image.

[0020] The image completion processing unit 402 requests image generation from the image generation unit 411 of the server 104 based on the image information set in the image input unit 404. The image generation unit 411 of the server 104, upon receiving the request for image generation, generates a completion image using the generation model 412 and sends the generated completion image to the client PC 102. The client PC 102, upon receiving the completion image, provides the completion image to the user from the image completion processing unit 402. The image completion processing unit 402 can also request image generation from the image generation unit 411 of the server 104, including the prompt information set in the prompt setting unit 403 along with the image information set in the image input unit 404.

[0021] The generative model 412 used by the image generation unit 411 to generate complementary images is a machine learning-based generative model that generates images using GANs (Generative Adversarial Networks), Stable Diffusion, and diffusion models used in DALL·E. Figure 5 is a diagram illustrating the image generation process using the generative model. When the generative model 501 receives an input 500 consisting of an input image and a text prompt, it can output an image as a product 502 that fits the "context" represented by the input 500. Furthermore, the generative model 501 can generate different images even when using the same input image and text prompt by changing the seed value of the random number generator used to generate initial values ​​during image generation.

[0022] Next, the functional blocks within the image output device 100 will be described. The ROM 201 contains a device information storage unit 406, a print job receiving unit 407, and a print execution unit 408. The device information storage unit 406 stores information such as the type and remaining amount of ink installed in the image output device 100, information such as the type and size of registered paper and paper feed paper, the main unit status information of the image output device 100, and the status information of print jobs. The print job receiving unit 407 receives print jobs sent from the client PC 102. The print execution unit 408 executes the printing process for the print jobs.

[0023] Furthermore, the information held by the device information holding unit 406 may be associated with the layout data DBs 400 and 410 and held on the client PC 102 and server 104 sides. This allows the client PC 102 and server 104 to generate layout data suitable for the image output device 100 to be used, if the image output device 100 to be used has been determined in advance.

[0024] Figure 6 shows an example of layout data stored in layout data DB600 and 410. A data table like the one shown in Figure 6 exists for each piece of layout data. The data table includes parameters such as ID600 for uniquely identifying the content, content601, content type602, layout coordinates603, content size604, content order605, and setting information606. Content601 is set to the address value where the data for each piece of content, such as text or images, is stored. Content type602 is set to information indicating the type of content. Layout coordinates603 is set to the coordinate value indicating the position of the content. Content size604 is set to a value indicating the size of the content. Content order605 is set to a value indicating the overlapping order of each piece of content, which is assigned a different content ID and placed on multiple different layers. Setting information606 is set to attribute values ​​for each piece of content, such as the color of the content. In addition to information about individual pieces of content, the layout data can also hold various setting information about the entire layout, such as the original size and data for variable printing. By registering "Overall" in Content 601, registering the setting type in Content Type 602, and registering the setting value in Setting Information 606, setting information for the entire layout can be stored in the layout data. Note that the information stored in the layout data is not limited to information about individual content or setting information for the entire layout; information of other parameter types may also be included in the layout data, or each parameter type may be stored in separate data.

[0025] Figure 7 shows an example of the layout data editing screen 700 displayed on client PC 102. Multiple templates are displayed in the template list 701, and the user can select the template that they think is closest to the completed layout data image from among the displayed templates. When the user selects a template, the selected template is displayed in the layout editing area 704. The template information displayed in the template list 701 may be obtained from the layout data DB 400 or 410, or from SNS or other external cloud services.

[0026] On the layout editing area 704, it is possible to edit each piece of content displayed by the layout data editing unit 401 and the data content editing unit 409, such as adjusting the position, cropping, and filling. To add content such as images and text to the layout editing area 704, users press the image add button 702 or the text add button 703 and specify the path to the content file as the import source. In addition to the image add button 702 and the text add button 703, buttons corresponding to the content type may also be added. Furthermore, in addition to the local storage of the client PC 102, it may be possible to specify the storage of SNS or other external cloud services as the content import source. It may also be possible to accept drag-and-drop addition of content to the layout editing area 704.

[0027] When the layout data editing unit 401 detects that the print execution button 705 has been pressed, it requests the print job transmission unit 405 to generate and send a print job in order to execute printing. The print job transmission unit 405, having received the request to generate and send a print job, generates a print job of the layout data displayed in the layout editing area 704 and sends the generated print job to the image output device A100 or image output device B101.

[0028] If there are missing parts in the input image added to the layout data editing screen 700, a complementary image can be generated to fill in those missing parts. The image complementary processing unit 402 sets the input image added by pressing the image add button 702 to the image input unit 404, and the image generation unit 411 sends the set input image to the server 104 and receives the complementary image from the server 104. The image complementary processing unit 402 provides the complementary image to the user by displaying it on the layout data editing screen 700. The image complementary processing may also be made executable from the context menu.

[0029] Figure 8 shows an example of a processing flow for generating and displaying a complementary image on the layout data editing screen 700. The flowchart shown in Figure 8 is implemented, for example, by the CPU 300 reading a program stored in ROM 301 into RAM 302 and executing it. When the client PC 102 adds an image, for example by pressing the image addition button 702 on the layout data editing screen 700, the image completion processing unit 402 starts this flow.

[0030] In S801, the image completion processing unit 402 sets an ID 600 that indicates the content set in the image input unit 404.

[0031] In S802, the image input unit 404 determines whether the content type 602 corresponding to the specified ID 600 is an "image". If the content type 602 is not an "image" (No in S802), the image completion processing unit 402 terminates the process. If the content type 602 is an "image" (Yes in S802), the process proceeds to S803.

[0032] In S803, the image completion processing unit 402 determines whether the input image is missing and identifies the missing portion. Missing parts in an input image refer to a state where a part of an object of a predetermined type, such as a person or an animal, is unnaturally missing, as shown in the input image in Figure 9. Here, if an object of a predetermined type exists in the input image, the contour of that object is extracted, and the linear portion of the extracted contour that satisfies the following missing part determination criteria is identified as the missing part. An example of the missing part determination criteria for an input image is whether at least a part of the contour of an object of a predetermined type, such as a person or an animal, coincides with one or more of the top, bottom, left, or right edges of the input image. That is, if multiple consecutive pixels constituting the contour of an object of a predetermined type are pixels at the edges of the input image and satisfy the above missing part determination criteria, it is determined that the image is missing. Alternatively, other missing part determination criteria may include, for example, performing a Hough transform to detect straight lines on the pixels constituting the contour and checking whether the parameters of the straight line have a peak that falls within a predetermined width, or finding a regression line and checking whether the coefficient of determination of that regression line is greater than or equal to a predetermined value. A linear portion determined to be a missing portion is defined as a linear portion where the variation in the positions of the pixels corresponding to that linear portion, calculated based on the pixel positions, is below a predetermined threshold. If the input image is not missing (No in S803), the image completion processing unit 402 terminates the process; if the input image is missing (Yes in S803), the process proceeds to S804.

[0033] In S804, the image completion processing unit 402 sets a completion region in the area outside the input image adjacent to the missing portion of the input image. For example, if the right side of the input image is missing, the width of the completion region will be 1 / 4 of the width of the input image, and the height of the completion region will be the same as the height of the input image. If the upper side of the input image is missing, the width of the completion region will be the same as the width of the input image, and the height of the completion region will be 1 / 4 of the height of the input image. Note that the completion region is not limited to the area set in this way, and may be a predetermined fixed size regardless of the size of the input image. Furthermore, the shape of the completion region may be changed according to the shape of the object that is determined to be missing. For example, as in the example in Figure 9, if the head of a person is missing, a circular completion region may be set around the head of the person, following the shape of the head. Of course, it goes without saying that the completion region can be any shape, not just rectangles or circles.

[0034] In S805, the image completion processing unit 402 requests the image generation unit 411 of the server 104 to generate a completed image based on the information set in the image input unit 404 and the completion region information.

[0035] In S806, the image generation unit 411 generates a complementary image using generational AI technology based on the generation model 412, and transmits the generated complementary image to the image completion processing unit 402 of the client PC 102, thereby ending the process.

[0036] Furthermore, the determination of whether or not to perform image interpolation in S802 may be made using a flag that indicates whether or not to perform image interpolation, in addition to the content type. For example, a flag may be set for each content, and if the flag is "TRUE", image loss detection and interpolation image generation may be performed, and if it is "FALSE", processing may be terminated.

[0037] As described above, in this embodiment, image interpolation is performed for images with missing parts, thus eliminating the need for the user to prepare another image without missing parts or to prepare correct data to specify how to interpolate the image. Furthermore, since the image interpolation function is built into the process of importing the input image, the user does not need to interrupt layout editing to perform image interpolation separately, allowing for uninterrupted layout editing.

[0038] <Embodiment 2> As described above, each piece of content displayed on the layout editing area 704 can be edited, including adjusting its position, cropping, and filling. Therefore, in this embodiment, when cropping each piece of content during content editing, complementary images are generated simultaneously.

[0039] Figure 10 is a flowchart illustrating an example of a process for generating interpolated images when content is cropped on the layout data editing screen 700. This flowchart is initiated by the image interpolation processing unit 402 when the user requests, for example, an operation to perform cropping on an image on the layout data editing screen 700. The same process as in Embodiment 1 described above will not be explained.

[0040] S1001 to S1002 are the same as S801 to S802.

[0041] In S1003, the image completion processing unit 402 performs a cropping process, or object extraction, on each piece of content. An object is a foreground object in an image, such as a person or an animal. Object extraction is the process of extracting the foreground of a person or animal from the input image, that is, removing the background.

[0042] In S1004, the image completion processing unit 402 determines whether the extracted object, such as a person or an animal, is missing. One example of how to determine whether the extracted object is missing is whether at least a part of the outline of the extracted object matches one or more edges of the input image, either top, bottom, left, or right, similar to Embodiment 1. If the object is not missing (No in S1004), the image completion processing unit 402 terminates the process. If the object is missing (Yes in S1004), the process proceeds to S1005.

[0043] S1005-S1007 are the same as S804-S806.

[0044] Furthermore, in S1006, the image used by the image completion processing unit 402 when requesting image generation from the image generation unit 411 of the server 104 may be either the image after object extraction, i.e., the input image with the background removed, or the input image before object extraction. Of course, if the input image before object extraction is used, it goes without saying that object extraction will be performed again before providing the completed image to the user. Also, if the image after object extraction is used, a white or black image may be used as the background. Of course, it goes without saying that the background will be removed again before providing the completed image to the user.

[0045] As described above, in this embodiment, if there are missing parts in the objects cut out from each content, image interpolation is performed, so, as with Embodiment 1, the user is saved the trouble of preparing another input image without missing parts. Furthermore, since the interpolation function is provided during the cropping process, the user does not need to interrupt layout editing to perform separate image interpolation processing, and can perform layout editing without interruption.

[0046] <Embodiment 3> In this embodiment, it is determined whether or not to generate a complementary image based on the placement position of the content.

[0047] Figure 11(a) shows an example where content 1101, with its right side missing, is placed in the center of layout area 1100. In this case, it is thought that filling in the missing portion of content 1101 would result in a more complete poster or flyer.

[0048] Figure 11(b) shows an example where content 1101, which is missing on the right side, is placed to the right of the layout area 1100. In this case, since the position of the edge of the layout area 1100 and the missing portion of content 1101 coincide, it is considered that there is no need to fill in the missing portion of content 1101. Furthermore, even if the missing portion of content 1101 is positioned outside the layout area 1100, it is considered that there is no need to fill in the missing portion of content 1101.

[0049] Figure 12 is a flowchart illustrating an example of a process that determines whether or not to generate a complementary image based on the placement position of the content in the layout data editing screen 700. Processes similar to those described in the above embodiment are omitted from the explanation. S1201 to S1203 are the same as S801 to S803.

[0050] In S1204, the image completion processing unit 402 obtains the image placement position based on the layout coordinates 603, which indicate the position of the content set in the image input unit 404, and the content size 604, which indicates the size of the content.

[0051] In S1205, the image completion processing unit 402 determines whether the location of the missing portion of the input image is within the layout area. That is, if content 901 with a missing right side is placed in the middle of the layout area 1100, as shown in Figure 11(a), S1205 determines that the missing portion is within the layout area 1100. If content 1101 is placed so that the missing portion is located at the right edge of the layout area 1100 or outside the layout area 1100, as shown in Figure 11(b), S1205 determines that the missing portion is not within the layout area 1100. If the result of the determination of whether the missing portion exists is "not present" (No in S1205), the image completion processing unit 402 terminates the process. If the result of the determination of whether the missing portion exists is "present" (Yes in S1205), the process proceeds to S1206.

[0052] In S1206, the image completion processing unit 402 sets the completion area in the same manner as in S804. When determining the missing portion of the input image and the placement position of the input image, the determination may be made based on the layout area 1100, or, in addition to the layout area 1100, information from the device information holding unit 406 from the image output device 100 may be obtained and the determination may be made based on the area where the image can be drawn.

[0053] Sections S1207 to S1208 are the same as sections S805 to S806.

[0054] As described above, in this embodiment, even if the input image has missing parts, unnecessary image generation processing can be avoided, thereby reducing computationally expensive processing and enabling a reduction in processing load and a reduction in user working time.

[0055] <Embodiment 4> The interpolated images generated by the generative model are not necessarily the images the user expected. On the other hand, as explained using Figure 5, it is possible to change the generated 502 by changing the seed value of the random number generator that generates the initial values ​​used in the generative model. Therefore, it is possible to generate multiple interpolated images and let the user choose which one to use.

[0056] Figure 13 shows an example of a processing flow for generating multiple complementary images in the layout data editing screen 700. The processing described in the above-described embodiment will not be explained further.

[0057] S1301-S1306 are the same as S801-S806.

[0058] In S1307, the image generation unit 411 changes the seed value of the random number generator used to generate initial values ​​used in the generation model used in the interpolation image generation process.

[0059] After performing S1307, return to S1305, and repeat S1305 to S1307 multiple times. The number of repetitions may be a predetermined fixed number, or it may be a number set by the user each time.

[0060] In S1308, the image interpolation processing unit 402 displays the generated interpolated images on the display 309.

[0061] In S1309, the image completion processing unit 402 receives input from the user to select one of the multiple completion images displayed, confirms the completion image to be used, and terminates the process.

[0062] As explained above, in this embodiment, by providing multiple complementary images for an input image with missing parts, it is possible to provide an image that better matches the user's intent.

[0063] <Embodiment 5> As shown in Figure 4, the image completion processing unit 402 can set text prompts to give instructions for image generation processing using text, as well as input images. By setting conditions using text as well as images, it is possible to generate images with higher accuracy. Therefore, when requesting image generation from the image generation unit 411, text information may be provided in addition to the image.

[0064] Figure 14 shows an example of a processing flow in the layout data editing screen 700, where prompt information is also passed to the image generation unit 411 when generating a complementary image. The processing described in the above-described embodiment will not be explained further.

[0065] S1401-S1404 are the same as S801-S804.

[0066] In S1405, the image completion processing unit 402 sets prompts as supplementary information for the generated image in the prompt setting unit 403. For example, as shown in Figure 8, if the head of a person is used as the completion area, the prompts will be text information such as "person," "head," or "hat." If the tail of an animal is used as the completion area, the prompts will be text information such as "animal," "dog," or "tail." The prompts may be estimated by the image completion processing unit 402 from the image information set in the image input unit 404, or they may be set based on user input.

[0067] In S1406, the image completion processing unit 402 requests the image generation unit 411 of the server 104 to generate a completed image based on the information set in the image input unit 404, the completion area information, and the information set in the prompt setting unit 403.

[0068] S1407 is the same as S806.

[0069] As explained above, in this embodiment, by providing the generation of a completed image using prompts for an image with missing parts, it is possible to provide an image that better matches the user's intent.

[0070] <Embodiment 6> The image generation unit 411 generates a interpolated image to fill in the missing parts of the input image, but in some cases, the generated interpolated image may also have missing parts. Therefore, it is desirable to determine whether the generated interpolated image has missing parts, and if so, to generate a re-interpolated image that fills in the missing parts of the interpolated image.

[0071] Figure 15 shows an example of a processing flow for determining whether or not there are missing parts in the generated interpolated image when generating an interpolated image on the layout data editing screen 700. The processing described in the above-described embodiment will not be explained further.

[0072] S1501-S1507 are the same as S801-S807.

[0073] In S1506, the image completion processing unit 402 determines whether or not there are missing parts in the generated completion image. If there are no missing parts (No in S1506), the image completion processing unit 402 terminates the process. If there are missing parts (Yes in S1506), the process returns to S1504.

[0074] In S1504, the image completion processing unit 402 sets a completion region in the image input unit 404 in the region adjacent to the missing portion of the completion image generated in S1506. Then, by executing S1505 to S1507 again, a re-completed image is generated in which the missing portion of the completion image has been completed.

[0075] As explained above, in this embodiment, by determining whether or not there are missing parts in the generated interpolated image, it is possible to provide the user with an interpolated image that does not contain any missing parts.

[0076] <Embodiment 7> The system may allow users to indicate whether or not they want to generate complementary images.

[0077] Figure 16 shows an example of a processing flow in the layout data editing screen 700, where the user is asked whether or not to generate a interpolated image if there are missing parts in the image. The processing described in the above-described embodiment will not be explained further.

[0078] S1601-S1603 are the same as S801-S803.

[0079] In S1604, the image completion processing unit 402 displays a dialog box to the user, allowing the user to select whether or not to generate a completed image, and accepts user input indicating whether or not to perform the completion process.

[0080] In S1605, the image interpolation processing unit 402 terminates processing if the user chooses not to perform interpolation processing (No in S1605), and proceeds to S1606 if the user chooses to perform interpolation processing (Yes in S1605).

[0081] In S1606, the image interpolation processing unit 402 sets the interpolation region, just as in S804.

[0082] S1607-S1608 are the same as S805-S806.

[0083] As explained above, in this embodiment, the user can choose whether or not to generate complementary images.

[0084] <Embodiment 8> When cropping content, some extracted objects may be cut off because they are hidden behind other objects. Figure 17 shows an example of an extracted object being cut off.

[0085] Figure 17(a) shows content in which part of a dog is hidden behind a tree. Figure 17(b) is an example showing the result of extracting the dog object from the content shown in Figure 17(a). In the result shown in Figure 17(b), the dog object has been extracted, but the image is missing where the tree was. In this embodiment, in such cases, as shown in Figure 17(c), a interpolated image is generated using the missing part of the dog object, i.e., the area where the tree was between the dog objects, as the interpolation region. This allows the user to generate the interpolated image they desire, eliminating the need for the user to prepare the image again.

[0086] Figure 18 shows an example of a processing flow in the layout data editing screen 700 where, during content cropping, if the same object is extracted as multiple objects, a interpolated image is generated with the area between the same objects as the interpolated region. The processing similar to that in the embodiment described above will not be explained further.

[0087] S1801-S1804 are the same as S901-S904.

[0088] In S1805, the image interpolation processing unit 402 determines whether there are multiple extracted objects. If there are no multiple extracted objects (No in S1805), the process proceeds to S1808; if there are multiple extracted objects (Yes in S1805), the process proceeds to S1806.

[0089] In S1806, the image interpolation processing unit 402 determines whether the extracted multiple objects are the same object. One example of a method for determining whether multiple objects are the same object is to determine that they are the same object if a portion of the contour of each of the extracted multiple objects has a shape that is close to a straight line, and these portions of the shape that are close to a straight line are located at a distance from each other and facing each other. As a criterion for determining whether the shape is close to a straight line, for example, a Hough transform is performed on the pixels that make up the contour to detect straight lines, and it is possible to determine whether the parameters of the straight line have a peak that falls within a predetermined width. Alternatively, a regression line may be found, and it may be determined whether the coefficient of determination of that regression line is greater than or equal to a predetermined value. In addition, when extracting objects in S1803, the user may be allowed to set the objects to be extracted, and the extracted objects set may be determined to be the same object. If the multiple objects are not the same (No in S1806), the process proceeds to S1808, and if the multiple objects are the same (Yes in S1806), the process proceeds to S1807.

[0090] In S1807, the image interpolation processing unit 402 sets an interpolation region in the area between the extracted identical objects.

[0091] S1808~S1810 are the same as S1005~S1007.

[0092] As described above, in this embodiment, when editing the cropped content, if there are multiple cropped objects and those multiple objects are the same object, image interpolation is performed. Therefore, as in Embodiment 1, the user is not required to prepare a different image again. Furthermore, by providing the interpolation function within the cropping function, the user does not need to perform the interpolation process separately and does not have to stop the layout editing work.

[0093] <Embodiment 9> The system may accept the interpolation region for generating the interpolated image from the user.

[0094] Figure 19 shows an example of a processing flow in the layout data editing screen 700, where, if there are missing parts in the image, the system accepts a interpolation region from the user to generate a interpolated image. The processing described in the above-described embodiment will not be explained further.

[0095] S1901-S1903 are the same as S801-S803.

[0096] In S1904, the image completion processing unit 402 displays a dialog box to allow the user to set the completion region for generating the completion image, and accepts user input to specify the completion region.

[0097] Figure 20 shows an example of a dialog box that allows the user to set the interpolation region for generating a interpolated image. The interpolation region setting dialog box 2000 includes a message 2001 prompting the user to set the interpolation region because there are missing parts in the input image, an image display area 2002 for setting the interpolation region for the input image, and an OK button 2003 for completing the setting.

[0098] Figure 20(a) shows an example of the startup of the interpolation area settings dialog 2000, with the OK button 2003 disabled. Figure 20(b) shows an example where the user has set the interpolation area 2004 to the outer area adjacent to the input image in the image display area 2002. In the example shown in Figure 20(b), the interpolation area 2004 is set, so the OK button 2003 is enabled.

[0099] Furthermore, when the interpolation area setting dialog 2000 in Figure 20(a) is launched, the interpolation area set by the image interpolation processing unit 402 may be displayed as the initial value of the interpolation area 2004, similar to S804. The user can edit the size of the interpolation area 2004.

[0100] S1905-S1906 are the same as S805-S806.

[0101] As explained above, in this embodiment, the user can set the interpolation region for generating the interpolation image. By setting the interpolation region, the user can set the size and location of the interpolation region according to their intentions.

[0102] <Embodiment 10> Whether or not to generate a complementary image may be determined based on the overlapping state with other content.

[0103] Figure 21 shows an example of overlapping content placed on different layers. In the example shown in Figure 21, the dog content 2101 and the triangle content 2102 are placed on different layers in layout data 2100. Also, the triangle content 2102 is placed in front of the dog content 2101. Furthermore, the right side of the dog content 2101 is missing.

[0104] Figure 21(a) shows an example where triangular content 2102 is placed to the left of dog content 2101. In this case, the missing portion of dog content 2101 is visible, so it is thought that filling in the missing portion would result in a more complete poster or flyer.

[0105] Figure 21(b) shows an example where triangular content 2102 is placed to the right of dog content 2101. In this case, since triangular content 2102 is placed on the missing portion of dog content 2101 so as to hide the missing portion, the missing portion of dog content 2101 becomes invisible, and therefore it is considered unnecessary to fill in the missing portion.

[0106] Figure 22 shows an example of a processing flow in the layout data editing screen 700 that determines whether or not to generate complementary images according to the overlapping state of the content. The processing similar to that described in the above embodiment will not be explained further.

[0107] S2201~S22034 are the same as S1201~S1204.

[0108] In S2205, the image completion processing unit 402 determines whether other content is superimposed in front of the input image that has a missing portion. If other content is superimposed in front of the input image (Yes in S2205), the process proceeds to S2206; if no other content is superimposed (No in S2205), the process proceeds to S2207.

[0109] In S2206, the image completion processing unit 402 determines the overlapping position of the missing portion of the image with other content. If the position of the missing portion of the input image matches that of the other content (No in S2206), the image completion processing unit 402 terminates the process. If the position of the missing portion of the image does not match that of the other content (Yes in S2206), the process proceeds to S2208.

[0110] In S2207, the image interpolation processing unit 402 performs the same placement position determination as in S1205.

[0111] In S2208, the image interpolation processing unit 402 sets the interpolation region, just as in S1206.

[0112] S2209~S2210 are the same as S1207~S1208.

[0113] As explained above, in this embodiment, even if there are missing parts in the input image, the decision of whether or not to perform interpolation is made based on the overlap state between the input image and other content, thus eliminating unnecessary image generation processing. Furthermore, image generation processing is computationally very expensive and, naturally, requires processing time. Eliminating unnecessary image generation processing leads to a reduction in computational costs and a reduction in user work time.

[0114] <Embodiment 11> As described in Embodiment 4, in addition to generating multiple interpolated images, multiple interpolation regions may be specified to generate multiple interpolated images.

[0115] Figure 23 shows an example of the processing flow for setting the interpolation area and generating multiple interpolation images in the layout data editing screen 700. The processing described above, similar to that in the embodiment, will not be explained further.

[0116] S2301-S2303 are the same as S1301-S1303.

[0117] In S2304, the image interpolation processing unit 402 sets the interpolation region and generates the interpolated image, just as in S1304 (S2305-S2307). At that time, the process of generating the interpolated image is repeated multiple times. The number of repetitions may be set to a fixed value, may be changed by the user, or may be set by the user each time the interpolated image is generated. Also, when setting the interpolation region repeatedly, the size, shape, and position of the interpolation region may be changed. Of course, the settings for the interpolation region may be set to a fixed value, or may be changed by the user.

[0118] S2308~S2309 are the same as S1308~S1309.

[0119] As explained above, in this embodiment, for an image with missing parts, a complementary image is generated for multiple complementary regions, and by providing the corresponding multiple complementary images, it is possible to provide an image that better matches the user's intent.

[0120] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0121] This disclosure includes the following configurations and methods:

[0122] [Configuration 1] An acquisition means for acquiring an input image, An extraction means for extracting objects from the aforementioned input image, A means for identifying the missing portion of the object extracted by the extraction means based on the shape of the object's outline, A generation means for generating a complementary image by filling in the missing portion identified in the input image, An information processing device characterized by having the following features.

[0123] [Configuration 2] The identification means identifies the linear portion as the missing portion if the contour of the extracted object includes a linear portion. The information processing device according to configuration 1, characterized by the above.

[0124] [Configuration 3] The identifying means determines that the portion of the contour that coincides with the edge of the input image is the straight line portion. The information processing apparatus according to configuration 2, characterized in that...

[0125] [Structure 4] The aforementioned straight portion is defined as a straight line where the variation in the position of the pixels relative to the straight line, calculated based on the position of the pixels corresponding to the straight portion, is below a predetermined threshold. The information processing apparatus according to configuration 3, characterized by the features described herein.

[0126] [Composition 5] The generation means sets the region outside the input image adjacent to the linear portion as a interpolation region for the input image. An information processing device according to any one of configurations 2 to 4, characterized by the above.

[0127] [Composition 6] The generation means, when the extraction means extracts multiple objects from the input image and the multiple objects are parts of the same object, sets a interpolation region for the input image between the multiple objects that constitute the same object. An information processing device according to any one of configurations 2 to 4, characterized by the above.

[0128] [Composition 7] The generation means determines that if the missing portions of the plurality of objects extracted by the extraction means are facing each other, the plurality of objects are part of the same object. The information processing device according to configuration 6, characterized by the features described therein.

[0129] [Structure 8] The generation means generates the interpolated image when an object cropping process is performed on the input image. An information processing device according to any one of configurations 1 to 7.

[0130] [Composition 9] The generation means generates the interpolated image if the missing portion is within a predetermined area. An information processing device according to any one of configurations 1 to 8.

[0131] [Configuration 10] The generation means generates a plurality of different complementary images for a single input image. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above.

[0132] [Composition 11] The generation means acquires a text prompt and generates the complementary image based on the acquired text prompt. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above.

[0133] [Composition 12] The extraction means extracts the object from the complementary image generated by the generation means, The aforementioned identification means identifies the missing portion of the object extracted from the complementary image, The generation means generates a re-completed image in which the missing portion identified in the completed image is completed. An information processing device according to any one of configurations 1 to 11, characterized by the above.

[0134] [Composition 13] Further equipped with a means for receiving user input, The generation means determines whether or not to generate the complementary image in response to the user input. An information processing apparatus according to any one of configurations 1 to 12, characterized by the above.

[0135] [Composition 14] Further equipped with a means for receiving user input, The generation means sets a completion area based on the user input. An information processing device according to any one of configurations 1 to 13, characterized by the above.

[0136] [Composition 15] The generation means does not generate the interpolation image if multiple input images are acquired by the acquisition means and other input images are placed on the missing portion. An information processing device according to any one of configurations 1 to 14, characterized by the above.

[0137] [Composition 16] The generation means generates a plurality of complementary images with different complementary regions. An information processing device according to any one of configurations 1 to 15, characterized by the above.

[0138] [Composition 17] The extraction means extracts objects having predetermined attributes. An information processing device according to any one of configurations 1 to 16, characterized by the above.

[0139] [Composition 18] The extraction means obtains user input specifying an object and sets the object to be extracted based on the user input. An information processing device according to any one of configurations 1 to 17, characterized by the above.

[0140] [Composition 19] The generation means generates the complementary image using a pre-machine-trained generation model. An information processing apparatus according to any one of configurations 1 to 18, characterized by the above.

[0141] [Configuration 20] Steps to obtain the input image, The steps include extracting objects from the aforementioned input image, The steps include identifying the missing portion of the object extracted by the extraction step based on the characteristics of the object's contour, The steps include generating a interpolated image by interpolating the missing portion identified in the input image, An information processing method characterized by having the following features.

[0142] [Composition 21] A program for causing a computer to function as an information processing device as described in any one of items 1 to 19.

Claims

1. An acquisition means for acquiring an input image, An extraction means for extracting objects from the aforementioned input image, A means for identifying the missing portion of the object extracted by the extraction means based on the shape of the object's outline, A generation means for generating a complementary image by filling in the missing portion identified in the input image, Equipped with, The identification means, when the extraction means extracts a plurality of objects from the input image and the plurality of objects are parts of the same object, identifies the region between the plurality of objects that constitute the same object as the missing portion. The generation means generates a interpolated image by filling in the missing portion identified in the input image. An information processing device characterized by the following:

2. The identification means identifies the linear portion as the missing portion if the contour of the extracted object includes a linear portion. The information processing apparatus according to feature 1.

3. The identifying means determines that the portion of the contour that coincides with the edge of the input image is the straight line portion. The information processing apparatus according to feature 2.

4. The aforementioned straight portion is defined as a straight line where the variation in the position of the pixels relative to the straight line, calculated based on the position of the pixels corresponding to the straight portion, is below a predetermined threshold. The information processing apparatus according to claim 3.

5. The aforementioned identifying means sets the region outside the input image adjacent to the linear portion as a interpolation region for the input image. The information processing apparatus according to feature 2.

6. The identification means determines that if the missing portions of the plurality of objects extracted by the extraction means are facing each other, the plurality of objects are part of the same object. The information processing apparatus according to feature 1.

7. The generation means generates the interpolated image when an object cropping process is performed on the input image. The information processing apparatus according to feature 1.

8. The information processing apparatus according to claim 1, characterized in that the generation means generates the interpolated image when the missing portion is within a predetermined area.

9. The generation means generates a plurality of different complementary images for a single input image. The information processing apparatus according to feature 1.

10. The generation means acquires a text prompt and generates the complementary image based on the acquired text prompt. The information processing apparatus according to feature 1.

11. The extraction means extracts the object from the complementary image generated by the generation means, The aforementioned identification means identifies the missing portion of the object extracted from the complementary image, The generation means generates a re-completed image in which the missing portion identified in the completed image is completed. The information processing apparatus according to feature 1.

12. Further equipped with a means for receiving user input, The information processing apparatus according to claim 1, characterized in that the identifying means determines whether or not to generate the complementary image in response to the user input.

13. Further equipped with a means for receiving user input, The aforementioned identification means sets a completion area based on the user input. The information processing apparatus according to feature 1.

14. The generation means does not generate the interpolation image if multiple input images are acquired by the acquisition means and other input images are placed on the missing portion. The information processing apparatus according to feature 1.

15. The generation means generates a plurality of complementary images with different complementary regions. The information processing apparatus according to feature 1.

16. The extraction means extracts objects having predetermined attributes. The information processing apparatus according to feature 1.

17. The extraction means obtains user input specifying an object and sets the object to be extracted based on the user input. The information processing apparatus according to feature 1.

18. The information processing apparatus according to claim 1, characterized in that the generation means generates the complementary image using a pre-machine-trained generation model.

19. The generated complementary image is displayed on the layout data editing screen. The information processing apparatus according to feature 1.

20. The aforementioned layout data editing screen is a screen for editing layout data related to printed materials or documents. The information processing apparatus according to feature 19.

21. Steps to obtain the input image, The steps include extracting objects from the aforementioned input image, The steps include identifying the missing portion of the object extracted by the extraction step based on the characteristics of the object's contour, The steps include generating a interpolated image by interpolating the missing portion identified in the input image, It has, The aforementioned identification step involves extracting multiple objects from the input image, and if the multiple objects are parts of the same object, identifying the region between the multiple objects that constitute the same object as the missing portion. The information processing method is characterized in that the generation step involves generating a interpolated image by filling in the missing portion identified in the input image.

22. A program for causing a computer to function as an information processing device according to any one of claims 1 to 20.

Images