Image processing systems, image processing methods, programs, and data structures

Abstract

To generate images that include objects and backgrounds more efficiently. The image processing method includes the steps of: acquiring a plurality of images to be processed, each representing an object along with the background of the object (S103); generating a background candidate image from each of the plurality of images to be processed, which does not include an image of the object represented in the image to be processed (S105); and storing at least a portion of the background candidate images generated from each of the plurality of images to be processed in a storage unit, associating the background candidate image with information indicating the type of object represented in the image to be processed from which it was generated (S106).

Description

Technical Field

[0001] The present invention relates to an image processing system, an image processing method, a program, and a data structure.

Background Art

[0002] For example, when displaying an image of an object on a website, in order to effectively convey the appeal of the object, an image in which the object is placed on an appropriate background is displayed. Furthermore, services are also provided that use generative AI to generate an image in which the object is placed on a background corresponding to the object.

Summary of the Invention

Problems to be Solved by the Invention

[0003] When generating an image including an object and a background using generative AI, the computing resources required for generation are large, and inconveniences such as deterioration of response are likely to occur.

[0004] An object of the present disclosure is to provide a technology that enables more efficient generation of an image including an object and a background.

Means for Solving the Problems

[0005] (1) A step of acquiring a plurality of processing target images each representing a background of the object together with the object, a step of generating a background candidate image not including an image of the object represented in the processing target image from each of the plurality of processing target images, and a step of storing at least a part of the background candidate images generated from each of the plurality of processing target images in a storage unit in association with information indicating the type of the object represented in the processing target image from which the background candidate image is generated. An image processing method including these steps.

[0006] (2) An image processing method in which, in (1), the steps of acquiring information indicating the pose of an object represented in each of the plurality of images to be processed, and storing in a storage unit at least a portion of the background candidate images generated from each of the plurality of images to be processed, in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and information indicating the pose of the object represented in the image to be processed.

[0007] (3) An image processing method comprising the step of obtaining the image size of an object represented in each of the plurality of images to be processed, wherein in the storage step, at least a portion of the background candidate images generated from each of the plurality of images to be processed are stored in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and the image size of the object included in the image to be processed.

[0008] (4) An image processing method comprising the step of obtaining the position of an image of an object contained in each of the plurality of images to be processed, wherein in the storage step, at least a portion of the background candidate images generated from each of the plurality of images to be processed are stored in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and the position of an image of the object represented in the image to be processed.

[0009] (5) An image processing method in which, in any of (1) to (4), in the step of acquiring the multiple images to be processed, the images of the multiple objects are input to a target image generation model, which is a machine learning model that generates an image in which the background is represented together with the object from the image of the object, and the output of the target image generation model is acquired to acquire the multiple images to be processed.

[0010] (6)(5) An image processing method comprising: in the step of acquiring the plurality of images to be processed, instructing the target image generation model to generate the plurality of images to be processed based on the original image of each of the plurality of objects in a given pose; and in the step of storing, storing in the storage unit at least a portion of the background candidate images generated from each of the plurality of images to be processed, in association with the given pose of the original image of the object from which the image to be processed was generated and information indicating the type of the object.

[0011] (7)(5) An image processing method in which, in the step of acquiring the plurality of images to be processed, for each of the plurality of objects, the target image generation model is instructed to generate an image to be processed that includes an image of the object and the background based on the original image of the object, and in the step of storing, at least a portion of the background candidate images generated from each of the plurality of images to be processed are stored in the storage unit in association with at least one of the position and size of the image of the object represented in the image to be processed and information indicating the type of the object.

[0012] (8) An image processing method in which, in any of (1) to (7), in the step of generating the background candidate image, the image of the object represented in the processing target image is removed from each of the plurality of processing target images, and the background area hidden by the image of the object is interpolated by an image editing model, which is a machine learning model that edits the input image, in order to obtain the background candidate image.

[0013] (9) An image processing system comprising: an image acquisition means for acquiring a plurality of images to be processed, each representing an object and the background of the object; an image generation means for generating a background candidate image from each of the plurality of images to be processed, which does not include an image of the object included in the image to be processed; and a storage processing means for storing at least a portion of the background candidate images generated from each of the plurality of images to be processed in a storage unit, in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated.

[0014] (10) A program for causing a computer to function as: an image acquisition means for acquiring a plurality of images to be processed, each representing an object and the background of the object; an image generation means for generating a background candidate image from each of the plurality of images to be processed, which does not include an image of the object included in the image to be processed; and a storage processing means for storing at least a portion of the background candidate images generated from each of the plurality of images in a storage unit, in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated.

[0015] (11) An image processing method comprising the steps of: acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; selecting one of a plurality of background candidate images, each generated from a plurality of processing target images that each represent the background of the object together with the object, and which does not include an image of the object included in the processing target image, from among the background candidate images stored in a storage unit that stores the background candidate image in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the posture of the object represented in the processing target image, based on the information indicating the type of the object to be superimposed and the information indicating the posture of the object to be superimposed; generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and outputting information of the generated image.

[0016] (12)(11) The storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, information indicating the pose of the object represented in the processing target image, and information indicating the size of the object represented in the processing target image, and in the generation step, generates an image in which an image of the object to be superimposed, which has been enlarged or reduced based on the size stored in association with the selected background candidate image, is superimposed on the selected background candidate image.

[0017] (13)(11) or (12), the storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, information indicating the pose of the object represented in the processing target image, and the position of the image of the object represented in the processing target image, and in the generation step, an image is generated in which the image of the object to be superimposed is superimposed on the selected background candidate image based on the position stored in association with the selected background candidate image, an image processing method.

[0018] (14) An image processing system comprising: an object acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; an image selection means for selecting one of a plurality of background candidate images, each generated from a plurality of processing target images that each represent the background of the object together with the object, and which do not include an image of the object included in the processing target image, from among the background candidate images stored in a storage unit that stores the background candidate image in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the posture of the object represented in the processing target image, based on the information indicating the type of object to be superimposed and the information indicating the posture of the object to be superimposed; a generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and an output means for outputting information of the generated image.

[0019] (15) Object acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; image selection means for selecting one of a plurality of background candidate images, each generated from a plurality of processing target images that each represent the background of the object together with the object, and which do not include an image of the object included in the processing target image, from among the background candidate images stored in a storage unit that stores the background candidate image in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the posture of the object represented in the processing target image, based on the information indicating the type of the object to be superimposed and the information indicating the posture of the object to be superimposed; generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; output means for outputting information of the generated image; and a program.

[0020] A data structure including: a plurality of background candidate images each generated from a respective one of a plurality of processing target images that each represent an object and the background of the object, the background candidate images not including an image of the object included in the processing target image; information indicating a type of the object represented in the processing target image from which the background candidate image was generated, the information being associated with each of the plurality of background candidate images; and information indicating a posture of the object represented in the processing target image.

Advantages of the Invention

[0021] According to the present invention, an image including an object and a background can be generated more efficiently.

Brief Description of the Drawings

[0022] [Figure 1] FIG. 1 is a diagram showing an example of elements related to an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing functions realized by the image processing system. [Figure 3] FIG. 3 is a flowchart showing an example of processing of the background generation system. [Figure 4] FIG. 4 is a diagram showing an example of information stored in the original storage unit. [Figure 5] FIG. 5 is a diagram showing an example of an original image of an object. [Figure 6] FIG. 6 is a diagram showing an example of a generated image with a background. [Figure 7] FIG. 7 is a diagram showing an example of a background candidate image generated from the image with a background. [Figure 8] FIG. 8 is a diagram showing an example of information stored in the candidate storage unit. [Figure 9] FIG. 9 is a flowchart showing an example of processing of the image superimposing system. [Figure 10] FIG. 10 is a diagram showing an example of a query image. [Figure 11] FIG. 11 is a diagram showing an example of an image in which the query image and the background are superimposed.

Embodiments for Carrying Out the Invention

[0023] Embodiments of the present invention will be described below with reference to the drawings. For components denoted by the same reference numerals, redundant descriptions will be omitted.

[0024] Figure 1 shows an example of elements related to an image processing system according to an embodiment of the present invention. The image processing system includes a background generation system 1 and an image overlay system 2. The background generation system 1 and the image overlay system 2 are implemented by one or more computers. The background generation system 1 and the image overlay system 2 do not necessarily have to be separate.

[0025] Background generation system 1 pre-generates and stores candidate background images based on the object type and orientation using generation AI. Image overlay system 2 then selects one of the candidate background images from the query image of an object that requires a background, and generates an image with a background by overlaying the object image onto the selected candidate background image.

[0026] Each of the background generation system 1 and the image overlay system 2 includes one or more computers (e.g., server computers). The background generation system 1 includes one or more processors 11, one or more storage 12, and one or more communication units 13. The image overlay system 2 includes one or more processors 21, one or more storage 22, and one or more communication units 23. The background generation system 1 may include multiple computers, each containing one or more processors 11, storage 12, and communication units 13, or it may include one computer having one or more processors 11 and storage 12. The image overlay system 2 may include multiple computers, each containing one or more processors 21, storage 22, and communication units 23, or it may include one computer having one or more processors 21 and storage 22. The background generation system 1 and the image overlay system 2 may be implemented on one or more virtual servers or container infrastructure.

[0027] Processors 11 and 21 operate according to programs (also called instruction codes) stored in storage devices 12 and 22. Processors 11 and 21 also control communication units 13 and 23. Processors 11 and 21 may include, for example, a CPU (Central Processing Unit), and may also include a GPU (Graphic Processing Unit) or an NPU (Neural Processing Unit). The above programs may be provided via the Internet or the like, or they may be provided stored on a computer-readable storage medium such as flash memory or a DVD-ROM.

[0028] The storage devices 12 and 22 are composed of memory elements such as RAM and flash memory, and external storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs). The storage devices 12 and 22 store the above-mentioned programs. In addition, the storage devices 12 and 22 store information and calculation results input from the processors 11 and 12 and the communication units 13 and 23.

[0029] The communication units 13 and 23 are communication interfaces that communicate with other devices, such as network interface cards. The communication units 13 and 23 are composed of integrated circuits, antennas, and communication terminals that implement wireless LANs and wired LANs, for example. Based on the control of the processors 11 and 21, the communication units 13 and 23 input information received from other devices via the network to the processors 11 and 21 and storage devices 12 and 22, and transmit the information to the other devices.

[0030] Note that the hardware configuration of background generation system 1 and image overlay system 2 is not limited to the examples above. For example, background generation system 1 and image overlay system 2 may include devices for reading computer-readable information storage media (e.g., optical disc drives or memory card slots) and devices for inputting and outputting data with external devices (e.g., USB ports). External devices may be input devices or output devices.

[0031] Next, the functions provided by the image processing system will be described. Figure 2 is a block diagram showing the functions realized by the image processing system. Functionally, the image processing system includes a background image acquisition unit 51, a background candidate generation unit 52, a placement acquisition unit 53, a storage processing unit 54, a query acquisition unit 56, a background selection unit 57, an overlay unit 58, an output unit 59, a background generation model 61, a background interpolation model 62, an original storage unit 71, and a candidate storage unit 72. The background image acquisition unit 51, the background candidate generation unit 52, the placement acquisition unit 53, and the storage processing unit 54 are realized by the background generation model 61, the background interpolation model 62, and mainly by the processor 11 executing programs corresponding to each function stored in the storage 12 and controlling the communication unit 13, etc. The query acquisition unit 56, the background selection unit 57, the overlay unit 58, and the output unit 59 are realized by the processor 21 executing programs corresponding to each function stored in the storage 22 and controlling the communication unit 23, etc. The original storage unit 71 is mainly realized by the storage 12. The candidate storage unit 72 may be mainly implemented by storage 12 or storage 22.

[0032] The original storage unit 71 stores the original image of an object, which is used to generate background candidate images, in association with the object's type (category) and orientation. The orientation indicates the orientation of the object in the original image.

[0033] The candidate storage unit 72 stores multiple background candidate images in association with the object type and orientation. The object type is the object type of the original image used to generate the background candidate image, and the orientation is the orientation of the object in the original image used to generate the background candidate image. The candidate storage unit 72 may further store multiple background candidate images in association with the position and size of the object image. The method for determining the position and size of the object will be described later.

[0034] The background image acquisition unit 51 acquires multiple background images, each representing an object along with the background of that object. The background image acquisition unit 51 also acquires information indicating the type of object represented in each of the multiple background images, and information indicating the pose of the object represented in each of the multiple background images.

[0035] The background generation model 61 is a machine learning model that generates and outputs an image in which the background is represented along with the input object image. The background image acquisition unit 51 instructs the background generation model 61 to generate a background image including the object image and background for each of the multiple objects, based on the original image of the object, and acquires the background image output by the background generation model 61. The background generation model 61 may be a model that constitutes an image generation tool such as Stable Diffusion, which includes an extension of Canny Edge in ControlNet. The background generation model 61 may be implemented on one or more computers different from the background generation system 1.

[0036] The background candidate generation unit 52 obtains a background candidate image from each of the multiple background images that does not contain an image of the object represented in that background image.

[0037] The background interpolation model 62 generates an image in which the image of an object is removed from the background image and the background area hidden by the image of the object is interpolated. The background interpolation model 62 may be input to an image from which the image of an object has been removed from the background image, or it may be input to an image with a background and information indicating the regions of the image of an object. The background candidate generation unit 52 instructs the generation of an image in which the background area is interpolated based on the background image and acquires the image output by the background interpolation model 62 as a background candidate image. The background interpolation model 62 may be a machine learning model that implements so-called inpainting using, for example, Lama's method. The background interpolation model 62 may be implemented on one or more computers different from the background generation system 1.

[0038] The placement acquisition unit 53 acquires the size of the image of the object represented in each of the multiple background images, and the position of the image of that object within the background image.

[0039] The memory processing unit 54 stores at least a portion of the background candidate images generated from each of the multiple background images in the candidate storage unit 72, associating them with information indicating the type of object represented in the background image from which the background candidate image was generated. The memory processing unit 54 stores at least a portion of the background candidate images in the storage unit, associating them with information indicating their type and information indicating the pose of the object represented in the background image from which the background candidate image was generated. The memory processing unit 54 may further store at least a portion of the background candidate images generated from each of the multiple background images, associating them with at least one of the size of the image of the object included in the background image and the position of the image of the object represented in the background image.

[0040] The query acquisition unit 56 acquires a query image, which is an image of the object to be superimposed, information indicating the type of the object, and information indicating the orientation of the object.

[0041] The background selection unit 57 selects one of the background candidate images stored in the candidate storage unit 72 based on information indicating the type of object to be superimposed and information indicating the orientation of that object.

[0042] The superimposition unit 58 generates an image in which the query image is superimposed on the selected background candidate image. The superimposition unit 58 may also generate an image in which an image of the target object, which has been enlarged or reduced based on the size stored in the candidate storage unit 72 in association with the selected background candidate image, is superimposed on the selected background candidate image. Alternatively, the superimposition unit 58 may also generate an image in which an image of the target object is superimposed on the selected background candidate image based on the position stored in the candidate storage unit 72 in association with the selected background candidate image.

[0043] The output unit 59 outputs information about the image in which the image of the target object is superimposed on the selected background candidate image.

[0044] Next, the processing of the background generation system 1 will be described. Figure 3 is a flowchart showing an example of the processing of the background generation system 1. Figure 3 shows the processing flow for generating background candidate images and storing them in the candidate storage unit 72.

[0045] Before the process shown in Figure 3 begins, the original image of the object is stored in the original storage unit 71, associated with the object's type (category) and orientation. Figure 4 shows an example of the information stored in the original storage unit 71. In Figure 4, the original storage unit 71 stores the image ID, type, and orientation in association with the image itself, rather than the image itself.

[0046] In Figure 4, for example, the original image with image ID "ORG1" is associated with the type "rice cooker" and orientation "front". Similarly, the original image with image ID "ORG2" is associated with the type "rice cooker" and orientation "side". Figure 5 shows an example of an original image of an object. Figure 5 shows an example of an original image with image ID "ORG1". In the original image of the object shown in Figure 5, the background of the object is plain. As shown in Figure 4, the orientation does not have to be a precise angle, but may be information indicating one of a predetermined number of candidate values.

[0047] The details of the process shown in Figure 3 will now be explained. First, the background image acquisition unit 51 acquires one of the one or more original images stored in the original storage unit 71 that have not yet been acquired (S101). The background image acquisition unit 51 also acquires the type and orientation of the object in the acquired original image from the original storage unit 71 (S102).

[0048] The background image acquisition unit 51 instructs the background generation model 61 to generate an image of the object with a background based on the acquired original image (S103). More specifically, the background image acquisition unit 51 instructs the background generation model 61 to generate an image of the object with a background, including the object's image and background, based on the object's original image, and acquires the background image output by the background generation model 61.

[0049] For example, the background image acquisition unit 51 may acquire the contour of the acquired original image using ControlNet's Canny Edge, and then input a command to Stable Diffusion (more specifically, realistic-vision based on Stable Diffusion) to generate a background image based on that contour. This command inputs an image based on the original image (e.g., an image of the contour) to the background generation model 61 that constitutes Stable Diffusion, thereby acquiring a background image. This command may also include at least a portion of the object type, orientation, and background text associated with the type, which are further input to the background generation model 61. In addition, the position and size of the objects in the image input to the background generation model 61 may be adjusted in advance, and the background generation model 61 may output an image in which the objects are placed in the same position and size as in the input image.

[0050] Figure 6 shows an example of a generated image with a background. The background image acquisition unit 51 and the background generation model 61 generate an image with a background that represents the object along with the object's image. Note that when using Canny Edge, the color of the object in the background image may differ from the original image, but this does not cause any problems as the object in the background image will be removed in a later process.

[0051] When an image with a background is generated, the placement acquisition unit 53 acquires the position and size of the object's image in the image with the background (S104). The placement acquisition unit 53 may acquire the position and size of the object's image by inputting the image with a background and the type of object into a trained machine learning model (e.g., Grounding DINO) that returns the position and size of the object's image in an image when given the image and the type of object, and obtaining its output. Alternatively, the position and size of the object's image may be acquired by performing a so-called Region Proposal technique on the image with a background. Alternatively, a so-called pattern matching technique may be used to search for a region of the image with a background that matches the original image, and the position and size of the searched region may be acquired.

[0052] Furthermore, if the background generation model 61 outputs an image in which objects are placed at the same position and size as the input image, the position and size of the objects, which have been pre-adjusted in the input image, may be obtained as the position and size of the object images in the background image.

[0053] The background candidate generation unit 52 then obtains a background candidate image from the background image that does not include the image of the object, and in which the background hidden by the image of the object has been interpolated (S105). More specifically, the background candidate generation unit 52 generates mask information that masks the area of ​​the object image to be deleted in the background image, based on the position and size of the obtained image of the object. The background candidate generation unit 52 inputs the background image and its mask information into the background interpolation model 62, and obtains the output image as the background candidate image.

[0054] The background interpolation model 62 may be a pre-trained machine learning model, for example, called Lama-big. The image output from the background interpolation model 62 may be input to another pre-trained machine learning model called a Refiner, which improves image quality (for example, a refiner provided by realistic-vision), and the background candidate generation unit 52 may acquire the output of that pre-trained model as a background candidate image.

[0055] Figure 7 shows an example of a background candidate image generated from an image with a background. The image shown in Figure 7 is an example of a background candidate image generated from the background image shown in Figure 6. Known generative AI techniques can be used to remove objects from an image with a background and generate a natural-looking background candidate image.

[0056] The background interpolation model 62 may be another type of trained machine learning model that interpolates the background. The background interpolation model 62 may be configured to receive an image from which the object image has been removed from the background image, instead of mask information and the background image, and the background candidate generation unit 52 may input the removed image to the background interpolation model 62.

[0057] When a candidate background image is acquired, the memory processing unit 54 stores the candidate background image in the candidate storage unit 72 in association with the type of acquired object, the orientation of the object, and the position and size of the object in the background image (S106). The memory processing unit 54 may store the candidate background image in the candidate storage unit 72 in association with at least some of the type of acquired object, the orientation of the object, and the position and size of the object in the background image.

[0058] Figure 8 shows an example of the information stored in the candidate storage unit 72. Instead of the background candidate image itself, the candidate storage unit 72 stores the image ID of the background candidate image, along with its type, orientation, object position, and object size, all associated with each other. An image with the image ID "BG1" is, for example, the image shown in Figure 7. In the example in Figure 8, the object position is indicated by x and y coordinates, and the object size is indicated by the number of pixels in the vertical and horizontal directions.

[0059] Here, the memory processing unit 54 does not have to store some of the acquired background candidate images in the candidate storage unit 72. For example, the memory processing unit 54 may determine whether the quality of the background candidate images is sufficient by some method, and store the background candidate images that are determined to be of sufficient quality in the candidate storage unit 72 in association with their type, orientation, position, size, etc. Alternatively, the memory processing unit 54 may store the acquired background candidate images in the candidate storage unit 72 and then delete images from the stored background candidate images that are determined to be of insufficient quality or that are determined to infringe intellectual property rights such as copyright.

[0060] When the process in S106 is executed, the background image acquisition unit 51 determines whether an unprocessed original image exists (S107). If an unprocessed original image exists (Y in S107), the processes from S101 onwards are repeatedly executed. If no unprocessed original image exists (Y in S107), the process in Figure 3 is terminated.

[0061] In Figure 3, the background generation system 1 processes multiple original images one by one in sequence, but the processes S101 to S106 may be executed simultaneously and in parallel for multiple original images. Also, the original images do not need to be stored in the original storage unit 71 in advance. In this case, the original image, type, and orientation may be input sequentially each time the S101 process is performed.

[0062] Note that the process shown in Figure 3 does not necessarily have to be automatically executed by the background generation system 1. In this case, the background generation system 1 may execute each of the processes from S101 to S106, and the start of each process and the input of the data necessary for the process may be manually controlled. After the process shown in Figure 3 is completed, background candidate images stored in the candidate storage unit 72 that are determined to be of insufficient quality or have intellectual property rights issues may be deleted. Because background candidate images are generated in advance, it is possible to make such determinations and delete inappropriate background candidate images. Furthermore, for the objects, types, and orientations associated with the deleted background candidate images, the process from S101 to S105 may be repeated until background candidate images without quality and intellectual property rights issues are generated, thereby storing background candidate images without quality and intellectual property rights issues in the candidate storage unit 72.

[0063] Next, we will explain the processing of the image superposition system 2. Figure 9 is a flowchart showing an example of the processing of the image superposition system 2. Figure 9 shows the processing flow for generating an image of an object with a background using a candidate background image.

[0064] Before the process shown in Figure 9 begins, the candidate storage unit 72 stores background candidate images associated with at least some of the following: the object type, the object's orientation, and the object's position and size in the background candidate image, through the process shown in Figure 3. Here, the candidate storage unit 72 accessed in the process of Figure 9 may be a copy of the storage unit 12 of the background generation system 1 to the storage unit 22 of the image overlay system 2, or it may be stored in the storage unit 12 or other external storage accessible from the image overlay system 2.

[0065] First, the query acquisition unit 56 acquires an image of the object to be added to the background (query image) from the user (S201). The query acquisition unit 56 also acquires information indicating the type and orientation of the object in the query image (S202). The query acquisition unit 56 may acquire the query image, type, and orientation entered by the user, for example, via a computer operated by the user. The query acquisition unit 56 may also acquire the type of object represented in the query image by applying a known Region Proposal technique to the query image. Alternatively, for example, the type and orientation of the object may be acquired from the query image using a Region Proposal machine learning model that has learned combinations of object type and orientation as a single class.

[0066] The background selection unit 57 selects one of the background candidate images stored in the candidate storage unit 72 based on the type and orientation of the object obtained for the query image (S203). In other words, the background selection unit 57 selects a background candidate image from among the multiple background candidate images stored in the candidate storage unit 72 that is associated with the same type and orientation as the one obtained for the query image. The background selection unit 57 may also select a background candidate image that is associated with the type and orientation that is most similar to the one obtained for the query image.

[0067] The background selection unit 57 then retrieves the position and size of objects in the background candidate image, which are stored in the candidate storage unit 72 in association with the selected background candidate image (S204).

[0068] Figure 10 shows an example of a query image. In the example in Figure 10, the object type and orientation are the same as in Figure 5. Therefore, the background candidate image selected from the query image in Figure 10 will be the one shown in Figure 7. The position and size described in the row for image ID "BG1" in Figure 8 are also obtained.

[0069] After the query image is obtained, the superimposing unit 58 extracts the object region (image) from the query image (S205). The superimposing unit 58 may extract the object region using a known segmentation technique, such as u2net or is-net.

[0070] The superimposing unit 58 resizes (enlarges or reduces) the region of the extracted object based on the size obtained for the background candidate image (S206). For example, the superimposing unit 58 may resize the region of the object so that the vertical size obtained for the background candidate image is the same as the vertical size of the resized region. The superimposing unit 58 may also resize the region of the object so that the horizontal size obtained for the background candidate image is the same as the horizontal size of the resized region, or it may resize it so that the area indicated by the size obtained for the background candidate image is the same as the area of ​​the resized region.

[0071] The superposition unit 58 superimposes the resized object region image and the selected background candidate image based on the position obtained for the background candidate image (S207). The superposition unit 58 superimposes the image of the object region in front of the background candidate image. The superposition may involve generating information for each pixel of the superimposed image, or it may involve generating metadata that logically superimposes the layer of the object region image and the layer of the background candidate image.

[0072] The superimposed image 58 may modify the superimposed image, which is a combination of the object region image and the selected background candidate image, using a machine learning model to make the superimposed image more natural. For example, the method described in the paper "PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations" may be used with a machine learning model to harmonize the background and object images and adjust shadows. This paper is publicly available at URL https: / / openaccess.thecvf.com / content / CVPR2023 / papers / Guerreiro_PCT-Net_Full_Resolution_Image_Harmonization_Using_Pixel-Wise_Color_Transformations_CVPR_2023_paper.pdf. Alternatively, the superimposed image 58, which is a combination of the object region image and the selected background candidate image, or an image modified using the method in this paper, may be input into another trained machine learning model called a Refiner to obtain an improved image.

[0073] When superimposition is performed, the output unit 59 outputs the superimposed image (S208). The output image may be the image that has undergone the aforementioned modifications. The output unit 59 may output the image by transmitting the image information to a computer operated by the user, or by storing the image information in a predetermined storage or database.

[0074] Figure 11 shows an example of an image in which a query image and a background are superimposed. In the example in Figure 11, the image of the object shown in Figure 10 and the candidate background image shown in Figure 7 are superimposed.

[0075] In this embodiment, by pre-generating candidate background images according to the type of object, the time from when the user inputs an image of an object (query image) to when an image with a background is generated can be significantly reduced. Furthermore, by pre-generating candidate background images according to the pose of the object, the generated image with a background can be made to look more natural.

[0076] In this embodiment, background candidate images are generated from images of objects with backgrounds. Images of objects with backgrounds have a high probability of having a natural relationship between the objects and the background. Here, the position and size of the objects in the image of objects with backgrounds are used for superimposing the objects in the query image. This prevents the position and size of the objects in the superimposed image from becoming unnatural. Furthermore, by removing objects from the image with a background and applying a technique to interpolate the areas hidden by the objects, it is possible to generate background candidate images that are robust to changes in the shape of the objects.

[0077] It should be noted that the present invention is not limited to the embodiments described above. For example, instead of generating an image with a background from the original image of the object, the background image acquisition unit 51 may acquire an image with a background in which the object is photographed together with the background. In this case, the image with a background is stored in the original storage unit 71 instead of the original image, and the processing from S104 to S106 in Figure 3 may be performed for each of the background images.

Claims

1. The steps include obtaining multiple images to be processed, each representing the background of the object, along with the object itself. The steps include generating a background candidate image from each of the multiple images to be processed, which does not include images of objects represented in the image to be processed, The steps include storing at least a portion of the background candidate images generated from each of the multiple images to be processed in a storage unit, associating the background candidate image with information indicating the type of object represented in the image to be processed from which it was generated, Includes, In the step of acquiring the multiple images to be processed, the images of each of the multiple objects are input to a target image generation model, which is a machine learning model that generates an image in which the background is represented together with the object from the image of the object, and the output of the target image generation model is obtained, thereby acquiring the multiple images to be processed. Image processing methods.

2. In the image processing method described in claim 1, The steps include obtaining information indicating the pose of an object represented in each of the aforementioned multiple images to be processed, In the storage step, at least a portion of the background candidate images generated from each of the multiple images to be processed are stored in the storage unit in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and information indicating the pose of the object represented in the image to be processed. Image processing methods.

3. In the image processing method described in claim 1, The process further includes the step of obtaining the image size of the object represented in each of the multiple images to be processed, In the aforementioned storage step, at least a portion of the background candidate images generated from each of the multiple images to be processed are stored in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and the size of the image of the object included in the image to be processed. Image processing methods.

4. In the image processing method described in claim 1, The process further includes the step of obtaining the image position of an object included in each of the plurality of images to be processed, In the step of causing the storage, at least a part of the background candidate images generated from each of the plurality of target images to be processed is stored in association with information indicating the type of the object represented in the target image from which the background candidate image is generated and the position of the image of the object represented in the target image to be processed. Image processing method.

5. In the image processing method according to claim 1, In the step of obtaining the plurality of target images to be processed, generation of the plurality of target images to be processed is instructed to the target image generation model based on the original images of the plurality of objects in respective given poses. In the step of causing the storage, at least a part of the background candidate images generated from each of the plurality of target images to be processed is stored in a storage unit in association with the given pose of the original image of the object from which the target image to be processed is generated and information indicating the type of the object. Image processing method.

6. In the image processing method according to claim 1, In the step of obtaining the plurality of target images to be processed, for each of the plurality of objects, generation of a target image to be processed including an image of the object and the background based on the original image of the object is instructed to the target image generation model. In the step of causing the storage, at least a part of the background candidate images generated from each of the plurality of target images to be processed is stored in a storage unit in association with at least one of the position and size of the image of the object represented in the target image to be processed and information indicating the type of the object. Image processing method.

7. In the image processing method according to claim 1, In the step of generating the background candidate image, from each of the plurality of target images to be processed, an image of the object represented in the target image to be processed is removed, and the background candidate image in which an area of the background hidden by the image of the object is interpolated by an image editing model which is a machine learning model for editing an input image is obtained. Image processing method.

8. Image acquisition means for acquiring a plurality of target images each representing an object and the background of the object; Image generation means for generating a background candidate image not including an image of the object included in each of the plurality of target images from each of the plurality of target images; A storage processing means that stores at least a portion of the background candidate images generated from each of the plurality of processing target images in a storage unit, in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated. Includes, The image acquisition means acquires the multiple images to be processed by inputting each image of the multiple objects into a target image generation model, which is a machine learning model that generates an image in which the background is represented together with the object from the image of the object, and obtaining the output of the target image generation model. Image processing system.

9. Image acquisition means for acquiring multiple processing target images that represent the object and the background of the object, respectively. Image generation means for generating a background candidate image from each of the plurality of images to be processed, which does not include images of objects contained in the image to be processed, and A storage processing means that stores at least a portion of the background candidate images generated from each of the plurality of processing target images in a storage unit, in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated. To make the computer function as, The image acquisition means acquires the multiple images to be processed by inputting each image of the multiple objects into a target image generation model, which is a machine learning model that generates an image in which the background is represented together with the object from the image of the object, and obtaining the output of the target image generation model. program.

10. The steps include obtaining an image of the object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the orientation of the object to be superimposed. A step of selecting one of several background candidate images, each generated from a plurality of processing target images that each represent the background of the object, and which do not include an image of the object included in the processing target image, from among the background candidate images stored in a storage unit that stores the background candidate image in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the pose of the object represented in the processing target image, based on information indicating the type of object to be superimposed and information indicating the pose of the object to be superimposed. The steps include generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image, The steps include outputting the information of the generated image, Image processing methods including [specific details omitted].

11. In the image processing method according to claim 10, The storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, information indicating the pose of the object represented in the processing target image, and information indicating the size of the object represented in the processing target image. In the generation step, an image of the object to be superimposed, which has been enlarged or reduced based on the size stored in association with the selected background candidate image, is superimposed on the selected background candidate image. Image processing methods.

12. In the image processing method according to claim 10, The storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, information indicating the pose of the object represented in the processing target image, and the position of the image of the object represented in the processing target image. In the generation step, an image is generated in which the image of the object to be superimposed is superimposed on the selected background candidate image, based on the position stored in association with the selected background candidate image. Image processing methods.

13. A means for acquiring an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the orientation of the object to be superimposed. Image selection means for selecting one of several background candidate images, each generated from a plurality of processing target images that each represent the background of the object, and which do not include an image of the object included in the processing target image, from among the background candidate images stored in a storage unit that stores the background candidate image in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the pose of the object represented in the processing target image, based on information indicating the type of object to be superimposed and information indicating the pose of the object to be superimposed. A generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image, An output means for outputting information of the generated image, Image processing methods including [specific details omitted].

14. A means for acquiring an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the orientation of the object to be superimposed. Image selection means selects one of several background candidate images stored in a storage unit that stores background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the pose of the object represented in the processing target image, based on information indicating the type of object to be superimposed and information indicating the pose of the object to be superimposed. A generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image, and Output means for outputting information of the generated image, A program that makes a computer function.

Images