Image processing system, image processing method, program, and data structure

Abstract

The present invention addresses the problem of generating an image including an object and a background more efficiently. Provided is an image processing method comprising: a step (S103) for acquiring a plurality of processing target images each representing an object with a background of the object; a step (S105) for generating a background candidate image not including an image of the object represented in the processing target image from each of the plurality of processing target images; and a step (S106) for storing at least a part of the background candidate images generated from each of the plurality of processing target images in a storage unit in association with information indicating a type of the object represented in the processing target image for which the background candidate image is generated.

Description

Image processing system, image processing method, program, and data structure

[0001] The present invention relates to an image processing system, an image processing method, a program, and a data structure.

[0002] For example, when displaying an image of an object on a website, the object is displayed against an appropriate background to effectively convey the appeal of the object. Furthermore, services are also provided that use generative AI to generate images in which the object is placed against a background appropriate for the object.

[0003] When generating an image that includes an object and a background using generation AI, a large amount of computing resources are required for generation, which can easily lead to inconveniences such as a decrease in response time.

[0004] An object of the present disclosure is to provide a technique that enables more efficient generation of an image that includes an object and a background.

[0005] (1) An image processing method including the steps of: acquiring a plurality of target images each representing an object together with a background of the object; generating a background candidate image from each of the plurality of target images, the background candidate image not including an image of the object represented in the target image; and storing at least a portion of the background candidate image generated from each of the plurality of target images in a memory unit in association with information indicating the type of object represented in the target image from which the background candidate image was generated.

[0006] (2) In (1), an image processing method includes a step of acquiring information indicating the posture of an object represented in each of the plurality of processing target images, and in the storing step, storing in a memory unit at least a portion of the background candidate images generated from each of the plurality of processing target images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and information indicating the posture of the object represented in the processing target image.

[0007] (3) In (1) or (2), the image processing method further includes a step of acquiring the image size of an object represented in each of the plurality of processing target images, and in the storing step, at least a portion of the background candidate images generated from each of the plurality of processing target images are stored in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and the image size of the object included in the processing target image.

[0008] (4) In any of (1) to (3), an image processing method further includes a step of acquiring the position of an image of an object included in each of the plurality of processing target images, and in the storing step, at least a portion of the background candidate images generated from each of the plurality of processing target images are stored in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated, and the position of the image of the object represented in the processing target image.

[0009] (5) In any one of (1) to (4), in the step of acquiring the plurality of processing target images, the image of each of the plurality of objects is input to a target image generation model, which is a machine learning model that generates an image that represents the object together with a background from an image of the object, and the plurality of processing target images are acquired by acquiring the output of the target image generation model.

[0010] (6) In (5), in the step of acquiring the plurality of processing target images, the target image generation model is instructed to generate the plurality of processing target images based on original images of the plurality of objects in given postures, and in the storing step, at least a portion of the background candidate images generated from each of the plurality of processing target images is stored in a memory unit in association with the given posture of the original image of the object from which the processing target image was generated and information indicating the type of the object.

[0011] (7) In (5), in the step of acquiring the plurality of processing target images, the step of instructing the target image generation model to generate a processing target image including an image of the object and the background based on an original image of the object for each of the plurality of objects, and in the step of storing, storing in a memory unit at least a portion of the background candidate images generated from each of the plurality of processing target images in association with at least one of the position and size of the image of the object represented in the processing target image and information indicating the type of the object.

[0012] (8) In any one of (1) to (7), in the step of generating the background candidate image, an image of an object represented in each of the plurality of processing target images is removed from the processing target images, and the background candidate image is obtained by interpolating the background area hidden by the image of the object using an image editing model, which is a machine learning model that edits input images.

[0013] (9) An image processing system including: an image acquisition means for acquiring a plurality of processing target images each representing an object together with the background of the object; an image generation means for generating a background candidate image from each of the plurality of processing target images that does not include an image of the object included in the processing target image; and a storage processing means for storing at least a portion of the background candidate image generated from each of the plurality of processing target images in a memory unit in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated.

[0014] (10) A program for causing a computer to function as an image acquisition means for acquiring a plurality of processing target images each representing an object together with the background of the object; an image generation means for generating a background candidate image from each of the plurality of processing target images that does not include an image of the object included in the processing target image; and a storage processing means for storing in a memory unit at least a portion of the background candidate image generated from each of the plurality of processing target images in association with information indicating the type of object represented in the processing target image from which the background candidate image was generated.

[0015] (11) An image processing method including the steps of: acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; selecting one of a plurality of background candidate images stored in a memory unit that stores the background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated and information indicating the posture of the object represented in the image to be processed; generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and outputting information about the generated image.

[0016] (12) In (11), the storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate images were generated, information indicating the posture of the object represented in the processing target image, and information indicating the size of the object represented in the processing target image, and in the generating step, an image is generated in which an image of the object to be superimposed, enlarged or reduced based on the size stored in association with the selected background candidate image, is superimposed on the selected background candidate image.

[0017] (13) In the image processing method of (11) or (12), the memory unit stores the plurality of background candidate images in association with information indicating the type of object represented in the processing target image from which the background candidate images were generated, information indicating the posture of the object represented in the processing target image, and the position of the image of the object represented in the processing target image, and in the generating step, generates an image in which the image of the object to be superimposed is superimposed on the selected background candidate image based on the position stored in association with the selected background candidate image.

[0018] (14) An image processing system including: a target acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; an image selection means for selecting one of a plurality of background candidate images stored in a memory unit that stores the background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated and information indicating the posture of the object represented in the image to be processed; a generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and an output means for outputting information on the generated image.

[0019] (15) A target acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the posture of the object to be superimposed; a plurality of background candidate images generated from each of a plurality of target images each representing an object together with the background of the object, and not including an image of the object included in the target image, the background candidate images being stored in a memory unit that stores the background candidate images in association with information indicating the type of object represented in the target image from which the background candidate images were generated and information indicating the posture of the object represented in the target image, based on the information indicating the type of the object to be superimposed and the information indicating the posture of the object to be superimposed; a generation means for generating an image in which an image of the object to be superimposed is superimposed on the selected background candidate image; and an output means for outputting information on the generated image; and a program.

[0020] (16) A data structure comprising: a plurality of background candidate images generated from a plurality of processing target images each representing an object together with the background of the object, and each not including an image of the object included in the processing target image; information associated with each of the plurality of background candidate images indicating the type of object represented in the processing target image from which the background candidate image was generated; and information indicating the posture of the object represented in the processing target image.

[0021] The present invention allows for more efficient generation of images that include objects and backgrounds.

[0022] FIG. 1 is a diagram illustrating an example of elements related to an image processing system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating functions realized by the image processing system. FIG. 3 is a flow diagram illustrating an example of processing by a background generation system. FIG. 4 is a diagram illustrating an example of information stored in an original storage unit. FIG. 5 is a diagram illustrating an example of an original image of an object. FIG. 6 is a diagram illustrating an example of a generated background-added image. FIG. 7 is a diagram illustrating an example of a background candidate image generated from the background-added image. FIG. 8 is a diagram illustrating an example of information stored in a candidate storage unit. FIG. 9 is a flow diagram illustrating an example of processing by an image superimposition system. FIG. 10 is a diagram illustrating an example of a query image. FIG. 11 is a diagram illustrating an example of an image in which a query image and a background are superimposed.

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Duplicate descriptions of components with the same reference numerals will be omitted.

[0024] 1 is a diagram showing an example of elements related to an image processing system according to an embodiment of the present invention. The image processing system includes a background generation system 1 and an image superimposition system 2. The background generation system 1 and the image superimposition system 2 are realized by one or more computers. Note that the background generation system 1 and the image superimposition system 2 do not have to be separate systems.

[0025] The background generation system 1 uses a generation AI to generate and store background candidate images according to the type and posture of an object in advance. The image superimposition system 2 then selects one of the background candidate images for a query image of an object that requires a background to be added, and generates an image with a background by superimposing the image of the object on the selected background candidate image.

[0026] Each of the background generation system 1 and the image superimposition system 2 includes one or more computers (e.g., server computers). The background generation system 1 includes one or more processors 11, one or more storages 12, and one or more communication units 13. The image superimposition system 2 includes one or more processors 21, one or more storages 22, and one or more communication units 23. The background generation system 1 may include multiple computers each including one or more processors 11, storages 12, and communication units 13, or may include a single computer having one or more processors 11 and storages 12. The image superimposition system 2 may include multiple computers each including one or more processors 21, storages 22, and communication units 23, or may include a single computer having one or more processors 21 and storages 22. The background generation system 1 and the image superimposition system 2 may be implemented on one or more virtual servers or container platforms.

[0027] The processors 11 and 21 operate according to programs (also referred to as instruction codes) stored in the storages 12 and 22. The processors 11 and 21 also control the communication units 13 and 23. The processors 11 and 21 include, for example, a CPU (Central Processing Unit) and may further include a GPU (Graphic Processing Unit) and an NPU (Neural Processing Unit). The programs may be provided via the Internet or the like, or may be provided by being stored in a computer-readable storage medium such as a flash memory or a DVD-ROM.

[0028] The storages 12 and 22 are configured with memory elements such as RAM and flash memory, and external storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs). The storages 12 and 22 store the above programs. The storages 12 and 22 also store information and calculation results input from the processors 11 and 12 and the communication units 13 and 23.

[0029] The communication units 13 and 23 are communication interfaces, such as network interface cards, that communicate with other devices. The communication units 13 and 23 are configured with integrated circuits, antennas, communication terminals, etc. that implement wireless LANs or wired LANs. Based on the control of the processors 11 and 21, the communication units 13 and 23 input information received from other devices via a network to the processors 11 and 21 or the storages 12 and 22, and transmit information to the other devices.

[0030] The hardware configurations of the background generation system 1 and the image superimposition system 2 are not limited to the above examples. For example, the background generation system 1 and the image superimposition system 2 may include a device for reading a computer-readable information storage medium (e.g., an optical disk drive or a memory card slot) or a device for inputting and outputting data to and from an external device (e.g., a USB port). The external device may be an input device or an output device.

[0031] Next, the functions provided by the image processing system will be described. FIG. 2 is a block diagram showing the functions realized by the image processing system. The image processing system functionally includes a background-added image acquisition unit 51, a background candidate generation unit 52, an arrangement acquisition unit 53, a storage processing unit 54, a query acquisition unit 56, a background selection unit 57, a superimposition unit 58, an output unit 59, a background generation model 61, a background interpolation model 62, an original storage unit 71, and a candidate storage unit 72. The background-added image acquisition unit 51, the background candidate generation unit 52, the arrangement acquisition unit 53, and the storage processing unit 54 are realized by the background generation model 61, the background interpolation model 62, and mainly by the processor 11 executing programs corresponding to each function stored in the storage 12 and controlling the communication unit 13, etc. The query acquisition unit 56, the background selection unit 57, the superimposition unit 58, and the output unit 59 are mainly realized by the processor 21 executing programs corresponding to each function stored in the storage 22 and controlling the communication unit 23, etc. The original storage unit 71 is mainly realized by the storage 12. The candidate storage unit 72 may be realized mainly by the storage 12 or the storage 22 .

[0032] The original image storage unit 71 stores original images of objects used to generate background candidate images, in association with the object's type (category) and orientation. The orientation indicates the orientation of the object in the original image.

[0033] The candidate storage unit 72 stores a plurality of background candidate images in association with the type and orientation of an object. The object type is the type of object in the original image used to generate the background candidate image, and the orientation is the orientation of the object in the original image used to generate the background candidate image. The candidate storage unit 72 may further store a plurality of background candidate images in association with the position and size of the object image. A method for determining the position and size of the object will be described later.

[0034] The background-added image acquisition unit 51 acquires a plurality of background-added images, each of which represents an object together with the background of the object. The background-added image acquisition unit 51 also acquires information indicating the type of object represented in each of the plurality of background-added images and information indicating the posture of the object represented in each of the plurality of background-added images.

[0035] The background generation model 61 is a machine learning model that generates and outputs an image that represents an object and a background from an input image of the object. The background-added image acquisition unit 51 instructs the background generation model 61 to generate a background-added image that includes an image of the object and a background based on the original image of the object for each of multiple objects, and acquires the background-added image output by the background generation model 61. The background generation model 61 may be a model that constitutes an image generation tool such as Stable Diffusion, which includes an extended function of Canny Edge in ControlNet. The background generation model 61 may be implemented on one or more computers different from the background generation system 1.

[0036] The background candidate generating unit 52 acquires, from each of the plurality of background-added images, a background candidate image that does not include an image of an object that is represented in that background-added image.

[0037] The background interpolation model 62 generates an image in which the image of the object is removed based on the background-added image and the background area hidden by the image of the object is interpolated in the background-added image. The background interpolation model 62 may receive an image in which the image of the object has been removed from the background-added image, or may receive the background-added image and information indicating the area of ​​the image of the object. The background candidate generation unit 52 instructs the generation of an image in which the background area has been interpolated based on the background-added image, and acquires the image output by the background interpolation model 62 as a background candidate image. The background interpolation model 62 may be a machine learning model that realizes so-called inpainting using, for example, the Lama method. The background interpolation model 62 may be implemented on one or more computers different from the background generation system 1.

[0038] The position acquisition unit 53 acquires the size of the image of the object depicted in each of the plurality of background-added images and the position of the image of the object in the background-added image.

[0039] The storage processing unit 54 stores at least some of the background candidate images generated from each of the plurality of background-added images in the candidate storage unit 72 in association with information indicating the type of object represented in the background-added image from which the background candidate image was generated. The storage processing unit 54 stores at least some of the background candidate images in the storage unit in association with the information indicating the type and information indicating the posture of the object represented in the background-added image from which the background candidate image was generated. The storage processing unit 54 may store at least some of the background candidate images generated from each of the plurality of background-added images in further association with at least one of the size of the image of the object included in the background-added image and the position of the image of the object represented in the background-added image.

[0040] The query acquisition unit 56 acquires a query image, which is an image of the target object to be superimposed, information indicating the type of the target object, and information indicating the posture of the target object.

[0041] The background selection unit 57 selects one of the background candidate images stored in the candidate storage unit 72 based on information indicating the type of the target object to be superimposed and information indicating the posture of the target object.

[0042] The superimposing unit 58 generates an image in which the query image is superimposed on the selected background candidate image. The superimposing unit 58 may generate an image in which an image of the target object enlarged or reduced based on the size associated with the selected background candidate image and stored in the candidate storage unit 72 is superimposed on the selected background candidate image. The superimposing unit 58 may also generate an image in which the image of the target object is superimposed on the selected background candidate image based on the position associated with the selected background candidate image and stored in the candidate storage unit 72.

[0043] The output unit 59 outputs information about an image in which the image of the target object is superimposed on the selected background candidate image.

[0044] Next, a description will be given of the processing of the background generation system 1. Fig. 3 is a flow diagram showing an example of the processing of the background generation system 1. Fig. 3 shows a processing flow relating to the processing of generating background candidate images and storing them in the candidate storage unit 72.

[0045] Before the process shown in Fig. 3 is started, it is assumed that original images of objects are stored in the original storage unit 71 in association with the object's type (category) and posture. Fig. 4 is a diagram showing an example of information stored in the original storage unit 71. In Fig. 4, the original storage unit 71 stores an image ID in association with the type and posture instead of the image itself.

[0046] In FIG. 4, for example, an original image with an image ID of "ORG1" is associated with the type "rice cooker" and the posture "front." An original image with an image ID of "ORG2" is associated with the type "rice cooker" and the posture "side." FIG. 5 is a diagram showing an example of an original image of an object. FIG. 5 shows an example of an original image with an image ID of "ORG1." In the original image of the object shown in FIG. 5, the background of the object is plain. As shown in FIG. 4, the posture does not need to be a strict angle, but may be information indicating which of a predetermined number of candidate values ​​it is.

[0047] The details of the process in Fig. 3 will be described. First, the background-added image acquisition unit 51 acquires one of one or more original images stored in the original storage unit 71 and not yet acquired (S101). The background-added image acquisition unit 51 also acquires the type and orientation of the object in the acquired original image from the original storage unit 71 (S102).

[0048] The background-added image acquisition unit 51 causes the background generation model 61 to generate a background-added image of the object based on the acquired original image (S103). More specifically, the background-added image acquisition unit 51 instructs the background generation model 61 to generate a background-added image including an image of the object and a background based on the original image of the object, and acquires the background-added image output by the background generation model 61.

[0049] For example, the background-added image acquisition unit 51 may acquire the contours of the acquired original image using the Canny Edge of ControlNet, and then input a command to Stable Diffusion (more specifically, realistic-vision based on Stable Diffusion) to generate a background-added image based on the contours, thereby inputting an image based on the original image (e.g., an image of the contours) into a background generation model 61 that constitutes Stable Diffusion and acquiring the background-added image. This command may further include at least a portion of the background text associated with the type, posture, and type of object that is input to the background generation model 61. Note that the position and size of the object in the image input to the background generation model 61 may be adjusted in advance, and the background generation model 61 may output an image in which the object is positioned at the same position and size as the input image.

[0050] 6 is a diagram showing an example of a generated background-added image. The background-added image, which expresses an image of an object along with a background corresponding to the object, is generated by the background-added image acquisition unit 51 and the background generation model 61. Note that when Canny Edge is used, the color of the object in the background-added image may differ from that of the original image, but this does not pose a problem because the object in the background-added image will be erased in subsequent processing.

[0051] Once the background-added image is generated, the layout acquisition unit 53 acquires the position and size of the object image in the background-added image (S104). The layout acquisition unit 53 may input the background-added image and the type of object into a trained machine learning model (e.g., Grounding DINO), which returns the position and size of the object image in response to the input of the image and the type of object, and acquire the output to acquire the position and size of the object image. Alternatively, the layout acquisition unit 53 may acquire the position and size of the object image by executing a so-called Region Proposal method on the background-added image. Alternatively, the layout acquisition unit 53 may use a so-called pattern matching method to search for an area in the background-added image that matches the original image, and acquire the position and size of the searched area.

[0052] In addition, when the background generation model 61 outputs an image in which an object is placed at the same position and size as the input image, the position and size of the object that have been previously adjusted in the input image may be obtained as the position and size of the image of the object in the background-added image.

[0053] The background candidate generation unit 52 then obtains a background candidate image from the background-added image, which does not include the object image but is an interpolated version of the background hidden by the object image (S105). More specifically, the background candidate generation unit 52 generates mask information for masking the area of ​​the object image to be deleted in the background-added image, based on the position and size of the obtained object image. The background candidate generation unit 52 inputs the background-added image and its mask information to the background interpolation model 62, and obtains the output image as the background candidate image.

[0054] The background interpolation model 62 may be, for example, a trained machine learning model called Lama-big. The image output from the background interpolation model 62 may be input to another trained machine learning model called a Refiner (for example, a refiner provided by realistic-vision) that improves image quality, and the background candidate generation unit 52 may acquire the output of the trained model as a background candidate image.

[0055] Fig. 7 is a diagram showing an example of a background candidate image generated from a background-added image. The image shown in Fig. 7 is an example of a background candidate image generated from the background-added image shown in Fig. 6. By using a known generation AI technology, it is possible to remove objects from the background-added image and generate a natural background candidate image.

[0056] The background interpolation model 62 may be another type of trained machine learning model that interpolates a background. The background interpolation model 62 may be configured to receive an image obtained by removing the image of the object from the background-added image instead of the mask information and the background-added image, and the background candidate generation unit 52 may input the removed image to the background interpolation model 62.

[0057] When the background candidate image is acquired, the storage processing unit 54 stores the background candidate image in the candidate storage unit 72 in association with the acquired type of object, the object's posture, and the position and size of the object in the background-added image (S106). The storage processing unit 54 may store the background candidate image in the candidate storage unit 72 in association with at least some of the acquired type of object, the object's posture, and the position and size of the object in the background-added image.

[0058] 8 is a diagram showing an example of information stored in the candidate storage unit 72. Instead of the background candidate images themselves, the candidate storage unit 72 stores the image IDs of the background candidate images in association with the type, posture, object position, and object size. An image with the image ID "BG1" is, for example, the image shown in FIG. 7. In the example of FIG. 8, the position of the object is indicated by x and y coordinates, and the size of the object is indicated by the number of dots in the vertical and horizontal directions.

[0059] Here, the storage processing unit 54 may not store some of the acquired background candidate images in the candidate storage unit 72. For example, the storage processing unit 54 may determine whether the quality of the background candidate images is sufficient using some method, and store the background candidate images that are determined to be of sufficient quality in the candidate storage unit 72 in association with their type, posture, position, size, etc. Alternatively, the storage processing unit 54 may store the acquired background candidate images in the candidate storage unit 72, and then delete any of the stored background candidate images that are determined to be of insufficient quality or that are determined to infringe intellectual property rights such as copyright.

[0060] After the process of S106 is executed, the background-added image acquisition unit 51 determines whether or not an unprocessed original image exists (S107). If an unprocessed original image exists (Y in S107), the process from S101 onward is repeatedly executed. If an unprocessed original image does not exist (Y in S107), the process of FIG. 3 ends.

[0061] 3, the background generation system 1 processes the multiple original images one by one in sequence, but the processes from S101 to S106 may be executed simultaneously in parallel for the multiple original images. Furthermore, the original images do not need to be stored in advance in the original image storage unit 71. In this case, the original image, type, and orientation may be input sequentially each time the process of S101 is executed.

[0062] The process shown in FIG. 3 does not have to be automatically executed by the background generation system 1. In this case, the background generation system 1 may execute each of the processes from S101 to S106, and the start of each process and the input of data required for the process may be manually controlled. Furthermore, after the process shown in FIG. 3 is completed, background candidate images stored in the candidate storage unit 72 that are determined to be of insufficient quality or have intellectual property rights issues may be deleted. Because background candidate images are generated in advance, such determinations and deletion of inappropriate background candidate images become possible. Furthermore, background candidate images that are free of quality and intellectual property rights issues may be stored in the candidate storage unit 72 by repeating the processes from S101 to S105 for the object, type, and posture associated with the deleted background candidate image until a background candidate image free of quality and intellectual property rights issues is generated.

[0063] Next, a description will be given of the processing of the image superimposition system 2. Fig. 9 is a flow diagram showing an example of the processing of the image superimposition system 2. Fig. 9 shows a processing flow relating to processing for generating an image of an object with a background using a background candidate image.

[0064] Before the process shown in Fig. 9 is started, the candidate storage unit 72 stores background candidate images associated with at least some of the object type, object posture, and object position and size in the background candidate image through the process shown in Fig. 3. Here, the candidate storage unit 72 accessed in the process of Fig. 9 may be copied from the storage 12 of the background generation system 1 to the storage 22 of the image superimposition system 2, or may be stored in the storage 12 or another external storage accessible from the image superimposition system 2.

[0065] First, the query acquisition unit 56 acquires an image of an object to which a background is to be added (query image) from the user (S201). The query acquisition unit 56 also acquires information indicating the type and orientation of the object in the query image (S202). The query acquisition unit 56 may acquire the query image, type, and orientation input by the user via, for example, a computer operated by the user. The query acquisition unit 56 may also acquire the type of object represented in the query image by applying a known Region Proposal technique to the query image. Alternatively, for example, the type and orientation of the object may be acquired from the query image using a Region Proposal machine learning model trained on a combination of object type and orientation as one class.

[0066] The background selection unit 57 selects one of the background candidate images stored in the candidate storage unit 72 based on the type and orientation of the object acquired for the query image (S203). In other words, the background selection unit 57 selects a background candidate image stored in association with the same type and orientation as those acquired for the query image from among the plurality of background candidate images stored in the candidate storage unit 72. The background selection unit 57 may select a background candidate image stored in association with the type and orientation most similar to those acquired for the query image.

[0067] The background selection unit 57 then acquires the position and size of the object in the background candidate image that is stored in the candidate storage unit 72 in association with the selected background candidate image (S204).

[0068] Fig. 10 is a diagram showing an example of a query image. In the example of Fig. 10, the type and orientation of the object are the same as those in Fig. 5. Therefore, the background candidate image selected from the query image of Fig. 10 is the one shown in Fig. 7. In addition, the position and size described in the row of image ID "BG1" in Fig. 8 are acquired.

[0069] After the query image is acquired, the superimposing unit 58 extracts an object region (image) from the query image (S205). The superimposing unit 58 may extract the object region using a known segmentation technique such as u2net or is-net.

[0070] The superimposing unit 58 resizes (enlarges or reduces) the region of the extracted object based on the size acquired for the background candidate image (S206). For example, the superimposing unit 58 may resize the region of the object so that the vertical size acquired for the background candidate image and the vertical size of the resized region are the same. The superimposing unit 58 may resize the region of the object so that the horizontal size acquired for the background candidate image and the horizontal size of the resized region are the same, or may resize so that the area indicated by the size acquired for the background candidate image and the area of ​​the resized region are the same.

[0071] The superimposing unit 58 superimposes the image of the resized object region on the selected background candidate image based on the acquired position of the background candidate image (S207). The superimposing unit 58 superimposes the image of the object region in front of the background candidate image. The superimposing may involve generating information about each pixel of the superimposed images, or may involve generating meta information that logically superimposes the layer of the image of the object region and the layer of the background candidate image.

[0072] To make the superimposed image more natural, the superimposing unit 58 may use a machine learning model to modify the image in which the image of the object region and the selected background candidate image are superimposed. For example, the method described in the paper "PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations" may be used to harmonize the background and object images and adjust shadows using a machine learning model. This paper is available at the URL https: / / openaccess.thecvf.com / content / CVPR2023 / papers / Guerreiro_PCT-Net_Full_Resolution_Image_Harmonization_Using_Pixel-Wise_Color_Transformations_CVPR_2023_paper.pdf. The superimposing unit 58 may also input the image in which the image of the object region and the selected background candidate image are superimposed or the image modified using the method described in this paper to another trained machine learning model called a refiner to obtain an image of improved quality.

[0073] Once the superposition is complete, the output unit 59 outputs the superimposed image (S208). The output image may be the image that has been corrected as described above. The output unit 59 may output the image by transmitting the image information to a computer operated by the user, or by storing the image information in a predetermined storage or database.

[0074] Fig. 11 is a diagram showing an example of an image in which a query image and a background are superimposed. In the example of Fig. 11, an image in which the image of the object shown in Fig. 10 and the background candidate image shown in Fig. 7 are superimposed is shown.

[0075] In this embodiment, by generating background candidate images according to the type of object in advance, the time from when a user inputs an image of the object (query image) to when an image with a background is generated can be significantly reduced. Also, by generating background candidate images according to the posture of the object in advance, the generated image with a background can be made more natural.

[0076] In this embodiment, a background candidate image is generated from an image of an object with a background. In an image of an object with a background, the relationship between the object and the background is likely to be natural. Here, the position and size of the object in the image of the object with a background are used to superimpose the object in the query image. This prevents the position and size of the object in the superimposed image from becoming unnatural. Furthermore, by applying a technique for removing the object from the image with a background and then interpolating the area hidden by the object, a background candidate image that is resistant to changes in the shape of the object can be generated.

[0077] The present invention is not limited to the above-described embodiment. For example, instead of generating a background-added image from an original image of an object, the background-added image acquisition unit 51 may acquire a background-added image in which the object is photographed together with the background. In this case, the background-added image may be stored in the original storage unit 71 instead of the original image, and the processes of S104 to S106 in FIG. 3 may be performed for each of the background-added images.

Claims

1. An image processing method comprising: a step of acquiring a plurality of target images each representing an object together with a background of the object; a step of generating a background candidate image from each of the plurality of target images that does not include an image of the object represented in the target image; and a step of storing in a storage unit at least a portion of the background candidate image generated from each of the plurality of target images in association with information indicating the type of object represented in the target image from which the background candidate image was generated.

2. An image processing method as claimed in claim 1, comprising: a step of acquiring information indicating the posture of an object represented in each of the plurality of images to be processed; and in the storing step, storing in a memory unit at least a portion of the background candidate images generated from each of the plurality of images to be processed in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and information indicating the posture of the object represented in the image to be processed.

3. An image processing method as defined in claim 1, further comprising a step of acquiring the size of an image of an object represented in each of the plurality of images to be processed, wherein in the storing step, at least a portion of the background candidate images generated from each of the plurality of images to be processed are stored in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and the size of the image of the object included in the image to be processed.

4. An image processing method as defined in claim 1, further comprising a step of acquiring the position of an image of an object included in each of the plurality of images to be processed, wherein in the storing step, at least a portion of the background candidate images generated from each of the plurality of images to be processed are stored in association with information indicating the type of object represented in the image to be processed from which the background candidate image was generated, and the position of the image of the object represented in the image to be processed.

5. An image processing method according to claim 1, wherein the step of acquiring the plurality of images to be processed comprises inputting the images of the plurality of objects into a target image generation model, which is a machine learning model that generates an image that represents an object together with a background from an image of the object, and acquiring the output of the target image generation model, thereby acquiring the plurality of images to be processed.

6. An image processing method as defined in claim 5, wherein, in the step of acquiring a plurality of processing target images, the target image generation model is instructed to generate the plurality of processing target images based on original images of the plurality of objects in given poses, and, in the storing step, at least a portion of the background candidate images generated from each of the plurality of processing target images is stored in a storage unit in association with the given pose of the original image of the object from which the processing target image was generated and information indicating the type of the object.

7. An image processing method as defined in claim 5, wherein in the step of acquiring a plurality of processing target images, for each of the plurality of objects, the target image generation model is instructed to generate a processing target image including an image of the object and the background based on an original image of the object, and in the storing step, at least a portion of the background candidate images generated from each of the plurality of processing target images is stored in a memory unit in association with at least one of the position and size of the image of the object represented in the processing target image and information indicating the type of the object.

8. An image processing method according to claim 1, wherein in the step of generating a background candidate image, an image of an object depicted in each of the plurality of processing target images is removed from that processing target image, and the background candidate image is obtained by interpolating the background area hidden by the image of the object using an image editing model, which is a machine learning model that edits input images.

9. An image processing system comprising: an image acquisition means for acquiring a plurality of target images each representing an object together with the background of the object; an image generation means for generating, from each of the plurality of target images, a background candidate image that does not include an image of the object included in the target image; and a storage processing means for storing in a storage unit at least a portion of the background candidate images generated from each of the plurality of target images in association with information indicating the type of object represented in the target image from which the background candidate image was generated.

10. A program for causing a computer to function as: an image acquisition means for acquiring a plurality of target images each representing an object together with the background of the object; an image generation means for generating, from each of the plurality of target images, a background candidate image that does not include an image of the object included in the target image; and a storage processing means for storing in a storage unit at least a portion of the background candidate images generated from each of the plurality of target images in association with information indicating the type of object represented in the target image from which the background candidate image was generated.

11. An image processing method comprising: a step of acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the attitude of the object to be superimposed; a step of selecting one of a plurality of background candidate images stored in a memory unit that stores the background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated and information indicating the attitude of the object represented in the image to be processed; a step of generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and a step of outputting information about the generated image.

12. An image processing method as defined in claim 11, wherein the storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated, information indicating the attitude of the object represented in the image to be processed, and information indicating the size of the object represented in the image to be processed, and the generating step generates an image in which an image of the object to be superimposed, enlarged or reduced based on the size stored in association with the selected background candidate image, is superimposed on the selected background candidate image.

13. An image processing method as defined in claim 11, wherein the storage unit stores the plurality of background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated, information indicating the attitude of the object represented in the image to be processed, and the position of the image of the object represented in the image to be processed, and the generating step generates an image in which the image of the object to be superimposed is superimposed on the selected background candidate image based on the position stored in association with the selected background candidate image.

14. An image processing method comprising: a target acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the attitude of the object to be superimposed; an image selection means for selecting one of a plurality of background candidate images stored in a memory unit that stores the background candidate images in association with information indicating the type of object represented in the image to be processed from which the background candidate images were generated and information indicating the attitude of the object represented in the image to be processed; a generation means for generating an image in which the image of the object to be superimposed is superimposed on the selected background candidate image; and an output means for outputting information on the generated image.

15. A program causing a computer to function as: target acquisition means for acquiring an image of an object to be superimposed, information indicating the type of the object to be superimposed, and information indicating the attitude of the object to be superimposed; image selection means for selecting one of a plurality of background candidate images stored in a memory unit that stores a plurality of background candidate images that are generated from a plurality of target images each representing an object together with the background of the object and that do not include an image of the object included in the target image, the background candidate images being stored in association with information indicating the type of object represented in the target image from which the background candidate images were generated and information indicating the attitude of the object represented in the target image; generation means for generating an image in which an image of the object to be superimposed is superimposed on the selected background candidate image; and output means for outputting information on the generated image.

16. A data structure including: a plurality of background candidate images generated from a plurality of target images each representing an object together with the background of the object, and each not including an image of the object included in the target image; and information associated with each of the plurality of background candidate images indicating the type of object represented in the target image from which the background candidate image was generated, and information indicating the posture of the object represented in the target image.

Images