Image generation system and image generation method

Abstract

To provide a mechanism for generating a composite image based on an image to which the influence of a light source is applied to an object and an image to which the influence of the light source is applied to a background.SOLUTION: An image generation system includes an acquisition unit that acquires a basic object image that is an image of an object, a light source setting unit that sets a virtual light source that illuminates the object, a background image generation unit that generates a final background image of a background including at least one of transmitted light from the object based on the virtual light source or a shadow of the object, an object image generation unit that generates a final object image of the object to which at least one of light or shade is applied based on the virtual light source, a composite image generation unit that generates a final composite image based on the final object image and the final background image, and a composite image output unit that outputs the final composite image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] [Related Applications] This application claims priority to Japanese Patent Application No. 2024-040621, entitled "Image Generation System and Image Generation Method," filed on March 15, 2024, the disclosure of which is incorporated herein by reference in its entirety. The present technology relates to an image generation system and an image generation method. [Background technology]

[0002] Japanese Patent Publication No. 2021-69004 (Patent Document 1) is a background technology in this technical field. This publication states that "an image processing device that generates a corrected image by compositing a captured image with a gloss component image created by a virtual light source includes a setting unit that sets a light source color of the virtual light source, a correction unit that corrects the light source color of the set virtual light source based on the color tone of the captured image, a gloss component generation unit that generates a gloss component image corresponding to the subject of the captured image in the light source color of the corrected virtual light source, and a gloss component applying unit that generates the corrected image by compositing the captured image with the generated gloss component image" (see Abstract). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-69004 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned Patent Document 1 describes a mechanism for correcting the three-dimensional effect of a subject in a captured image without creating an unnatural appearance even when the color tone of the captured image is changed. However, the above-mentioned Patent Document 1 does not consider a mechanism for generating a composite image based on an image in which the effect of a light source is applied to a target object and an image in which the effect of a light source is applied to a background. Therefore, the present technology provides a mechanism for generating a composite image based on an image in which the effect of a light source is applied to an object and an image in which the effect of a light source is applied to a background. [Means for solving the problem]

[0005] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is an image generation system characterized by having an acquisition unit that acquires a basic object image, which is an image of an object; a light source setting unit that sets a virtual light source that shines light on the object; a background image generation unit that generates a final background image of a background having at least one of transmitted light from the object based on the virtual light source or a shadow of the object; an object image generation unit that generates a final object image of the object by applying at least one of a light and a shadow effect to the object based on the virtual light source; a composite image generation unit that generates a final composite image based on the final object image and the final background image; and a composite image output unit that outputs the final composite image. [Effects of the Invention]

[0006] According to the present technology, it is possible to provide a mechanism for generating a composite image based on an image in which the effect of a light source is applied to an object and an image in which the effect of a light source is applied to a background. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an example of an explanatory diagram illustrating an image generation system 20 and the like connected to a network 100. [Figure 2] FIG. 2 shows an example of the hardware configuration of the image generation system 20. As shown in FIG. [Figure 3] FIG. 3 shows an example of the hardware configuration of the user terminal 30. [Figure 4] FIG. 4 is an example of a data structure 400 . [Figure 5] FIG. 5 is an example of an image generation flow 500. [Figure 6] FIG. 6 shows an example of a flow 600 for processing an input image and generating a basic object image. [Figure 7] FIG. 7 shows an example of a flow 700 for obtaining light source setting information. [Figure 8] FIG. 8 shows an example of a flow 800 for generating a final background image using light source information. [Figure 9] FIG. 9 is an example of a flow 900 for generating a final object image using light source setting information. [Figure 10] FIG. 10 is a first explanatory diagram illustrating an example of a light source setting. [Figure 11] FIG. 11 is a second explanatory diagram illustrating an example of a light source setting. [Figure 12] FIG. 12 is a third example of an explanatory diagram illustrating light source settings. [Figure 13] FIG. 13 is an example of an explanatory diagram for explaining the final composite image. [Figure 14] FIG. 14 shows another example of the hardware configuration of the image generation system 20. In FIG. [Figure 15] FIG. 15 is an example of another image generation flow 1500. [Figure 16] FIG. 16 shows an example of a flow 1600 for generating a constraint image. [Figure 17] FIG. 17 is an example of another flow 1700 for generating a constraint image. [Figure 18] FIG. 18 is an example of a generation flow 1800 for a generated image using the fifth generation AI. [Figure 19] FIG. 19 is an example of another generation flow 1900 for a generated image using the fifth generation AI. [Figure 20] FIG. 20 shows an example of a generation flow 2000 for generating a generated image when there are multiple objects. [Figure 21] FIG. 21 is an explanatory diagram illustrating an example of depth information of an object and a background. [Figure 22]FIG. 22 is an explanatory diagram illustrating an example of layer information of an object and a background. [Figure 23] FIG. 23 is an explanatory diagram illustrating an example of the constraint condition image generation process and the constraint condition image. [Figure 24] FIG. 24 is an example of an explanatory diagram illustrating the arrangement of virtual light sources. [Figure 25] FIG. 25 is another example of an explanatory diagram illustrating the arrangement of virtual light sources. [Figure 26] FIG. 26 is an example of an explanatory diagram for explaining selection of an object to which a lighting effect is applied. [Figure 27] FIG. 27 is an example of an explanatory diagram for explaining selection of a lighting effect to be applied to an object. [Figure 28] FIG. 28 is another example of an explanatory diagram for explaining selection of an object to which a lighting effect is applied. [Figure 29] FIG. 29 is an example of an explanatory diagram for explaining adjustment of a generated image. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, the embodiments will be described with reference to the drawings. Note that in each drawing, components having the same functions may be designated by reference numerals and redundant description may be omitted.

[0009] FIG. 1 is an example of an explanatory diagram illustrating an image generation system 20 and the like connected to a network 100. In the configuration of FIG. 1, an image generation system 20 is connected to a user terminal 30 via a network 100 .

[0010] The image generation system 20 in FIG. 1 is an example of a system that performs various processes of image information, which will be described later. The specific configuration of the image generation system 20 will be described in more detail with reference to FIG.

[0011] The image information is an image including an object such as a commodity, a product, a person, or a building. The image information may be an image taken using actual photographic equipment, an image of a two-dimensional or three-dimensional virtual object created using a computer, or an image generated using generation AI, etc.

[0012] The image generation system 20 can be realized, for example, as a standalone image generation terminal 20a that can be directly operated by a user or the like, or as an image generation server 20b that can be accessed by a user or the like via the cloud.

[0013] Furthermore, the image generation system 20 may be realized in other configurations. For example, the image generation system 20 may be realized in a state in which the individual components that make up the image generation system 20 are present in separate devices or separate locations.

[0014] Each terminal (image generation terminal 20a) and server (image generation server 20b) of the image generation system 20 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, a wristwatch, or clothing. They may also be stationary or portable computers, or servers located on the cloud or a network. In terms of functionality, they may be VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, or MR (Mixed Reality) terminals. Alternatively, they may be a combination of multiple of these terminals. For example, a combination of one smartphone and one wearable terminal can logically function as a single terminal. Other information processing terminals may also be used.

[0015] Each terminal (image generation terminal 20a) and server (image generation server 20b) of image generation system 20 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as RAM (Random Access Memory), an auxiliary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), flash memory, etc., a communication control unit such as a network card, wireless communication module, or mobile communication module, input devices such as a touch panel, keyboard, mouse, voice input, and input based on motion detection captured by a camera unit, and an output device such as a monitor or display. Note that the output device may also be a device or terminal that transmits information to be output to an external monitor, display, printer, device, etc.

[0016] The main memory stores various programs, applications, etc. (modules), and the processor executes these programs and applications to realize the various functional elements of the overall system. These modules may be implemented in hardware, such as by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

[0017] In this specification, each module is described as a subject that performs processing, but in reality, the processing is performed by a processor that processes various programs, applications, etc. (modules). Various databases (DBs) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. There are no limitations on how the database is implemented; for example, it can be a database management system, spreadsheet software, or a text file such as XML or JSON.

[0018] The user terminal 30 in FIG. 1 is a terminal that has functions such as photographing, creating, using, editing, viewing, processing, displaying, and outputting image information of objects such as merchandise, products, people, and buildings. The specific configuration of the user terminal 30 will be described in more detail with reference to FIG.

[0019] A user of the user terminal 30 can use the user terminal 30, for example, a smartphone 30a, a tablet 30b, a laptop PC 30c, a desktop PC 30d, a camera 30e, VR / AR goggles 30f, etc. to capture, create, use, edit, view, process, display, output, etc. image information.

[0020] The user of the user terminal 30 is, for example, a still cameraman, photographer, designer, retoucher, or other person who takes, creates, uses, and edits image information. In another example, the user of the user terminal 30 is a person such as a person in charge of an advertising agency that handles product advertisements and the like.

[0021] A user can use the image generation system 20 and multiple user terminals 30 to cross-sectionally use the same image information. For example, a user can transmit image information captured by a camera 30e to the image generation system 20 via the network 100.

[0022] The image generation system 20 can transmit generated image information generated from the image information transmitted from the camera 30e via the network 100 to the tablet 30b and laptop PC 30c.

[0023] Therefore, when taking a picture in a photography studio, for example, the user can not only check the actual photography information but also check the generated image information generated by the image generation system 20. This allows the user to take pictures while checking a wider variety of shooting patterns than before.

[0024] Furthermore, one or more of the elements constituting the image generation system 20, namely, one or more image generation terminals 20a, image generation server 20b, and one or more user terminals 30, may be installed in different countries. Furthermore, the image generation system 20 may be realized by one or more computers, any of which may be installed in different countries.

[0025] The components and modules of the image generation system 20 and the user terminal 30 described below with reference to FIGS. 2 and 3 can be added, omitted, integrated, or substituted depending on the embodiment.

[0026] The components and modules of the image generation system 20 described using Fig. 2 and the user terminal 30 described using Fig. 3 may be configured to exist in a distributed manner or stored on the network 100 in order to control and process the entire system. For example, each module of the image generation system 20 described later using Fig. 2 may be stored in a separate device or the like.

[0027] FIG. 2 shows an example of the hardware configuration of the image generation system 20. As shown in FIG. The image generation system 20 includes a main memory device 201, an auxiliary memory device 202, a processor 203, an input device 204, an output device 205, a communication control unit 206, and the like.

[0028] The main memory device 201 of the image generation system 20 stores programs and applications such as an acquisition module 211, an object extraction module 212, a 3D object generation module 213, a light source setting module 214, a background image generation module 215, an object image generation module 216, a composite image generation module 217, an evaluation module 218, and an output module 119. The processor 203 executes these programs and applications to realize each functional element of the image generation system 20.

[0029] The acquisition module 211 is an example of an “acquisition unit” and can acquire image information to be processed in the image generation system 20 . The acquisition module 211 can acquire, for example, input image information input by a user or predetermined object image information (basic object image) extracted by an object extraction module 212 described later as image information to be processed.

[0030] The object extraction module 212 is an example of an "object extraction unit" and can create and extract predetermined object image information, which will be described in detail below, from any image information containing an object.

[0031] The object extraction module 212 can also create and extract predetermined object image information by upscaling it from arbitrary image information. Specific processes for creating and extracting predetermined object image information will be described later with reference to FIG.

[0032] When multiple objects are included in any image information, the object extraction module 212 can extract each object separately or all of them together. The object extraction module 212 can also create and extract predetermined object image information from any video information that includes an object.

[0033] The predetermined object image information is, for example, an image of an object included in any image information. The predetermined object image information is, in particular, image information obtained by cutting out the outer contour of the object from any image information, or image information obtained by cutting out the background, which is the area other than the object, from any image information, etc.

[0034] In the predetermined object image information, the effect of the light source on the object contained in any image information, for example the effect of light and shadow, may be weakened. In particular, it is preferable that the image information of a specified object has removed reflected light and shade from the surface of the object contained in any image information, shadows extending from the object contained in any image information, and transmitted light from the object contained in any image information.

[0035] In this specification, the term "shade" primarily refers to an area within the contours of an object or on the surface of an object that is not illuminated by light from a light source, or that is darker and less affected by light from the light source than other areas.

[0036] In this specification, the term "shadow" primarily refers to the area outside the contours of an object or outside the object itself, which is not illuminated by light from a light source, or is darker and less affected by light from the light source than other areas.

[0037] These terms may be used interchangeably, or they may be unified into one term, in which case the unified term will have the same meaning as both terms.

[0038] For example, when a shadow occurring on the surface of an object and a shadow extending from the object to the ground or the like are continuous in image information, they may be collectively expressed as a "shadow," "shadow," or "shadow."

[0039] The object extraction module 212 can execute each process related to "extraction" in the following description. The object extraction module 212 may have, for example, a learning model for extraction in each process. In the following description, the above-mentioned predetermined object image information may also be referred to as a "basic object image."

[0040] The 3D object generation module 213 is an example of a "3D object generation unit", and generates a 3D object of an object (hereinafter also referred to as a "three-dimensional object") based on a basic object image using a first generation AI 213a. Specific processing for generating a 3D object of the target object will be described later with reference to FIG. In the following explanation, the term "generative model" may be used instead of "generative AI."

[0041] The light source setting module 214 is an example of a "light source setting unit" and sets a virtual light source that irradiates the object with light. Specific processing for setting the virtual light source will be described later with reference to FIGS. 5, 7, and 10 to 12.

[0042] The background image generation module 215 is an example of a "background image generation unit" and generates final background image information, which will be described later. Specific processing for generating the final background image information will be described later with reference to FIG.

[0043] The object image generation module 216 is an example of an "object image generation unit" and generates a final object image, which will be described later. Specific processing for generating the final object image information will be described later with reference to FIG.

[0044] The composite image generation module 217 is an example of a "composite image generation unit" and generates final composite image information, which will be described later. Specific processing for generating the final composite image information will be described later with reference to FIG.

[0045] The evaluation module 218 evaluates the final composite image information based on, for example, the type of object, the lighting settings, and so on. The evaluation module 218 performs evaluation on the degree of agreement with light source settings preferred by customers, etc., obtained from purchasing trend information, etc., described below, depending on the type of product, which is the target object, for example. The evaluation module 218 may also output preferred light source settings to the light source setting module 214, for example, for each type of object.

[0046] The output module 219 (hereinafter sometimes referred to as the "composite image output module 219") is an example of a "composite image output unit" and can output (display) light source settings, final object image information, final background image information, composite image information, etc. to an output device 205 such as a monitor. The output module 219 can further output (transmit) the final composite image information to the user terminal 30 or the like as the image generation result. The output module 219 can output (transmit) the final composite image information so that the final composite image information is displayed on an output device 305 such as a monitor of the user terminal 30, for example.

[0047] In another configuration, at least one of the modules may be stored in the user terminal 30 or in a further terminal connected to the image generation system 20 via the network 100.

[0048] For example, the processing performed by the object extraction module 212, such as the processing of creating basic object image information from arbitrary image information, and the processing performed by the background image generation module 215, such as the processing of generating arbitrary background image information, may be performed by the user terminal 30.

[0049] In yet another configuration, a module illustrated as one module or one sub-module in FIG. 2 may be implemented as a module executed in cooperation with another terminal other than the image generation system 20, such as a user terminal 30, or a further terminal connected to the image generation system 20 via the network 100, or as a module executed in cooperation with multiple other terminals.

[0050] Similarly, in the configuration of Figure 2, the various information stored in the auxiliary storage device 202 may be stored in the user terminal 30 or in yet another terminal connected to the image generation system 20 via the network 100.

[0051] The auxiliary storage device 202 of the image generation system 20 stores setting information 221, user management information 222, image information 223, virtual light source information 224, purchasing tendency information 225, content information 226, and the like.

[0052] Each piece of information stored in the secondary storage device 202 may be stored at least partially, or at least temporarily, in the primary storage device 201, for example, as part of each module that uses it.

[0053] The setting information 221 may include at least a part of the setting information of the user terminal 30 in addition to the setting information for each module of the image generation system 10 stored in the main storage device 201 .

[0054] The user management information 222 is information about users who use the image generation system 20 using, for example, the user terminal 30, such as authentication information and registration information of each individual user. The image generation system 20 can use user management information 222 to personalize the individual functions it provides for each user or object, for example by switching the virtual light source information 224 of available virtual light sources.

[0055] The image information 223 is, for example, information such as input image information input by the user, final background image information, final object image information, and final composite image information for each project or document.

[0056] Light source information 224 is information such as the type and specifications of virtual light sources that can be set, the type and placement of light sources set by the user for each project or document, and the type and placement of light sources that are preset according to the type of object or customer preferences.

[0057] The content information 226 may be, for example, information relating to an object such as a product, information relating to a background image, information relating to text, information relating to an arrangement, information relating to a representation of an entire scene, including image information, etc. Specifically, the content information 226 may be any derived information used for reference, such as background images, object images, or explanatory text, illustrations, images, and the like. The derived information may be monochrome, depth, texture, normal maps, textual variations, and encoded versions thereof. The content information 226 may be information about content such as a preset background image for compositing or a background image for compositing that can be purchased by the user.

[0058] FIG. 3 shows an example of the hardware configuration of the user terminal 30. The user terminal 30 includes a main memory device 301, an auxiliary memory device 302, a processor 303, an input device 304, an output device 305, a communication control unit 306, and the like.

[0059] The main memory device 301 of the user terminal 30 stores programs and applications such as an image processing module 311, and the processor 303 executes these programs and applications.

[0060] The image processing module 311 can perform processes such as capturing, creating, editing, processing, displaying, and outputting image information. This allows the user to use and view the image information, for example, display, create, modify (edit), share, and the like.

[0061] The user terminal 30 can display image information captured or created by the image processing module 311 on an output device 305, such as a display, accept changes or modifications to the image information data using an input device 304, such as a keyboard, mouse, LCD pen tablet, flat-type pen tablet, or other input support device, and output, for example, transmit, the image information to an image generation system 20, etc. via a communication control unit 306 or the network 100.

[0062] The user terminal 30 can use any program, software, application, etc. that processes data files in, for example, JPG (JPEG) format, RAW format, PSD format, etc. as image information.

[0063] The auxiliary storage device 302 of the user terminal 30 stores setting information 321, user information 322, image information 323, and the like. The setting information 321 includes setting information for the image processing module 311 of the user terminal 30 stored in the main storage device 301 .

[0064] The image processing module 311 can store captured images, created images, etc. as image information 323. The image processing module 311 can transmit the image information 323 to the image generation system 20 via the communication control unit 306 and the network 100 .

[0065] FIG. 4 is an example of a data structure 400 . The data structure 400 is an example of the structure of data that the image generation system 10 processes. The image generation system 10 has, for example, for each document, a document ID, a user ID, a creation date, an update date, an original image, a generated object image, a generated background image, light source information, background instructions, object information, background information, etc. The specific sample values ​​and outlines of each item shown in FIG. 4 are examples of information for each item, and are not limited to these.

[0066] The image generation process executed by the image generation system 20 will be described below with reference to FIGS. The steps of each flow and subflow described using Figures 5 to 9 may be executed in reverse order, executed in parallel, incorporated into another flow, or used together in different flows or subflows, as long as no technical inconsistencies arise.

[0067] FIG. 5 is an example of an image generation flow 500. The image generation process by the image generation system 20 will be described below using an image generation flow 500.

[0068] In the following explanation, an example of image generation will be described in which input image information acquired from the image generation system 20 and the user terminal 30 is used, but the source and storage location of the input image information are not limited to the user terminal 30.

[0069] In the image generation system 20, The acquisition module 211 acquires a basic object image, which is an image of an object. The light source setting module 214 sets a virtual light source that illuminates the object. A background image generation module 215 generates a final background image of the background having at least one of transmitted light from the object and a shadow from the object based on a virtual light source. An object image generation module 216 generates a final object image of the object by applying light and / or shadow effects to the object based on the virtual light source. A composite image generation module 217 generates a final composite image based on the final object image and the final background image. A composite image output module 219 outputs the final composite image. Each will be explained in detail below.

[0070] When the image generation process is started in the image generation system 20, the acquisition module 211 of the image generation system 20 acquires input image information to be processed from the user terminal 30 (S510).

[0071] The object extraction module 212 and the 3D object generation module 213 of the image generation system 20 process the input image information sent from the acquisition module 211 and generate a basic object image (S520). The processing of the input image information and the processing of generating the basic object image will be described in detail with reference to FIG.

[0072] The light source setting module 214 of the image generation system 20 acquires light source setting information, which is setting information for a virtual light source that irradiates light onto the object included in the basic object image (S530). The virtual light source may be set based on a setting input operation by the user, or may be set based on setting information stored in advance as virtual light source information 224 of the image generation system 20. The process of acquiring the light source setting information will be described in detail with reference to FIG.

[0073] The background image generation module 215 of the image generation system 20 applies the light source setting information, in particular, applies at least one of light and shadow based on the virtual light source, and generates a final background image (S540).

[0074] For example, the final background image is an image that has at least one of transmitted light from an object based on a virtual light source and a shadow from the object. The process of generating the final background image will be described in detail with reference to FIG.

[0075] The object image generation module 216 of the image generation system 20 generates a final object image to which the light source setting information has been applied, in particular to which light and / or shadow effects based on the virtual light source have been applied (S550).

[0076] For example, the final object image is an image in which at least one of the effects of light and shade is applied to the object based on the virtual light source, i.e., the effect of making some parts of the object brighter or darker than other parts of the object.

[0077] The acquisition module 211 acquires the final background image generated by the background image generation module 215 and the final object image generated by the object image generation module 216, and sends them to the composite image generation module 217 of the image generation system 20. The composite image generation module 217 creates a final composite image from the final background image and the final object image, and the output module 219 of the image generation system 20 outputs the final composite image (S560).

[0078] FIG. 6 shows an example of a flow 600 for processing an input image and generating a basic object image. The flow 600 of processing an input image and generating a basic object image corresponds to step S520 described with reference to FIG. The object extraction module 212 executes a process of extracting an object or a process of removing the background from any input image information acquired by the acquisition module 211 (S610).

[0079] The object extraction module 212 generates an object image that adjusts for the effect of at least one of light and shadow on the object separated from the input image information in step S610 (S610).

[0080] The object extraction module 212 can adjust the above effects, for example, to reduce the effect of shadows on the object surfaces separated from the input image information, or in some cases to eliminate the shadows on the object surfaces.

[0081] The object extraction module 212 can adjust the above effects, for example, to reduce the effect of reflected light on the object surface. The object extraction module 212 can adjust the above effect, for example, so that the brightness and darkness of each range of the object separated from the input image information becomes approximately uniform. The object extraction module 212 can adjust the above effects, for example, to remove color information for each range of the object separated from the input image information.

[0082] The object extraction module 212 adjusts the contours of the object (S630) for the object image of the object separated from the input image information in step S610 or the object image in which the effects of at least one of light and shadow in each area of ​​the object have been adjusted in step S620.

[0083] The object extraction module 212 adjusts the contour of the object, for example, so that the contour becomes a continuous straight line or curve, or so that a zigzag contour becomes a straight line or curve.

[0084] The object extraction module 212 generates a basic object image from the object image that has been subjected to at least the process of step S610, preferably the processes of steps S610 to S630 (S640).

[0085] Similarly, in the image generation system 20, the object extraction module 212 can extract an object from the input image or remove a background from the input image, or reduce the effect of at least one of light or shadow on the object in the input image, and extract a basic object image from the input image, as described above.

[0086] The 3D object generation module 213 generates a 3D object of the object from the basic object image (S650). The 3D object generation module 213, for example, inputs a basic object image to a first generation AI 213a and generates a three-dimensional object of the object. The generated three-dimensional object is not limited to information in OBJ format, PLY format, or the like. The generated three-dimensional object may be, for example, a full 3D object or other forms of information that allow for a 3D approximation, such as depth, texture, normal maps, etc.

[0087] The object extraction module 212 outputs the basic object image, and the 3D object generation module 213 outputs the 3D object of the object to, for example, the acquisition module 211 and the output module 219, respectively (S660). It should be noted that the object extraction module 212 may output the basic object image, for example, after generating the basic object image in step S640 and before step S650.

[0088] The acquisition module 211 can send the acquired basic object images and 3D objects to other modules such as the background image generation module 215 and the object image generation module 216, or store them in the auxiliary storage device 202. The output module 219 can perform processing to output (display) the acquired basic object image and 3D object to the output device 205 such as a monitor.

[0089] FIG. 7 shows an example of a flow 700 for obtaining light source setting information. In the image generating system 20 described below, the acquisition module 211 can acquire light source setting information according to a flow 700 for acquiring light source setting information. In the image generation system 20, the 3D object generation module 213 inputs the basic object image into a first generation AI to generate a three-dimensional object of the object. The light source setting module 214 performs the following on the three-dimensional object generated using the first generation AI: The three-dimensional position of the virtual light source, Types of virtual light sources, The three-dimensional direction of the light from the virtual light source, The color of the light from the virtual light source, The number of virtual light sources, Set at least one adjustable virtual light source.

[0090] The flow 700 for obtaining light source setting information corresponds to step S530 described with reference to FIG. The light source setting information described below can be, for example, setting information input by the user, setting information preset as virtual light source information 224 in the auxiliary storage device 202, or the like.

[0091] The light source setting module 214 can also acquire virtual light source information 224 according to the object by, for example, inputting the type of object into a table or a learning model. The virtual light source information 224 according to the object may be, for example, light source information corresponding to a light source setting that is frequently applied to the object or a light source setting of an image that is associated with better sales performance on an e-commerce site using an image of the object.

[0092] The light source setting module 214 acquires position information, e.g., coordinate information, of a virtual light source that shines light (hereinafter sometimes referred to as "virtual light") onto an object in a basic object image or a 3D object of an object created from the basic object image (hereinafter both are collectively referred to as "3D object") (S710).

[0093] The light source setting module 214 acquires the type information of the virtual light source (S720). Information on the type of virtual light source can be, for example, information about the light source (light emitter) such as natural light or lighting equipment, the type of light source's emission time such as instantaneous light (strobe) or constant light, information about the presence or absence of equipment involved in the diffusion and reflection of light such as diffusers, umbrellas, and reflectors, or information about combinations of these.

[0094] The light source setting module 214 acquires orientation information of the virtual light source, particularly, direction information of the virtual light emitted from the virtual light source toward the 3D object (S730). The direction information of the virtual light source can be, for example, vector information of a vector pointing from the center of the virtual light source to the center of the 3D object, or coordinate information of each of the centers. The direction information of the virtual light can be, for example, vector information or coordinate information that defines the direction of the rotation axis (central axis) of the virtual light that is emitted in a cone shape with the virtual light source as the apex.

[0095] The light source setting module 214 acquires the intensity information of the virtual light source (S740). The intensity information of the virtual light source is, for example, information about the intensity of the light emitted from the virtual light source.

[0096] The light source setting module 214 acquires color information of the virtual light source (S750). The color information of the virtual light source is, for example, information about the saturation, color temperature, hue, brightness, etc. of the virtual light emitted from the virtual light source.

[0097] The light source setting module 214 outputs the light source setting information acquired in at least one of steps S710 to S750 to, for example, the acquisition module 211 or the output module 219 (S760).

[0098] The light source setting module 214 may output the acquired light source setting information, for example, during or after each step from step S710 to step S750. In such a configuration, for example, the output module can output (display) information that reflects the acquired light source setting information each time, as exemplified in FIGS.

[0099] FIG. 8 shows an example of a flow 800 for generating a final background image using light source information. The flow 800 for generating a final background image using light source information corresponds to step S540 described with reference to FIG.

[0100] The background image generation module 215 obtains at least one of the basic object image or the 3D object generated in the processing of the input image and generation of the basic object image step S520 (S810).

[0101] In addition, instead of step S810, the background image generation module 215 may perform processing similar to steps S610 to S660 of the input image processing and basic object image generation flow 600 described using Figure 6, and generate at least one of a basic object image or a 3D object.

[0102] The background image generation module 215 performs a predetermined map process (mapping process) on the 3D object (S820). The predetermined map processing is a process for adding details to the surface information of the 3D object, for example, by adding information about irregularities.

[0103] Examples of predetermined map processing that can be used include a bump map, a normal map, a texture map, a gloss map, and a displacement map.

[0104] The background image generation module 215 acquires the light source setting information acquired by the light source setting module 214 in the light source setting information acquisition step S530, for example, directly from the light source setting module 214 or indirectly from the acquisition module 211 (S830).

[0105] The background image generation module 215 applies virtual light from a virtual light source based on the light source setting information after map processing to the 3D object, and generates an image to which light and shadow are applied (S840). An image with light and shadow applied may be, for example, an image in which light and shadow are applied to the 3D object of the subject and light and shadow are applied to the background, or an image in which light and shadow are applied only to the background.

[0106] The background image generation module 215 may output the image with light and shadow applied generated in step S840, for example, an image with light and shadow applied only to the background, as the final background image in step S870 described below.

[0107] An image to which light and shadow have been applied can be, for example, an image that reflects the reflective effect of reflected light on a three-dimensional object, the reflective effect of transmitted light on a three-dimensional object, or the shading effect of shadows within the contours of a three-dimensional object.

[0108] In images that reflect these effects, the object has areas that are brighter than other areas based on reflected or transmitted light, and areas that are darker than other areas due to shadows caused by the direction of the virtual light from the virtual light source and the shape of the object.

[0109] The light and shadow applied image can be an image that includes light incident on an object, light transmitted from an object, and shadows outside the object's outline. The background image generation module 215 can selectively determine the light and shadow to be applied to a plurality of virtual light sources.

[0110] The background image generation module 215 inputs the image generated in step S840, or the image generated in step S840 and background generation conditions to the fourth image and condition generation AI 215a or the like (S850).

[0111] The background generation conditions are the conditions for the final background image generated in step S860. The background generation conditions include, for example, prompts, instructions, styles, templates, etc. related to the background. The fourth generation AI 215a is an AI model that has, for example, a function of generating an image from text information, or a function of generating an image corresponding to an input condition in a specific area of ​​an image.

[0112] The background image generation module 215 uses the fourth generation AI 215a etc. to generate a final background image from the image generated in step S840 or from the image generated in step S840 and background generation conditions (S860).

[0113] The final background image may be, for example, an image including only the background or an image including the background and the object, and including lighting and shadows corresponding to the image generated in step S840, at least for the background.

[0114] The background image generation module 215 can generate a final background image that includes an image area of ​​a light source, such as the sun or a lighting device, at the position of at least one virtual light source located within the field of view.

[0115] The final background image may be, for example, an image including only the background or an image including the background and the object, and including lighting and shadows corresponding to the image generated in step S840, at least for the background.

[0116] As mentioned above, the final background image does not necessarily have to contain the subject. The final background image may be a single color image, such as white, black, green, or blue. The final background image may be, for example, an image in which the area of ​​the object has been masked or an image in which the area of ​​the object has been cut out.

[0117] Using the above method, transmitted light and shadows from objects contained in the input image can be removed from the final background image, and incident light on the object, transmitted light from the object, and shadows from the object based on the light source setting information can be included.

[0118] The background image generation module 215 outputs the final background image to, for example, the acquisition module 211 or the output module 219 (S870). The output module 219 can output (display) the generated final background image, for example, as illustrated in FIG.

[0119] As described above, in the image generation system 20, the background image generation module 215 generates a final background image having at least one of incident light on the object, transmitted light from the object, and shadows outside the object's contour based on a virtual light source.

[0120] Similarly, in the image generation system 20, the background image generation module 215 generates a final background image in which light emitters are placed in a range that includes the position where the virtual light source is set, in the same manner as described above.

[0121] FIG. 9 is an example of a flow 900 for generating a final object image using light source setting information. The flow 900 for generating a final object image using light source information corresponds to step S550 described with reference to FIG.

[0122] Note that the flow 900 for generating a final object image using light source setting information includes steps corresponding to at least some of the steps included in the flow 600 for processing an input image and generating a basic object image, which are described using FIG. 6.

[0123] Corresponding steps can share processing results with each other, and corresponding steps do not need to be performed separately. Furthermore, when one of the corresponding processes is executed, the other corresponding process does not have to be executed.

[0124] In the flow 900 for generating a final object image using light source setting information, the 3D object generation module 213 acquires the basic object image generated in step S640 of the flow 600 for processing an input image and generating a basic object image described using Figure 6 (S910).

[0125] The 3D object generation module 213 generates a 3D object of the object from the basic object image (S920), similar to step S650 of the flow 600 for processing the input image and generating the basic object image described using Figure 6, and outputs it to, for example, the acquisition module 211 or the output module 219. As described above, if a 3D object of the object has already been generated in the flow 600 for processing an input image and generating a basic object image, step S920 does not need to be executed.

[0126] The target image generation module 216 applies the light source setting information acquired in the light source setting information acquisition flow 700 described with reference to FIG. 7 to the target 3D object (S930).

[0127] The object image generation module 216 generates a first intermediate object image to which light and shadow have been applied (S940) based on processing similar or equivalent to steps S820 to S840 of the final background image generation flow 800 to which light source information has been applied as described using Figure 8.

[0128] The first intermediate object image may be, for example, a projected image in which light and shadow are applied to the 3D object of the object and light and shadow are applied to the background, or a projected image in which light and shadow are applied only to the 3D object of the object. The object image generation module 216 may generate a plurality of first intermediate object images as projection images from a plurality of angles.

[0129] As described above, in the image generation system 20, the object image generation module 216 generates a virtual light source-based Reflection light effect on 3D objects, Transparency effect of light transmission on 3D objects, Shadow effect within the contours of three-dimensional objects, a first intermediate object image including at least one of: The reflected light effect is an effect related to the brightness of the reflected light of the virtual light from the virtual light source on the surface of the object. The transmitted light effect is the effect related to the brightness of transmitted light that appears on the surface of an object through the object from a light source inside or outside the object. The shadow effect is an effect related to the darkness of the shadow areas that appear on the surface of an object where virtual light from a virtual light source does not strike.

[0130] Similarly, in the image generation system 20, the object image generation module 216 generates, based on the setting of a virtual light source for the three-dimensional object, a projection image of the three-dimensional object as a first intermediate object image, which has at least one of transmitted light from the three-dimensional object or a shadow of the three-dimensional object and applies at least one of the effects of light or shadow to the three-dimensional object. Transmitted light from a three-dimensional object is light that passes through the object and hits the ground, etc. A shadow of a three-dimensional object is a dark area that appears on the ground, etc.

[0131] The object image generation module 216 inputs the base object image and the first intermediate object image into the second generative model 216a (S950). The second generation AI 216a is an AI model that has, for example, a function of generating a new image by making a specific area of ​​the image correspond to a condition.

[0132] The object image generation module 216 generates a second intermediate object image from the base object image and the first intermediate object image using the second generative model 216a (S960). In the second intermediate object image, the area of ​​the image of the 3D object of the object included in the first intermediate object image is replaced with the image of the object included in the base object image.

[0133] In the second intermediate object image, in areas where light and shadow effects are applied to the 3D object contained in the first intermediate object image, the same effects are reflected in the image of the replaced object.

[0134] Similarly, in the image generation system 20, the object image generation module 216 inputs the basic object image and the projection image into a second generation AI to generate a second intermediate object image having at least one of transmitted light from the object based on a virtual light source or a shadow of the object, and applying at least one of the effects of light or shadow to the object based on the virtual light source.

[0135] The object extraction module 212 or the object image generation module 216 processes the second intermediate object image by performing processes such as upscaling, background removal, segmentation, clustering, and classification of objects, etc., and generates a third intermediate object image (S970).

[0136] The object image generation module 216 inputs the input image and the third intermediate object image to the third generative model 216b (S980). The third generation AI 215b is an AI model that has, for example, a function to apply a style, including, for example, shadow, lighting, shape, color, etc., from one image to another image, a function to generate a new image by matching a specific area of ​​an image to a condition, or a function to replace a specific area of ​​one image with a corresponding area of ​​another image. The third generative model 216b, for example, takes an image (the main input image), generates aspects, such as characteristics or features, from another image, and regenerates the image (the main input image) including those aspects as effects. Effects include effects related to shadow patterns, shapes, colors, styles, etc. The third generative model 216b can apply or reproduce these effects on another input image.

[0137] The object image generation module 216 uses the third generation model 216b to generate a final object image by applying light and shadow to the object from the input image and the processed second intermediate object image, and outputs the final object image to, for example, the acquisition module 211 or the output module 219 (S990). The object image generation module 216 can output (display) the generated final object image, for example, as illustrated in FIGS.

[0138] As described above, in the image generation system 20, the object extraction module 212 or the object image generation module 216 generates a third intermediate object image based on the process of extracting the object from the second intermediate object image or removing the background from the second intermediate object image, and the object image generation module 216 inputs the input image and the third intermediate object image into a third generation AI and can generate, as a final object image, an image in which at least one effect of light or shadow is applied to the object based on a virtual light source, and in which the color of the object is different from that of the third intermediate object image.

[0139] In the final object image, color information corresponding to the color information of the object included in the input image is added to the object included in the second intermediate object image. The added color information, etc. includes not only the color information of the object but also information about the texture, shape, etc. This information may be defined as style information, etc. In the final object image, in areas where the effects of light and shadow are applied to the object included in the second intermediate object image, the image of the object to which color information, style information, etc. corresponding to the input image have been added also reflects effects that are the same as, similar to, or derived from those effects. As a result, in the final object image, color effects are also applied to the effects of light and shadow.

[0140] Therefore, the final object image can reflect light and shadow effects that are different from those in the input image and are based on light source setting information, while maintaining the color and shape of the object contained in the input image.

[0141] FIG. 10 is a first explanatory diagram illustrating an example of a light source setting. FIG. 10A is an example of a setting screen 1000 for setting conditions for generating a final composite image. FIG. 10(B) is another example of the first setting area 1010 in FIG. 10(A). FIG. 10C is yet another example of the first setting area 1010 in FIG. 10A.

[0142] The output module 219 can display the setting screen 1000 on the output device 205 of the image generation system 20. In addition, the output module 219 can display the setting screen 1000 on the output device 305 of the user terminal 30 connected to the image generation system 20 via the network 100.

[0143] In the example of FIG. 10(A), a setting screen 1000 has a first setting area 1010, a second setting area 1020, and a third setting area 1030. The output module 219 can display an image of the object 1040 or the like in the first setting area 1010 .

[0144] The output module 219 can display, as an image of the object 1040, each image, such as the input image, basic object image, or image generated from them, or a 3D object, as described using step S520 of the image generation flow 500 in Figure 5 and the input image processing and basic object image generation flow 600 in Figure 6.

[0145] The output module 219 can also display an image of an object illuminated by a virtual light source in response to input settings such as a virtual light source, as will be described later. The output module 219 can switch between images of the object to be displayed in the first setting area 1010, or can display multiple images side by side, for example, a basic object image and an image of the object illuminated by a set virtual light source. This allows the user to visually compare the object in the input image with the object that reflects the current light source settings, making it easier for the user to determine more preferable settings.

[0146] The output module 219 displays in the second setting area 1020 the type and material of the virtual light source (light) to be reflected in the final object image and the final composite image. The types of virtual light sources (lights) to be reflected include, for example, spotlights, natural light, and diffuse light. The elements are generated separately from the target object in the final composite image, and include, for example, a stand, a body part such as fingers, a plant, a flower or petal, fruit, food, drink, and the like.

[0147] The output module 219 can switch between displaying and not displaying the second setting area 1020 in response to an input operation in a third setting area 1030, which will be described later, for example. The output module 219 displays in the third setting area 1030 an input acceptance section for "template," "add material," "created image," and "image generation" to be reflected in the final object image, final background image, and final composite image.

[0148] In the example of FIG. 10, "Desert and Mountains" is selected as the "Template" in the third setting area 1030. In the image information 223 and content information 226 in the auxiliary storage device 202, multiple "template" candidates are stored according to, for example, the category of each object and the time of generation and publication of the final composite photograph.

[0149] When the acquisition module 211 acquires a selection input for "add material" in the third setting area 1030 by operating a mouse or the like, the output module 219 can display or expand the second setting area 1020.

[0150] The acquisition module 211 can acquire the conditions for generating the final background image, in the example of FIG. 10, "perfumed and surrounded by beautiful nature," in response to the user's text input in the "image to be created" field.

[0151] When the acquisition module 211 acquires an operation input for "image generation", such as a click input on the "generate" area, the modules 211 to 219 for creating the final composite image cooperate to start the process of generating the final composite image.

[0152] In the example of FIG. 10(B), the output module 219 displays a first setting section (first setting icon) 1050 and a second setting section (second setting icon) 1060 in the first setting area 1010. First setting unit 1050 can display setting information relating to the color and brightness (intensity) of the virtual light of the virtual light source, and can accept setting changes.

[0153] Second setting unit 1060 can display setting information relating to the position, direction, angle, etc. of the virtual light source, and can accept setting changes. 10(B), the virtual light source corresponding to the setting information of second setting unit 1060 is positioned behind the object. Shadow 1070 (1070a) in Fig. 10(B) is generated by background image generation module 215 or object image generation module 216 in association with virtual light from the virtual light source positioned behind the object.

[0154] In the example of Figure 10(B), the bottleneck portion of the object 1040 has a brighter area within the finely dashed ellipse due to the effect of reflected light 1041 and a darker area within the coarsely dashed ellipse due to the effect of shadow 1042.

[0155] The reflected light effect 1041 and the shadow effect 1042 are produced by a light source other than the virtual light source located at a position substantially directly behind the object corresponding to the setting information of the second setting unit 1060.

[0156] The different light source may be, for example, the light source used when capturing the input image or a different virtual light source that has been set. In the example of FIG. 10(B), the shadow cast by the object 1040 based on this different light source is not displayed.

[0157] The background image generation module 215 can selectively determine whether each virtual light source it sets affects only the background or both the background and the object in the final background image or final composite image.

[0158] The background image generation module 215 can selectively determine whether the effect of each virtual light source set on the final background image or the final composite image is to generate only shadows, only shades, only light, or a combination of at least two of shadows, shades, and light.

[0159] Similarly, the object image generation module 216 can selectively determine whether each virtual light source it sets affects only the object or both the object and the background in the final object image or the final composite image.

[0160] The object image generation module 216 can selectively determine whether the effect of each virtual light source set on the final object image or the final composite image is to generate only shadows, only shades, only light, or a combination of at least two of shadows, shades, and light.

[0161] As described above, in the image generation system 20, the light source setting module 214 can independently adjust the light and / or shadow effect applied to the final object image and the transmitted light and / or shadow of the final background image for the same virtual light source.

[0162] By configuring at least one of the background image generation module 215 or the object image generation module 216 as described above, the user can generate an image that reflects lighting effects that are optically impossible or difficult to achieve in actual photography.

[0163] Furthermore, even in cases where lighting settings would conventionally result in dark or light areas in the background that would need to be corrected by retouching or the like in order to achieve a desired lighting effect on an object, it is now possible to generate a final composite image that does not include dark or light areas that require such correction.

[0164] In FIG. 10(C), the virtual light source corresponding to the setting information of second setting unit 1060 is set at a position different from that in the example of FIG. 10(B), that is, at a position diagonally behind the object. Accordingly, the shadow 1070 (1070b) in Figure 10(C) is generated by the background image generation module 215 or the object image generation module 216 so as to correspond to virtual light from a virtual light source positioned diagonally behind the object, and is tilted more to the right than the shadow 1070 (1070a) in Figure 10(B).

[0165] Instead of generating shadows and the like using the 3D objects described above or below, the background image generation module 215 or the object image generation module 216 can also generate shadows, shades, reflective surfaces, and the like for the image of a two-dimensional object based on the position of the virtual light source, the positional relationship with the object 1040, the two-dimensional shape in the image of the object, and the like.

[0166] In such a configuration, the output module 219 can, for example, change only the direction in which the displayed shadow 1070 extends, or change only the position of the object's reflective surface in response to the position setting input, depending on the setting input for the position of the virtual light source.

[0167] FIG. 11 is a second explanatory diagram illustrating an example of a light source setting. FIG. 11 shows another example of the first setting area 1010 described with reference to FIGS.

[0168] In the example of FIG. 11, the first setting section 1050 includes a color setting section 1110 and an intensity setting section 1120 . The acquisition unit can acquire input information such as a click operation on the color setting unit 1110 or the intensity setting unit 1120 by the user, for example.

[0169] In FIG. 11(A), the minimum brightness is set in the intensity setting section 1120. In FIG. 11(B), the maximum brightness is set in the intensity setting section 1120. 11(A) and 11(B), the position of the virtual light source is the same, but the direction of irradiation of the virtual light is different, and accordingly, the directions in which shadows 1070c and 1070d extend from object 1040 are different.

[0170] In FIG. 11(C), a different light color from that in FIGS. 11(A) and 11(B) is set in the color setting section 1110, and maximum brightness is set in the intensity setting section 1120. When the acquisition unit acquires input information such as a click operation on the color setting unit 1110 by a user, the output module 219 displays a new light source color setting unit 1111, for example. The user can set a desired light source color in the light source color setting section 1111.

[0171] FIG. 12 is a third example of an explanatory diagram illustrating light source settings. FIG. 12A is another example of a setting screen 1000 for setting conditions for generating a final composite image. FIG. 12(B) is another example of the first setting area 1010 in FIG. 12(A). FIG. 12(C) is yet another example of the first setting region 1010 in FIG. 12(A).

[0172] The explanation of the setting screen 1000 using FIG. 10 also applies to the setting screen 1000 shown in FIG. 12, so a repeated explanation will be omitted. In the example of FIG. 12(A), output module 219 displays virtual light source 1200 in first setting area 1010 in a manner that makes it easy to grasp the spatial positional relationship with respect to object 1040.

[0173] To this end, in the example of FIG. 12(A), the output module 219 displays a plurality of first depth effect lines 1210 extending in the depth direction and a plurality of second depth effect lines 1220 extending in the left-right direction in the first setting area 1010. In the example of FIG. 12(A), the first depth effect line 1210 and the second depth effect line 1220 are shown more emphasized than in the examples of FIG. 12(B) and FIG. 12(C).

[0174] In the following description using FIG. 12, an example will be described in which the image or the like of the object 1040 is a 3D object of the object 1040. By using a 3D object of the object 1040, it is also possible to set the position of the light source, for example, to a position inside the object 1040 in a virtual three-dimensional space.

[0175] The transmitted light and its effects that pass through transparent materials such as glass can be reproduced based on actual photographic data and physical calculations under various conditions that vary in the physical properties of the transparent material, the intensity, color, and direction of the transmitted light.

[0176] In the examples of Figures 12(B) and 12(C), the first setting section 1050 has a position setting section 1240 and an angle setting section 1250 in addition to the color setting section 1110 and intensity setting section 1120 described using Figure 11. The acquisition unit can acquire input information such as a click operation on the position setting unit 1240 or the angle setting unit 1250 by the user, for example.

[0177] In FIG. 11(B), the position setting section 1240 is selected, and the position information setting section 1211 is displayed. The user can set a desired light source position in the position information setting section 1211 .

[0178] In FIG. 11(C), the angle setting section 1250 is selected, and the angle information setting section 1221 is displayed. The user can set a desired light source angle, such as the irradiation direction of virtual light emitted from the virtual light source, in angle information setting section 1221.

[0179] FIG. 13 is an example of an explanatory diagram for explaining the final composite image. FIG. 13A is an example of a generation screen 1300 for generating a final composite image. FIG. 13B shows an example of an editing screen 1310 for editing the generated final composite image. FIG. 13C is an example of an output screen 1320 on which the generated or edited final composite image is output.

[0180] In FIG. 13A, a generation screen 1300 has a final object image display area 1301, a final composite image display area 1302, and a condition input area 1303. In the final composite image display area 1302, at least one final composite image 1313, and preferably a plurality of final composite images 1313, are displayed.

[0181] In the condition input area 1303, for example, various conditions for generating a final background image can be input. In the condition input area 1303, for example, information about the template used to generate the final background image can be displayed.

[0182] Information about the template used to generate the final background image includes, for example, the template search word input section, template classification, recommended templates that are pre-set or suggested by a learning model, etc., depending on the type of object 1040 and the characteristics of the final object image.

[0183] In FIG. 13B, an editing screen 1310 has a final composite image editing area 1311 and an editing instruction input area 1312. In the editing instruction input area 1312, it is possible to accept input of editing instructions for editing the generated final composite image 1313.

[0184] The user can input text information such as "There is perfume and it is surrounded by beautiful nature" as an editing instruction. The user can also select recommended editing instructions suggested by the image generation system 20 and input the editing instructions.

[0185] In the example of FIG. 13(B), the upper range of the final composite image 1313 is selected as an edit selection range 1314. For example, when the user inputs an editing instruction to "erase" the edit selection range 1314, the composite image generation module 217 generates a final composite image in which the edit selection range 1314 has been erased.

[0186] Alternatively, if an editing instruction to "erase" is input for the editing selection range 1314, the background image generation module 215 regenerates the final background image with the editing selection range 1314 erased, and the composite image generation module 217 can regenerate the final composite image using the regenerated final background image.

[0187] In such a configuration, the background image generation module 215 may execute processing according to the final background image generation flow 800 described using FIG. 8 from an intermediate step, for example, from step S850 of "inputting the generated image and background generation conditions into the fourth generation AI, etc."

[0188] The final composite image editing area 1311 can also display the editing history. The output module 219 can switch the final composite image 1313 displayed in the final composite image editing area 1311 in response to a selection input for an image or the like displayed as the editing history.

[0189] In response to the switching of the display of the final composite image 1313, the output module 219 can switch the display content of, for example, the edit instruction input area 1312 so as to display edit instructions for the final composite image 1313 being displayed.

[0190] This allows the user to easily check how their input instructions have been reflected in the edits, and also makes it easier for the user to adjust the content of their edit instructions, such as the text information they input, to get closer to the desired edit result.

[0191] The output screen 1320 in Figure 13(C) displays the final composite image 1313 that has been generated or generated and edited, its "Image Details" information, areas "Image Download" and "PSD Download" that accept download instruction input for downloading the output screen 1320 and its related files, and an area "Edit" that accepts input to resume editing for editing the final composite image 1313 again.

[0192] For example, when the acquisition module 211 acquires a click operation or the like on "edit" by the user, the output module 219 can transition the display content to the edit screen 1310 of FIG. 13(B) again.

[0193] FIG. 14 shows another example of the hardware configuration of the image generation system 1400(20). The image generation system 1400 can be used in addition to, instead of, or in cooperation with the image generation system 20 described with reference to FIG. 2 and the like. Furthermore, each module of the image generation system 1400 described below may be incorporated into the image generation system 20 described with reference to FIG. 2 and the like. In the following description, explanations of modules and the like similar to the modules of the image generating system 20 described using FIG. 2 and the like will be omitted.

[0194] The image generation system 1400 includes a main memory device 1401, an auxiliary memory device 1402, a processor 1403, an input device 1404, an output device 1405, a communication control unit 1406, and the like.

[0195] The main memory device 1401 of the image generation system 1400 stores programs and applications such as an acquisition module 1411, an object identification module 1412, a constraint condition image generation module 1413, a light source setting module 1414, a replacement background image generation module 1415, a light source effect application module 1416, and an output module 1417. The processor 1403 executes these programs and applications to realize the various functional elements of the image generation system 1400 .

[0196] The object identification module 1412 is an example of an "object identification unit" and identifies an object in an input image including the object and a background. Note that the object extraction module 212 of the image generation system 20 described using FIG. 2 etc. may function as the object identification module 1412.

[0197] The object identification module 1412 may be capable of identifying an object count, which is the number of objects identified in the input image. For example, in a configuration in which an image generation system 1400 described with reference to FIG. 14 and an image generation system 20 described with reference to FIG. 2 cooperate with each other, If the number of objects is a predetermined value, for example, 1, the composite image generation module 217 outputs the final composite image; If the number of objects is greater than a predetermined value, for example, two or more, the lighting effects application module 1416 may output a generated image.

[0198] The object identification module 1412 may include a main object selection module 1412a, which selects a main object from the identified objects when the number of objects is greater than a predetermined value, for example, two or more.

[0199] The constraint condition image generation module 1413 is an example of a "constraint condition image generation unit" and generates a constraint condition image by a constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source. Hereinafter, the constraints may be referred to as a constraint image, and the constraint generation process may be referred to as a constraint image generation process.

[0200] The condition image generation module 1413 generates a one-channel image as a constraint condition image based on calculation processing such as ray-casting processing and normal mapping processing. A specific example of the constraint image generation process will be described later.

[0201] The constraint condition image generation module 1413 includes a depth information acquisition module 1413a, a 3D object generation module 1413b, and a depth image generation module 1413c.

[0202] The depth information acquisition module 1413a acquires depth information that indicates the depth of pixels that make up the input image. The depth information acquisition module 1413a identifies the correspondence between the three-dimensional coordinate information of the points or surfaces that make up the three-dimensional object generated by the 3D object generation module 1413b described below and the two-dimensional coordinates and depth information of the pixels that make up the input image.

[0203] The 3D object generation module 1413b generates three-dimensional objects of the objects identified in the input images. The depth image generation module 1413c generates a depth image from the depth information obtained for pixels (points) or groups of pixels (regions) of the input image.

[0204] The constraint image generation module 1413 Three-dimensional light source position information, three-dimensional object position information composed of two-dimensional object position information in an input image corresponding to a three-dimensional object and object depth information which is depth information associated with the object position information; Based on this, a constraint condition image is generated that represents the lighting effect of a virtual light source on a three-dimensional object, which is calculated based on three-dimensional light source position information set by a light source setting module 1414 described below and three-dimensional object position information.

[0205] Additionally or alternatively, the constraint image generation module 1413 may: Three-dimensional light source position information, three-dimensional object position information composed of two-dimensional object position information of the identified object in the input image and object depth information which is depth information associated with the object position information; Based on this, a constraint image is generated that represents the lighting effect of a virtual light source on the identified object, which is calculated based on three-dimensional light source position information and three-dimensional object position information set by the light source setting module 1414 described below.

[0206] The constraint condition image generation module 1413 generates the constraint condition image using a three-dimensional object for the main object selected by the main object selection module 1412a, and generates the constraint condition image using a depth image for objects other than the main object.

[0207] When the number of objects identified by the object identification module 1412 is two or more, the constraint condition image generation module 1413 can switch, for each object, whether or not to include the lighting effect of a virtual light source in the constraint condition image.

[0208] When two or more virtual light sources are set by the light source setting module 1414, the constraint image generation module 1413 can switch, for each virtual light source, whether or not to use the three-dimensional information of the virtual light source in the constraint generation process.

[0209] Next, the light source setting module 1414 functions in the same manner as the light source setting module 214 of the image generation system 20 described with reference to FIG. 2 and the like. The light source setting module 1414 can set three-dimensional light source position information for a virtual light source, which is composed of two-dimensional light source position information on the input image and light source depth information, which is depth information of the virtual light source associated with the light source position information.

[0210] For example, the light source setting module 1414 may: Input of two-dimensional position information on the input image, An input that assigns depth information to input position information; Based on You can set the position of the virtual light source. The input for assigning depth information is, for example, a numerical value input, a slide bar operation, or an operation for placing a virtual light source object in an image.

[0211] For example, the light source setting module 1414 displays a light source object representing the set virtual light source in the input image. For example, the light source setting module 1414 changes at least one of the size, shape, shade, brightness, and type of the light source object in response to an input that assigns depth information, and displays the light source object.

[0212] For example, the light source setting module 1414 may: When the virtual light source is located in front of the object, When the virtual light source is located inside the object, The virtual light source is located behind the object. At least one of the size, shape, shade, brightness, and type is changed to display the light source object.

[0213] Next, the replacement background image generation module 1415 is an example of a "background generation unit" and generates a generated background that is different from the background of the input image. The replacement background image generation module 1415 receives an input image and a background prompt that specifies the conditions for generating the generated background. The replacement background image generation module 1415 inputs the input image and background prompt to a sixth generation AI, which is a background generation AI, and obtains an image (generated background image) in which the background of the input image is replaced with the generated background.

[0214] The replacement background image generation module 1415 can use, as the sixth generation AI, any generation AI that receives an image or a prompt as input, executes image generation processing, and outputs a generated image. The replacement background image generation module 1415 can be used as a sixth generation AI by combining or switching between a plurality of arbitrary generation AIs.

[0215] Next, lighting effect application module 1416 is an example of a "lighting effect application unit", which inputs the input image and the constraint condition image to a fifth generation AI and generates a generated image to which a lighting effect based on a virtual light source is applied. The lighting effect based on the virtual light source is at least one of transmitted light from an object, a shadow of the object, the effect of light on the object, and the effect of shadow on the object.

[0216] The objects identified in the input image may include at least a first object and a second object. In that case, the lighting effect applied by the lighting effect application module 1416 is: Light from a first object striking a second object, the shadow of the first object falling on the second object, the effect of light from the first object on the second object; The shadow effect on the second object due to the shadow of the first object, At least one of the following is true.

[0217] Additionally, the light effect application module 1416 can obtain a light source prompt that specifies the lighting behavior of the virtual light source, and input the image in which the background is replaced with the generated background, the constraint image, and the light source prompt into a fifth generation AI to generate a generated image.

[0218] FIG. 15 is an example of another image generation flow 1500. The image generation system 20 described using FIG. 2 and the image generation system 1400(20) described using FIG. 14 can cooperate or be combined to selectively or switch between the image generation flow 500 described using FIG. 5 and another image generation flow 1500 described using FIG. 15.

[0219] The steps of each flow and subflow explained below using Figures 15 to 20 may be executed in reverse order, in parallel, incorporated into another flow, or used together in different flows or subflows, as long as no technical discrepancies arise.

[0220] When another image generation flow 1500 is started in the image generation system 1400, the acquisition module 1411 of the image generation system 1400 acquires input image information (hereinafter also referred to as input image) to be processed from the user terminal 30 (S1510).

[0221] The object identification module 1412 of the image generation system 1400 identifies objects, such as goods, products, people, buildings, etc., contained in the input image (S1520). The object identification module 1412 of the image generation system 1400 may, for example, segment or classify the identified objects, and associate them with content information stored in the auxiliary storage device 1402 based on the segmentation or classification.

[0222] Image generation system 1400 has, for example, content information corresponding to the segmentation or classification of objects, such as preset background images, background settings, background prompts, preset virtual light source settings, virtual light source prompts, etc., generated by an image retoucher or AI prompter. This allows the image generation system 1400 to additionally provide processing that was previously performed by a retoucher, AI prompter, etc. as a content service.

[0223] The light source setting module 1414 of the image generation system 1400 acquires light source setting information, which is setting information for a virtual light source that irradiates light onto an object included in an input image (S1530). The virtual light source may be set based on a setting input operation by the user, or may be set based on setting information stored in advance as virtual light source information 1424 of image generation system 1400. The process of acquiring light source setting information in the alternative image generation flow 1500 is the same as the process of acquiring light source setting information 700 (S530) described with reference to FIGS. 5 and 7, and therefore will not be described again.

[0224] The constraint image generation module 1413 of the image generation system 1400 generates a constraint image by a constraint generation process based on the three-dimensional information on the object identified in the input image and the three-dimensional information on the virtual light source (S1540). The process of generating the constraint image will be described in detail with reference to FIGS.

[0225] The constraint generation process is, for example, a ray-casting process, a normal mapping process, etc., but is not limited to these. The constraint generation process may be any process that can calculate values ​​corresponding to various effects caused by light rays from the three-dimensional positional relationship between the three-dimensional positional information of the virtual light source and the three-dimensional positional information of each point or area of ​​the object.

[0226] The lighting effect application module 1416 of the image generation system 1400 inputs the input image and the constraint image to the fifth generation AI 1416a, generates a generated image to which the lighting effect based on the virtual light source is applied (S1550), outputs the generated generated image (S1560), and ends the other image generation flow 1500. Note that in another example of the image generation system 1400, it is possible to return to S1530, for example, to adjust the output generated image, and accept re-input of virtual light source information 1424 by the user. The process of generating a generated image using the fifth generation AI will be described in detail with reference to FIGS.

[0227] The fifth generation AI 1416a can add or subtract the influence of light, particularly the influence of the lighting effect due to a virtual light source, to various features and concepts included in the input image. Adding the influence of a lighting effect means, for example, adding or strengthening (increasing) shadows, reflected light, etc. to the input image. Subtracting the influence of a lighting effect means, for example, removing or weakening (reducing) shadows, reflected light, etc. from the input image.

[0228] The fifth generation AI 1416a can, in particular, generate an image in which the influence of the lighting effect of a virtual light source is added or subtracted from the input image while limiting changes in various light-related features and concepts contained in the input image to a range that maintains consistency.

[0229] FIG. 16 shows an example of a flow 1600 for generating a constraint image. The depth information acquisition module 1413a of the constraint condition image generation module 1413 acquires depth information of each point of the acquired input image (S1610). Depth information is information also called depth information. The term "obtain" may be read as "extract," "calculate," "estimate," etc. The constraint image generation module 1413 can obtain depth information for each point in the input image using any method. The depth information will be described in detail with reference to FIG.

[0230] The depth information of the image is a measurement value assigned to each point or region of the input image, and may be information that is included in the input image in advance. The depth information of the image may be an estimated value estimated by any estimation method for each point or region of the input image.

[0231] For example, the depth information acquisition module 1413a may use deep learning-based monocular depth estimation or the like to acquire the depth information. The depth information acquisition module 1413a can thereby use deep learning to obtain an estimate of depth information from a monocular image. The depth information acquisition module 1413a can acquire depth information of the input image using any convolutional neural network (CNN) or transformer model.

[0232] For example, the depth information acquisition module 1413a can use a Depth from Defocus (DfD) method to estimate depth information based on the out-of-focus (blur) of an input image in order to acquire depth information.

[0233] For example, the depth information acquisition module 1413a can estimate depth information based on the geometric features of the background and objects of the input image, such as straight lines that appear in perspective, to acquire depth information.

[0234] The 3D object generation module 1413b of the constraint image generation module 1413 generates a 3D object of the identified object based on the contour, segmented information, classified information, etc. of the identified object (S1620).

[0235] The 3D object generation module 1413b can input an input image to the first generation AI 213a and generate a three-dimensional object of the target object, similarly to the 3D object generation module 213 described with reference to FIG. 6, for example.

[0236] For example, based on the segmented information or classified information, the 3D object generation module 1413b may appropriately select a 3D object that corresponds to the information and is stored in the content information 1426.

[0237] For example, if the identified object is a wine glass, the 3D object generation module 1413b can select a suitable one from the 3D objects of wine glasses stored in the content information 1426, and in some cases convert it into a shape corresponding to the input image for use.

[0238] The generated three-dimensional object is not limited to information in OBJ format, PLY format, or the like. The generated three-dimensional object may be, for example, a full 3D object or other forms of information that allow for a 3D approximation, such as depth, texture, normal maps, etc.

[0239] The constraint condition image generation module 1413 associates the depth information of the virtual light source and the depth information of the 3D object with the depth information of the input image (S1630). As a result, the constraint image generation module 1413 can place the virtual light source and the 3D object in the virtual three-dimensional coordinates of the input image based on the position of the virtual light source placed in the input image (two-dimensional coordinates consisting of the length and width of the input image) and the depth position of the virtual light source in the input image, and the position of the object identified in the input image (two-dimensional coordinates consisting of the length and width of the input image) and the depth position of the 3D object in the input image.

[0240] The constraint image generation module 1413 executes a constraint generation process based on the three-dimensional position information of the 3D object and the three-dimensional position information of the virtual light source (S1640).

[0241] By using the three-dimensional position information of a 3D object and the three-dimensional position information of a virtual light source, it is possible to simulate with high precision the effect of light (rays) emitted from the virtual light source and hitting each point on a 3D object, as well as the effect of light passing through an object and hitting the background, such as the ground or walls.

[0242] This allows three-dimensional lighting effects (light conditions) to be applied to the object based on the object's depth information and the position, color, and intensity of the virtual light source. Each virtual light source, based on its position, color, and intensity, imparts a specific lighting effect (e.g., brightness of reflected light, darkness of shadows, etc.) to each pixel in the input image or depth image. In the conditional image, the intensity of this lighting effect is based on the relative position of the virtual light source and the object associated with the depth information.

[0243] For example, the constraint image generation module 1413 can perform processing based on techniques such as ray-casting, UV map, normal map, ray-tracing, beam-tracing, and photon-mapping as constraint generation processing to simulate light effects.

[0244] The three-dimensional position information of a 3D object is composed of two-dimensional object position information of the 3D object in the input image and object depth information, which is depth information associated with the object position information. Note that the three-dimensional position information of a 3D object may also be interpreted as three-dimensional position information of each point or surface of the 3D object.

[0245] The constraint image generation flow 1600 using a 3D object can generate a constraint image with higher accuracy than another constraint image generation flow 1700 described below. In the constraint image generation flow 1600, the constraint image generation module 1413 can generate a constraint image by finely controlling, for example, the density and gradient of shadows.

[0246] The three-dimensional position information of the virtual light source is composed of two-dimensional light source position information of each point of the virtual light source in the input image and light source depth information, which is depth information of the virtual light source associated with the light source position information. Note that the three-dimensional position information of the virtual light source may also be interpreted as three-dimensional position information of each point or each surface of the virtual light source.

[0247] The constraint condition image generation module 1413 generates a constraint condition image of the input image based on the constraint condition generation process (S1650), and outputs the generated constraint condition image (S1660). Constraint condition images are, for example, one-channel images whose shades vary depending on the lighting effect of a virtual light source, normal map images, albedo images, shadow map images, transmission map images, specular highlight map images, silhouette images, environmental occlusion maps, and reflection map images.

[0248] In such a one-channel image, for example, areas corresponding to shadows on an object caused by a virtual light source or shadows extending from the object are displayed dark, while areas corresponding to the reflected light on the object caused by the virtual light source (bright parts of the object) are displayed light.

[0249] FIG. 17 is an example of another flow 1700 for generating a constraint image. The constraint condition image generation module 1413 can switch between the constraint condition image generation flow 1600 described using Figure 16 and another constraint condition image generation flow 1700 described below depending on the scene, for example, depending on the number of objects included in the input image.

[0250] The depth information acquisition module 1413a of the constraint condition image generation module 1413 acquires depth information of each point of the acquired input image (S1710). The depth information is the same information as that explained using Fig. 16, and the method of acquiring the depth information and the like will be omitted by referring to the explanation of Fig. 16.

[0251] The depth information acquisition module 1413a extracts depth information for each point in the range of the object identified in the input image from the depth information for each point in the input image (S1720). Note that the acquisition of depth information for each point in the input image in S1710 and the acquisition of depth information for each point in the range of the object in S1720 may be performed in the same step. As a result, the constraint condition image generation module 1413 can obtain three-dimensional position information of the object by associating the depth information of the object with the two-dimensional position of the object in the input image.

[0252] The depth image generation module 1413c generates a depth image based on the depth information acquired from the input image and the depth information acquired from the input image and associated with the identified object (S1730). Note that the depth image generation module 1413c may generate a depth image only from the depth information acquired from the input image, and then identify and segment the object in the depth image.

[0253] The constraint image generation module 1413 associates the depth information of the virtual light source with the depth information of the input image (S1740). As a result, the constraint image generation module 1413 can place a virtual light source in the virtual three-dimensional coordinates of the input image based on the position of the virtual light source placed in the input image (two-dimensional coordinates consisting of the length and width of the input image) and the depth position of the virtual light source within the input image.

[0254] The constraint image generation module 1413 executes a constraint generation process based on the three-dimensional position information of the identified object and the three-dimensional position information of the virtual light source (S1750). By using the three-dimensional position information of the object based on the depth information obtained from the input image and the three-dimensional position information of the virtual light source, it is possible to simulate with low computational load the effects of light (rays) emitted from the virtual light source and hitting each point on the object, and light that passes through the object and hits the background, such as the ground or walls.

[0255] This allows three-dimensional lighting effects (light conditions) to be applied to the object based on the object's depth information and the position, color, and intensity of the virtual light source. Each virtual light source, based on its position, color, and intensity, imparts a specific lighting effect (e.g., brightness of reflected light, darkness of shadows, etc.) to each pixel in the input image or depth image. In the conditional image, the intensity of this lighting effect is based on the relative position of the virtual light source and the object associated with the depth information.

[0256] For example, the constraint image generation module 1413 can perform processing based on techniques such as ray-casting, UV map, normal map, ray-tracing, beam-tracing, and photon-mapping as constraint generation processing to simulate light effects.

[0257] The three-dimensional position information of an object is composed of two-dimensional object position information of each point of the object in the input image and object depth information, which is depth information associated with the object position information. The three-dimensional position information of the virtual light source is composed of two-dimensional light source position information of each point of the virtual light source in the input image and light source depth information, which is depth information of the virtual light source associated with the light source position information.

[0258] The constraint condition image generation module 1413 generates a constraint condition image of the input image based on the constraint condition generation process (S1760), and outputs the generated constraint condition image (S1770). The constraint image is, for example, a one-channel image whose shade varies depending on the lighting effect of a virtual light source.

[0259] In such a one-channel image, for example, areas corresponding to shadows on an object caused by a virtual light source or shadows extending from the object are displayed dark, while areas corresponding to the reflected light on the object caused by the virtual light source (bright parts of the object) are displayed light.

[0260] FIG. 18 is an example of a generation flow 1800 for a generated image using the fifth generation AI. When the generation flow 1800 (S1550) of a generated image using the fifth generation AI 1416a starts, the lighting effect application module 1416 inputs the input image and the constraint condition image to the fifth generation AI 1416a (S1810). The fifth generation AI 1416a can also receive, as input, various prompts related to the lighting effects of the virtual light source, in addition to the input image and the constraint condition image.

[0261] The fifth generation AI 1416a generates a generated image to which a lighting effect has been applied based on the input image and the constraint condition image (S1820), and outputs the generated image to which the lighting effect has been applied (S1830). More specifically, the fifth generation AI 1416a generates a generated image by replacing the light and shadow features included in the input image with features corresponding to the lighting effect included in the constraint condition image through conditional generation, while retaining predetermined features (hereinafter also referred to as feature amounts) of the input image, such as shape features of the object or background.

[0262] The fifth generation AI 1416a uses, for example, a one-channel constraint condition image as a constraint condition, and generates shading for points and areas that are expressed darkly in the constraint condition image, based on the input image and settings for each feature. The fifth generation AI 1416a uses, for example, a one-channel constraint condition image as a constraint condition, based on the input image and settings for each feature, and generates light such as reflected light, emitted light, transmitted light, etc. at points and areas that are faintly represented in the constraint condition image.

[0263] The fifth generation AI 1416a as described above uses the constraint condition image as a light condition image (image representing light conditions) that expresses the conditions of light to be applied to the texture of the object. The fifth generation AI 1416a may also select features of the input image to retain.

[0264] The fifth generation AI 1416a can, for example, switch between replacing the features corresponding to the above-mentioned lighting effects on the entire input image and replacing the features corresponding to the above-mentioned lighting effects on a portion of the input image, depending on, for example, instructions input from the user.

[0265] When the fifth generation AI 1416a generates a generated image by applying a lighting effect based on a constraint image to an input image, it can limit the difference between the input image and the generated image to which the lighting effect has been applied, for example, for each feature, the difference between the feature contained in the input image and the feature of the generated image to which the lighting effect has been applied, so that it does not exceed a set threshold.

[0266] The fifth generation AI 1416a can perform calculations to apply changes in lighting conditions while preserving the characteristics of the input image. This allows for consistency between the input image and the generated image to which the lighting effects are applied. As a result, the user will feel less uncomfortable when the lighting effects are applied.

[0267] For example, in order to maintain consistency between the input image and the generated image to which the lighting effect has been applied, the fifth generation AI 1416a can set features that are not allowed to change, features that are allowed to change, the amount of change that is allowed, the rate of change that is allowed, etc. for each step in applying the lighting effect of the virtual light source.

[0268] In this case, the fifth generation AI 1416a further determines whether or not there is a change in a feature that is not allowed to be changed, and whether or not the amount or rate of change of the changed feature is within an allowable range. The fifth generation AI 1416a executes generation processing as appropriate according to the determination result so that the features that are not allowed to change do not change, and so that the amount and rate of change of the changed features fall within an allowable range.

[0269] Even when the fifth generation AI 1416a also receives various prompts regarding the lighting effects of the virtual light source, the fifth generation AI 1416a performs generation processing in accordance with the prompts as appropriate, as long as consistency is maintained between the input image and the generated image to which the lighting effects have been applied, as described above.

[0270] The fifth generation AI 1416a is Images, information on the types of features and concepts contained in the images, the magnitude of the feature quantities and concept quantities, and combinations of these. Information on the image before and after the change, the change in various features between the image before and the image after the change, the rate of change in the feature amounts, etc., and combinations thereof, An image before the change, an image after the change, information on whether the difference between them is acceptable, and a combination thereof; Relationships between various characteristics, such as the relationship between the material and transparency of an object, and the relationship between the material and the attenuation of light passing through the material. This is a learning model that has been trained using the following.

[0271] The various features are features related to the object or the entire image, such as shape, volume, color, brightness, material, transmittance (transparency), surface treatment, type of object, type of background, contrast, focal length, angle of view, exposure (degree of overexposure or underexposure, etc.), interference (for example, the object slightly penetrating the ground), style, reflection, viewpoint height, symmetry, etc.

[0272] By learning using this information, for example, by associating, paying attention to, and configuring the associations between images and various features, the fifth generation AI 1416a can generate images that distinguish between features to be changed and features to be retained.

[0273] FIG. 19 is an example of another generation flow 1900 for a generated image using the fifth generation AI. The light source effect application module 1416 can switch between, replace, or execute another generation flow 1900 (S1550) of a generated image using the fifth generation AI and the generation flow 1800 of a generated image using the fifth generation AI described using Figure 18.

[0274] When another generation flow 1900 of a generated image using the fifth generation AI is started, the light source effect application module 1416 inputs the input image, the constraint image, and the light source prompt obtained at any timing to the fifth generation AI 1416a (S1910).

[0275] The fifth generation AI 1416a generates a generated image to which the lighting effects and instructions from the light source prompts are applied, based on the input image, constraint condition image, and light source prompt (S1920).

[0276] Regarding the generation of a generated image applying instructions by light source effects and light source prompts, the description regarding Fig. 18 is used and the description will be omitted. As described above, the fifth generation AI 1416a can generate a generated image even without input of a light source prompt.

[0277] The lighting effect application module 1416 determines whether or not the generated image generated in S1920 needs to be adjusted based on, for example, an input from the user (S1930). If the generated image is not to be adjusted (No in S1930), the lighting effect application module 1416 finalizes the generated image, outputs the finalized generated image (S1980), and ends another generation flow 1900 of the generated image using the fifth generation AI.

[0278] When adjusting the generated image (Yes in S1930), the lighting effect application module 1416 acquires, for example, a light source prompt for adjustment (S1940). At this timing, the lighting effect application module 1416 may also acquire, for example, adjusted light source setting information for a virtual light source. The lighting effect application module 1416 can acquire the adjusted light source setting information using the methods described with reference to FIGS. 7 and 15.

[0279] The lighting effect application module 1416 inputs the light source prompt for adjustment to the fifth generation AI 1416a (S1950). The lighting effect application module 1416 generates a generated image to which the lighting effect and instructions from the light source prompt for adjustment have been applied, based on the input lighting prompt for adjustment and at least one of the input image and constraint image input in S1910 and the generated image generated in S1920 (S1960).

[0280] For example, when the fifth generation AI 1416a generates an adjusted generated image based on an input light source prompt for adjustment and the generated image generated in S1920, it changes only the features related to the light source prompt for adjustment among the various features contained in the generated image generated in S1920 to generate the adjusted generated image.

[0281] As a result, the lighting effect application module 1416 can provide the user with an adjusted generated image in which the desired adjustments have been made while retaining the characteristics of the generated image generated in S1920, except for the characteristics related to the lighting prompt.

[0282] The lighting effect application module 1416 determines whether or not to end the adjustment of the adjusted generated image generated in S1960 based on, for example, an input from the user (S1970).

[0283] If the generated image is not to be finished (No in S1970), the lighting effect application module 1416 executes the processes from S1940 to S1960 again. If the generated image is to be terminated (Yes in S1970), the light source effect application module 1416 finalizes the generated image, outputs the finalized generated image (S1980), and terminates another generation flow 1900 of the generated image using the fifth generation AI.

[0284] FIG. 20 shows an example of a generation flow 2000 for generating a generated image when there are multiple objects. The image generation system 1400 can appropriately select or switch between the image generation flow 500 described using FIG. 5, another image generation flow 1500 described using FIG. 15, and the image generation flow 2000 for generating images when there are multiple objects described using FIG. 20.

[0285] When the image generation system 1400 starts the generation flow 2000 for a generated image when there are multiple objects, the acquisition module 1411 of the image generation system 1400 acquires an input image to be processed from the user terminal 30 (S2010).

[0286] The object identification module 1412 of the image generation system 1400 identifies objects, such as goods, products, people, buildings, etc., contained in the input image (S2020). The light source setting module 1414 of the image generation system 1400 acquires light source setting information, which is setting information for a virtual light source that irradiates light onto an object included in an input image (S2030). The processing from S2010 to S2030 is the same as the processing from S1510 to S1530 in FIG. 15, so a description thereof will be omitted.

[0287] The object identification module 1412 determines whether a main object is present in the input image (S2040). The object identification module 1412 determines the main object based on, for example, an input by a user.

[0288] If there is no main object (No in S2040), the lighting effect application module 1416 generates a constraint image that summarizes non-main objects (S2070). The light effect application module 1416 An input image; A constraint image summarizing non-major objects generated by S2070, and Based on this, a generated image is generated using the fifth generation model (S2080), and the generated image is output (S2090), thereby completing the generated image generation flow 2000 when there are multiple objects. The processing in S2070 and S2080 is similar to the processing in S1540 and S1550 in FIG. 15, and therefore a description thereof will be omitted.

[0289] If there are main objects (Yes in S2040), the constraint image generation module 1413 generates a constraint image for each main object (S2050). The processing in S2050 is the same as the processing in S1540 in FIG. 15, so a description thereof will be omitted.

[0290] The object identification module 1412 determines whether or not a non-primary object is present in the input image (S2060). The object identification module 1412 determines the non-primary object based on, for example, an input by a user.

[0291] If there is no non-primary object (No in S2060), the lighting effect application module 1416 An input image; Constraint condition images generated for each main object generated by S2050, Based on this, a generated image is generated using the fifth generation model (S2080), and the generated image is output (S2090), thereby completing the generated image generation flow 2000 when there are multiple objects. The processing in S2070 and S2080 is similar to the processing in S1540 and S1550 in FIG. 15, and therefore a description thereof will be omitted.

[0292] If non-main objects exist (Yes in S2060), the lighting effect application module 1416 generates a constraint image that summarizes the non-main objects (S2070). The light effect application module 1416 An input image; Constraint condition images generated for each main object generated by S2050, A constraint image summarizing non-major objects generated by S2070, and Based on this, a generated image is generated using the fifth generation model (S2080), and the generated image is output (S2090), thereby completing the generated image generation flow 2000 when there are multiple objects. The processing in S2070 and S2080 is similar to the processing in S1540 and S1550 in FIG. 15, and therefore a description thereof will be omitted.

[0293] The light source effect application module 1416 generates a generated image according to the generated image generation flow 2000 when there are multiple objects, thereby applying light source effects precisely to the main object and applying light source effects with low computational load to non-main objects. This allows the lighting effects application module 1416 to efficiently use computing resources to generate the generated image.

[0294] When the constraint image generation module 1413 generates a plurality of constraint images for a plurality of objects, it may integrate or synthesize at least some of them and output them to the lighting effect application module 1416 .

[0295] FIG. 21 is an explanatory diagram illustrating an example of depth information of an object and a background. FIG. 21A is an example of an input image 2100. FIG. 21(B) is an example of a virtual top view 2150 of the input image 2100.

[0296] The input image 2100 includes an object 2110 and a background 2120 . In other words, the input image 2100 is made up of an object 2110 region and a background 2120 region.

[0297] The input image 2100 includes, as objects 2110, a first object 2100a, a second object 2100b, and a third object 2100c. For example, the first object 2100a is the primary object, and the second object 2100b and the third object 2100c are secondary objects.

[0298] The input image 2100 includes a floor 2120a and a wall 2120b as a background 2120. Dashed line II is a reference line of the input image. Dashed line II (hereinafter also referred to as reference line II) may be associated with the position of the camera that captured the input image 2100, or may be considered to be the position of the camera that captured the input image 2100. The dashed line II-II is the boundary between the floor 2120a and the wall 2120b.

[0299] The first depth information 2101 (2101a, 2101b, 2101c, 2101d) shown in Figure 21(B) is information regarding the depth length (distance) from the reference line II of the input image to each object 2110 (2110a, 2110b, 2110c) or wall 2120b, and can be obtained from the input image using learning-based monocular depth estimation, DfD methods, geometric methods, etc.

[0300] The second depth information 2102 (2102a, 2102b, 2102c) shown in Figure 21(B) is information regarding the depth length of each object 2110 (2110a, 2110b, 2110c), and can be obtained from the input image using learning-based monocular depth estimation, geometric methods, etc.

[0301] In FIG. 21(B), for ease of explanation, the first depth information and the second depth information are represented by lines perpendicular to the reference line II. The first depth information and the second depth information may be information assigned from an arbitrary point to each point of the input image.

[0302] In FIG. 21B, for the sake of simplicity, one piece of first depth information 2101 and one piece of second depth information 2102 are shown for one object 2110 and one piece of background 2120. The first depth information 2101 and the second depth information 2102 can be assigned to each point (each pixel) of the object 2110 or background 2120 in the input screen, or to each range made up of a plurality of pixels.

[0303] FIG. 22 is an explanatory diagram illustrating an example of layer information of an object and a background. The object identification module 1412 can generate a layered image 2200 in which an object 2110 and a background 2120 are layered based on the object identification result. Alternatively, the constraint condition image generation module 1413 can generate a layered image 2200 in which the object 2110 and the background 2120 are layered, based on the object identification result and depth information acquired from the input image.

[0304] The layer image 2200 is an image obtained by layering the object 2110 and the background 2120 of the input image 2100. The layered image 2200 includes an object layer 2210 (2210a, 2210b, 2210c) and a background layer 2220 (2220a, 2220b).

[0305] First depth information 2101 and second depth information 2102 are assigned to each point of the first object layer 2210a, second object layer 2210b, third object layer 2210c, floor layer 2220a, and wall layer 2220b.

[0306] By using the object layer 2210 (2210a, 2210b, 2210c) and background layer 2220 (2220a, 2220b) having the first depth information 2101 and the second depth information 2102, the constraint condition image generation module can generate a constraint condition image with a lower computational load than when using an input image that identifies the object, etc. as is.

[0307] FIG. 23 is an explanatory diagram illustrating an example of the constraint condition image generation process and the constraint condition image. FIG. 23(A) is an example of an original input image 2100. In the input image 2100 shown in FIG. 23(A), light from the light source at the time the input image 2100 was captured is shining on the object 2110 from the left side of the input image 2100.

[0308] The brightness of the first range 2111 is due to the reflection of light hitting the object 2110. The darkness of the second range 2112 is due to a shadow cast by the object 2110. The darkness of the third range 2113 is due to a shadow cast by the object 2110.

[0309] FIG. 23B is an example of a depth image 2310 generated based on the depth information included in the input image 2100. The object 2110 and the background 2120 can still be easily identified in the depth image 2310. For example, it can be identified that the contour shape of the object 2110 in the depth image 2310 is almost the same as the contour shape of the object 2110 in the input image 2100.

[0310] In the depth image 2310, the reflection of light hitting the object 2110, the shadow cast on the object 2110, the shadow extending from the object 2110, etc. are greatly weakened. In other words, in the depth image 2310, much of the reflected light and shadow caused by the light source when the original input image 2100 was captured has been eliminated.

[0311] The depth image 2310 includes information about the three-dimensional shape of the object 2110 and the like. The constraint condition image generation module 1413 can generate, as the depth image 2310, image information that excludes information about reflected light and shadows caused by a light source when the original input image 2100 was captured.

[0312] FIG. 23C is an example of a processed image 2320 obtained by performing a predetermined process on the depth image 2310. The constraint image generation module 1413 can execute a process of simplifying the depth image 2310 into simpler image information as a predetermined process. The constraint condition image generation module 1413 can execute, for example, a process of emphasizing the contour of the object 2110, a process of flattening the curved surface of the object 2110, and the like.

[0313] The constraint image generation module 1413 can perform constraint generation processing based on techniques such as Ray-Casting, UV Map, Normal Map, Ray-Tracing, Beam-Tracing, and Photon-Mapping on the depth image 2310 shown in Figure 23(B) and the processed image 2320 shown in Figure 23(C).

[0314] When the constraint condition generation process is executed on the depth image 2310 shown in FIG. 23(B), the constraint condition image generation module 1413 can generate a more accurate constraint condition image.

[0315] When the constraint condition image generation module 1413 performs the constraint condition image generation process on the processed image 2320 shown in Figure 23 (C), it can generate the constraint condition image with a lower computational load and a faster processing speed by performing the constraint condition image generation process approximately.

[0316] The constraint image generation module 1413 can switch between performing the constraint image generation process on the depth image 2310 or the constraint image generation process on the processed image 2320 depending on user input, such as a switch input to prioritize accuracy or processing speed.

[0317] The constraint condition image generation module 1413 can adjust various setting values ​​for the generation process of the depth image 2310 and the processed image 2320. This makes it easier to reduce the influence of light when the input image 2100 was captured, which is included in the depth image 2310 and the processed image 2320.

[0318] FIG. 23D is an example of a constraint image 2330. FIG. 23(D) shows a schematic constraint image 2330. The constraint image 2330 is an image obtained by the constraint image generation process executed by the constraint image generation module 1413 .

[0319] In Figure 23(D), the constraint condition image generation module 1413 sets a virtual light source 2400 (1200) that shines light on the object 2110 from the right of at least one of the depth image 2310 shown in Figure 23(B) and the processed image 2320 shown in Figure 23(C), and performs constraint condition image generation processing using coordinate information of the set virtual light source 2400 (1200), etc.

[0320] The darkness of the fourth region 2114 in the constraint image 2330 is due to a shadow cast on the object 2110 by the light from the virtual light source 2400 . The brightness of the fifth region 2115 of the constraint image 2330 is due to the reflection of light from the virtual light source 2400 off the object 2110 . The darkness of the sixth region 2116 in the constraint image 2330 is due to a shadow cast by the object 2110 illuminated by the virtual light source 2400 .

[0321] In the constraint image 2330, the background 2120 (floor 2120a and floor wall 2120b) is brightest on the right side and gradually becomes darker from the right to the left side, based on the light from the virtual light source 2400 set on the right side of the constraint image 2330.

[0322] The fifth generation AI 1416a generates a generated image by applying light source effects from the virtual light source 2400, as well as light and shadow-related features such as the brightness and color of the fourth range 2114, fifth range 2115, and sixth range 2116, while maintaining predetermined features of the input image 2100, such as the shape of the object 2110 and the shape of the background 2120, in accordance with the constraint condition image 2330.

[0323] Additionally or alternatively, the fifth generation AI 1416a can also apply a lighting effect to the generated image, such as a gradient of light and shade that darkens from right to left, like the background 2120 (floor 2120a and floor wall 2120b), in accordance with the constraint image 2330.

[0324] The fifth generation AI 1416a can generate a generated image in which the influence of the lighting effect of a virtual light source is added or subtracted from the input image 2100, while keeping changes in various light-related features and concepts contained in the input image 2100 within a range that maintains consistency.

[0325] FIG. 24 is an example of an explanatory diagram illustrating the arrangement of virtual light sources. FIG. 24A shows an example in which an object represented by a virtual light source 2400 (hereinafter referred to as the “virtual light source 2400”) is arranged in a layer image 2300. FIG. 24(B) shows an example in which virtual light source 2400 of FIG. 24(A) is placed at a position corresponding to virtual top view 2150 described with reference to FIG. 21(B).

[0326] The lighting effect application module 1416 can also perform at least some of the steps of the alternative image generation flow 1500 described with reference to FIG. 15 using the layer image 2300 as an input image.

[0327] The image generation system 1400 may perform each step of another image generation flow 1500 based on the current virtual light source 2400 placement, for example, before determining the placement of the virtual light source 2400 during user input of the placement of the virtual light source 2400. This allows the image generation system 1400 to display to the user information such as that shown in Figures 24(A) and 24(B) based on the current placement of the virtual light source 2400 to assist the user in inputting the placement of the virtual light source 2400.

[0328] In FIG. 24A, a virtual light source 2400 is placed in a layer image 2200. In FIG. 24A, a shadow 2410 of an object is further applied to the layer image 2300 as a light source effect of a virtual light source 2400.

[0329] A shadow 2410a of the first object extends from the first object layer 2210a corresponding to the first object 2110a, a shadow 2410b of the second object extends from the second object layer 2210b corresponding to the second object 2110b, and a shadow 2410c of the third object extends from the third object layer 2210c corresponding to the third object 2110c. In FIG. 24(B), arrows represent light from virtual light source 2400 toward virtual top views 2450 (2450a, 2450b, 2450c) of the first to third objects.

[0330] FIG. 25 is another example of an explanatory diagram illustrating the arrangement of virtual light sources. In Fig. 25, virtual positions where virtual light source 2400 (1200) can be placed are shown in virtual top view 2150 described with reference to Fig. 21(B). Note that the reference numerals for the first to third objects are the same as those in Fig. 21(B) and are therefore omitted.

[0331] FIG. 25(A) shows a virtual light source 2500 placed within an object. A first virtual light source 2500a positioned within the first object 2110a emits light within the first object 2110a, and the light passes through the first object 2110a and travels toward the reference line I-I as shown by the arrow.

[0332] A second virtual light source 2500b located within the first object 2110a emits light within the first object 2110a, and the light passes through the first object 2110a and travels toward the second object 2110b as shown by the arrow.

[0333] Light that passes through the first object 2110a and strikes the second object 2110b is reflected from the surface of the second object 2110b and travels toward the reference line II as indicated by the arrow. The reflection on the surface of the second object 2110b is set to be directed toward the reference line II, which is different from the relationship between the angle of incidence and the angle of reflection in physical light.

[0334] A virtual light source 2500c located within the second object 2110b emits light within the second object 2110b, and the light passes through the second object 2110b and travels toward the reference line II as shown by the arrow. In FIG. 25(A), the light traveling toward the reference line II is indicated by an arrow extending perpendicular to the reference line II, but it may also be understood as traveling toward a single point.

[0335] The constraint condition image generation module 1413 and the light source effect application module 1416 can simulate with high accuracy the light source effect of the virtual light source 2500 placed within the object based on the depth information of the object 2110 and information regarding the material and transparency of the object 2110.

[0336] FIG. 25(B) shows a virtual light source 2510 placed between the objects. Light emitted from a first virtual light source 2510a positioned between a first object 2110a and a second object 2110b travels from the outside of the first object 2110a into the inside of the first object 2110a, passes through the first object 2110a, exits the outside of the first object 2110a, and travels toward the reference line I-I as shown by the arrow.

[0337] Light emitted from a second virtual light source 2510b positioned between the first object 2110a and the second object 2110b is reflected from the surface of the second object 2110b and travels toward the reference line II as shown by the arrow.

[0338] The light source settings module 1414 can display a virtual top view 2150 to assist in inputting the placement of the virtual light source 2510. For example, the user can easily input the depth position of virtual light source 2510 by placing the object of virtual light source 2510 on virtual top view 2150 by, for example, dragging and dropping the object with a mouse or the like.

[0339] The position of virtual light source 2500 placed within the object and the position of virtual light source 2510 placed between the objects shown in Figure 25 can be set based on the depth information (first depth information 2101, second depth information 2102) described in Figure 21, etc.

[0340] The light source setting module 1414 can output a virtual top view 2150, shown in FIGS. 21 and 25, to the user for positioning the virtual light source 2400. The user can place a light source object of virtual light source 2400 on virtual top view 2150. The user can also easily adjust the position of the light source object on virtual top view 2150 by dragging it with a mouse or the like.

[0341] FIG. 26 is an example of an explanatory diagram for explaining selection of an object to which a lighting effect is applied. FIG. 26(A) shows a selection screen 2600 for selecting an object to which a lighting effect is to be applied. The selection screen 2600 for the object to which the lighting effect is to be applied includes an object selection area 2610 (2610a) and the layer image 2200 described with reference to FIGS.

[0342] In the object selection area 2610a, a list of objects including object Obj1, object Obj2, and object Obj3, which are objects to which a lighting effect can be applied, is displayed. Note that the objects Obj1, Obj2, and Obj3 correspond to the first object 2110a, the second object 2110b, and the third object 2110c, respectively.

[0343] The user can select and input from the list of objects to which the lighting effect of the virtual light source 2400 is to be applied. In FIG. 26(A), objects Obj1, Obj2, and Obj3 are selected as objects to which a lighting effect is to be applied.

[0344] In FIG. 26(A), for objects Obj1, Obj2, and Obj3, The effect of reflected light from a virtual light source 2400 1041, Shadow effect 1042 caused by virtual light source 2400, Shadows 2410a, 2410b, 2410c of an object caused by a virtual light source 2400 is applied.

[0345] In FIG. 26(B), from the list of objects in the object selection area 2610b, only object Obj1 has been selected as the object to which the lighting effect is to be applied, and is displayed in text that is more emphasized than objects Obj2 and Obj3.

[0346] In FIG. 26(B), for only object Obj1, The effect of reflected light from a virtual light source 2400 1041, Shadow effect 1042 caused by virtual light source 2400, A shadow 2410a of an object caused by a virtual light source 2400; is applied.

[0347] The image generation system 1400 can execute the steps of another image generation flow 1500 based on the currently selected object in response to a user's selection input of an object to which a lighting effect is to be applied. As a result, the image generation system 1400 can switch and display information such as that shown in Figure 25(A) or Figure 25(B) depending on the currently selected object while the user is selecting an object to which a lighting effect is to be applied.

[0348] FIG. 27 is an example of an explanatory diagram for explaining selection of a lighting effect to be applied to an object. FIG. 27(A) shows a selection screen 2700 for selecting a lighting effect to be applied to an object. FIG. 27 shows a selection screen 2700 for selecting a lighting effect to be applied to an object, as an example of a screen changed from the selection screen 2600 (2600a, 2600b) for selecting an object to which a lighting effect is to be applied, which was described using FIG.

[0349] The lighting effect selection area 2710a in FIG. 27(A) contains a list of lighting effects that can be applied to the object Obj1 in addition to the object selection area 2610b in FIG. 26(B). The user can select Light, shade and shadow Light and Shadow Light and Shadow Light only Shadow only Shadow only From this, you can easily select the lighting effect to apply to the object Obj1. In this list, "light" means "reflected light" from an object.

[0350] In Figure 27(A), in the light source effect selection area 2710a, only light is selected as the light source effect to be applied to the object Obj1, and accordingly, in the layer image 2200, a reflected light effect 1041 is applied to the first object layer 2210a.

[0351] In FIG. 27(B), in the light source effect selection area 2710b, only shadow is selected as the light source effect to be applied to the object Obj1, and accordingly, the shadow 2410a of the first object extending from the first object layer 2210a is applied to the layer image 2200.

[0352] In FIG. 27, in the light source effect selection area 2710, the object Obj1, which is the selected object, and the selected light source effect are displayed in highlighted text. Note that the list of available lighting effects shown in lighting effect selection area 2710 can be collapsed to hide it. When the list of available lighting effects is hidden in Figure 27, lighting effect selection area 2710 transitions to object selection area 2610 in Figure 26.

[0353] In response to a user's selection of a lighting effect to apply, the image generation system 1400 can execute the steps of another image generation flow 1500 based on the currently selected lighting effect. As a result, the image generation system 1400 can switch between and display information such as that shown in Figure 27(A) or Figure 27(B) depending on the currently selected lighting effect while the user is selecting and inputting the lighting effect to be applied.

[0354] FIG. 28 is another example of an explanatory diagram for explaining selection of an object to which a lighting effect is applied. 28(A) and 28(B), a first virtual light source 2830 and a second virtual light source 2840 are placed on layer image 2200. In FIG.

[0355] Another selection screen 2800a for an object to which a lighting effect is to be applied in FIG. 28(A) includes a first object selection area 2810a, a second object selection area 2820a, and the layer image 2200.

[0356] In first object selection area 2810a in Fig. 28(A), object Obj1 is selected as an object to which the lighting effect of first virtual light source 2830 is to be applied. Similarly, in second object selection area 2820a in Fig. 28(A), object Obj1 is selected as an object to which the lighting effect of second virtual light source 2840 is to be applied. In FIG. 28(A), in light source effect selection area 2710 (not shown) for first virtual light source 2830 and second virtual light source 2840, "light only" is selected as the light source effect to apply.

[0357] In layer image 2200 in FIG. 28(A), effect 2850 of reflected light by first virtual light source 2830 and effect 2860 of reflected light by second virtual light source 2840 are applied to first object layer 2210a. For ease of explanation, in FIG. 28(A), effect 2850 of reflected light by first virtual light source 2830 and effect 2860 of reflected light by second virtual light source 2840 are each surrounded by dashed lines.

[0358] Another selection screen 2800b of an object to which a lighting effect is to be applied in FIG. 28(B) includes a first object selection area 2810a, a second object selection area 2820a, and the layer image 2200.

[0359] In first object selection field 2810a in Fig. 28(B), object Obj1 is selected as an object to which the lighting effect of first virtual light source 2830 is to be applied. Note that in Fig. 28(B), in lighting effect selection field 2710 (not shown) related to first virtual light source 2830, "light and shadow" is selected as the lighting effect to be applied.

[0360] In second object selection field 2820a in Figure 28(B), object Obj3 is selected as an object to which to apply the lighting effect of second virtual light source 2840. Note that in Figure 28(B), in lighting effect selection field 2710 (not shown) for second virtual light source 2840, "shadow only" is selected as the lighting effect to apply.

[0361] The generated image, which is the final output image, includes a shadow 2410a of the first object and a shadow 2410c of the third object applied as lighting effects by the first virtual light source 2830 and the second virtual light source 2840, as shown in FIG. 28(B), but does not include the first virtual light source 2830 and the second virtual light source 2840. This allows the user to easily apply lighting effects that are difficult to reproduce in an actual photography studio, such as different shadow directions for each object 2110.

[0362] The image generation system 1400 can execute the steps of another image generation flow 1500 based on the currently selected object or the currently selected lighting effect in response to a user's input selecting an object to apply a lighting effect to or a lighting effect to apply. As a result, while the user is selecting and inputting the object to which the lighting effect is to be applied or the lighting effect to be applied, the image generation system 1400 can switch and display information such as that shown in Figure 28(A) or Figure 28(B) depending on the currently selected object or the currently selected lighting effect.

[0363] FIG. 29 is an example of an explanatory diagram for explaining adjustment of a generated image. FIG. 29 shows an adjustment screen 2900 for a generated image 2910 . The adjustment screen 2900 for the generated image 2910 includes the generated image 2910, a light source position adjustment information input area 2920, and a light source effect adjustment prompt input area 2930.

[0364] The generated image 2910 is an image generated in the step (S1920) of generating a generated image applying a lighting effect by the fifth generation AI in another generation flow 1900 of a generated image using the fifth generation AI described using, for example, Figure 19.

[0365] The generated image 2910 includes a generated background 2911 consisting of a generated floor 2911a and a generated wall 2911b. In the generated image 2910, a shadow 2410a of the first object 2110a is applied to the first object 2110a as a lighting effect by the virtual light source 2400.

[0366] Light source position adjustment information input area 2920 accepts an input operation by the user to adjust the position where virtual light source 2400 is to be placed. The user can adjust the position where virtual light source 2400 is placed by moving the sliders left and right, up and down, and front and back.

[0367] As the user moves the slider between "left" and "right," the virtual light source 2400 moves left and right within the generated image 2910. As the user moves the slider between "up" and "down," virtual light source 2400 moves up or down within generated image 2910.

[0368] As the user moves the slider between "before" and "after," virtual light source 2400 changes in generated image 2910, as does positioned virtual light source 2940. For example, when the user moves the slider to "forward," virtual light source 2400 changes to virtual light source 2940a moved forward. In this case, there is no change in the two-dimensional position information corresponding to each point in the generated image assigned to virtual light source 2400.

[0369] For example, when the user moves the slider to "rear," virtual light source 2400 changes to virtual light source 2940b, which is moved backward. In this case, too, there is no change in the two-dimensional position information corresponding to each point in the generated image assigned to virtual light source 2400.

[0370] Lighting effect adjustment prompt input area 2930 accepts user input in the form of text to adjust aspects of virtual light source 2400 . The lighting effect adjustment prompt input field 2930 in FIG. 29 includes the following as a lighting effect adjustment prompt: First object Lighten the shadows Darken the shadows Warm the shadows The following is entered.

[0371] The lighting effect application module 1416 inputs these lighting adjustment prompts to the fifth generation AI, and the fifth generation AI makes adjustments to the generated image 2910 in accordance with the prompts (S1960). The image generation system 1400 requires the following adjustments, which require time and equipment costs to achieve simultaneously in actual photography in a photography studio, etc.: Lighten the shadow (weakening the shadow effect 1042), Darken the shadows, Change the color of the shadow Such adjustments can be performed on the adjustment screen 2900 of the generated image 2910 with relatively simple operations.

[0372] Lighting effect adjustment prompt input area 2930 can also accept prompts to recreate or refine the lighting effects that virtual light source 2400 has on the generated image.

[0373] In response to a user's selection of a lighting effect to apply, the image generation system 1400 can execute the steps of another image generation flow 1500 based on the currently selected lighting effect. As a result, the image generation system 1400 can switch between and display information such as that shown in Figure 27(A) or Figure 27(B) depending on the currently selected lighting effect while the user is selecting and inputting the lighting effect to be applied.

[0374] It should be noted that the present technology is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present technology is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0375] Each module disclosed in the above embodiments may be configured by combining multiple sub-modules. Furthermore, some or all of the operations or functions performed by one module may be performed or realized by another module.

[0376] The present application not only provides technology relating to the system described using the image generation system 20, 1400 as an example, but also, in another aspect, can provide technology relating to an image generation method. The present technology also provides a program for causing a server to execute each step in the image generation method. This program may consist of a single program or two or more subprograms. The program may also be a program for causing the server to execute any one or more of the steps.

[0377] The present technology also provides a program for causing a computer to execute each step in the image generation method. This program may consist of a single program, or may consist of two or more subprograms. The program may also be a program for causing a computer to execute any one or more of the steps.

[0378] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0379] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0380] Furthermore, one or more of the one or more servers and one or more user terminals 30 constituting the image generation system 20, 1400 may be installed in different countries. Furthermore, the image generation system 20, 1400 may be realized by one or more computers, any of which may be installed in different countries.

[0381] The above-described embodiments disclose at least the configurations described in the claims. The present invention also includes at least one of the following [Configuration 1] to [Configuration 20], or any combination of two or more thereof. [Configuration 1] an acquisition unit that acquires a basic object image, which is an image of an object; a light source setting unit that sets a virtual light source that irradiates the object with light; a background image generation unit that generates a final background image of a background having at least one of transmitted light from the object based on the virtual light source and a shadow of the object; an object image generation unit that generates a final object image of the object by applying at least one of light and shadow effects to the object based on the virtual light source; a composite image generation unit that generates a final composite image based on the final object image and the final background image; a composite image output unit that outputs the final composite image; having Image generation system. [Configuration 2] The method further includes a three-dimensional object generation unit that inputs the basic object image into a first generation AI and generates a three-dimensional object of the object, The light source setting unit, for the three-dimensional object generated using the first generation AI, the three-dimensional position of the virtual light source; the type of the virtual light source; a three-dimensional irradiation direction of the light from the virtual light source; the color of the light of the virtual light source; the number of virtual light sources; and setting the virtual light source to be adjustable in at least one of the following ways: 2. The image generation system according to claim 1. [Configuration 3] The object image generation unit generates an image based on the virtual light source. the reflection light effect of the reflected light on the three-dimensional object; the transmission light effect of the transmitted light on the three-dimensional object; a shadow effect within the contours of said three-dimensional object; generating a first intermediate object image, the first intermediate object image including at least one of: 3. The image generation system according to configuration 2. [Configuration 4] the object image generation unit generates, as the first intermediate object image, a projection image of the three-dimensional object that has at least one of transmitted light from the three-dimensional object and a shadow of the three-dimensional object and applies at least one of a light effect and a shadow effect to the three-dimensional object, based on the setting of the virtual light source for the three-dimensional object. 4. The image generation system according to configuration 3. [Configuration 5] The object image generation unit inputting the base object image and the projection image into a second generation AI; generating a second intermediate object image having at least one of transmitted light from the object and a shadow of the object based on the virtual light source, and applying at least one of a light effect and a shadow effect to the object based on the virtual light source; 5. The image generation system according to configuration 4. [Configuration 6] The apparatus further includes an object extraction unit that extracts the basic object image from the input image by performing at least one of extracting the object from the input image or removing a background from the input image, and reducing the effect of at least one of light and shadow on the object in the input image. 6. The image generation system according to configuration 5. [Configuration 7] the object extraction unit or the object image generation unit generates a third intermediate object image based on a process of extracting the object from the second intermediate object image or a process of removing a background from the second intermediate object image; The object image generation unit inputting the input image and the third intermediate object image into a third generation AI; generating, as the final object image, an image in which at least one of light and shadow effects is applied to the object based on the virtual light source, and in which the color of the object is different from that of the third intermediate object image; 7. The image generation system according to configuration 6. [Configuration 8] the background image generation unit generates the final background image having at least one of incident light on the object, transmitted light from the object, and a shadow outside the contour of the object, based on the virtual light source; 8. The image generation system according to any one of configurations 1 to 7. [Configuration 9] the background image generation unit generates the final background image in which a light-emitting object is arranged in a range including a position where the virtual light source is set. 8. The image generation system according to any one of configurations 1 to 7. [Configuration 10] The light source setting unit can individually adjust, with respect to the same virtual light source, at least one of the effects of the light or the shadow applied to the final object image and at least one of the transmitted light or the shadow of the final background image. 8. The image generation system according to any one of configurations 1 to 7. [Configuration 11] an object identification unit that identifies the object in an input image including the object and the background; a constraint condition image generation unit that generates a constraint condition image by a constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source; a lighting effect application unit that inputs the input image and the constraint condition image to a fifth generation AI and generates a generated image to which a lighting effect based on the virtual light source is applied; and The lighting effect is transmitted light from the object; the shadow of the object; the effect of light on said object; the effect of shadows on said object; At least one of 2. The image generation system according to claim 1. [Configuration 12] a depth information acquisition unit that acquires depth information representing depths of pixels that constitute the input image; a three-dimensional object generator that generates a three-dimensional object of the target object; and the depth information acquisition unit identifies a correspondence relationship between three-dimensional coordinate information of points or surfaces constituting the three-dimensional object and two-dimensional coordinates and depth information of pixels constituting the input image; the light source setting unit sets three-dimensional light source position information for the virtual light source, the three-dimensional light source position information being composed of two-dimensional light source position information on the input image and light source depth information which is depth information of the virtual light source associated with the light source position information; The constraint condition image generation unit the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information in the input image corresponding to the three-dimensional object and object depth information which is depth information associated with the object position information; and generating the constraint condition image representing the lighting effect of the virtual light source on the three-dimensional object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on the above. 12. The image generation system according to claim 11. [Configuration 13] a depth image generating unit that generates a depth image from the depth information acquired for the pixels of the input image; The constraint condition image generation unit the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information of the identified object in the input image and object depth information which is depth information associated with the object position information; and generating the constraint condition image representing the lighting effect of the virtual light source on the identified object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on the above. 13. The image generation system according to claim 12. [Configuration 14] the object identification unit identifies the number of objects that are identified in the input image; If the number of objects is a predetermined value, the composite image output unit outputs the final composite image; If the number of objects is greater than the predetermined value, the lighting effect application unit outputs the generated image; further comprising a main object selecting unit that selects a main object from the identified objects when the number of objects is greater than the predetermined value; For the main object, the constraint image generation unit generates the constraint image using the three-dimensional object; For objects other than the main object, the constraint condition image generation unit generates the constraint condition image using the depth image. 14. The image generation system according to claim 13. [Configuration 15] If the number of objects is two or more, The constraint condition image generation unit generates, for each of the objects, whether the lighting effect of the virtual light source is to be included in the constraint image; It is possible to switch between 15. The image generation system according to claim 14. [Configuration 16] When the number of virtual light sources to be set is two or more, The constraint condition image generation unit, for each of the virtual light sources, whether or not to use three-dimensional information of the virtual light source in the constraint condition generation process; It is possible to switch between 16. The image generation system according to claim 15. [Configuration 17] the objects identified in the input image include at least a first object and a second object; The lighting effect is light from the first object impinging on the second object; a shadow of the first object falling on the second object; the effect of light on the second object due to light from the first object; a shadow effect on the second object due to a shadow of the first object; At least one of 17. The image generation system according to claim 16. [Configuration 18] the constraint image is a one-channel image; 18. The image generation system according to claim 17. [Configuration 19] a background generation unit that generates a generated background different from the background of the input image; the background generation unit receives a background prompt that specifies a generation condition for the input image and the generated background; the background generation unit inputs the input image and the background prompt to a background generation AI, and obtains an image in which the background of the input image is replaced with the generated background; the light source effect application unit receives a light source prompt that specifies a lighting mode of the virtual light source; The lighting effect application unit inputs the image in which the background is replaced with the generated background, the constraint condition image, and the light source prompt to the fifth generation AI, and generates the generated image. 19. The image generation system according to claim 18. [Configuration 20] The light source setting unit inputting two-dimensional position information on the input image; an input for assigning depth information to the input position information; Based on setting the position of the virtual light source; 20. The image generation system according to claim 19. [Configuration 21] the light source setting unit displays a light source object representing the set virtual light source within the input image; The light source setting unit changes at least one of size, shape, shade, brightness, and type in response to an input that assigns the depth information, and displays the light source object. 21. The image generation system according to claim 20. [Configuration 22] The light source setting unit When the set virtual light source is located in front of the object, When the set virtual light source is located inside the object, The set virtual light source is located behind the object; and displaying the light source object by changing at least one of the size, shape, shade, brightness, and type. 22. The image generation system according to claim 21. [Configuration 23] A program for causing a computer to realize the functions of each part of the image generation system described in any one of configurations 1 to 22. [Configuration 24] A basic object image is obtained, which is an image of the object; setting a virtual light source that irradiates the object with light; generating a final background image of a background having at least one of transmitted light from the object or a shadow of the object based on the virtual light source; generating a final object image of the object by applying at least one of light and shadow effects to the object based on the virtual light source; generating a final composite image based on the final object image and the final background image; outputting the final composite image. Image generation method. [Configuration 25] inputting the basic object image into a first generation AI to generate a three-dimensional object of the object; For the three-dimensional object generated using the first generation AI, the three-dimensional position of the virtual light source; the type of the virtual light source; a three-dimensional irradiation direction of the light from the virtual light source; the color of the light of the virtual light source; the number of virtual light sources; and setting the virtual light source to be adjustable in at least one of the following ways: 25. The image generating method according to claim 24. [Configuration 26] generating, as a first intermediate object image, a projected image of the three-dimensional object that has at least one of transmitted light from the three-dimensional object and a shadow of the three-dimensional object and applies at least one of a light effect and a shadow effect to the three-dimensional object based on the setting of the virtual light source for the three-dimensional object; inputting the base object image and the projection image into a second generation AI; generating a second intermediate object image having at least one of transmitted light from the object and a shadow of the object based on the virtual light source, and applying at least one of a light effect and a shadow effect to the object based on the virtual light source; generating a third intermediate object image based on the extraction of the object from the second intermediate object image or the removal of the background from the second intermediate object image; inputting the input image and the third intermediate object image into a third generation AI; generating, as the final object image, an image in which at least one of light and shadow effects is applied to the object based on the virtual light source, and in which the color of the object is different from that of the third intermediate object image; 26. The image generating method according to claim 25. [Configuration 27] generating the final background image having at least one of incident light on the object, transmitted light from the object, and shadow outside the outline of the object based on the virtual light source; 27. The image generating method according to any one of aspects 24 to 26. [Configuration 28] identifying the object in an input image comprising the object and the background; obtaining depth information representing the depth of pixels constituting the input image; generating a three-dimensional object of the subject; Identifying a correspondence between three-dimensional coordinate information of points or surfaces constituting the three-dimensional object and two-dimensional coordinates and depth information of pixels constituting the input image; setting three-dimensional light source position information for the virtual light source, the three-dimensional light source position information being composed of two-dimensional light source position information on the input image and light source depth information which is depth information of the virtual light source associated with the light source position information; A constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source is performed. the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information in the input image corresponding to the three-dimensional object and object depth information which is depth information associated with the object position information; represents the lighting effect of the virtual light source on the three-dimensional object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on Generate a constraint image; The input image and the constraint condition image are input to a fifth generation AI, and a generated image is generated by applying a lighting effect based on the virtual light source; The lighting effect is transmitted light from the object; the shadow of the object; the effect of light on said object; the effect of shadows on said object; At least one of 25. The image generating method according to claim 24. [Configuration 29] identifying the object in an input image comprising the object and the background; obtaining depth information representing the depth of pixels constituting the input image; generating a depth image from the acquired depth information; A constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source is performed. the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information of the identified object in the input image and object depth information which is depth information associated with the object position information; represents the lighting effect by the virtual light source on the identified object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, Generate a constraint image; The input image and the constraint condition image are input to a fifth generation AI, and a generated image is generated by applying a lighting effect based on the virtual light source. 29. The image generating method according to claim 28. [Configuration 30] identifying an object count, which is the number of objects identified in the input image; If the number of objects is a predetermined value, the composite image output unit outputs the final composite image; If the number of objects is greater than the predetermined value, the lighting effect application unit outputs the generated image; If the number of objects is greater than the predetermined value, selecting a main object from the identified objects; generating the constraint image for the primary subject using the three-dimensional object; For objects other than the main object, the constraint image is generated using the depth image. 30. The image generating method according to claim 29. [Configuration 31] obtaining a background prompt specifying the input image and the generated background; inputting the input image and the background prompt into a background generation AI to obtain an image in which the background of the input image is replaced with a generated background different from the background of the input image; obtaining a light source prompt that specifies a lighting aspect of the virtual light source; inputting the image in which the background has been replaced with the generated background, the constraint image, and the light source prompt into the fifth generation AI to generate the generated image; 31. The image generation system of claim 30. [Configuration 32] Acquire input of two-dimensional position information on the input image; acquiring an input for assigning depth information to the input position information; setting the position of the virtual light source based on the two-dimensional position information and the depth information. 32. The image generating method according to claim 31. [Configuration 33] displaying a light source object representing the set virtual light source within the input image; and displaying the light source object by changing at least one of size, shape, shade, brightness, and type in response to an input for assigning the depth information. 33. The image generating method according to claim 32. [Explanation of symbols]

[0382] 211...acquisition module, 214...light source setting module, 215...background image generation module, 216...object image generation module, 217...synthesized image generation module, 218...output module, 1411...acquisition module, 1413...constraint condition image generation module, 1414...light source setting module, 1415...replaced background image generation module, 1416...light source effect application module, 1417...output module

Claims

1. an acquisition unit that acquires a basic object image, which is an image of an object; a light source setting unit that sets a virtual light source that irradiates the object with light; a background image generation unit that generates a final background image of a background having at least one of transmitted light from the object based on the virtual light source and a shadow of the object; an object image generation unit that generates a final object image of the object by applying at least one of light and shadow effects to the object based on the virtual light source; a composite image generation unit that generates a final composite image based on the final object image and the final background image; a composite image output unit that outputs the final composite image; having Image generation system.

2. The method further includes a three-dimensional object generation unit that inputs the basic object image to a first generation AI and generates a three-dimensional object of the object; The light source setting unit, for the three-dimensional object generated using the first generation AI, the three-dimensional position of the virtual light source; the type of the virtual light source; a three-dimensional irradiation direction of the light from the virtual light source; the color of the light of the virtual light source; the number of virtual light sources; and setting the virtual light source to be adjustable in at least one of the following ways: The image generation system of claim 1 .

3. The object image generation unit generates an image based on the virtual light source. the reflection light effect of the reflected light on the three-dimensional object; a transmission light effect of the transmitted light on the three-dimensional object; a shadow effect within the contours of said three-dimensional object; generating a first intermediate object image, the first intermediate object image including at least one of: The image generation system of claim 2 .

4. the object image generation unit generates, as the first intermediate object image, a projection image of the three-dimensional object, based on the setting of the virtual light source for the three-dimensional object, the projection image having at least one of transmitted light from the three-dimensional object and a shadow of the three-dimensional object, and applying at least one of a light effect and a shadow effect to the three-dimensional object. The image generation system of claim 3 .

5. The object image generation unit inputting the base object image and the projection image into a second generating AI; generating a second intermediate object image having at least one of transmitted light from the object and a shadow of the object based on the virtual light source, and applying at least one of a light effect and a shadow effect to the object based on the virtual light source; The image generation system of claim 4 .

6. The image processing device further includes an object extraction unit that extracts the basic object image from the input image by performing at least one of extracting the object from the input image or removing a background from the input image, and reducing the effect of at least one of light and shadow on the object in the input image. The image generation system of claim 5 .

7. the object extraction unit or the object image generation unit generates a third intermediate object image based on a process of extracting the object from the second intermediate object image or a process of removing a background from the second intermediate object image; The object image generation unit inputting the input image and the third intermediate object image into a third generating AI; generating, as the final object image, an image in which at least one of a light effect and a shadow effect is applied to the object based on the virtual light source, and in which the color of the object is different from that of the third intermediate object image; The image generation system of claim 6.

8. the background image generation unit generates the final background image having at least one of incident light on the object, transmitted light from the object, and a shadow outside the contour of the object, based on the virtual light source; 8. An image generation system according to any one of claims 1 to 7.

9. the background image generation unit generates the final background image in which a light-emitting object is arranged in a range including a position where the virtual light source is set.

8. An image generation system according to any one of claims 1 to 7.

10. The light source setting unit can individually adjust, with respect to the same virtual light source, at least one of the effects of the light or the shadow applied to the final object image and at least one of the transmitted light or the shadow of the final background image.

8. An image generation system according to any one of claims 1 to 7.

11. an object identification unit that identifies the object in an input image including the object and the background; a constraint image generation unit that generates a constraint image by a constraint generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source; a lighting effect application unit that inputs the input image and the constraint condition image to a fifth generation AI and generates a generated image to which a lighting effect based on the virtual light source is applied; and The lighting effect is transmitted light from the object; the shadow of the object; the effect of light on said object; the effect of shadows on said object; At least one of The image generation system of claim 1 .

12. a depth information acquisition unit that acquires depth information representing depths of pixels that constitute the input image; a three-dimensional object generator that generates a three-dimensional object of the target object; and the depth information acquisition unit identifies a correspondence relationship between three-dimensional coordinate information of points or surfaces constituting the three-dimensional object and two-dimensional coordinates and depth information of pixels constituting the input image; the light source setting unit sets three-dimensional light source position information for the virtual light source, the three-dimensional light source position information being composed of two-dimensional light source position information on the input image and light source depth information which is depth information of the virtual light source associated with the light source position information; The constraint condition image generation unit the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information in the input image corresponding to the three-dimensional object and object depth information which is depth information associated with the object position information; generating the constraint condition image representing the lighting effect caused by the virtual light source on the three-dimensional object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on the The image generation system of claim 11.

13. a depth image generating unit that generates a depth image from the depth information acquired for the pixels of the input image; The constraint condition image generation unit the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information of the identified object in the input image and object depth information which is depth information associated with the object position information; and generating the constraint condition image representing the lighting effect of the virtual light source on the identified object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on the above. The image generation system of claim 12.

14. the object identification unit identifies the number of objects that are identified in the input image; If the number of objects is a predetermined value, the composite image output unit outputs the final composite image; If the number of objects is greater than the predetermined value, the lighting effect application unit outputs the generated image; further comprising a main object selecting unit that selects a main object from the identified objects when the number of objects is greater than the predetermined value; Regarding the main object, the constraint condition image generation unit generates the constraint condition image using the three-dimensional object; For objects other than the main object, the constraint condition image generation unit generates the constraint condition image using the depth image. The image generation system of claim 13.

15. If the number of objects is two or more, The constraint condition image generation unit generates, for each of the objects, whether the lighting effect of the virtual light source is to be included in the constraint image; It is possible to switch between The image generation system of claim 14.

16. When the number of the virtual light sources to be set is two or more, The constraint condition image generation unit, for each of the virtual light sources, whether or not three-dimensional information of the virtual light source is to be used in the constraint condition generation process; It is possible to switch between 16. The image generation system of claim 15.

17. the objects identified in the input image include at least a first object and a second object; The lighting effect is light from the first object impinging on the second object; a shadow of the first object striking the second object; the effect of light on the second object due to light from the first object; a shadow effect on the second object due to a shadow of the first object; At least one of 17. The image generation system of claim 16.

18. the constraint image is a one-channel image; 18. The image generation system of claim 17.

19. a background generation unit that generates a generated background different from the background of the input image; the background generation unit receives a background prompt that specifies a generation condition for the input image and the generated background; The background generation unit inputs the input image and the background prompt to a background generation AI, and obtains an image in which the background of the input image is replaced with the generated background; the light source effect application unit receives a light source prompt that specifies a lighting mode of the virtual light source; The lighting effect application unit inputs the image in which the background is replaced with the generated background, the constraint condition image, and the light source prompt to the fifth generation AI, and generates the generated image.

20. The image generation system of claim 18.

20. The light source setting unit inputting two-dimensional position information on the input image; an input for assigning depth information to the input position information; Based on setting the position of the virtual light source; 20. The image generation system of claim 19.

21. the light source setting unit displays a light source object representing the set virtual light source within the input image; The light source setting unit changes at least one of size, shape, shade, brightness, and type in response to an input for assigning the depth information, and displays the light source object.

21. The image generation system of claim 20.

22. The light source setting unit When the set virtual light source is located in front of the object, When the set virtual light source is located inside the object, The set virtual light source is located behind the object; and displaying the light source object by changing at least one of the size, shape, shade, brightness, and type.

22. The image generation system of claim 21.

23. A program for causing a computer to realize the functions of each unit of the image generating system according to any one of claims 1 to 22.

24. A basic object image is obtained, which is an image of the object; setting a virtual light source that irradiates the object with light; generating a final background image of a background having at least one of transmitted light from the object or a shadow of the object based on the virtual light source; generating a final object image of the object by applying at least one of light and shadow effects to the object based on the virtual light source; generating a final composite image based on the final object image and the final background image; outputting the final composite image. Image generation method.

25. inputting the basic object image into a first generation AI to generate a three-dimensional object of the object; For the three-dimensional object generated using the first generation AI, the three-dimensional position of the virtual light source; the type of the virtual light source; a three-dimensional irradiation direction of the light from the virtual light source; the color of the light of the virtual light source; the number of virtual light sources; and setting the virtual light source to be adjustable in at least one of the following ways:

25. The image generating method of claim 24.

26. generating, as a first intermediate object image, a projection image of the three-dimensional object having at least one of transmitted light from the three-dimensional object and a shadow of the three-dimensional object based on the setting of the virtual light source for the three-dimensional object, and applying at least one of a light effect and a shadow effect to the three-dimensional object; inputting the base object image and the projection image into a second generating AI; generating a second intermediate object image having at least one of transmitted light from the object and a shadow of the object based on the virtual light source, and applying at least one of a light effect and a shadow effect to the object based on the virtual light source; generating a third intermediate object image based on the extraction of the object from the second intermediate object image or the removal of the background from the second intermediate object image; inputting the input image and the third intermediate object image into a third generating AI; generating, as the final object image, an image in which at least one of a light effect and a shadow effect is applied to the object based on the virtual light source, and in which the color of the object is different from that of the third intermediate object image; 26. The image generating method of claim 25.

27. generating the final background image having at least one of incident light on the object, transmitted light from the object, and shadow outside the outline of the object based on the virtual light source; 27. The image generating method according to any one of claims 24 to 26.

28. identifying the object in an input image comprising the object and the background; obtaining depth information representing the depth of pixels constituting the input image; generating a three-dimensional object of the subject; Identifying a correspondence between three-dimensional coordinate information of points or surfaces constituting the three-dimensional object and two-dimensional coordinates and depth information of pixels constituting the input image; setting three-dimensional light source position information for the virtual light source, the three-dimensional light source position information being composed of two-dimensional light source position information on the input image and light source depth information which is depth information of the virtual light source associated with the light source position information; A constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source is performed. the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information in the input image corresponding to the three-dimensional object and object depth information which is depth information associated with the object position information; represents the lighting effect of the virtual light source on the three-dimensional object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, based on Generate a constraint image; The input image and the constraint condition image are input to a fifth generation AI, and a generated image is generated by applying a lighting effect based on the virtual light source; The lighting effect is transmitted light from the object; the shadow of the object; the effect of light on said object; the effect of shadows on said object; At least one of 25. The image generating method of claim 24.

29. identifying the object in an input image comprising the object and the background; obtaining depth information representing the depth of pixels constituting the input image; generating a depth image from the acquired depth information; A constraint condition generation process based on three-dimensional information about the object identified in the input image and three-dimensional information about the virtual light source is performed. the three-dimensional light source position information; three-dimensional object position information composed of two-dimensional object position information of the identified object in the input image and object depth information which is depth information associated with the object position information; represents the lighting effect of the virtual light source on the identified object, the lighting effect being calculated based on the three-dimensional light source position information and the three-dimensional object position information, Generate a constraint image; The input image and the constraint condition image are input to a fifth generation AI, and a generated image is generated to which a lighting effect based on the virtual light source is applied.

29. The image generating method of claim 28.

30. identifying an object count, which is the number of objects identified in the input image; If the number of objects is a predetermined value, the composite image output unit outputs the final composite image; If the number of objects is greater than the predetermined value, the lighting effect application unit outputs the generated image; If the number of objects is greater than the predetermined value, selecting a main object from the identified objects; generating the constraint image for the main subject using the three-dimensional object; For objects other than the main object, the constraint image is generated using the depth image.

30. The image generating method of claim 29.

31. obtaining a background prompt specifying the input image and the generated background; inputting the input image and the background prompt into a background generation AI to obtain an image in which the background of the input image is replaced with a generated background different from the background of the input image; obtaining a light source prompt that specifies a lighting aspect of the virtual light source; inputting the image in which the background has been replaced with the generated background, the constraint image, and the light source prompt into the fifth generation AI to generate the generated image; 31. The image generating method of claim 30.

32. Acquire input of two-dimensional position information on the input image; acquiring an input for assigning depth information to the input position information; setting a position of the virtual light source based on the two-dimensional position information and the depth information; 32. The image generating method of claim 31.

33. displaying a light source object representing the set virtual light source within the input image; and displaying the light source object by changing at least one of size, shape, shade, brightness, and type in response to an input for assigning the depth information.

33. The image generating method of claim 32.

Images