Method, server and computer program for generating relit image on basis of object image

Abstract

A method for generating a relit image on the basis of an object image, according to various embodiments of the present invention for solving the described problem, is disclosed. The method comprises the steps of: acquiring a source original image; acquiring image characteristic information on the basis of the source original image; and generating a relit image on the basis of the source original image, the image characteristic information and target lighting information, wherein the relit image can be an image reflecting realistic human skin tone, texture and shadow effects under target lighting conditions, the image having lighting effects different from that of the source original image.

Description

Method for generating a re-illuminated image based on an object image, server and computer program

[0001] Various embodiments of the present invention relate to a method for relighting an image, and more particularly, to a method, server, and computer program for changing the lighting conditions of a digital image.

[0002]

[0003] Digital image processing and computer graphics are rapidly advancing in modern society, leading to a continuously increasing demand for high-quality digital content. Amidst this trend, relighting techniques, which modify the lighting conditions of portrait photography, are receiving particular attention. Relighting techniques play a crucial role in reproducing portraits in more realistic and diverse lighting environments in diverse applications, including filmmaking, digital art, and virtual reality.

[0004] In fields such as virtual reality and augmented reality, combining individual object images captured under various lighting conditions to create a unified scene can be essential. Applying relighting techniques can transcend the original lighting conditions of each object and provide a natural lighting consistency across the entire scene. This can provide a more immersive experience for users and significantly enhance the realism of virtual environments.

[0005] However, particularly in portraiture, the task of realistically transforming subjects under various lighting conditions remains a technically challenging problem with poorly defined parameters due to complex lighting effects and the diverse characteristics of the subject. Conventional methodologies have actively leveraged information from 3D facial models, exploited the inherent characteristics of the image, and in some cases interpreted the problem as a style transition. However, these methods have shown limitations in adequately handling subtle phenomena such as complex non-Lambertian effects.

[0006] In these circumstances, light stage technology has been proposed as a method to capture the diversity of lighting. By precisely recording a subject's response under various lighting conditions, light stage technology has shown great promise in capturing the detailed reflective characteristics of a subject under varying lighting conditions. However, practical application of this technology presents practical challenges, including the need for highly specialized equipment and the significant time and effort required for large-scale data collection.

[0007] Recent advances in deep learning technology have offered solutions to these problems. Relighting methods utilizing neural networks trained on light stage data have made significant progress in automating the relighting of portraits based on lighting changes. However, this approach still has limitations in fully simulating complex light interactions, such as the non-Lambertian effect.

[0008] Accordingly, there is a continuous demand in the industry for research and development of more advanced modeling techniques and algorithms.

[0009] (Prior art literature)

[0010] (Patent Document)

[0011] Patent Publication No. 10-2022-0117324

[0012]

[0013] The problem to be solved by the present invention is to provide a method for changing the lighting conditions of a portrait image in a natural and realistic manner in response to the aforementioned background technology.

[0014] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0015]

[0016] A method for generating a re-illumination image based on an object image according to one embodiment of the present invention for solving the above-described problem is disclosed. The method may include the steps of acquiring a source original image, acquiring image characteristic information based on the source original image, and generating a re-illumination image based on the source original image, the image characteristic information, and target illumination information.

[0017] In an alternative embodiment, the re-illuminated image may be characterized as an image in which the lighting effect is changed compared to the source original image, reflecting the characteristics of the object under the target lighting conditions.

[0018] In an alternative embodiment, the re-lit image may be characterized as an image that reflects realistic human skin tones, textures, and shadow effects under target lighting conditions.

[0019] In an alternative embodiment, the step of obtaining the image characteristic information includes the step of extracting a foreground image from the source original image through a foreground extraction model and the step of performing reverse rendering on the extracted foreground image to obtain the image characteristic information, wherein the image characteristic information may include at least one of a normal map, an albedo map, information about roughness, information about reflectivity, and lighting condition information as information about physical and optical properties of a surface corresponding to the foreground image.

[0020] In an alternative embodiment, the step of performing the reverse rendering to obtain the image characteristic information may include the step of deriving the normal map corresponding to the source original image using a normal map generation model, the step of deriving the lighting condition information corresponding to the source original image using a lighting condition inference model, the step of generating diffuse shading based on the normal map and the lighting condition information, the step of generating the albedo map based on the diffuse shading, and the step of obtaining information about the roughness and the reflectance corresponding to the source original image based on the source original image, the normal map, and the albedo map.

[0021] In an alternative embodiment, the step of generating the albedo map based on the diffuse reflection shading includes the steps of processing the source original image and the diffuse reflection shading as inputs to a diffuse reflection model to obtain a diffuse reflection render, and the step of generating the albedo map based on the diffuse reflection shading and the diffuse reflection render, wherein the diffuse reflection model includes a network function pre-trained to output a diffuse reflection render based on the diffuse reflection shading corresponding to the source original image, and the diffuse reflection render may be characterized as a final image generated by combining the diffuse reflection shading and the albedo map, and being an image in which a diffuse reflection effect in which light is evenly spread in all directions from a surface is visually expressed.

[0022] In an alternative embodiment, the step of obtaining information about the roughness and the reflectivity corresponding to the source original image based on the source original image, the normal map, and the albedo map includes the step of processing the source original image, the normal map, and the albedo map as inputs to a specular reflection model to obtain information about the roughness and the reflectivity, wherein the specular reflection model may include a network function pre-learned to obtain specular reflection information including information about the roughness and the reflectivity by inferring specular reflection elements of a surface based on microsurface theory.

[0023] In an alternative embodiment, the step of generating a re-lighting image includes the steps of generating a diffuse render and a specular render based on the normal map, the albedo map, the information about the roughness, the information about the reflectivity, and the target illumination information, the step of generating an initial re-lighting image based on the diffuse render and the specular render, and the step of processing the initial re-lighting image as an input of a rendering model to generate the re-lighting image, wherein the rendering model is a pre-trained neural network model based on an integrated loss related to a weighted sum of a reconstruction loss, a perceptual loss, an adversaria loss, and a specular loss, wherein the reconstruction loss is a loss related to a pixel-level difference between an original image and a result image predicted corresponding to the original image, the perceptual loss is a loss related to a feature difference between the original image and the result image, the adversaria loss is a loss related to a difference between the original image and the result image judged by a discriminator model, and the specular loss is a loss related to a difference between the original image and the result image. The loss may be a weighted loss of the above reconstruction loss using the specular information.

[0024] In an alternative embodiment, the method further comprises the steps of constructing a learning data set based on a plurality of optical stage data, generating a plurality of reconstructed images corresponding to each of a plurality of source original images included in the learning data set by utilizing an image reconstruction model, and augmenting the learning data set based on the plurality of reconstructed images, wherein the image reconstruction model may be trained to generate a reconstructed image corresponding to an input image by reflecting the perceptual loss and the adversarial loss to a reconstruction loss regarding a difference between each source original image and each reconstructed image corresponding to each source original image.

[0025] In an alternative embodiment, the image reconstruction model may be characterized in that it is trained to generate multiple reconstructed images by utilizing dynamic masking that dynamically adjusts one or more patches of different sizes to different regions of the input image.

[0026] According to another embodiment of the present invention, a server for performing a method for generating a re-illumination image based on an object image is disclosed. The server includes a memory storing one or more instructions and a processor for executing one or more instructions stored in the memory, wherein the processor can perform the above-described method for generating a re-illumination image based on an object image by executing the one or more instructions.

[0027] According to another embodiment of the present invention, a computer program stored on a computer-readable recording medium is disclosed. The computer program is coupled with a computer, which is hardware, to perform a method for generating a re-illumination image based on an object image.

[0028] Other specific details of the present invention are included in the detailed description and drawings.

[0029]

[0030] According to various embodiments of the present invention, real-time image processing and lighting adjustment can be facilitated by generating a re-illuminated image corresponding to the original source image. This allows users to preview the visual effects of an image under various lighting conditions, which can be particularly useful in fields such as video production, game development, and virtual reality. In other words, the ability to quickly simulate various lighting conditions not only contributes to increased productivity but also expands the scope of creative visual expression.

[0031] Furthermore, the present invention utilizes an architecture that integrates a physics-based approach and a self-supervised pre-training framework to enhance the quality of re-illuminated images while maintaining the consistency of the original image. This configuration generates images that are closer to reality, enhancing the user experience and providing more realistic visual results.

[0032] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0033]

[0034] FIG. 1 is an exemplary diagram schematically illustrating a system for implementing a method for generating a re-illumination image based on an object image according to one embodiment of the present invention.

[0035] Figure 2 illustrates exemplary images re-illuminated through various lighting environments.

[0036] FIG. 3 is a hardware configuration diagram of a server that performs a method for generating a re-illumination image based on an object image according to one embodiment of the present invention.

[0037] FIG. 4 illustrates a flowchart exemplarily showing a method for generating a re-illumination image based on an object image related to one embodiment of the present invention.

[0038] FIG. 5 is an exemplary diagram illustrating an original image and essential properties corresponding to the original image in accordance with one embodiment of the present invention.

[0039] FIG. 6 is a diagram exemplarily illustrating an architecture for performing a method for generating a re-illumination image based on an object image related to one embodiment of the present invention.

[0040] FIG. 7 is an exemplary diagram illustrating a process for generating a re-illumination image related to one embodiment of the present invention.

[0041] FIG. 8 illustrates images to which a relighting effect is applied according to a method for generating a relighting image based on an object image according to one embodiment of the present invention.

[0042] FIG. 9 is an exemplary diagram illustrating dynamic masking of an image reconstruction model related to one embodiment of the present invention.

[0043]

[0044] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate an understanding of the present invention. However, it will be apparent that these embodiments may be practiced without these specific details.

[0045] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0046] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0047] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0048] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0049] The description of the disclosed embodiments is provided to enable those skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Therefore, the present invention is not limited to the disclosed embodiments. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0050] In this specification, the term "computer" refers to any type of hardware device including at least one processor, and may also be understood to encompass software components operating on the hardware device, depending on the embodiment. For example, the term "computer" may be understood to encompass, but is not limited to, smartphones, tablet PCs, desktops, laptops, and all user clients and applications running on each device.

[0051] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0052] Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited thereto, and at least some of each step may be performed by different devices depending on the embodiment.

[0053]

[0054] FIG. 1 is an exemplary diagram schematically illustrating a system for implementing a method for generating a re-illumination image based on an object image according to one embodiment of the present invention.

[0055] As illustrated in FIG. 1, a system according to embodiments of the present invention may include a server (100), a user terminal (200), an external server (300), and a network (400). The components illustrated in FIG. 1 are exemplary, and additional components may exist or some of the components illustrated in FIG. 1 may be omitted. The server (100), the external server (300), and the user terminal (200) according to embodiments of the present invention may exchange data for the system according to embodiments of the present invention via the network (400).

[0056] The network (400) according to embodiments of the present invention can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0057] Additionally, the network (400) presented herein can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0058] The network (400) according to embodiments of the present invention may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network (400) may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth. The technologies described in this specification may be used not only in the networks mentioned above, but also in other networks.

[0059] According to an embodiment of the present invention, a server (100) (hereinafter referred to as 'server (100)') that performs a method for generating a re-illumination image based on an object image can generate and provide a re-illumination image that reflects changes in lighting conditions for a source original image. Specifically, the server (100) can extract essential image properties (i.e., image characteristic information) from the source original image and generate a re-illumination image that satisfies target lighting conditions based on the extracted properties. This process may include separating the foreground and background, extracting essential characteristic information such as a normal map and an albedo map, and re-rendering the image under target lighting based on the extracted information. That is, the server (100) can implement a realistic re-illumination effect of an object under lighting conditions specified by a user through the aforementioned processing process.

[0060] In the present invention, a re-lit image may refer to an image modified to reflect new lighting conditions specified by a user. The re-lit image may be an image that reflects the characteristics of an object under the target lighting conditions. For example, the object may be a person or an object, but is not limited thereto, and may include various subjects such as landscapes or animals. For example, a re-lit image may be a result that naturally expresses a person's skin tone, texture, shadows, etc. under conditions where the direction, intensity, color, etc. of the lighting have been changed. For example, a re-lit image, as illustrated in FIG. 2, allows the user to experience realistic changes under various lighting conditions compared to the original image. This may be applied to a portrait image to simulate a lighting environment that does not actually exist or to create a specific time zone or special atmosphere.

[0061] In one embodiment, the server (100) can generate reconstructed images with altered illumination according to the user's request by utilizing an architecture that integrates a physics-based approach and a self-supervised pre-training framework.

[0062] In a specific embodiment, the server (100) can precisely simulate the interaction of light with surface microfacets that take into account spatially varying roughness and reflectivity through a re-illumination image generation model, which is a physics-based model. For example, the re-illumination image generation model may be a physics-based model based on the Cook-Torrance reflection model, and can provide a re-illumination image that reflects a high level of realism by maximizing realism.

[0063] In addition, the server (100) can implement a framework utilizing an image reconstruction model to overcome the limitations of optical stage data, which is generally difficult to acquire. For example, the image reconstruction model can adopt a Multi-Masked Autoencoder (MMAE) or a similar self-supervised learning method, thereby enabling learning from unlabeled data. MMAE can learn important features from various parts of an input image by applying various masks, and based on this, can achieve precise image reconstruction. Through this, the server (100) can extract deep image characteristics without data labeling and utilize these in the relighting image generation process, thereby generating more sophisticated and realistic results. In other words, the server (100) can perform learning from unlabeled data using an image reconstruction model, thereby improving the realism of the final relighting image generated. This has the advantage of enabling specific relighting tasks to be performed more effectively through fine-tuning without relying on optical stage data. In other words, the server (100) can learn from unlabeled data through a self-supervised pre-training framework and, based on this, provide realistically re-illuminated images tailored to user needs. Consequently, the server (100) can utilize an architecture that integrates a physics-based approach and a self-supervised pre-training framework to automate the re-illumination of images that reflect lighting changes in various real-world scenarios, thereby providing users with enhanced realism or realism. A more detailed description of the object image-based re-illumination image generation method performed by the server will be described below with reference to FIG. 4.

[0064] Although only one server (100) is illustrated in FIG. 1, it will be apparent to those skilled in the art that more servers may also fall within the scope of the present invention and that the server (100) may include additional components. Specifically, the server (100) may be comprised of multiple computing devices. In other words, a set of multiple nodes may constitute the server (100).

[0065] According to one embodiment of the present invention, the server (100) may be a server that provides a cloud computing service. More specifically, the server (100) may be a server that provides a cloud computing service, a type of Internet-based computing that processes information using another computer connected to the Internet rather than the user's computer. The cloud computing service may be a service that stores data on the Internet and allows users to access it anytime and anywhere via an Internet connection without having to install necessary data or programs on their own computers. Furthermore, the data stored on the Internet can be easily shared and transmitted with simple operations and clicks. Furthermore, the cloud computing service may be a service that not only stores data on a server on the Internet, but also allows users to perform desired tasks using the functions of web-based application programs without having to install separate programs. Furthermore, the cloud computing service may be a service that allows multiple people to simultaneously share and work on documents. Furthermore, the cloud computing service may be implemented in at least one of the following forms: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, and a container-based cloud server. In other words, the server (100) of the present invention may be implemented in at least one of the above-described cloud computing services. The specific description of the cloud computing service described above is merely an example, and may include any platform that constructs the cloud computing environment of the present invention.

[0066] The user terminal (200) according to an embodiment of the present invention may refer to any type of node(s) in a system having a mechanism for communicating with the server (100). The user terminal (200) is a terminal capable of receiving an optimized re-illumination image through information exchange with the server (100), and may refer to a terminal carried by a user.

[0067] For example, a user terminal (200) can perform tasks such as uploading an original image to a server (100) via a smartphone or tablet, and downloading an image with changed lighting (i.e., a re-illuminated image) from the server. During this process, the user can check in real time the image with highly realistic re-illumination effects applied, processed by the server (100), and experiment with various lighting settings as needed.

[0068] According to an embodiment, when the server (100) receives a source original image (e.g., an original portrait image) from a user terminal (200), it can extract essential image property information based on the extracted image property information and generate a re-illuminated image according to target lighting conditions. For example, the source original image uploaded from the user terminal (200) is first separated into its main object through a foreground extraction model within the server (100), and then undergoes a reverse rendering process to identify essential properties such as a normal map, an albedo map, roughness, and reflectance. This information and new lighting conditions selected by the user are input into a re-illuminated image generation model, so that a re-illuminated image with a realistic re-illuminated effect applied under the final modified lighting conditions can be generated. Through this process, the user can simulate an object under various lighting environments, which can be utilized for artistic creation or practical purposes.

[0069] A user terminal (200) may refer to any type of entity(ies) in a system having a mechanism for communicating with a server (100). For example, such a user terminal (200) may include a personal computer (PC), a notebook, a mobile terminal, a smart phone, a tablet PC, a wearable device, etc., and may include all types of terminals capable of connecting to a wired / wireless network. In addition, the user terminal (200) may include any server implemented by at least one of an agent, an Application Programming Interface (API), and a plug-in. In addition, the user terminal (200) may include an application source and / or a client application.

[0070] In one embodiment, the external server (300) may be connected to the server (100) via a network (400), and may provide various information / data necessary for the server (100) to perform a method for generating a re-illumination image based on an object image, or may receive, store, and manage result data derived by performing the method for generating a re-illumination image based on an object image. For example, the external server (300) may be a storage server separately provided outside the server (100), but is not limited thereto.

[0071] Additionally, in the embodiment, information stored in the external server (300) may be utilized as learning data, verification data, and test data for training the artificial neural network of the present invention. That is, the external server (300) may store data for training the artificial intelligence model of the present invention. The server (100) of the present invention may construct multiple learning data sets based on the information received from the external server (300). The server (100) may generate multiple artificial intelligence models by performing learning on one or more network functions through each of the multiple learning data sets.

[0072] The external server (300) may be a digital device equipped with a processor, memory, and computing power, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The external server (300) may be a web server that processes services. The types of servers described above are merely examples, and the present disclosure is not limited thereto. Hereinafter, with reference to FIG. 3, the hardware configuration of the server (100) that performs the method for generating a re-illumination image based on an object image will be described.

[0073]

[0074] FIG. 3 is a hardware configuration diagram of a server that performs a method for generating a re-illumination image based on an object image according to one embodiment of the present invention.

[0075] Referring to FIG. 3, a server (100) that performs a method for generating a re-illumination image based on an object image according to an embodiment of the present invention may include one or more processors (110), a memory (120) that loads a computer program (151) executed by the processor (110), a bus (130), a communication interface (140), and a storage (150) that stores the computer program (151). Here, only components related to the embodiment of the present invention are illustrated in FIG. 3. Therefore, a person skilled in the art to which the present invention pertains may understand that other general components may be further included in addition to the components illustrated in FIG. 3.

[0076] According to one embodiment of the present invention, the processor (110) can typically process the overall operation of the server (100). The processor (110) can process signals, data, information, etc. input or output through the components described above, or can operate an application program stored in the memory (120) to provide or process appropriate information or functions to a user or a user terminal.

[0077] Additionally, the processor (110) may perform operations for at least one application or program for executing a method according to embodiments of the present invention, and the server (100) may have one or more processors.

[0078] According to one embodiment of the present invention, the processor (110) may be configured with one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device.

[0079] The processor (110) can read a computer program stored in the memory (120) to provide a method for generating a re-illumination image based on an object image according to one embodiment of the present invention.

[0080] In various embodiments, the processor (110) may further include a Random Access Memory (RAM, not shown) and a Read-Only Memory (ROM, not shown) that temporarily and / or permanently store signals (or data) processed within the processor (110). In addition, the processor (110) may be implemented in the form of a system on chip (SoC) that includes at least one of a graphics processing unit, RAM, and ROM.

[0081] The memory (120) stores various data, commands, and / or information. The memory (120) can load a computer program (151) from the storage (150) to execute methods / operations according to various embodiments of the present invention. When the computer program (151) is loaded into the memory (120), the processor (110) can perform the method / operation by executing one or more instructions constituting the computer program (151). The memory (120) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0082] The bus (130) provides communication functions between components of the server (100). The bus (130) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0083] The communication interface (140) supports wired and wireless Internet communication of the server (100). Furthermore, the communication interface (140) may support various communication methods other than Internet communication. To this end, the communication interface (140) may be configured to include a communication module well known in the technical field of the present invention. In some embodiments, the communication interface (140) may be omitted.

[0084] Storage (150) can non-temporarily store a computer program (151). When performing a re-illumination image generation process based on an object image through a server (100), storage (150) can store various information necessary to provide the re-illumination image generation process based on an object image.

[0085] Storage (150) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art to which the present invention pertains.

[0086] The computer program (151) may include one or more instructions that, when loaded into the memory (120), cause the processor (110) to perform a method / operation according to various embodiments of the present invention. That is, the processor (110) may perform the method / operation according to various embodiments of the present invention by executing the one or more instructions.

[0087] In one embodiment, the computer program (151) may include one or more instructions that cause a method for generating a re-illumination image based on an object image, the method comprising the steps of: obtaining a source original image; obtaining image characteristic information based on the source original image; and generating a re-illumination image based on the source original image, the image characteristic information, and target illumination information.

[0088] The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains.

[0089] The components of the present invention may be implemented as a program (or application) to be executed in combination with a computer as hardware and stored on a medium. The components of the present invention may be implemented as software programming or software elements, and similarly, the embodiments may be implemented in a programming or scripting language such as C, C++, Java, assembler, etc., including various algorithms implemented as a combination of data structures, processes, routines, or other programming components. Functional aspects may be implemented as an algorithm executed on one or more processors. Hereinafter, a method for generating a re-illumination image based on an object image performed by a server (100) will be described in detail with reference to FIGS. 4 to 9.

[0090]

[0091] Figure 4 illustrates a flowchart exemplifying a method for generating a re-illumination image based on an object image, according to one embodiment of the present invention. The steps illustrated in Figure 4 may be rearranged as needed, and at least one step may be omitted or added. In other words, the following steps are merely exemplary of the present invention, and the scope of the present invention is not limited thereto.

[0092] According to one embodiment of the present invention, a method for generating a re-illumination image based on an object image may include a step (S100) of acquiring a source original image. In an embodiment, acquiring the source original image may be receiving or loading data stored in a memory (120). Acquiring the source original image may be receiving or loading the source original image from another storage medium, another computing device, or a separate processing module within the same computing device based on a wired / wireless communication means. For example, the source original image may be acquired from a user's smartphone, tablet, or digital camera, and may be directly uploaded to the server (100) from these devices, or may be indirectly transmitted via cloud-based storage, email, social media platforms, etc. The acquired source original image may be stored in the memory (120) of the server (100) and prepared for a processing process. The user may transmit the source original image to the server via a wired / wireless network, and the server may receive the source original image and proceed with the re-illumination process.

[0093] In an embodiment, the source original image may be an original image relating to a specific object. The object may be a person, an animal, an object, or a landscape. Specifically, the source original image may be an image captured in various subjects and environments, representing the characteristics of an individual object. For example, this may include a portrait of an individual or group, a photograph of wild animals or pets, a natural landscape or cityscape, or a detailed photograph of a specific object.

[0094] In one embodiment, the source original image may refer to an original portrait image. For example, the source original image may be a photograph containing the face of the user or another person. The source original image may be a photograph taken by the user, or an image obtained from a digital archive, a photo sharing platform, or social media. Such a source original image may be used as input to a relighting process to change lighting conditions.

[0095] According to one embodiment of the present invention, a method for generating a re-illumination image based on an object image may include a step (S200) of obtaining image characteristic information based on a source original image.

[0096] In an embodiment, the step of obtaining image characteristic information may include the step of extracting a foreground image from a source original image through a foreground extraction model and the step of performing reverse rendering on the extracted foreground image to obtain image characteristic information.

[0097] According to one embodiment, the image characteristic information may include at least one of information about a normal map, an albedo map, roughness, reflectivity, and lighting information, which is information about physical and optical properties of a surface corresponding to the foreground image.

[0098] In an embodiment, referring to FIG. 5, data for each of normal, albedo, roughness, and reflectivity corresponding to a source original image may represent various physical characteristics of the image. This characteristic information precisely expresses the directionality, color and texture of the surface, the roughness of the surface, and the degree of light reflection for each pixel or image area, and based on this, more sophisticated and realistic relighting effects can be implemented. The server (100) can utilize this information to determine how to illuminate the image under target lighting conditions, and ultimately generate a relighting image that reflects the lighting effect desired by the user.

[0099] In one embodiment, the foreground extraction model may be a neural network model trained to extract the foreground (e.g., a main object or person) from the background by separating it from an input image (e.g., a source original image). The foreground extraction model may be generated through a supervised learning process utilizing various image data sets, and in this process, a method for accurately distinguishing the foreground and background within an image may be pre-learned. As an example, the foreground extraction model may be a model related to a matting network. The matting network may be a neural network model that finely adjusts the boundary between the foreground and background for each pixel of the image and precisely extracts foreground objects from the image using an alpha matting technique. The foreground extraction model may separate the foreground from an image provided by a user with high precision, and thereafter, the server (100) may obtain image characteristic information based on the separated foreground image.

[0100] In an embodiment, the server (100) may perform reverse rendering on the extracted foreground image to obtain image characteristic information. In one embodiment, reverse rendering may refer to the process of reversely calculating and inferring the physical and optical properties of a surface from an image. This process may analyze multiple factors, such as lighting, reflectance, texture, and geometric shape of the extracted foreground image, to obtain information about how the image may be generated in the real world. Through reverse rendering, the server (100) precisely extracts essential characteristic information of the image, such as a normal map, an albedo map, and lighting information, and such image characteristic information may be utilized in a subsequent re-illumination process.

[0101] According to one embodiment of the present invention, the step of obtaining image characteristic information by performing reverse rendering may include the step of deriving a normal map corresponding to a source original image by utilizing a normal map generation model, the step of deriving lighting condition information corresponding to the source original image by utilizing a lighting condition inference model, the step of generating diffuse shading based on the normal map and the lighting condition information, the step of generating an albedo map based on the diffuse shading, and the step of obtaining information on roughness and reflectance corresponding to the source original image based on the source original image, the normal map, and the albedo map.

[0102] To explain in more detail, the server (100) can generate a normal map based on a foreground image extracted from a source original image through a normal map generation model. Here, the normal map generation model may be NormalNet, which identifies the slope and direction of the surface corresponding to each pixel in the image and generates a normal map based on the same, but is not limited thereto. The normal map generation model can analyze the physical shape and structure from the source image using a deep learning technique and convert the result into a normal map containing surface direction information.

[0103] In an embodiment, a normal map may refer to a visual representation representing the geometric shape and directionality of a surface. Each pixel of the normal map contains a unit normal vector n, which may represent the directionality of the corresponding surface point. In other words, the normal map represents the directionality of each point forming the surface of a three-dimensional object as a two-dimensional image, and the color value of each pixel encodes each component of the vector representing the directionality of the surface. A normal map generated in this manner can be used to implement more realistic lighting effects by taking the directionality of the surface into account during a subsequent relighting process.

[0104] Additionally, in the embodiment, the server (100) can infer the lighting conditions of the foreground image extracted from the source original image by utilizing the lighting condition inference model. The lighting conditions of the foreground image may refer to the direction, intensity, color, and distribution of the actual or virtual lighting source applied when capturing the source original image. Based on these lighting conditions and the normal map, a diffuse reflection shadow can be generated, which can contribute to imparting realistic depth and texture to the image during the relighting process.

[0105] In one embodiment, the lighting condition inference model is a model for accurately identifying and analyzing various lighting effects within a source image, and may be, but is not limited to, Illum Net, which identifies lighting characteristics and infers lighting conditions based on the characteristics. The Illum Net model utilizes a deep learning-based algorithm to extract lighting-related information from an image and can infer the lighting conditions under which the image was taken. In one embodiment, the lighting condition information obtained through the lighting condition inference model can be utilized to adjust and optimize new lighting effects to be applied to an image when generating a re-illuminated image in the future.

[0106] In one embodiment, image rendering may refer to the process of visually representing a generated image to a user. The primary goal of image rendering may be to generate a visual representation that accurately simulates the interaction between light and a surface. The interaction between light and a surface may be defined by the following rendering equation, which can be calculated by considering factors such as material properties, lighting characteristics, and the position of the viewer.

[0107] (Formula 1)

[0108] The rendering equation is given by: L = Ω + ... i (l) can mean the cumulative result. Here, L o (v) represents the irradiance, or light intensity, perceived by an observer in direction v, which may be related to the degree of light reflection and absorption on the surface of the object. f(v,l) may be the bidirectional reflectance distribution function (BRDF), which describes the reflective properties of the surface. The BRDF defines the interaction between the observation direction v and the incident light direction l, which allows us to determine the direction in which light is incident on and reflected from the surface.

[0109] In one embodiment, for a BRDF, represented by f(v, l), it is intended to describe how light reflects from an opaque surface, and since surfaces inherently exhibit both diffuse and specular reflection, for the BRDF, diffuse reflection (f d ) and specular reflection (f s ) consists of two components, which can be expressed through the following formula.

[0110] (Formula 2)

[0111] In one embodiment, the diffuse component acts to evenly disperse light, providing a consistent lighting effect regardless of the viewing angle. This allows the object to maintain a similar brightness regardless of the viewing angle, creating a more natural impression. Conversely, the specular component creates a reflective effect that varies depending on the viewing angle and light direction. This creates sparkling highlights on the surface of the object, which can be essential for achieving the realism seen in photographs and videos. Specular reflections are more pronounced on smoother surfaces and can provide important details for realistic image rendering.

[0112] In some embodiments, a diffuse reflection model related to diffuse reflection may include a Lambertian reflection model. The Lambertian reflection model assumes that a surface reflects light equally in all directions, which can be expressed as follows:

[0113] (Formula 3)

[0114] Here, σ represents the albedo, which can represent the intrinsic color and brightness of a surface. Dividing the albedo by π may be to normalize the reflectance.

[0115] In one embodiment, the Oren-Nayar model can be used as a diffuse reflection model in addition to the Lambert model. The Oren-Nayar model can more realistically represent light scattering by using surface roughness as an additional parameter.

[0116] In an embodiment, a specular reflection model related to specular reflection may include the Cook-Torrance model, which is based on microfacet theory. The Cook-Torrance model assumes a surface to be composed of numerous small, mirror-like microfacets. The Cook-Torrance model can precisely render the specular reflectance of a surface by introducing a roughness parameter α, which can be expressed as follows:

[0117] (Formula 4)

[0118] D(h,α) is the microfacet distribution function, G(v,l,α) is the geometric attenuation factor, and F(v,h,f0) can be a Fresnel term that describes the variation of reflectance with the observation angle. These factors can be used to calculate the orientation of the microfacets, the shadowing and masking effects, and the variation of reflectance with the observation angle, respectively. For example, by utilizing these factors, we can precisely calculate how the microfacets on each surface are arranged, how shadowing and masking occur when light reaches the surface, and how the reflectance changes with the position of the observer.

[0119] In the Cook-Torrance model, a lower value of α results in a smoother surface, producing sharper and more distinct specular highlights, while a higher value of α results in a rougher surface, resulting in more diffuse reflections. Therefore, by adjusting different values ​​of α, the Cook-Torrance model can effectively describe a wide range of specular reflectances.

[0120] According to one embodiment of the present invention, an integrated rendering formula can be derived based on the basic rendering equation (i.e., Equation 1) by including the diffuse and specular components of the BRDF, which can be expressed as follows.

[0121] (Formula 5)

[0122] Here, E(l) represents the incident environmental illumination from a given direction l, and f(v,l) represents the light reflection characteristics defined by the BRDF. In other words, by mathematically modeling the complex interaction between light and the surface, more realistic images can be created.

[0123] According to an embodiment of the present invention, in order to make the concept of the above-described (Formula 5) more clear, a rendering function R can be defined, which can be expressed as follows.

[0124] (Formula 6)

[0125] Here, E can represent the light intensity, which represents the intensity of the light source in the surrounding environment. n can be the surface normal vector, which can define the direction the surface is facing. σ, α, and f0 can represent the intrinsic color and brightness of the surface, the surface roughness, and the surface's fundamental Fresnel reflectance, respectively. By using a rendering function that considers these factors, the image formation process can be more effectively simplified and more realistic images can be generated under various lighting conditions and surface characteristics.

[0126] According to one embodiment, the step of generating an albedo map based on the diffuse reflection shading may include the step of processing a source original image and the diffuse reflection shading as inputs to a diffuse reflection model to obtain a diffuse reflection render, and the step of generating the albedo map based on the diffuse reflection shading and the diffuse reflection render.

[0127] In an embodiment, the diffuse model may include a pre-trained network function to output a diffuse render based on diffuse shading corresponding to the source original image.

[0128] In a specific embodiment, a diffuse reflection render can be generated based on a diffuse reflection shading and an albedo map, but accurate estimation of the albedo map may be difficult due to ambiguity in surface color and material properties, so the present invention can preferentially infer a diffuse reflection render and obtain an albedo map based on the diffuse reflection render.

[0129] To be more specific, the server (100) can generate a diffuse reflection shadow based on a normal map corresponding to the source original image and lighting condition information corresponding to the source original image. That is, the diffuse reflection shadow can be generated based on the normal map corresponding to the source original image and lighting condition information. In the process of generating the diffuse reflection shadow, the normal map is used to calculate the incident angle of light and the directionality of the surface, and the lighting condition information can represent the distribution and intensity change of light for the source image. By combining the two pieces of information, the server (100) can generate a diffuse reflection shadow that visually expresses the diffuse reflection effect occurring on the surface under various lighting conditions.

[0130] Specifically, the server (100) can obtain a normal map corresponding to the source original image by utilizing a normal map generation model (NormalNet). In addition, the server (100) can obtain lighting condition information from the source original image by utilizing a lighting condition inference model (e.g., IllumNet). Here, the lighting condition information may include information corresponding to the lighting conditions inferred from the source original image. Thereafter, the server (100) can generate a diffuse reflection shadow in which light is incident on a surface and spreads evenly in various directions based on the extracted normal map and lighting condition information.

[0131] In addition, in an embodiment, the server (100) can generate a diffuse render by utilizing the generated diffuse shading and the source original image. According to an embodiment, the server (100) can process the diffuse shading and the source original image as inputs to a pre-trained network function to output a diffuse render. In one embodiment, the pre-trained network function may be a DiffuseNet. The operating principle of the DiffuseNet is to analyze the characteristics of the diffuse shading and the source original image, and to generate a final diffuse render based on the interaction between light and the surface, and the diffuse render (L) excessive in the process src,Odiff(v)) can be expressed through the following formula.

[0132] (Formula 7)

[0133] Here, σ / π represents the diffuse BRDF, Esrc(l) represents the lighting environment corresponding to the source image, and (n·l) can represent the incident light intensity according to the angle that the lighting makes with the surface normal. Ω can be a hemisphere representing all possible lighting directions on the surface.

[0134] The above equation (Equation 7) can illustrate how a diffuse reflection renderer actually models the diffuse reflection effect, where light from a source image strikes a surface and spreads in multiple directions. Based on this equation, DiffuseNet calculates the complex interaction between light from a source image and the surface within its algorithm, thereby generating realistic diffuse reflection renders.

[0135] That is, a diffuse reflection render is a final image generated by combining a diffuse reflection shading and an albedo map, and can be characterized as an image that visually expresses the diffuse reflection effect in which light is evenly spread in all directions from a surface.

[0136] In one embodiment, the diffuse reflection model can generate an albedo map based on diffuse reflection shading and diffuse reflection render.

[0137] In general, accurate estimation of an albedo map can be complicated by the ambiguity of surface color and material properties, as well as shadow effects. To solve this problem, the server (100) of the present invention utilizes a method of first deriving a diffuse reflection render and then inferring an albedo map based on this. That is, the diffuse reflection render is first derived using the deep learning model, Diffuse ReflectionNet, and then the albedo map can be derived based on the derived diffuse reflection render and the generated diffuse reflection shading. Through this process, the true surface color unaffected by lighting and shadows can be more accurately predicted, which has the advantage of significantly improving albedo prediction in various real-world scenarios.

[0138] In addition, according to one embodiment, the step of obtaining information about roughness and reflectivity corresponding to the source original image based on the source original image, the normal map, and the albedo map includes the step of obtaining information about roughness and reflectivity by processing the source original image, the normal map, and the albedo map as inputs to a specular reflection model, and the specular reflection model may include a network function pre-learned to obtain specular reflection information including information about roughness and reflectivity by inferring specular reflection elements of a surface based on microsurface theory.

[0139] In one embodiment, the specular reflection model may include a pre-trained network function based on microsurface theory that can be used to accurately infer the specular reflection characteristics of a surface and obtain specular reflection information. The pre-trained network function included in the specular reflection model may be Specular Net. Specular Net is an advanced deep learning algorithm designed to model and infer the specular reflection characteristics of complex surfaces.

[0140] Specular Net uses source original images, normal maps, and albedo maps as input. When data is input to Specular Net, information about various specular properties, including surface roughness and reflectivity, can be inferred. Specular Net can infer specular properties, including surface roughness and reflectivity, from the input data through complex nonlinear transformations and pattern recognition techniques. Surface roughness is an important factor in determining the degree of dispersion of highlights generated when light reflects from a surface, and reflectivity indicates how well a surface reflects light, which can be closely related to the intrinsic properties of a material.

[0141] In embodiments, the Specular Net can output a final image or attribute data reflecting specular reflection characteristics based on inferred surface roughness and reflectance information. This output data (e.g., specular reflection information) can be utilized in applications such as image rendering, material recognition, or visual effect generation.

[0142] Referring to FIG. 6, first, the server (100) can generate a normal map by processing a foreground image extracted from a source original image as input to a normal map generation model. In addition, the server (100) can infer lighting condition information corresponding to the extracted foreground image by utilizing a lighting condition inference model.

[0143] Thereafter, the server (100) can generate a diffuse reflection shadow based on the normal map and lighting condition information. In the process of generating the diffuse reflection shadow, the normal map is used to calculate the incident angle of light and the directionality of the surface, and the lighting condition information can represent the distribution and intensity changes of light for the source image. By combining the two pieces of information, the server (100) can generate a diffuse reflection shadow that visually expresses the diffuse reflection effect occurring on the surface under various lighting conditions.

[0144] In addition, the server (100) can generate a diffuse render by utilizing the diffuse shading and the source original image. In an embodiment, the server (100) may be characterized by preferentially inferring the diffuse render based on the diffuse shading by utilizing a neural network model, and inferring the albedo based on the inferred diffuse render. Specifically, the server (100) may generate the diffuse render by processing the diffuse shading and the source original image as inputs of the diffuse net. The diffuse render may be characterized by being a final image generated by combining the diffuse shading and the albedo map, and being an image in which the diffuse reflection effect in which light spreads evenly in all directions from the surface is visually expressed. The server (100) of the present invention utilizes a method of preferentially deriving the diffuse render and inferring the albedo map based on the diffuse render. That is, by utilizing the deep learning model, DiffuseNet, a diffuse reflection render can be first derived, and then an albedo map can be derived based on the derived diffuse reflection render and the generated diffuse reflection shading. This process allows for more accurate predictions of true surface colors, unaffected by lighting and shadows, which has the advantage of significantly improving albedo predictions in various real-world scenarios.

[0145] In addition, the server (100) can obtain specular information regarding specular reflection characteristics based on the original source image, normal map, and albedo map by utilizing the specular reflection model. The server (100) can process the original source image, normal map, and albedo map as inputs to the specular reflection model (e.g., Specular Net) to output a final image or attribute data in which specular reflection characteristics are reflected based on roughness and reflectance information.

[0146] Through the aforementioned process, the server (100) can obtain image characteristic information corresponding to the original source image, i.e., information regarding a normal map, an albedo map, roughness, and reflectance. The server (100) can generate a re-illuminated image based on this image characteristic information and the target lighting conditions (i.e., the target lighting conditions).

[0147] According to one embodiment of the present invention, a method for generating a re-illumination image based on an object image may include a step (S300) of generating a re-illumination image based on a source original image, image characteristic information, and target illumination information.

[0148] In an embodiment, the target lighting information may be a Target HDRI (High Dynamic Range Imaging), which relates to the direction, color, intensity, etc. of light used to illuminate a specific scene or object. A Target HDRI contains high dynamic range lighting information measured in a real-world environment, which can be very useful for simulating and reproducing realistic lighting conditions. For example, by using a high-resolution HDRI recorded in various real-world lighting environments, such as sunrise, sunset, cloudy days, and indoor lighting, as the Target Lighting Information, the rendering model can precisely reproduce how an object or scene would appear under such lighting.

[0149] According to one embodiment, the step of generating a re-lighting image may include the steps of generating a diffuse render and a specular render based on a normal map, an albedo map, information about roughness, information about reflectivity, and target lighting information, the step of generating an initial re-lighting image based on the diffuse render and the specular render, and the step of processing the initial re-lighting image as an input of a rendering model to generate the re-lighting image.

[0150] More specifically, referring to FIG. 7, the server (100) can generate a diffuse render and a specular render based on a normal map, an albedo map, information about roughness, and information about reflectivity.

[0151] Here, diffuse reflection renders can simulate the phenomenon of light being scattered and reflected in multiple directions from an object's surface. This can provide a more realistic visual effect by reflecting the object's texture and roughness. Diffuse reflection is important in determining the surface's basic color and texture, and albedo maps and roughness information can be used to adjust reflectivity.

[0152] Additionally, specular reflection renders can simulate the phenomenon where light strikes an object's surface at a specific angle and is reflected back at the same angle. This is crucial for expressing the shine or glossiness of an object and can be derived using information about the object's reflectivity and roughness. Specular reflection is most noticeable on smooth, glossy surfaces, and its visual effect can vary significantly depending on the direction of the light and the viewing position of the object.

[0153] The server (100) can generate an initial re-illuminated image by combining a diffuse render and a specular render. In this step, target lighting information (e.g., target HDRI (High Dynamic Range Imaging)) is used to determine the direction, intensity, color, etc. of light, and the generated renders can contribute to predicting the visual response of an object under such lighting conditions.

[0154] In one embodiment, the server (100) may generate an initial re-lit image based on physically based rendering (PBR) principles (e.g., following the Cook-Torrance model). In an embodiment, the initial re-lit image may be a PBR render. A PBR render may be an image with realistic lighting effects by modeling the interaction between lighting and objects based on actual physical laws.

[0155] In a specific embodiment, the server (100) can derive diffuse and specular renders under target lighting based on (Equation 3) and (Equation 4), and combine the diffuse and specular renders to generate a PBR rendering, i.e., an initial re-illuminated image.

[0156] The initial relighting image provides an approximation of how the object will appear in the new lighting environment, and the image can be input into a rendering model to form the basis for generating the final relighting image. The server (100) can process the initial relighting image as input into the rendering model to generate the final relighting image.

[0157] In embodiments, the rendering model may further refine the initial re-lit image and adjust it to new lighting conditions while maintaining consistency with the original image.

[0158] In a specific embodiment, the rendering model may be a neural network model that analyzes the characteristics of a source original image and generates a re-illuminated image under new lighting conditions based on the characteristics. The rendering model may be pre-trained to output an image in an environment with changed lighting by comprehensively analyzing and processing the output of a specular reflection model, a normal map, an albedo map, and target lighting condition information as input.

[0159] In an embodiment, the rendering model can generate a realistic reconstructed image corresponding to the original image but with changed lighting conditions by being pre-trained based on an integrated loss involving a weighted sum of reconstruction loss, perceptual loss, adversaria loss, and specular loss.

[0160] Here, the reconstruction loss is a loss regarding the pixel-level difference between the original image and the predicted result image corresponding to the original image, the perception loss is a loss regarding the feature difference between the original image and the result image, the adversarial loss is a loss regarding the difference between the original image and the result image judged by the discriminator model, and the specular loss may be a loss obtained by weighting the reconstruction loss using specular information.

[0161] In other words, the rendering model can relight the original image under new lighting conditions by leveraging complex input data and various loss functions. The rendering model can generate a relighted image based on the initial relighted image. During this process, the model minimizes the direct pixel value difference between the original and relighted images through reconstruction loss, and reduces the visual characteristic difference between the two images through perceptual loss, resulting in more natural results. Furthermore, the adversarial loss is used to make the generated image so natural that it is difficult to distinguish it from reality, and the specular loss can more accurately simulate the light reflection characteristics of reflective and glossy surfaces. In this way, the pre-trained rendering model through a complex learning method can effectively mimic various real-world lighting conditions.

[0162] In summary, the server (100) can generate a diffuse render and a specular render based on normal map, albedo map, roughness, and reflectance information using a physically based rendering (PBR) approach utilizing the Cook-Torrance model. In addition, based on the renders, an initial re-lighting image, i.e., a PBR render, under target lighting conditions can be generated. In addition, the server (100) can generate a re-lighting image that is more realistic by improving the image in terms of brightness, specular details, etc. using the rendering model, as can be confirmed through FIG. 8. That is, the final re-lighting image (neural render) is a result of a neural network, and while it is based on a PBR render, it can capture and express even finer details that are difficult to capture with only the Cook-Torrance model.

[0163] According to one embodiment of the present invention, a method for generating a re-illumination image based on an object image may further include a step of constructing a learning data set based on a plurality of light stage data, a step of generating a plurality of reconstructed images corresponding to each of a plurality of source original images included in the learning data set by utilizing an image reconstruction model, and a step of reinforcing the learning data set based on the plurality of reconstructed images.

[0164] The image reconstruction model may be characterized in that it is trained to generate a reconstructed image corresponding to an input image by reflecting perceptual loss and adversarial loss in the reconstruction loss regarding the difference between each source original image and each reconstructed image corresponding to each source original image.

[0165] More specifically, the server can build a learning dataset by collecting portrait images taken under various lighting conditions using light stage data. Light stage data is data generated through a light stage facility, and is a high-resolution record of the three-dimensional shape of an object or person and the optical properties of its surface. A light stage is a specialized photography studio consisting of multiple lighting devices (such as LED lights and spotlights) that can provide illumination from various directions and a high-resolution camera. By placing the subject in the center and applying illumination at various angles and intensities, the appearance of light reflected from the surface of the subject is captured from various angles. However, since such light stage data requires special photography equipment and settings, it may be difficult to obtain in a general environment.

[0166] To overcome this, the server (100) can generate multiple reconstructed images corresponding to each of the acquired optical stage data, and reinforce the learning data set through the generated reconstructed images.

[0167] Specifically, the server (100) can process each optical stage data as input to an image reconstruction model to generate multiple reconstructed images corresponding to each optical stage data.

[0168] In an embodiment, the image reconstruction model of the present invention may be a neural network model trained to generate a reconstructed image by utilizing dynamic masking that dynamically adjusts one or more patches of different sizes to different areas of an input image.

[0169] The image reconstruction process of the image reconstruction model of the present invention goes beyond a simple patch-based approach and employs a more dynamic and flexible masking technique to reflect complex image characteristics and various lighting conditions. For example, rather than simply applying a patch of a certain size to an input image and reconstructing the corresponding patch portion to generate a reconstructed image, the image reconstruction model can utilize dynamic masking to reconstruct various regions of the image with various sizes and shapes. As illustrated in Figure 9, by utilizing overlapping patches of various sizes, outpainting masks, and free-form masks, the image reconstruction model can reconstruct specific regions of the image in greater detail. For example, small patches can be applied to areas with detailed features, such as the eyes or mouth of a face, to achieve high-resolution reconstruction. Conversely, larger patches can be used for backgrounds or less important areas, allowing for a wider area to be processed simultaneously. By utilizing dynamic masks that determine the size and shape of patches for each region in this way, the model can preserve important features of the image while maintaining overall harmony, and generate reconstructed images of a wider variety of shapes.

[0170] This dynamic masking method has the advantage of allowing the model to more precisely handle important elements within the image, such as changes in lighting or details of specific objects, resulting in more realistic and natural relighting effects.

[0171] In other words, image reconstruction models can generate reconstructed images by considering not only the overall composition of the image but also the details of specific areas. For example, they enable neural networks to learn how specific parts of a face or background elements change depending on lighting. Furthermore, dynamic masking can be used to selectively emphasize certain areas of an image or to apply different processing to specific areas to maximize the relighting effect.

[0172] In particular, the image reconstruction model of the present invention is trained by applying reconstruction loss that minimizes the difference between each source original image and the corresponding reconstructed image, and perceptual loss and adversarial loss to preserve the texture and details of the image and produce more natural results. Therefore, the generated image is not only reproduced very closely to the original, but can also include subtle visual effects according to lighting changes. Through this process, the neural network can understand more complex image characteristics and various lighting conditions, and effectively generate reconstructed images with realistic lighting effects.

[0173] In other words, as the image reconstruction model acquires a large number of reconstructed images and strengthens the training data set, the neural network gains more information to learn. As the diversity and scale of the training data set increase, the neural network learns more sophisticated pattern recognition and image processing capabilities, enabling it to create realistic lighting effects. This has the advantage of allowing the neural network to preserve fine image detail while generating natural relighting effects under various lighting conditions.

[0174] In summary, the server (100) can learn from unlabeled data through a self-supervised pre-training framework and, based on this, provide realistically re-illuminated images tailored to user needs. That is, the server (100) utilizes an architecture that integrates a physics-based approach and a self-supervised pre-training framework to automate the re-illumination of portrait images that reflect lighting changes in various real-world scenarios, thereby providing users with enhanced realism.

[0175]

[0176] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be comprised of a set of interconnected computational units, generally referred to as "nodes." These "nodes" may also be referred to as "neurons." A neural network comprises at least one node. The nodes (or neurons) comprising a neural network may be interconnected by one or more "links."

[0177] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network, it is possible to identify latent structures in data. That is, it is possible to identify latent structures of photos, text, videos, voices, and music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.). A deep neural network can include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, etc. The description of the above-mentioned deep neural network is merely an example, and the present invention is not limited thereto.

[0178] Neural networks can be trained using at least one of supervised learning, unsupervised learning, and semi-supervised learning. Neural network training aims to minimize output errors. Training involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network in a direction that reduces the error. Supervised learning uses training data with the correct answer labeled for each training data (i.e., labeled training data). Unsupervised learning, on the other hand, may not have the correct answer labeled for each training data. For example, in the case of supervised learning for data classification, the training data may be data in which each training data category is labeled. Labeled training data is input to a neural network, and an error can be calculated by comparing the output (categories) of the neural network with the labels of the training data. For example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the backward direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle.For example, in the early stages of training a neural network, a high learning rate can be used to quickly allow the neural network to achieve a certain level of performance, thereby improving efficiency, while in the later stages of training, a low learning rate can be used to improve accuracy.

[0179] The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains.

[0180] The components of the present invention may be implemented as programs (or applications) and stored on a medium to be executed in conjunction with a computer, which is hardware. The components of the present invention may be implemented as software programs or software elements. Similarly, the embodiments may be implemented in a programming or scripting language such as C, C++, Java, or an assembler, including various algorithms implemented as a combination of data structures, processes, routines, or other programming components. Functional aspects may be implemented as algorithms that are executed on one or more processors.

[0181] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as “software”), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0182] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, wireless channels and various other media capable of storing, retaining, and / or carrying instructions and / or data.

[0183] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present invention based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0184] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method performed on one or more processors of a computing device, Step of obtaining the source original image; A step of obtaining image characteristic information based on the above source original image; and A step of generating a re-illuminated image based on the source original image, the image characteristic information, and the target illumination information; comprising; A method for generating a re-illuminated image based on an object image.

2. In paragraph 1, The above re-illuminated image is an image that reflects the characteristics of the object under the target lighting conditions, and is characterized in that it is an image in which the lighting effect is changed compared to the source original image. A method for generating a re-illuminated image based on an object image.

3. In paragraph 2, The above re-illuminated image is characterized in that it is an image reflecting realistic human skin tone, texture and shadow effects under target lighting conditions. A method for generating a re-illuminated image based on an object image.

4. In paragraph 1, The step of obtaining the above image characteristic information is: A step of extracting a foreground image from the source original image using a foreground extraction model; and A step of performing reverse rendering on the extracted foreground image to obtain the image characteristic information; The above image characteristic information is, Information about the physical and optical properties of the surface corresponding to the foreground image, including at least one of a normal map, an albedo map, information about roughness, information about reflectivity, and information about lighting conditions. A method for generating a re-illuminated image based on an object image.

5. In paragraph 4, The step of performing the above reverse rendering to obtain the image characteristic information is: A step of deriving the normal map corresponding to the source original image by utilizing a normal map generation model; A step of deriving the lighting condition information corresponding to the source original image by utilizing a lighting condition inference model; A step of generating diffuse shading based on the above normal map and the above lighting condition information; A step of generating the albedo map based on the above diffuse reflection shading; and A step of obtaining information about the roughness and the reflectance corresponding to the source original image based on the source original image, the normal map, and the albedo map; comprising; A method for generating a re-illuminated image based on an object image.

6. In paragraph 5, The step of generating the albedo map based on the above diffuse reflection shading is: A step of obtaining a diffuse reflection render by processing the above source original image and the above diffuse reflection shading as inputs to a diffuse reflection model; and A step of generating the albedo map based on the above diffuse reflection shading and diffuse reflection render; The above diffuse reflection model includes a pre-trained network function to output a diffuse reflection render based on the diffuse reflection shading corresponding to the source original image, The above diffuse reflection render is a final image generated by combining the diffuse reflection shading and the albedo map, and is characterized by being an image in which the diffuse reflection effect in which light spreads evenly in all directions from the surface is visually expressed. A method for generating a re-illuminated image based on an object image.

7. In paragraph 5, The step of obtaining information about the roughness and the reflectance corresponding to the source original image based on the source original image, the normal map, and the albedo map is as follows: A step of processing the source original image, the normal map, and the albedo map as inputs to a specular reflection model to obtain information about the roughness and the reflectivity, The above specular reflection model includes a network function pre-trained to obtain specular reflection information including information about the roughness and the reflectivity by inferring the specular reflection elements of the surface based on the microsurface theory. A method for generating a re-illuminated image based on an object image.

8. In paragraph 4, The steps to create a re-illuminated image are: A step of generating a diffuse reflection render and a specular reflection render based on the normal map, the albedo map, the roughness information, the reflectivity information, and the target lighting information; A step of generating an initial re-illuminated image based on the above diffuse reflection render and the above specular reflection render; and A step of generating the re-illumination image by processing the initial re-illumination image as an input of a rendering model; The above rendering model is, It is a pre-trained neural network model based on the integrated loss related to the weighted sum of reconstruction loss, perceptual loss, adversaria loss, and specular loss. The above reconstruction loss is a loss regarding a pixel-level difference between an original image and a result image predicted corresponding to the original image, the above perception loss is a loss regarding a characteristic difference between the original image and the result image, the above adversarial loss is a loss regarding a difference between the original image and the result image judged by a discriminator model, and the above specular loss is a loss obtained by weighting the above reconstruction loss using specular information. A method for generating a re-illuminated image based on an object image.

9. In paragraph 8, The above method, A step of constructing a learning data set based on multiple optical stage data; A step of generating a plurality of reconstructed images corresponding to each of a plurality of source original images included in a learning data set by utilizing an image reconstruction model; and A step of reinforcing a learning data set based on the above plurality of reconstructed images; further comprising: The above image reconstruction model is, Characterized in that it is learned to generate a reconstructed image corresponding to an input image by reflecting the perceptual loss and the adversarial loss in the reconstruction loss regarding the difference between each source original image and each reconstructed image corresponding to each source original image. A method for generating a re-illuminated image based on an object image.

10. In paragraph 9, The above image reconstruction model is, characterized in that it is trained to generate multiple reconstructed images by utilizing dynamic masking that dynamically adjusts one or more patches of different sizes to different areas of the input image. A method for generating a re-illuminated image based on an object image.

11. Memory that stores one or more instructions; and A processor comprising: a processor that executes one or more instructions stored in the memory; The processor executes one or more of the instructions, A device for performing the method of claim 1.

12. A computer program stored on a computer-readable recording medium that is combined with a computer as hardware and can perform the method of claim 1.

Images