A method based on object lighting editing

Through the lighting data set based on physical rendering and the multi-scale attention mechanism, combined with the lighting exchange mechanism, the problem of insufficient generalization ability of object lighting editing in the existing technology is solved, and efficient and accurate lighting editing effect is achieved, avoiding reverse rendering Technology dependence.

CN114004803BActive Publication Date: 2024-11-12WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111269555.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-12
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

In the prior art, the generalization ability of object lighting editing methods is not strong, and they rely heavily on inverse rendering technology, making it difficult to efficiently and accurately model the lighting information of objects.

Method used

Using physical rendering to build a large-scale, high-quality lighting data set, through multi-scale attention mechanism and lighting exchange mechanism, learn the shadow information of real objects and realize lighting editing of virtual objects, combining heavy lighting networks and background lighting networks for lighting Exchange and fusion of features.

Benefits of technology

It realizes the accuracy and efficiency of object lighting editing, generates lighting harmonious images, avoids the dependence of inverse rendering technology, and improves the generalization ability of lighting editing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114004803B_ABST
    Figure CN114004803B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of augmented reality, and provides an object-based lighting editing method. First, a physical rendering technology is proposed to construct a large-scale, high-quality data set for object lighting editing tasks; the constructed synthetic data and its corresponding lighting information are used as network input, and the shadow information of real objects in the scene is learned through a multi-scale attention mechanism to provide reference and guidance for the shadow generation of virtual objects inserted in the scene; by utilizing the lighting exchange mechanism, the lighting information of the virtual foreground object inserted in the scene is effectively exchanged with the background lighting information to achieve lighting fusion of the foreground and background; finally, a more realistic scene lighting harmony image is achieved and output through generative adversarial learning. The present invention can avoid any inverse rendering technology and simply, robustly and effectively realize object lighting editing and achieve lighting harmony of the scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality, and in particular relates to a method based on object illumination editing, which directly adopts an end-to-end algorithm to realize the object illumination editing. Background Art

[0002] At present, the commonly used object-based lighting editing methods are divided into the following categories: 1. Using traditional methods, by modeling the geometry, material and lighting information of objects and scenes, to achieve object re-lighting, such as the technology proposed in the paper "Karschet al". Although this method can achieve good editing effects, it is heavily dependent on the inverse rendering information such as geometry and lighting of objects and scenes. Once the geometry, lighting and other information are estimated inaccurately, it will cause poor lighting editing results; 2. Using deep learning technology to achieve object and scene fusion. This type of method mainly focuses on the problem of color consistency between objects and backgrounds in the scene. This type of method only considers the color consistency between objects and scenes, and fails to truly solve the problem of editing the lighting information of objects (color temperature, light intensity and shadows of objects). In the existing technology, there is a lack of a method that avoids any inverse rendering technology and realizes object lighting editing simply, robustly and effectively. Summary of the invention

[0003] The present invention is to address the deficiencies of the prior art and propose a method based on object lighting editing, which solves the problems that the existing methods have poor generalization ability for object lighting editing, are heavily dependent on inverse rendering technology, and are difficult to efficiently and accurately model the lighting information of objects.

[0004] The present invention provides a method for object illumination editing, comprising the following steps:

[0005] Step 1: First, build a large-scale, high-quality lighting dataset based on physical rendering.

[0006] Step 2: Divide each type of data according to the designed network input requirements, and finally obtain the corresponding network input in the form of 6-tuples for each type of data.

[0007] Step 3: Extract the input 6-tuple data features and learn the shadow information of real objects in the scene through a multi-scale attention mechanism to derive reasonable shadows of virtual objects.

[0008] Step 4: Through the illumination exchange mechanism, the illumination information of the inserted virtual object is exchanged and merged with the illumination information of the background where it is located.

[0009] Step 5: synthesize the reasonable shadow information and lighting fusion information of the virtual object described in Step 3 and Step 4 to obtain the lighting harmony image after the object lighting is edited and output it.

[0010] Preferably, step 1 comprises the following sub-steps:

[0011] Step 1.1: First, download and obtain 2686 HDR panorama data with various real scenes from the Laval dataset and the Internet. At the same time, select 7 different 3D models from the Internet, namely bunny, lucy, buddha and other three character models.

[0012] Step 1.2: For each panoramic image, capture any 8 limited viewing angles as different background images of illumination data, and finally obtain 22,256 background images.

[0013] Step 1.3: According to the different three-dimensional models and background images of different scenes described in step 1.1 and step 1.2, different three-dimensional models are rendered under various lighting conditions and embedded into the background images through physically based rendering, and finally 89,898 lighting composite images are obtained.

[0014] Preferably, step 2 includes the following sub-steps:

[0015] Step 2.1: According to the illumination data set described in step 1, obtain the mask of the object, the mask of the background, the illumination images corresponding to the object and the background, and the illumination harmony image for each type of data.

[0016] Step 2.2: According to the data components described in step 2.1, each data is combined into 6-tuples respectively, and finally the network input of each data is obtained in the form of 6-tuples. The 5-tuple of data includes non-illuminated harmonious image, object mask, background mask, object illumination information, background illumination information, and illuminated harmonious image.

[0017] Preferably, step 3 includes the following sub-steps:

[0018] Step 3.1: First, use the 6-tuple illumination data as the input information of the neural network and perform feature extraction on it.

[0019] Step 3.2: Based on the extracted features described in step 3.1, a multi-scale attention mechanism with three different scale convolution kernels (1×1, 3×3, and 5×5) is used to learn the shadow information of the reference objects in the real scene to derive reasonable shadows of virtual objects.

[0020] Preferably, step 4 includes the following sub-steps:

[0021] Step 4.1: The re-illumination network first extracts the overall features of the input non-illuminated harmonious image, that is, the feature contains the overall illumination features and non-illuminated features of the image.

[0022] Step 4.2: The background illumination network extracts the illumination information of the input background image and finally obtains the illumination features of the background image.

[0023] Step 4.3: According to the overall features of the image described in step 4.1, multiply it by the mask of the object pixel by pixel, and finally obtain the overall features of the object, that is, the illumination features and non-illumination features of the object.

[0024] Step 4.4: Based on the illumination features of the background described in step 4.2 and the illumination features of the object described in step 4.3, the illumination features are exchanged through an illumination exchange mechanism. The specific exchange process is as follows: the background illumination features finally extracted in the encoder in step 4.2 are represented by a 512-dimensional vector.

[0025] The features described in step 4.3 share a 1024-dimensional vector feature representation, of which 512 dimensions are the non-illumination features of the object, and the other 512-dimensional vector representation is the illumination features of the object. In the illumination exchange part, the illumination features of the object are exchanged with the illumination features of the background.

[0026] Preferably, step 5 includes the following sub-steps:

[0027] Step 5.1: According to the features after illumination exchange described in step 4.4, decode them, fuse the illumination information with the non-illumination information, and finally obtain the features after illumination fusion.

[0028] Step 5.2: According to the illumination fusion features described in step 5.1 and the virtual object shadow features described in step 3.2, the decoder of the re-illumination network is used to decode and finally obtain a harmonious image of the scene after illumination editing.

[0029] The advantages of the present invention are as follows: firstly, a large-scale, high-quality data set based on physical rendering technology is proposed for object lighting editing tasks. The data set can fully and comprehensively reflect lighting information, including information such as object shadows, color temperature, and light illumination darkness, which can better meet the lighting editing purpose of the present invention and achieve a more ideal lighting editing effect.

[0030] Secondly, the present invention proposes a multi-scale attention mechanism and an illumination exchange mechanism, which guides objects to generate reasonable virtual shadows through the multi-scale attention mechanism, and uses the illumination exchange mechanism to achieve illumination exchange between foreground objects and backgrounds, thereby completing the illumination editing of objects. Through the object-based illumination editing method of the present invention, the scene illumination harmony of a single image can be achieved, which has the advantages of accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1A flow chart for preparing a training data set for training a network model provided in an embodiment of the present invention.

[0032] Figure 2 A schematic diagram of an overall network design of an image-based object lighting editing method provided by an embodiment of the present invention.

[0033] Figure 3 This is an experimental effect diagram of an image-based object lighting editing method provided in this embodiment. DETAILED DESCRIPTION

[0034] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] The present invention provides an image-based object lighting editing method, which mainly includes the following steps:

[0036] Step 1: First, build a large-scale, high-quality lighting dataset based on physical rendering.

[0037] Step 2: Divide each type of data according to the designed network input requirements, and finally obtain the corresponding network input in the form of 6-tuples for each type of data.

[0038] Step 3: Extract the input 6-tuple data features and learn the shadow information of real objects in the scene through a multi-scale attention mechanism to derive reasonable shadows of virtual objects.

[0039] Step 4: Through the illumination exchange mechanism, the illumination information of the inserted virtual object is exchanged and merged with the illumination information of the background where it is located.

[0040] Step 5: synthesize the reasonable shadow information and lighting fusion information of the virtual object described in Step 3 and Step 4 to obtain the lighting harmony image after the object lighting is edited and output it.

[0041] The present invention will be further described below in conjunction with the accompanying drawings.

[0042] refer to Figure 1-Figure 3 The present embodiment provides an image-based object lighting editing method, which mainly includes the following steps:

[0043] First, we download and obtain HDR panorama data with various real scenes from the Laval dataset and the Internet, and for each panorama, we capture any 8 limited perspectives as different background images for illumination data; at the same time, we select 7 different 3D models such as bunny, lucy, buddha and other three character models on the Internet, and then render the different 3D models under various illumination conditions through physical rendering (Blender) and embed them into the background image, and finally obtain 89898 illumination composite images of size 256×256. The existing models have been expanded to 60, and 60 models can also be used for data production.

[0044] from Figure 1 As can be seen from the figure, taking the 3D model Buddha as an example, the 3D model is rendered under the condition of lighting information (d) and inserted into the scene with the background image (e), and finally the rendered lighting harmony image (f) as the network input is obtained, and its size is 256×256. Figure 1 (a), (b), and (c) represent the 2D image of the 3D model, the mask of the 2D object, and the non-illuminated and harmonious synthetic image, respectively.

[0045] The acquired 6-tuple data [non-illuminated harmonious image, object mask, object illumination information, scene illumination information, illuminated harmonious image] is used as the input of the network, and its features are extracted through the neural network.

[0046] According to the extracted features, the shadow information and illumination information are learned respectively through the multi-scale attention mechanism and illumination exchange mechanism, and the final illumination-harmonious image is generated.

[0047] from Figure 2 As can be seen in the figure, the present invention proposes an end-to-end scene inference generative adversarial network SI-GAN, which models the shadow and illumination of objects, aiming to solve the problems that the existing methods have weak generalization ability for object illumination editing, rely heavily on inverse rendering technology, and are difficult to efficiently and accurately model the illumination information of objects. The network structure involved consists of two branch networks, one is the re-illumination network, and the other is the illumination feature extraction network. Its working principle is as follows:

[0048] First, the network inputs the 6-tuple data information obtained by steps 1 and 2. The 6-tuple data information includes non-illuminated harmonious image, object mask, scene mask, object illumination information, scene illumination information, and illuminated harmonious image; then the re-illumination network and the background illumination prediction network respectively extract features of the non-illuminated harmonious image in the input 6-tuple information.

[0049] At the final output of the re-illumination network encoding, the features extracted by the re-illumination network in (a) are multiplied pixel by pixel with the mask of the object to obtain the illumination features and non-illumination features of the object. At the same time, the illumination estimation network obtains background information by multiplying the input non-illumination harmony image by the background mask pixel by pixel and performs illumination estimation to obtain background illumination. After (b), the encoder output of the re-illumination network outputs 1024-dimensional features to represent the illumination features of the object, 512-dimensional features, and 512-dimensional features of non-illumination. The encoder of the illumination estimation network finally outputs a 512-dimensional feature vector to represent the background illumination.

[0050] The 512-dimensional illumination features of the object (c) are exchanged with the 512-dimensional illumination features of the background output by the illumination extraction network through the illumination exchange mechanism.

[0051] The decoder of the re-illumination network decodes the features of the exchanged illumination image, and makes full use of the multi-scale attention module in the entire feature extraction and fusion process to learn the reference shadows of real objects in the background to guide the shadow generation of the target object, while using the original input illumination harmony image as a constraint to continuously iterate and optimize the learning accuracy of the network. The specific constraint algorithm expression is:

[0052] L total =β1L illu +β2L noillu +β3L per +β4L adv

[0053] The values ​​of β1, β2, β3, and β4 are 25.0, 6.0, 0.04, and 0.5, respectively. total is the total loss of the network, L illu is the light loss term, L noillu is the non-light loss term, L per is the perceptual loss term, L adv To combat loss.

[0054] L illu and L noillu The loss term is mainly used to assist the illumination exchange mechanism to achieve the fusion of background illumination information and object non-illumination information. illu With L noillu is defined as follows:

[0055]

[0056] in, They are respectively the object illumination image and illumination image predicted by the network, Y OI , Y BI They are represented by their corresponding real images respectively.

[0057]

[0058] in, It is represented as the non-illumination features of the same object under different illumination conditions, N nonillu Represented as a per-pixel quantity of the non-illuminated feature map.

[0059] Finally, the network obtains and outputs a lighting-harmonious image that is a fusion of background lighting features and object non-lighting features.

[0060] Figure 3 A set of experimental results based on the object lighting editing method are visualized. (a), (b), (c), and (d) represent the two-dimensional object inserted into the scene, the scene with the object inserted, the non-lighting harmonious image, and the lighting harmonious image, respectively. The non-lighting harmonious image is the original input image of the network, and the lighting harmonious image is the output image of the network. It can be seen from the experimental results that the edited object not only produces a reasonable shadow, but also the lighting information of the object is consistent with the lighting information of its background. The experimental results show that the image-based object lighting editing method can achieve a good lighting harmony effect.

[0061] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for editing illumination based on an image object, characterized in that: include: Step 1: Build a large-scale, high-quality lighting dataset based on physical rendering; Step 2: Divide each type of data according to the designed network input requirements, and finally obtain the corresponding network input in the form of 6 tuples for each type of data; Step 3: Extract the input 6-tuple data features and learn the shadow information of real objects in the scene through a multi-scale attention mechanism, so as to derive the reasonable shadow of the virtual object; Step 4: Through the illumination exchange mechanism, the illumination information of the inserted virtual object is exchanged and merged with the illumination information of the background where it is located, which specifically includes: Step 4.1, the re-illumination network first extracts the overall features of the input non-illumination harmonious image, that is, the feature contains the overall illumination features and non-illumination features of the image; Step 4.2, the background illumination network extracts the illumination information of the input background image, and finally obtains the illumination features of the background image; Step 4.3, according to the overall feature of the image described in step 4.1, multiply it by the mask of the object pixel by pixel, and finally obtain the overall feature of the object, that is, the illumination feature and the non-illumination feature of the object; Step 4.4, according to the illumination characteristics of the background described in step 4.2 and the illumination characteristics of the object described in step 4.3, the illumination characteristics are exchanged through an illumination exchange mechanism; Step 5: synthesize the reasonable shadow information and illumination fusion information of the virtual object in step 3 and step 4 to obtain the illumination harmony image after the object illumination is edited and output, which specifically includes: Step 5.1, according to the features after illumination exchange described in step 4.4, decode them by a decoder, fuse the illumination information with the non-illumination information, and finally obtain the features after illumination fusion; Step 5.2: According to the illumination fusion features described in step 5.1 and the virtual object shadow features described in step 3.2, the decoder of the re-illumination network is used to decode and finally obtain a harmonious image of the scene after illumination editing.

2. The method for editing illumination based on image objects according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 1.1, first download and obtain a number of HDR panorama data with various real scenes from the Laval dataset and the Internet; at the same time, select 7 different 3D models on the Internet; Step 1.2: For each panorama, capture any 8 limited viewing angles as different illumination data background images; Step 1.3: According to the different three-dimensional models and background images of different scenes described in step 1.1 and step 1.2, different three-dimensional models are rendered under various lighting conditions through physically based rendering and embedded into the background image to obtain a lighting composite image.

3. The method for editing illumination based on image objects according to claim 2, characterized in that: The step 2 includes the following sub-steps: Step 2.1, according to the illumination data set described in step 1, respectively obtain the mask of the object, the mask of the background, the illumination images corresponding to the object and the background, and the illumination harmony image for each data; Step 2.2: According to the data components described in step 2.1, each data is combined into 6-tuples respectively, and finally the network input of each data is obtained in the form of 6-tuples. The data 6-tuple includes non-illuminated harmonious image, object mask, background mask, object illumination information, background illumination information and illuminated harmonious image.

4. The method for editing illumination based on image objects according to claim 3, characterized in that: The step 3 includes the following sub-steps: Step 3.1, firstly using the 6-tuple illumination data as input information of the neural network, and performing feature extraction on it; Step 3.2: Based on the extracted features described in step 3.1, a multi-scale attention mechanism with three different scale convolution kernels of 1 × 1, 3 × 3, and 5 × 5 is used to learn the shadow information of the reference objects in the real scene to derive reasonable shadows of virtual objects.