A method and system for reconstructing the appearance of a continuous frame opaque material surface

Abstract

The application provides a continuous frame opaque material surface appearance reconstruction method and system, obtains material surface image data of the same opaque material at different times; based on the material surface image data, a pre-trained deep learning model is used to obtain a bidirectional reflectance distribution function map of the opaque material in a continuous time domain; based on the bidirectional reflectance distribution function map, a rendering equation is used to calculate a rendering image corresponding to the map, and a reconstruction result of continuous frames is obtained. The application has the ability to provide a 6D space change bidirectional reflectance distribution function reconstruction result of continuous frame space and angle, and the reconstruction result has good time and space quality.

Description

Technical Field

[0001] This invention belongs to the field of image reconstruction technology, specifically relating to a method and system for reconstructing the appearance of opaque material surfaces in consecutive frames. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Reconstructing the appearance of real-world materials from 2D images is a common problem in computer graphics. For opaque materials, their surface material appearance can be obtained by modeling using a 6D spatially varying bidirectional reflectance distribution function (SVBRDF) of space and angle.

[0004] According to the inventors, many recent works have focused on using deep learning networks to solve the BRDF reconstruction problem of a single image, and these works have achieved very good results for static materials. However, they only focus on the BRDF reconstruction of a single image, meaning they can only reconstruct the material information represented by the input image at a single time, and cannot predict the material information of the input image at other times. Therefore, using this type of network cannot directly obtain the BRDF reconstruction result of dynamic materials whose surface appearance changes over time. Although this single-frame reconstruction network can be used in conjunction with video frame interpolation methods, that is, first interpolating the input image to obtain the appearance image at an intermediate time, and then feeding it into the single-frame reconstruction network to obtain the BRDF reconstruction result at the intermediate time, the BRDF reconstruction of a single image is inherently an ill-posed problem. The output results of different frames obtained by this method may have temporal inconsistencies. This inconsistency is not conducive to post-rendering and material editing, and may cause visual jumps in the final rendering result. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method and system for reconstructing the appearance of opaque material surfaces across continuous frames. This invention has the capability to provide reconstruction results of the bidirectional reflection distribution function for 6D spatial variations in space and angle across continuous frames, and the reconstruction results possess both good temporal and spatial quality.

[0006] According to some embodiments, the present invention adopts the following technical solution:

[0007] A method for reconstructing the appearance of an opaque material surface in consecutive frames includes the following steps:

[0008] Acquire surface image data of the same opaque material at different times;

[0009] Based on the material surface image data, a pre-trained deep learning model is used to obtain the bidirectional reflection distribution function map of the opaque material in the continuous time domain.

[0010] Based on bidirectional reflection distribution function mapping, the rendering image corresponding to the mapping is calculated using rendering equations to obtain the reconstruction results of consecutive frames.

[0011] As an alternative implementation, the deep learning model is a fully convolutional deep learning network, which includes a feature extraction module, a reweighting module, and a decoder module connected in sequence.

[0012] As a further limitation, the feature extraction module is used to extract feature information of each input material image, the reweighting module is used to learn the correspondence between the time series and the feature sequence to obtain feature information at a specific time, and the decoder module is used to decode the material features at a specific time into a bidirectional reflection distribution function map.

[0013] As an alternative implementation, the training loss function of the deep learning model includes two parts: reconstruction loss and temporal domain loss, which are used to measure the reconstruction quality of the results in the spatial domain and the temporal domain, respectively.

[0014] As a further constraint, different weights are assigned to the reconstruction loss of the reconstruction features at different time points, with the weight increasing the closer the time distance to the keyframe.

[0015] As an alternative implementation, the training process of the deep learning model includes:

[0016] Using the known corresponding material rendering dataset, procedural textures are superimposed on the four channels of the bidirectional reflection distribution function of the static material to generate several dynamic material data.

[0017] The dataset is divided into a training set and a test set. The bidirectional reflectance distribution function texture and rendering results of the material images in the training set are used as the real results. The relative error between the real results and the network output results is calculated by training the loss function to determine the degree of approximation between the reconstruction results and the real results.

[0018] As an alternative implementation, the specific process of obtaining the bidirectional reflectance distribution function map of the opaque material in the continuous time domain includes: cyclically inputting a continuous time series into a trained deep learning model to obtain the bidirectional reflectance distribution function map corresponding to that time.

[0019] A system for reconstructing the appearance of an opaque material surface in consecutive frames includes:

[0020] The image acquisition module is configured to acquire image data of the same opaque material surface at different times;

[0021] The mapping module is configured to obtain the bidirectional reflectance distribution function mapping of the opaque material in the continuous time domain based on the material surface image data and using a pre-trained deep learning model.

[0022] The reconstruction module is configured to use a bidirectional reflection distribution function map and a rendering equation to calculate the rendered image corresponding to the map, thereby obtaining the reconstruction results of consecutive frames.

[0023] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.

[0024] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This invention has the ability to provide continuous frame SVBRDF reconstruction results. It integrates SVBRDF reconstruction and frame interpolation into the same deep learning network, introduces a time dimension to SVBRDF reconstruction, and makes it better applicable to dynamic material reconstruction. The reconstruction results have good temporal and spatial quality at the same time.

[0027] Compared with existing single-frame reconstruction and frame interpolation methods, this invention maintains better temporal consistency of consecutive frames while ensuring the spatial quality of reconstruction.

[0028] This invention has important applications in fields such as material rendering, and can be applied to material interpolation, dynamic material prediction, etc.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a flowchart of the method of the present invention;

[0032] Figure 2 Diagram of deep learning model architecture;

[0033] Figure 3 Images of the material to be predicted at different times;

[0034] Figure 4The BRDF texture obtained from network reconstruction;

[0035] Figure 5 This is a diagram showing the rendering results of the predicted consecutive frames. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] A method for reconstructing the appearance of an opaque material surface in consecutive frames, such as Figure 1 As shown, it includes the following steps:

[0040] Step (1): Obtain material surface image data of the same material at different times, and mark the corresponding time information, such as... Figure 3 As shown;

[0041] Step (2): Based on the pre-trained deep learning model and the material surface image, obtain the BRDF map of the material in the continuous time domain. Figure 4 BRDF textures for two adjacent time points obtained from network reconstruction;

[0042] Step (3): Based on the given rendering equation and BRDF map, calculate the rendered image corresponding to the map to obtain the reconstruction results of consecutive frames. Figure 5 This is the rendering result for ten consecutive frames.

[0043] In step (2), constructing the pre-trained deep learning model includes:

[0044] Step (2-1): Deep learning model construction. The deep learning model provided in this embodiment is a fully convolutional deep learning network, and the overall framework is as follows: Figure 2As shown, the network comprises three parts: feature extraction, reweighting module, and decoder module, which output the BRDF map corresponding to the input material at any time. The network input consists of M material images of the same material at different times and their corresponding times, and the output is a BRDF reconstruction result of a time series of length N (in this embodiment, M=5 and N=5 are set during training). Figure 3 For the 5 material images input, Figure 4 This is the output of the reconstructed consecutive frames.

[0045] The deep learning model proposed in step (2-1) specifically includes:

[0046] Step (2-1-a): Feature Extraction Module. This embodiment uses the feature extraction module in an existing two-stream network to extract feature information from each input material image, converting five 256*256 images into 640*128*128 convolutional layers. This module can be replaced by the feature extraction module in any BRDF reconstruction network.

[0047] Step (2-1-b): Reweighting Module. This module aims to learn the correspondence between the time series and the feature sequences output from step (2-1-a) to obtain feature information at a specific time. The specific architecture of this module is as follows: Figure 2 As shown, the input is a 640*128*128 convolutional layer, along with the time information corresponding to each input image and the output time value.

[0048] The specific processing flow of the reweighting module described in step (2-1-b) includes:

[0049] Step (2-1-b-1): Combine the time information corresponding to the input material image and the time information output by the network into a 1×(n+1) vector;

[0050] Step (2-1-b-2): Use a fully connected layer to convert the vector from step (2-1-b-1) into a 1×64 implicit vector z;

[0051] Step (2-1-b-3): Perform convolution operations on the input features to gradually compress the number of feature layers in order to obtain the interpolation results of the features.

[0052] In this embodiment, three convolutional layers are used to perform convolution operations, with output channels of 512, 256, and 128 respectively. The kernel size is 3, the stride is 1, and the padding edge is 1. The Leaky ReLU activation function is used after each convolution operation.

[0053] Step (2-1-b-4): After each convolution operation, the distributions μ and σ of the vector z are extracted. The existing adaptive normalization method adaIN is used to remap the features onto the distribution of the time vector, thereby establishing the relationship between time and features. A single feature compression process can be defined as:

[0054] F′:=adaIN(conv θ (F), θ z)

[0055] Where F represents the input features of each layer, and F′ represents the output features. T is an affine transformation used to obtain μ and σ of z. z is a 1*64-dimensional implicit vector representing the time vector, conv represents the convolution operation, and θ is the network parameters in the convolutional layer.

[0056] The formula for adaIN is:

[0057]

[0058] Step (2-1-b-4): After the last convolution and normalization operations, output a 128*128*128 feature information;

[0059] Step (2-1-c): Decoder Module. This embodiment uses the existing decoder module in a dual-stream network to decode the material features at a specific time into BRDF maps. The 128*128*128 feature information is decoded into four 3*256*256 BRDF maps: diffuse map, normal map, specular map, and roughness map. This module can be replaced by any decoder module.

[0060] Step (2-2): Loss Function Construction. Unlike the traditional BRDF reconstruction loss function, this embodiment constructs a new training loss function, comprising two parts: reconstruction loss and temporal domain loss, used to measure the reconstruction quality in the spatial and temporal domains, respectively. Our loss function is defined as:

[0061] L=λ recon L recon +λ temp L temp

[0062] Where λ recon , λ temp It is used to balance the weight of each item. In this embodiment, λ recon =1,λ temp =1. Reconstruction loss L recon The primary measure is the similarity between the reconstructed result and the true value, with the time-domain loss L. temp It is mainly used to measure the degree of temporal consistency between adjacent frames.

[0063] The specific definitions of reconstruction loss and temporal domain loss mentioned in step (2-2) are as follows:

[0064] Step (2-2-a): Reconstruction loss L recon Definition. To minimize spatial quality loss while ensuring temporal consistency of the output results, we fine-tuned the network's input and output. Taking N images representing the same material at different times as input, the network sequentially outputs two input images and the BRDF maps of the images between them. For example, given inputs F1, F3, F5, F7, F9, the network sequentially outputs F... 1-3 F 3-5 F 5-7 F 7-9 F 9-10 We assign different weights to the loss function for reconstructing features at different time points; the closer the time distance to the keyframe (input image), the greater the weight. recon The definition is as follows:

[0065]

[0066] Where t i ∈T, representing the time information corresponding to the input image. I is the input image sequence, and G... t L represents the BRDF map corresponding to time t obtained from network reconstruction. sr λ represents the reconstruction loss of a single image. t λ represents the loss weight corresponding to time t. t The definition is as follows:

[0067]

[0068] Among them, t i , t i+1 Let d(t) represent the time points corresponding to two adjacent input images. i , t i+1 ) represents time t i , t i+1 The distance between them, min i Represents time t and all input times t i The minimum distance.

[0069] d(t i , t i+1 The formula for ) is:

[0070] d(t i , t i+1 )=|t i -t i+1 |

[0071] Lsr (G t (I)) is defined as follows:

[0072] L sr (G t (I))=L map (G t (I))+L render (G t (I))

[0073] Where L map and L render G t (I) and the corresponding true value of L1 loss.

[0074] Step (2-2-b): Time domain loss L temp The definition of . The design of the reweighting module in the network has already ensured the temporal consistency of the output results to a certain extent. To further solve this problem, we also introduce the time domain loss L. temp This is used to measure stability between consecutive frames. It is defined as follows:

[0075]

[0076] Where G(I) represents the BRDF map of image I reconstructed by the network, GT(I) represents the true BRDF map of image I, and t i and t i+1 G represents the time corresponding to adjacent input frames, respectively. t (I) and GT t (I) represent the reconstructed BRDF texture and the real BRDF texture at time t, respectively, φ(x) t The sum of pixel-wise interpolations between the images corresponding to time t and t+1 is defined by the following formula:

[0077] φ(t)=x t+1 -x t

[0078] Where x t+1 and x t+1 The image steps (2-3) corresponding to time t+1 and time t are as follows: Construction of the training dataset. In order to generate a large number of realistic training data and their corresponding labels (BRDF maps), this embodiment is based on a publicly available large material rendering dataset. Procedural textures are superimposed on the four channels of the BRDF of static materials to generate about fifty frames of dynamic material data.

[0079] Step (2-4): Model Training. Divide the dataset obtained in step (2-3) into training and test sets in a 4:1 ratio. Use the BRDF textures and rendering results of the material images in the training set as the real results. Calculate the relative error between the real results and the network output results using the loss function given in step (2-2) to determine the degree of approximation between the reconstructed results and the real results.

[0080] In this embodiment, the Adam optimizer is used as the training optimizer, and the training learning rate is set to 10. -4 The attenuation rate is 10. -5 The batch size is set to 2.

[0081] Step (2-5): Obtain the SVBRDF reconstruction results of consecutive frames. We cyclically input a continuous time series into the trained network to obtain the BRDF map corresponding to that time. Figure 3 For the 5 material images input, Figure 4 This is the output of the reconstructed consecutive frames.

[0082] As can be seen, the method provided in this embodiment maintains better temporal consistency of consecutive frames while ensuring the spatial quality of the reconstruction.

[0083] This embodiment applies to opaque materials, including but not limited to wood, steel, ironware, fabric, and many other materials. No exhaustive list will be provided here.

[0084] It should be noted that the above is only an illustration of a specific embodiment. The setting values ​​and setting ranges of all parameters can be adjusted or changed according to different materials and different prediction / reconstruction needs.

[0085] The following product examples are also provided:

[0086] A system for reconstructing the appearance of an opaque material surface in consecutive frames includes:

[0087] The image acquisition module is configured to acquire image data of the same opaque material surface at different times;

[0088] The mapping module is configured to obtain the bidirectional reflectance distribution function mapping of the opaque material in the continuous time domain based on the material surface image data and using a pre-trained deep learning model.

[0089] The reconstruction module is configured to use a bidirectional reflection distribution function map to calculate the rendered image corresponding to the map using the rendering equation, thereby obtaining the reconstruction results of consecutive frames.

[0090] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.

[0091] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reconstructing the appearance of an opaque material surface in consecutive frames, characterized in that, Includes the following steps: Acquire surface image data of the same opaque material at different times; Based on the material surface image data, a pre-trained deep learning model is used to obtain the bidirectional reflection distribution function map of the opaque material in the continuous time domain. Based on bidirectional reflection distribution function mapping, the rendering equation is used to calculate the rendered image corresponding to the mapping and obtain the reconstruction results of consecutive frames. The deep learning model is a fully convolutional deep learning network, which includes a feature extraction module, a reweighting module and a decoder module connected in sequence. The feature extraction module is used to extract feature information of each input material image; the reweighting module is used to learn the correspondence between the time series and the feature series to obtain feature information at a specific time; and the decoder module is used to decode the material features at a specific time into a bidirectional reflection distribution function map. The training loss function of the deep learning model includes two parts: reconstruction loss and temporal domain loss, which are used to measure the reconstruction quality of the results in the spatial domain and the temporal domain, respectively. Different weights are assigned to the reconstruction loss of the reconstruction features at different time points, with the weight being greater the closer the time distance to the keyframe.

2. The method for reconstructing the appearance of an opaque material surface in consecutive frames as described in claim 1, characterized in that, The training process of the deep learning model includes: Using the known corresponding material rendering dataset, procedural textures are superimposed on the four channels of the bidirectional reflection distribution function of the static material to generate several dynamic material data. The dataset is divided into a training set and a test set. The bidirectional reflectance distribution function texture and rendering results of the material images in the training set are used as the real results. The relative error between the real results and the network output results is calculated by training the loss function to determine the degree of approximation between the reconstruction results and the real results.

3. The method for reconstructing the appearance of an opaque material surface in consecutive frames as described in claim 1, characterized in that, The specific process of obtaining the bidirectional reflectance distribution function map of the opaque material in the continuous time domain includes: cyclically inputting a continuous time series into the trained deep learning model to obtain the bidirectional reflectance distribution function map corresponding to that time.

4. A system for reconstructing the appearance of an opaque material surface in consecutive frames, characterized in that, include: The image acquisition module is configured to acquire image data of the same opaque material surface at different times; The mapping module is configured to obtain the bidirectional reflectance distribution function mapping of the opaque material in the continuous time domain based on the material surface image data and using a pre-trained deep learning model. The reconstruction module is configured to use a bidirectional reflection distribution function map to calculate the rendered image corresponding to the map using the rendering equation, thereby obtaining the reconstruction results of consecutive frames. The deep learning model is a fully convolutional deep learning network, which includes a feature extraction module, a reweighting module and a decoder module connected in sequence. The feature extraction module is used to extract feature information of each input material image; the reweighting module is used to learn the correspondence between the time series and the feature series to obtain feature information at a specific time; and the decoder module is used to decode the material features at a specific time into a bidirectional reflection distribution function map. The training loss function of the deep learning model includes two parts: reconstruction loss and temporal domain loss, which are used to measure the reconstruction quality of the results in the spatial domain and the temporal domain, respectively. Different weights are assigned to the reconstruction loss of the reconstruction features at different time points, with the weight being greater the closer the time distance to the keyframe.

5. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1-3.

6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the method according to any one of claims 1-3.

Images