A method and system for collecting plane material database based on neural network

Abstract

This invention discloses a neural network-based method and system for collecting a planar material database. The method includes a training phase, an acquisition phase, and a reconstruction phase. The present invention designs a neural network comprising an optimized illumination pattern component, a gate network component, an expert network component, and a nonlinear mapping network component. Using information from a target material photographed under the illumination pattern, the gate network adaptively selects the optimal expert network to predict a latent vector representation of the material's spatially independent reflective properties. The latent vector is further optimized under the constraints of a set of photographs taken under other illumination patterns. Finally, the latent vector is restored to a high-dimensional luminance vector through a nonlinear mapping network and fitted into a BRDF model, whose parameters are stored as a texture map. This method can robustly, efficiently, and effectively collect near-planar anisotropic SVBRDFs.

Description

Technical Field

[0001] The present invention relates to a method and system for collecting a plane material database based on a neural network, and belongs to the fields of computer graphics and computer vision. Background Art

[0002] High-quality material appearance reflects the complex physical interactions between objects and light, typically expressed as a six-dimensional spatially varying bidirectional reflectance distribution function (SVBRDF). It is a core issue in computer graphics and vision, with important applications in fields such as cultural heritage, e-commerce, computer games, and filmmaking. In computer graphics, high-quality digital material appearance can realistically render the complex physical appearance of objects that varies with position, lighting, and viewing angle. Conversely, in computer vision, material appearance can help machines better understand the real world from images.

[0003] Over the past few decades, due to the growing demand for accurate and diverse digital appearance in academia and industry, great efforts have been made to build high-value material reflectance databases. Some pioneering works have been born, including actual collection of BRDFs / SVBRDFs and synthetic SVBRDFs. However, the amount of publicly available collected material reflectance data is still limited, which hinders the development of related research in this data-driven era. For example, Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-driven Reflectance Model. ACM Trans. Graph. 22, 3 (July 2003), 759–769. published 100 measured isotropic BRDFs data, which is still used in many studies 20 years after its publication.

[0004] The main reason for this is the technical difficulty of acquiring large-scale datasets using existing techniques. Despite high acquisition quality, densely sampling the six-dimensional physical space of a single SVBRDF is extremely time-consuming and therefore not scalable to building a large database. Methods based on strong priors trade reconstruction quality for acquisition efficiency. When the prior does not hold, the quality of the results cannot be guaranteed. Even the most advanced work, with both high-quality and efficient lighting multiplexing techniques, has unsatisfactory quality when recreating the appearance of some challenging materials such as brushed metal and polished wood. Summary of the Invention

[0005] The present invention aims to address the deficiencies of the prior art and provide a method for large-scale acquisition of near-planar high-dimensional materials. The method can robustly, efficiently and effectively acquire near-planar anisotropic SVBRDFs.

[0006] The present invention designs a neural network comprising a lighting pattern optimization component, a gate network, an expert network, and a nonlinear mapping network. Using information from photos of the target material taken under the lighting pattern, the gate network adaptively selects the optimal expert network to predict a latent vector representation of the material's spatially independent reflectance properties. This latent vector is further optimized under the constraints of a set of photos taken under other lighting patterns. Finally, the latent vector is restored to a high-dimensional luminance vector through a nonlinear mapping network, fitted to a BRDF model, and its parameters stored as a texture map.

[0007] This method can use a single perspective or multiple perspectives to shoot the target material; the photo alignment method for multiple perspectives is not limited to a specific method, and other methods that can perform pixel matching on photos taken by two cameras at two perspectives are also applicable; the method of outputting probabilities of the gate network in this method is not limited to a specific method, and other methods that can output a set of probabilities with a sum of 1 are also applicable, such as the Softmax function; the lighting conditions of the photos used to optimize the latent vector in this method are not limited to a certain set of specific lighting patterns, such as the linear light source pattern used in this method, and other lighting patterns that can be illuminated by the acquisition device are also applicable, such as point light source patterns, surface light source patterns, etc.; the neural network is not limited to a fully connected network; the expression of the material properties of the object is not limited to the Lumitexel vector, the fitted BRDF model is not limited to the GGX BRDF model, and the fitting method is not limited to the differentiable neural fitting used in this method, and other fitting methods are also applicable, such as the traditional numerical method L-BFGS-B.

[0008] According to a first aspect of this specification, there is provided a method for collecting a plane material database based on a neural network, the method comprising a training phase, an acquisition phase, and a reconstruction phase;

[0009] The training phase includes the following steps:

[0010] (1) Obtain the parameters of the acquisition device and generate the acquisition results of the simulated camera as training data;

[0011] (2) Using the generated training data, a neural network is trained, wherein the characteristics of the neural network are as follows:

[0012] The input of the neural network is the Lumitexel vector under all observation directions;

[0013] The first part of the neural network is a linear fully connected layer, which is used to simulate the lighting pattern used in actual acquisition and transform the Lumitexel vector into the acquisition result of the corresponding camera;

[0014] The second part of the neural network consists of a gate network and multiple expert networks. The gate network takes the data collected by all cameras as input and outputs the probability of selecting a set of expert networks. Each expert network takes the data collected by all cameras as input and predicts the latent vector representation of the material in the latent space.

[0015] The third part of the neural network is a nonlinear mapping network, which is used to recover high-dimensional material information based on the latent vector;

[0016] The acquisition phase includes the following steps:

[0017] (1) Neural network illumination pattern acquisition: The acquisition device illuminates the target near-plane sample in sequence according to a set of illumination patterns, and all cameras obtain a set of photos respectively;

[0018] (2) Material optimization illumination pattern acquisition: The acquisition device sequentially illuminates the target near-plane sample according to a set of preset linear illumination patterns to obtain a set of photos taken by the main camera;

[0019] The reconstruction phase includes the following steps:

[0020] (1) Material prediction: Based on the photos collected in the acquisition phase (1), the results of samples collected by all cameras under different lighting patterns are used as the input of the gate network of the neural network. The gate network outputs the probability of a group of expert networks being selected. The expert network with the highest probability takes the results collected by all cameras as input and predicts the latent vector of high-dimensional material information;

[0021] (2) Material tuning: Based on the photos collected in the acquisition phase (2), the predicted latent vector is used as the initial value and restored to the luminance vector through a nonlinear mapping network. Based on the linear relationship between luminance and the luminous intensity of the light source, the acquisition process is simulated by vector multiplication. The results of the samples collected by the main camera under different linear illumination patterns are used as the target to tune the latent vector;

[0022] (3) Material fitting: The optimized latent vector is used to restore the high-dimensional material information Lumitexel through a nonlinear mapping network; for each near-plane sample, a differentiable rendering neural network is trained to obtain the sample's GGX BRDF model parameters and local coordinate system as the material acquisition result of the near-plane sample.

[0023] Furthermore, the acquisition device is installed with at least one camera facing the target near-plane material. When there are multiple cameras, the camera facing the target near-plane material is set as the main camera, and the remaining cameras are secondary cameras.

[0024] Furthermore, each value of the Lumitexel describes the reflected light intensity of the sampling point for the incident light from each light source along a certain observation direction. Lumitexel is linearly related to the luminous intensity of the light source and is simulated using a linear fully connected layer.

[0025] Furthermore, the gate network of the neural network is composed of several sub-gate networks. The number of sub-gate networks is related to the method of outputting probability of the gate network. For the method of encoding the expert network index in binary code, the gate network is composed of log2n one-bit sub-gate networks, where n is the number of expert networks. A sub-gate network takes the acquisition results of all cameras as input and outputs g(b), which represents the probability that the b-th bit of the binary index of the expert network is 1. The probability of the expert network with index a being selected is formally expressed as follows:

[0026]

[0027] Among them, a b Indicates the b-th bit of a.

[0028] Furthermore, the loss function of the neural network is as follows:

[0029] The Lumitexel vector of the sampling point under the main camera observation angle is m p , the hidden vector output by the expert network with index a is the vector m after the nonlinear mapping network is passed through a , the loss function Loss of the material feature part is expressed as follows:

[0030]

[0031] Where n is the number of expert networks, Pr(s) is the probability of the expert network with index a being selected, is a nonlinear mapping function that acts on each dimension of the vector, where l represents the light source l;

[0032] After training, the illumination matrix is transformed as the illumination pattern according to the parameters of the linear fully connected layer.

[0033] Furthermore, for the case where the total number of cameras is greater than one, the acquisition stage also includes a mapping acquisition step: the acquisition device illuminates the target near-plane sample with full white light, and each camera obtains a low-dynamic range photo. The photo of the primary camera and the photo of a secondary camera to be aligned are used as input to obtain the corresponding relationship between the pixels of the near-plane sample in the two photos.

[0034] Furthermore, the mapping collection step is specifically as follows:

[0035] Based on the augmented reality markers on the photos taken by the camera, the homography matrix between each secondary camera photo and the primary camera photo is calculated, and the secondary camera photos are transformed to the observation angle of the primary camera;

[0036] Compute dense SIFT feature vectors of the primary camera photo and the transformed secondary camera photo;

[0037] Taking the photo taken by the primary camera as the benchmark, we perform block matching calculation based on the dense SIFT feature vectors of the two photos to find the corresponding pixel on the secondary camera photo for each pixel on the primary camera photo.

[0038] Furthermore, in the material fitting step, the characteristics of the differentiable rendering neural network are as follows:

[0039] For each valid texture coordinate of the near-plane sample, the input of the network is a high-dimensional neural parameter vector, which is an optimizable variable;

[0040] The network consists of multiple layers of nonlinear fitting networks represented by fully connected layers;

[0041] The output of the network is the GGX BRDF model parameters and the local coordinate system. The error between the Lumitexel vector rendered by the GGX BRDF model parameters and the local coordinate system and the Lumitexel vector output by the nonlinear mapping network is used to optimize the high-dimensional neural parameter vector. The GGX BRDF model parameters and the local coordinate system after optimization are used as the material collection result of the near plane sample.

[0042] Furthermore, the acquisition stage also includes: the acquisition device illuminates the target near-plane sample from the bottom according to a preset lighting pattern, obtains a photo taken by the main camera, and calculates the transparency of the near-plane sample in the reconstruction stage based on the obtained photo.

[0043] According to a second aspect of this specification, a system for collecting a plane material database implemented by the above method is provided, comprising:

[0044] Preparation module: obtains the parameters of the acquisition device and generates the acquisition results of the simulated camera as training data; uses the generated training data to train the neural network;

[0045] Acquisition module: used for neural network lighting pattern acquisition and material optimization lighting pattern acquisition;

[0046] Restoration module: The samples collected by all cameras under different lighting patterns are used as input, the trained neural network is loaded, the material feature latent vector is predicted, the latent vector is then optimized based on the photos used for tuning, and the coordinate system and material parameters are fitted using a differentiable rendering neural network.

[0047] The beneficial effects of the present invention are as follows: the method of the present invention proposes a method for collecting near-planar anisotropic SVBRDFs robustly, with high quality and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A three-dimensional schematic diagram of a collection device in an embodiment of the present invention;

[0049] Figure 2 This is an external front view of a collection device in an embodiment of the present invention;

[0050] Figure 3 This is an external side view of a collection device in an embodiment of the present invention;

[0051] Figure 4 This is a side view of the interior of a collection device in an embodiment of the present invention;

[0052] Figure 5 This is an expanded diagram of a collection device in an embodiment of the present invention;

[0053] Figure 6 Flowchart of the collection method according to the embodiment of the present invention;

[0054] Figure 7 Schematic diagram of a neural network structure according to an embodiment of the present invention;

[0055] Figure 8 The lighting pattern of the mapped photo is collected and calculated for the embodiment of the present invention, and the grayscale value represents the luminous intensity;

[0056] Figure 9 A partial display of the illumination pattern obtained by an embodiment of the present invention, where the grayscale value represents the luminous intensity;

[0057] Figure 10 Partial display of a linear illumination pattern according to an embodiment of the present invention, where the grayscale value represents the luminous intensity;

[0058] Figure 11 This is a diagram showing the illumination pattern required for calculating transparency according to an embodiment of the present invention, where the grayscale value represents the luminous intensity;

[0059] Figure 12 Lumitexel vector results restored by the system using the embodiment of the present invention;

[0060] Figure 13 This is the material attribute result of the sampled object restored using the system according to the embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings.

[0062] The present invention proposes a method for collecting near-planar anisotropic SVBRDFs on a large scale, robustly, with high quality and high efficiency. The specific implementation can be divided into the following steps:

[0063] 1. Training phase, which includes the following steps:

[0064] 1. Generate training data

[0065] The acquisition device is equipped with at least one camera facing the target near-plane material. If there are multiple cameras, the camera facing the target near-plane material is set as the primary camera, and the remaining cameras are secondary cameras. The parameters of the acquisition device are obtained, including the distance and angle between the light source and the origin of the sampling space, the characteristic curve of the light source, the distance and angle between the camera and the origin of the sampling space, and the intrinsic and extrinsic parameters of the camera. These parameters are used to generate acquisition results that simulate actual cameras as training data. The rendering model used to generate training data in this embodiment is the GGX model, and the generation formula is as follows:

[0066]

[0067] ″″″

[0068] Among them, f r (ω i ,ω o ; P) is about ω i ,ω o The four-dimensional reflection function, ω i Indicates the direction of incident light in the world coordinate system, ω o Indicates the direction of the emitted light in the world coordinate system, ω i is the incident direction in the local coordinate system, ω o is the emission direction in the local coordinate system, ω h is the halfway vector in the local coordinate system. P contains the parameter information of the sampling point, including the material parameters n, t, α of the sampling point x , α y ,ρ d ,ρ s , where n represents the normal vector in the world coordinate system, t represents the x-axis direction of the local coordinate system of the sampling point in the world coordinate system, and n and t are used to transform the incident direction and the outgoing direction from the world coordinate system to the local coordinate system. x , αy represents the roughness coefficient, ρ d represents diffuse reflectivity, ρ s represents the specular reflectivity, ρ d and ρ s In single channel, it is a scalar, and in color, it is three scalars. and D GGX is the microsurface distribution term, F is the Fresnel term, G GGX Represents the shadow coefficient function.

[0069] 2. Using the generated training data, Figure 7 The neural network shown is trained. The characteristics of the neural network are as follows:

[0070] (1) The observation value B of a sampling point p on the object surface in the photo, the reflection function f r The relationship between the light intensity of each light source can be described as:

[0071]

[0072] Where I represents the luminous information of each light source l, including: the spatial position x of the light source l l , the normal vector n of the light source l l , the luminous intensity I(l) of the light source l, P contains the parameter information of the sampling point p, including: the spatial position x of the sampling point p , material parameter n p ,t,α x , α y ,ρ d ,ρ s Ψ(x l ,·) describes the light intensity distribution of light source l under different incident directions, V represents x l For x p The binary function of visibility, (·) + It is the dot product operation of two vectors, and negative values will be truncated to 0. r (ω′ i ;ω′ o ,P) is ω′ o When fixed, about ω′ i The two-dimensional reflection function of .

[0073] The input of the neural network is the Lumitexel vector sampled in the main camera observation direction, denoted as m p (l; P) and the Lumitexel vector sampled in the secondary camera viewing direction, denoted as m s(l; P), where each value describes the reflected light intensity of the sampling point for the incident light from each light source along a certain observation direction. Lumitexel is linearly related to the luminous intensity of the light source and is simulated by a linear fully connected layer;

[0074]

[0075]

[0076] in, and are the viewing directions of the primary and secondary cameras, respectively.

[0077] (2) The first part of the neural network is the linear fully connected layer, which is used to simulate the illumination pattern used in actual acquisition and transform the Lumitexel vector into the acquisition result of the corresponding camera; the linear fully connected layer is composed only of the parameter matrix and does not include the nonlinear activation function. The parameter matrix of the linear fully connected layer is trained by the following formula:

[0078] W l =f W (W raw )

[0079] Among them, W raw is the parameter to be trained; W l Is the lighting matrix, for a single-channel light source, the size is 1×n l , for colored light sources, the size is k×n l ;n l is the length of vector m, i.e. the sampling accuracy of Lumitexel, k is the number of illumination patterns; f W is a mapping for W raw The transformation is performed so that the generated lighting matrix can correspond to the possible luminous intensity of the light source.

[0080] The linear fully connected layer is represented as follows:

[0081] y1=m·W l

[0082] Among them, y1 is the output of the first layer network.

[0083] (3) The second part of the neural network includes a gate network and multiple expert networks. The gate network takes the acquisition results of all cameras obtained in step (2) as input and outputs the probability of a set of expert networks being selected. Each expert network takes the acquisition results of all cameras as input and predicts the latent vector representation of the material in the latent space.

[0084] The gate network can directly output the probability of a group of expert networks being selected through the softmax function, or construct a multi-sub-gate network structure; in this embodiment, the gate network is composed of γ sub-gate networks, and the value of γ is related to the method of outputting the probability of the gate network. For the method of encoding the expert network index in binary code, the gate network is composed of log2n one-bit sub-gate networks, where n is the number of expert networks. In this embodiment, n is 128, but not limited to 128. Each sub-gate network takes the output of the first part of the neural network as input and outputs g(b), which represents the probability that the b-th bit of the binary index of the expert network is 1. The probability of the expert prediction network with binary index a being selected is formally expressed as follows:

[0085]

[0086] Among them, a b Represents the bth bit.

[0087] Each expert network takes the acquisition results of all cameras as input and predicts the latent vector representation of the material in the latent space;

[0088]

[0089]

[0090] in, i is the mapping function of the i-th layer network, W i is the parameter matrix of the i-th layer network, b i is the offset vector of the i-th layer network, y i is the output of the i-th layer network, Z A With Z S Represents the albedo part and the shape part respectively, input and is the collection result of all cameras, and are the maximum number of layers of the two branches, and the output Z A and Z S They are 8-dimensional albedo latent vector and 48-dimensional shape latent vector, which are combined into a 56-dimensional latent vector, expressed as follows:

[0091] Z=concat[Z A ,Z S ]

[0092] In the above formula, Z is represented under a single channel. When representing a three-channel material, Z is expanded to the following form:

[0093] Z 3c =concat[Z A_R ,Z A_G ,ZA_B ,Z S ]

[0094] The dimensions of the albedo latent vector and shape latent vector are not limited to 8 and 48.

[0095] (4) The third part of the neural network is the nonlinear mapping network after the expert network, which is used to recover the high-dimensional material information based on the latent vector, and is expressed as follows:

[0096] y i =f i (y i-1 W i +b i ),n r ≥i≥n l

[0097] Among them, f i is the mapping function of the i-th layer network, W i is the parameter matrix of the i-th layer network, b i is the offset vector of the i-th layer network, y i is the output of the i-th layer network, and the input is the hidden vector output by the expert network, n r is the maximum number of layers in the network.

[0098] (5) The loss function of the neural network is designed as follows:

[0099] The Lumitexel vector of the sampling point under the main camera observation angle is m p , the hidden vector output by the expert network with index a is output as vector m after the nonlinear mapping network a , where m a With m p The length is the same, and the loss function of the material feature part is expressed as follows:

[0100]

[0101] in, is a nonlinear mapping function that acts on each dimension of the vector. In actual use, a logarithmic function or other compression range functions can be used; n is the number of expert networks.

[0102] 3. After the training is completed, the parameters W of the linear fully connected layer of the network are raw Take out, through the formula W l =f W (W raw ) is transformed as the lighting pattern.

[0103] 2. Collection Phase

[0104] The acquisition stage can be further divided into mapping acquisition and material acquisition, and the material acquisition can be further divided into the neural network lighting pattern acquisition stage, the material optimization lighting pattern acquisition stage and the transparency lighting pattern acquisition stage.

[0105] 1. Mapping acquisition phase

[0106] When the total number of cameras is greater than one, the acquisition device illuminates the target near-plane sample with full white light. Each camera obtains a low-dynamic-range photo. Using the photo as input, the pixel correspondence between the near-plane sample in the two photos is obtained as follows:

[0107] (1) Based on the augmented reality markers ARTags on the photos taken by the camera, the homography matrix from each secondary camera photo to the primary camera photo is calculated, and the secondary camera photos are transformed to the observation angle of the primary camera. The transformation process can be described as:

[0108] H = findHomography(Img p ,Img s )

[0109] Among them, findHomography is the function for calculating the homography matrix, Img p and Img s are the photos taken by the primary camera and the secondary camera respectively, and H is the calculated homography matrix;

[0110] Img′ s =warpPerspective(Img s ,H)

[0111] Among them, warpPerspective is the perspective transformation function, Img′ s This represents the result of transforming the photo taken by the secondary camera to the viewing angle of the primary camera.

[0112] (2) Calculate the dense SIFT feature vectors of the photos taken by the primary camera and the transformed photos taken by the secondary camera;

[0113] (3) Taking the photo taken by the primary camera as the benchmark, perform block matching PatchMatch calculation based on the dense SIFT feature vectors of the two photos to find the corresponding pixel on the secondary camera photo for each pixel on the primary camera photo.

[0114] 2. Material collection stage

[0115] (1) Neural network illumination pattern acquisition: The acquisition device illuminates the target near-plane sample according to a set of illumination patterns, and all cameras obtain a set of photos respectively; the illumination pattern corresponds to the first layer illumination matrix part W of the network. l ;

[0116] (2) Material optimized illumination pattern acquisition: The acquisition device sequentially illuminates the target near-plane sample according to a set of preset optimized illumination patterns to obtain a set of photos taken by the main camera; the optimized illumination pattern can adopt a linear illumination pattern;

[0117] (3) The acquisition device illuminates the target near-plane sample from the bottom according to a preset lighting pattern to obtain a photo taken by the main camera.

[0118] III. Reconstruction Phase

[0119] The reconstruction stage can be further divided into the material prediction stage, the material tuning stage, the material fitting stage and the transparency calculation stage.

[0120] 1. Material prediction stage

[0121] All cameras collect several groups of photos where p and s1...s n Represents the main camera and all secondary cameras respectively, k represents k illumination patterns, first calculate the mapping relationship between each secondary camera photo and the main camera, and transform the photo into For the valid texture coordinates on the sampling plane sample, find its pixel value in all camera photos and form a vector Take v as the input of the gate network in the neural network. The gate network outputs the probability of a set of expert networks being selected. The expert network with the largest probability is selected to predict the latent vector of high-dimensional material information.

[0122] 2. Material tuning stage

[0123] Based on the photos collected in the material acquisition stage (2), the predicted latent vector is used as the initial value and restored to the Lumitexel vector through a nonlinear mapping network. According to the linear relationship between Lumitexel and the luminous intensity of the light source, the acquisition process is simulated by vector multiplication. The results of the samples collected by the main camera under different linear lighting patterns are used as the target to tune the latent vector.

[0124] Specifically, according to the photos collected in the material collection stage (2) {Li p1 ,Li p2 ,…,Li pj}, j represents the jth linear light source illumination pattern. Taking the above predicted latent vector as the initial value, first convert the photos of the samples captured by the main camera under different linear illumination patterns into grayscale photos, and for each valid texture coordinate u on the sampled near plane sample, tune its 56-dimensional latent vector:

[0125]

[0126] in, Represents the pixel value corresponding to the valid texture coordinate u in the grayscale photo, Represents the latent vector prediction value of the effective texture coordinate u output by the neural network, LT represents the nonlinear mapping part in the neural network, WL j Represents the jth linear lighting pattern, and the optimization result is

[0127] Next, Make three copies of the photo. p1 ,Li p2 ,…,Li pj}, optimize the R, G, and B channels separately The 8-dimensional latent vector part representing the reflectivity Fix the part of the latent vector that represents the shape The expression is as follows:

[0128]

[0129]

[0130]

[0131] in They represent the R, G, and B channel values of the pixel corresponding to the effective texture coordinate u in the photo, and the optimization result is recorded as

[0132] Next, As a shared part, optimize and It is expressed as follows:

[0133]

[0134]

[0135]

[0136] The optimization result is recorded as Synthesize the latent vector representing the three channels:

[0137]

[0138] 3. Material fitting stage

[0139] The optimized latent vector can be used to recover the high-dimensional material information Lumitexel through a nonlinear mapping network. For each near-plane sample, a differentiable rendering neural network is trained to obtain the GGX BRDF model parameters of the sample. The characteristics of the differentiable rendering neural network are as follows:

[0140] (1) For each valid texture coordinate of a near-plane sample, the network input is a high-dimensional neural parameter vector, which is an optimizable variable;

[0141] (2) The network consists of multiple layers of nonlinear fitting networks represented by fully connected layers;

[0142] (3) The output of the network is the GGX BRDF model parameters and the local coordinate system. Based on the error between the Lumitexel vector rendered by the GGX BRDF model parameters and the local coordinate system and the Lumitexel vector output by the nonlinear mapping network, the high-dimensional neural parameter vector is optimized. The GGX BRDF model parameters and the local coordinate system after optimization are used as the material acquisition result of the near-plane sample. Specifically, the loss function of the neural network is designed as follows:

[0143]

[0144] in, The three single-channel Lumitexels are synthesized into a three-channel Lumitexel. θ is the high-dimensional neural parameter input of the network, G is the nonlinear fitting network, G(θ) outputs the GGX BRDF model parameters and local coordinate system parameters, R is the traditional rendering equation, and G(θ) is rendered as a three-channel Lumitexel under the main camera perspective. The network error is propagated through the gradient, which will optimize the parameters in θ and G. When the network converges, G(θ) is saved as a texture map as the fitting result.

[0145] 4. Transparency calculation stage

[0146] Based on the photos collected in the material collection stage (3), the transparency of the near plane samples is calculated.

[0147] Specifically, a method already disclosed in the art (Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3 (2003), 749–

[0148] 758.) The transparency of the near-plane sample is calculated based on two photos. Both photos are illuminated by bright white light from the light source at the bottom of the acquisition device. One photo is taken before the sample is placed on the sample placement table, and the other is taken after the sample is placed. The transparency of each valid texture coordinate on the sample is determined by the quotient of its pixel value in the photo with the sample and the photo without the sample.

[0149] Specifically, during the material prediction, material tuning, and transparency calculation processes in the reconstruction phase, the photos taken by the camera must first be flat-field corrected, dedistorted, and color corrected.

[0150] The following is a specific example of a collection device system. Figure 1 For the three-dimensional display of the system instance, Figure 2 This is the external front view of the system instance. Figure 3 This is the external side view of the system example. Figure 4 This is the internal side view of the system instance. Figure 5 This is a diagram of the system's internal structure. The acquisition device consists of a six-sided light panel with a camera mounted on the top and another on the side for image acquisition. A sliding drawer is installed 10 cm from the bottom light panel. The drawer houses a sample platform. During acquisition, the near-surface sample can be changed by sliding the drawer in and out. The top camera, referred to as the primary camera, looks at the center of the sample platform at a 90-degree angle. The side cameras, referred to as secondary cameras, look at the center of the sample platform at a 45-degree angle. The six-sided light panel is densely packed with LEDs: 4096 on each of the top and bottom panels, and 2048 on each of the four side panels, for a total of 16,384. The LEDs are controlled by an FPGA, which adjusts the brightness and duration of the light.

[0151] The following is an example of a data acquisition system using the method of the present invention. The system is generally divided into the following modules:

[0152] Preparation module: Provides a data set for network training. This part uses the GGX model. By inputting a set of BRDF parameters, the spatial position of the point, and the positions of two cameras, two reflection situations can be obtained. The network training part uses the Pytorch open source framework and uses the Adam optimizer for training. The network structure is as follows Figure 7 As shown in the figure, each rectangle represents a layer of neurons, and the number inside the rectangle indicates the number of neurons in that layer. The leftmost layer is the input layer, and the rightmost layer is the output layer. The solid arrows between layers indicate full connections.

[0153] Acquisition module: equipment such as Figure 1 、 2 , 3, 4, and 5, the specific structure has been described above.

[0154] Restoration module: Use the pixel position of the sample in the photo and the calibrated drawer position to estimate the geometric model of the sample, use the model to calculate the geometric model of the sample with texture coordinates, load the trained neural network, and predict the material feature latent vector for each vertex on the near-plane sample geometric model with texture coordinates. Then, optimize the latent vector based on the photo used for tuning, and use a differentiable fitting network to fit the coordinate system and material parameters.

[0155] Figure 6 This is the workflow of this embodiment. First, generate training data, randomly sample 200 million sets of material parameters, render the Lumitexels corresponding to the two cameras, take 80% as the training set, and the rest as the validation set. When training the network, use the Xavier method to initialize the parameters, and the learning rate is 1e-4. The illumination pattern is single-channel light, and the size of the illumination matrix is (64,16384). After the training is completed, the illumination matrix is taken out and transformed into an illumination pattern. The parameters of each column specify the luminous intensity of the light source at that position. Figure 7 The following process is as follows: 1. The acquisition device light board presses Figure 8 The light pattern emits light, and two cameras shoot the object at the same time, obtaining two shooting results; the light board of the acquisition device presses Figure 9 The light pattern emits light, and two cameras shoot the object at the same time to obtain a set of shooting results; the light board of the acquisition device presses Figure 10 The light pattern emits light, and the main camera shoots the object to obtain a set of shooting results; the light board of the acquisition device presses Figure 11 The light pattern emits light, and the main camera shoots the object to obtain a shooting result. 2. For the geometric model of the sampled plane object, use Isochart to obtain the geometric model with texture coordinates. 3. According to the light pattern Figure 8 The mapping relationship between the secondary camera pixels and the primary camera pixels is calculated for the two photos below; the secondary camera is placed in the lighting pattern according to the mapping relationship Figure 9 Convert the photo to the main camera perspective; load the neural network and extract the lighting pattern for each vertex on the geometric model with texture coordinates Figure 9 The pixel values of the next two camera photos are used as the input of the second part of the neural network to recover the latent vector. Figure 10 The hidden vector output by the network is optimized based on the photos taken by the main camera under Figure 11 5. Train a differentiable fitting network to fit the rendering coordinate system and roughness, specular reflectivity, and diffuse reflectivity to each vertex on the sample.

[0156] Figure 12Two Lumitexel vectors in the validation set are recovered using the above system. The left column is m p , the right column is the corresponding m a .

[0157] Figure 13 The material properties of the sampled objects are restored by scanning the material appearance using the above system. The first row represents the sampled objects. Three components, the second row represents the sampled objects Three components, the third row represents the roughness coefficient α of the sampled object x , α y , the gray value represents the numerical value, the fourth line represents the transparency coefficient of the sample, and the gray value represents the numerical value.

[0158] The above description is merely a preferred embodiment, and the present invention is not limited to the above embodiments. Any method that achieves the technical effects of the present invention by the same means shall fall within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementations are possible.

Claims

1. A method for collecting a plane material database based on a neural network, characterized in that: It includes training phase, acquisition phase and reconstruction phase; The training phase includes the following steps: (1) Obtain the parameters of the acquisition device and generate the acquisition results of the simulated camera as training data; The acquisition device is equipped with at least one camera facing the target near-plane material. If there are multiple cameras, the camera facing the target near-plane material is set as the primary camera, and the remaining cameras are secondary cameras. (2) Using the generated training data, a neural network is trained, wherein the characteristics of the neural network are as follows: The input of the neural network is the Lumitexel vector under all observation directions; The first part of the neural network is a linear fully connected layer, which is used to simulate the lighting pattern used in actual acquisition and transform the Lumitexel vector into the acquisition result of the corresponding camera; The second part of the neural network consists of a gate network and multiple expert networks. The gate network takes the data collected by all cameras as input and outputs the probability of selecting a set of expert networks. Each expert network takes the data collected by all cameras as input and predicts the latent vector representation of the material in the latent space. The third part of the neural network is a nonlinear mapping network, which is used to recover high-dimensional material information based on the latent vector; The acquisition phase includes the following steps: (1) Neural network illumination pattern acquisition: The acquisition device illuminates the target near-plane sample in sequence according to a set of illumination patterns, and all cameras obtain a set of photos respectively; (2) Material optimization illumination pattern acquisition: The acquisition device sequentially illuminates the target near-plane sample according to a set of preset linear illumination patterns to obtain a set of photos taken by the main camera; The reconstruction phase includes the following steps: (1) Material prediction: All cameras collect several sets of photos where p and s1...s n Represents the main camera and all secondary cameras respectively, k represents the kth lighting pattern, first calculate the mapping relationship between each secondary camera photo and the main camera, and transform the photo into For the valid texture coordinates on the sampling plane sample, find its pixel value in all camera photos and form a vector Take v as the input of the gate network in the neural network. The gate network outputs the probability of a set of expert networks being selected. The expert network with the highest probability takes all camera acquisition results as input and predicts the latent vector of high-dimensional material information. (2) Material tuning: Based on the photos collected in the acquisition phase (2), the predicted latent vector is used as the initial value and restored to the luminance vector through a nonlinear mapping network. Based on the linear relationship between luminance and the luminous intensity of the light source, the acquisition process is simulated by vector multiplication. The results of the samples collected by the main camera under different linear illumination patterns are used as the target to tune the latent vector; (3) Material fitting: The optimized latent vector is used to restore the high-dimensional material information Lumitexel through a nonlinear mapping network; for each near-plane sample, a differentiable rendering neural network is trained to obtain the sample's GGX BRDF model parameters and local coordinate system as the material acquisition result of the near-plane sample.

2. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: Each value of the Lumitexel describes the reflected light intensity of the sampling point for the incident light from each light source along a certain observation direction. Lumitexel is linearly related to the luminous intensity of the light source and is simulated by a linear fully connected layer.

3. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: The gate network of the neural network is composed of several sub-gate networks. The number of sub-gate networks is related to the method of outputting probability of the gate network. For the method of encoding the expert network index in binary code, the gate network is composed of log2n one-bit sub-gate networks, where n is the number of expert networks. A sub-gate network takes the acquisition results of all cameras as input and outputs g(b), which represents the probability that the b-th bit of the binary index of the expert network is 1. The probability of the expert network with index a being selected is formally expressed as follows: Among them, a b Indicates the b-th bit of a.

4. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: The loss function of the neural network is as follows: The Lumitexel vector of the sampling point under the main camera observation angle is m p , the hidden vector output by the expert network with index a is the vector m after the nonlinear mapping network is passed through a , the loss function Loss of the material feature part is expressed as follows: Where n is the number of expert networks, Pr(a) is the probability of the expert network with index a being selected, is a nonlinear mapping function that acts on each dimension of the vector, where l represents the light source l; After training, the illumination matrix is transformed as the illumination pattern according to the parameters of the linear fully connected layer.

5. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: For the case where the total number of cameras is greater than one, the acquisition stage further includes a mapping acquisition step: the acquisition device illuminates the target near-plane sample with full white light, and each camera obtains a low-dynamic range photo. The photo of the primary camera and the photo of a secondary camera to be aligned are used as input to obtain the pixel correspondence between the near-plane samples in the two photos.

6. The method for collecting a plane material database based on a neural network according to claim 5, characterized in that: The mapping acquisition steps are specifically as follows: Based on the augmented reality markers on the photos taken by the camera, the homography matrix between each secondary camera photo and the primary camera photo is calculated, and the secondary camera photos are transformed to the observation angle of the primary camera; Compute dense SIFT feature vectors of the primary camera photo and the transformed secondary camera photo; Taking the photo taken by the primary camera as the benchmark, we perform block matching calculation based on the dense SIFT feature vectors of the two photos to find the corresponding pixel on the secondary camera photo for each pixel on the primary camera photo.

7. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: In the material fitting step, the characteristics of the differentiable rendering neural network are as follows: For each valid texture coordinate of the near-plane sample, the input of the network is a high-dimensional neural parameter vector, which is an optimizable variable; The network consists of multiple layers of nonlinear fitting networks represented by fully connected layers; The output of the network is the GGX BRDF model parameters and the local coordinate system. The error between the Lumitexel vector rendered by the GGX BRDF model parameters and the local coordinate system and the Lumitexel vector output by the nonlinear mapping network is used to optimize the high-dimensional neural parameter vector. The GGX BRDF model parameters and the local coordinate system after optimization are used as the material collection result of the near plane sample.

8. The method for collecting a plane material database based on a neural network according to claim 1, characterized in that: The acquisition phase also includes: the acquisition device illuminates the target near-plane sample from the bottom according to a preset lighting pattern to obtain a photo taken by the main camera, and calculates the transparency of the near-plane sample in the reconstruction phase based on the obtained photo.

9. A system for collecting a plane material database implemented by the method according to any one of claims 1 to 8, characterized in that: The system includes: Preparation module: obtains the parameters of the acquisition device and generates the acquisition results of the simulated camera as training data; uses the generated training data to train the neural network; Acquisition module: used for neural network lighting pattern acquisition and material optimization lighting pattern acquisition; Restoration module: The samples collected by all cameras under different lighting patterns are used as input, the trained neural network is loaded, the material feature latent vector is predicted, the latent vector is then optimized based on the photos used for tuning, and the coordinate system and material parameters are fitted using a differentiable rendering neural network.

Images