A method of capturing fabric appearance from a single image

Abstract

The application discloses a method for capturing fabric appearance from a single image, proposes a forward fabric shading model and an inverse process framework for estimating model parameters, including a neural network and a differentiable rendering-based optimization strategy, the forward shading model regards fabric yarns as curved and twisted cylinders, and uses a microflake-based BRDF model to shade them, the model can generate realistic rendering effects on a large class of woven fabrics, and is suitable for differentiable rendering. Real fabric data is captured using a simple and inexpensive fabric capture configuration, a neural network trained on a large synthetic dataset generated using the forward shading model is used to estimate the parameters of the forward model from a single captured image, and then the parameters are further optimized through differentiable rendering to produce more accurate results. The method realizes high-quality reconstruction of the captured fabric sample, and can be used for efficient rendering and further editing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer graphics rendering, in particular, a method for capturing fabric appearance from a single image. BACKGROUND

[0002] Capturing the appearance of fabric is an important task in many applications of realistic rendering, from indoor scenes to virtual characters. Previous work ranges from applying general shading models that are not suitable for fabric, to data-driven methods that require expensive equipment and large amount of data to scan fabric, none of which can efficiently convert a single fabric photo into a high-quality reconstruction that can achieve compact storage and efficient rendering.

[0003] In recent years, many methods have been proposed for recovering material appearance. Early methods use integrated capture systems with controlled lighting and camera arrays to capture spatially-varying bidirectional reflectance distribution functions (SVBRDF), which are very accurate but require expensive equipment and a lengthy capture process. Another method is for artists to manually create materials in a material editing software, which requires trained artists and is a complex and time-consuming process. More lightweight methods often take a single or multiple images as input, based on some specific prior knowledge to optimize the SVBRDF capture process. With the development of deep learning technology, different neural network structures and training strategies have been proposed in academia. For the acquisition of SVBRDF per pixel of a single image, Li et al. proposed a convolutional neural network on a limited labeled SVBRDF training pair. Deschaintre et al. combined U-Net and fully connected networks to extract features from each stage, while using a rendering loss to optimize training, and later proposed a method to fine-tune a plane under ambient lighting conditions. Guo et al. proposed a dual-stream network structure, with a highlight-aware convolution module that can handle highlights in the image to eliminate their effects. Zhao et al. proposed a generative adversarial network to handle the recovery and synthesis of SVBRDF. Henzler used a latent space vector to handle the recovery of SVBRDF, which can be further edited. Hu et al. proposed an inverse procedural modeling framework, training an unsupervised clustering model to automatically select the most suitable procedural model, and then using a CNN to map the input image to the material parameters in the model. Shi et al. used a similar method to capture materials into a production-level procedural material model using a differentiable Adobe Substance material graph. These methods are all designed for general materials, which are not ideal for fabric. The present application follows a similar framework, but with more improvements designed for high-precision fabric reconstruction.

[0004] The most commonly used material representation model is the microfacet model, which assumes that the surface of an object is composed of many microsurfaces with different orientations, and each microsurface is absolutely smooth, which can represent the appearance of metal, plastic and other surfaces. Most of the previous material restoration methods use the microfacet model, but fabric is usually woven by yarns, and the microfacet model cannot model fabric realistically, so most of the previous material restoration methods do not include fabric material, or the reconstructed fabric appearance is not realistic. Jakob first proposed the microflake model to represent anisotropic media, which is suitable for modeling volumetric data. The model represents the object as volumetric data with microflakes distributed inside, which can represent fabric, leaves and hair materials. Recently, Wang et al. proposed the SpongeCake model, which defines each layer of the model as a volumetric medium represented by microflakes. The model of the present application is further modified on the basis of Wang to enhance the realism of the fabric. SUMMARY

[0005] The present application aims to provide a method for capturing the appearance of fabric from a single image, avoiding the collection of heavy data sets and achieving high-quality fabric appearance reconstruction.

[0006] To achieve the above-mentioned purpose, the present application provides a method for capturing the appearance of fabric from a single image, comprising the following steps:

[0007] Step one, build a fabric capture scene, take pictures of the fabric, and perform preprocessing operations such as cropping and adjusting resolution;

[0008] Step two, edit the fabric yarn weaving pattern and export the corresponding normal and direction maps;

[0009] Step three, use the forward shading model, sample other parameters of the model randomly according to the maps exported in step two, and generate a training data set;

[0010] Step four, construct a network model, train on the data set generated in step three, and obtain a trained network model;

[0011] Step five, input the measured fabric image into the trained network model to obtain a predicted parameter set, and use the inverse process framework to optimize the parameters.

[0012] Step (1.1) is specifically, place a cylinder on the plane, the cylinder radius 3.5 cm, length 10 cm, the fabric sample is wrapped around the cylinder; In front of the cylinder placed a mobile phone as a camera, cylinder in the center of the camera view position; In front of the cylinder placed another mobile phone, using its flash as a point light source; Vertically placed a gray card behind the cylinder, using the camera to shoot the gray card to adjust the exposure, white balance parameters, fixed shooting parameters;

[0013] Step (1.2) is specifically, using the fixed shooting parameters of step (1.1) to shoot the fabric photo, the image obtained by shooting is converted to a png format file, the edge of the image is cut to keep the middle part, and the resolution is adjusted to 300x800 to obtain the final image.

[0014] Step two is specifically, configuring the weaving method of the yarn in the fabric weaving pattern file, specifying the type, center position, length and width, twisting angle, bending angle and other attributes of the yarn, modeling the yarn as a curved and twisted cylinder, and calculating the normal map n and direction map t of the yarn according to the attributes, and the calculation formulas are respectively:

[0015]

[0016]

[0017] Wherein u, v respectively represent the horizontal coordinate and vertical coordinate of the yarn, and ψ represents the twisting angle of the yarn, and the function R is the component of the initial vertical vector rotating along the x or y axis.

[0018] Step three is specifically, randomly sampling the roughness a of the yarn, the height field scaling factor β, the highlight color k d , diffuse reflection color k s , Lambertian term weight w, height field noise factor U s , color noise factor U n , warp and weft repetition times s to obtain a group of parameters P; Write a program to establish the same simulation scene as the measured fabric described in step one, use the forward shading model, use the parameter group P to render to obtain an image M, and the image and the rendering parameter group are used as a sample of the data set; At least 2000 samples are generated for different normal maps and direction maps to form a training data set. The forward shading model rendering formula is:

[0019]

[0020]

[0021]

[0022] wherein

[0023] wherein q=h T S -1 h, and

[0024] wherein f r is the shading point rendering formula, is the specular reflection term, is the diffuse reflection term, omega i is the incident angle of the shading point, omega o is the exit angle of the shading point, omega m is the macroscopic surface normal of the shading point, omega n is the microscopic surface normal of the shading point, D(h) is the normal distribution function, G(w i , w o ) is the shadow-occlusion function, h is the half vector of w i and w o , S is the microflake distribution matrix; according to the color noise factor U n Berlin noise is generated and multiplied with the highlight color, according to the height field noise factor U s Berlin noise is generated and multiplied with the height field scaling factor beta, the height field scaling factor beta scales omega n , and thus adds noise to the rendered image.

[0025] The fourth step is specifically using a pre-trained VGG-19 neural network to extract the Gram matrix of the image, and connecting the Gram matrix with a two-layer 256-node fully connected neural network to form a training network, wherein the VGG does not participate in weight updating during training; for an input image M, first, subtract the image mean value mean pixel by pixel, and then divide by the image standard deviation std to obtain a preprocessed image M':

[0026]

[0027] The image M' is input into the training network to obtain an output parameter P', and the loss is calculated using a mean absolute error loss function:

[0028]

[0029] The training network is trained until the loss tends to be stable and does not decrease any more, and the final network model is obtained.

[0030] Step five is specifically, the captured image M is input into the network model trained in step four, to obtain the predicted parameter group P, using the forward shading model to render to obtain the image M'; using a two-dimensional normal distribution function to create an attenuation mask, simulating the light attenuation effect of the mobile phone flash, multiplying the attenuation mask and the image M' to obtain the image M"; inputting the images M and M" into the pre-trained VGG network respectively to obtain their respective Gram matrices, using the mean absolute error loss function to calculate the style loss L g :

[0031]

[0032] Downsample the images M and M" to 8x16 resolution, and use the mean absolute error loss function to calculate the style loss L c :

[0033]

[0034] For the height field scaling factor β in the predicted parameter group P, according to its mean μ β and standard deviation σ β , the L p :

[0035]

[0036] The loss function L opt includes the style loss L g , the color loss L c and the prior loss L p , and each loss is added with different weights, w1 and w2 are the weights corresponding to the losses:

[0037] L opt = L g +w1L p +w2L c

[0038] For the repeat times of warp and weft, the height field noise factor and the color noise factor, the optimization strategy is to perform a random disturbance once every 5 optimizations, the repeat times of warp and weft are increased or decreased by 10 within one hundred optimizations, increased or decreased by 5 for 100-150 optimizations, and increased or decreased by 2 after 150 optimizations; the height field noise factor and the color noise factor are increased or decreased by 1 each time the random disturbance is performed; after 200 iterations of optimization, the final optimized parameters P opt .

[0039] Compared with the prior art, the present application has the following beneficial effects: (1) by using a coloring method based on a microflake model, a real fabric appearance can be rendered, high-quality rendering is generated at the yarn level, while keeping simple implementation, supporting efficient differentiable rendering and generation of synthetic data; (2) the reverse framework of the present application can optimize fabric parameters that match the synthetic data and the captured data, realize high-quality fabric appearance capture, and further editing can be performed. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a method for capturing fabric appearance from a single image according to the present application.

[0041] Figure 2 A fabric capture scene configuration according to step one of the method of the present application.

[0042] Figure 3 A yarn modeling schematic according to step two of the method of the present application.

[0043] Figure 4 A refinement flowchart according to step four of the method of the present application.

[0044] Figure 5 A network structure schematic according to step four of the method of the present application.

[0045] Figure 6 A refinement flowchart according to step five of the method of the present application.

[0046] Figure 7 A reconstruction effect diagram on two kinds of fabrics using the method of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific implementation methods described herein are only used to explain the present application and do not limit the present application.

[0048] As shown in Figure 1 , a method for capturing fabric appearance from a single image according to the present application includes the following steps:

[0049] Step one, build a fabric capture scene, take fabric images, and perform preprocessing operations, including the following substeps:

[0050] (1.1) as shown in Figure 2As shown, a cylinder with a radius of 3.5 cm and a length of 10 cm is placed on a plane, and the fabric sample is wrapped around the cylinder; a mobile phone is placed 12 cm in front of the cylinder to act as a camera, with a height of 8 cm from the plane; another mobile phone is placed 37 cm in front of the cylinder to act as a point light source, with a height of 36 cm from the plane; a gray card is placed vertically behind the cylinder, and the camera is used to shoot the gray card to adjust the exposure and white balance parameters, and the shooting parameters are fixed;

[0051] (1.2) Use fixed shooting parameters to shoot the fabric, convert the captured image to a png format file, crop the edges to retain the middle part, and adjust the resolution to 300x800 to obtain the final image.

[0052] Step two, configure the weaving method of the yarn in the fabric weaving pattern file, specify the type, center position, length and width, twist angle, bending angle and other attributes of the yarn, and model the yarn as a curved and twisted cylinder, Figure 3 The geometric modeling diagram of the yarn is shown, and the normal map n and the direction map t of the yarn are calculated according to these attributes, and the calculation formulas are respectively:

[0053]

[0054]

[0055] Where u, v represent the horizontal and vertical coordinates of the yarn respectively, ψ represents the twist angle of the yarn, and the function R is the component of the initial vertical vector rotating along the x or y axis.

[0056] Step three, randomly sample the roughness α of the yarn, the height field scaling factor β, the highlight color k d , diffuse reflection color k s , Lambertian term weight w, height field noise factor U s , color noise factor U n , warp and weft repetition number s to obtain a group of parameters P; write a program to establish the same simulation scene as the measured fabric described in step one, use the forward shading model, and use the parameter group P to render an image M. The image and the rendering parameter group are used as a sample of the data set; 2000 samples are generated for different normal maps and direction maps respectively to form the training data set. The forward shading model rendering formula is:

[0057]

[0058]

[0059]

[0060] wherein

[0061] wherein q = h T S -1 h, and

[0062] wherein f r is the shading point rendering formula, is the specular reflection term, is the diffuse reflection term, ω i is the incident angle of the shading point, ω o is the exit angle of the shading point, ω m is the macroscopic surface normal of the shading point, ω n is the microscopic surface normal of the shading point, D(h) is the normal distribution function, G(w i , w o ) is the shadow-occlusion function, h is the half vector of w i and w o , S is the microflake distribution matrix; the color noise factor U n is used to generate the Berlin noise multiplied by the highlight color, the height field noise factor U s is used to generate the Berlin noise multiplied by the height field scaling factor β, and the height field scaling factor β scales ω n , thereby adding noise to the rendered image.

[0063] Step four, a Gram matrix of the image is extracted using a pre-trained VGG-19 neural network, and a two-layer 256-node fully connected neural network is connected to form a training network, and the VGG is not involved in weight updating during training, and the network structure diagram is as shown in Figure 5 For an input image M, first, subtract the image mean value mean pixel by pixel, and then divide by the image standard deviation std to obtain the preprocessed image M':

[0064]

[0065] The image M' is input into the training network to obtain the output parameter P', and the loss is calculated using the mean absolute error loss function:

[0066]

[0067] The training network is trained until the loss tends to be stable and no longer decreases, and the final network model is obtained.

[0068] Step 5: Input the captured image M into the network model trained in Step 4 to obtain the prediction parameter set P. Render the image M′ using the forward coloring model. Create an attenuation mask using a two-dimensional normal distribution function to simulate the light attenuation effect of a mobile phone flash. Multiply the attenuation mask with image M′ to obtain image M″. Input images M and M″ into the pre-trained VGG network to obtain their respective Gram matrices. Calculate the style loss L using the mean absolute value error loss function. g :

[0069]

[0070] Images M and M″ are downsampled to 8×16 resolution, and the style loss L is calculated using the mean absolute value error loss function. c :

[0071]

[0072] For the height field scaling factor β in the predicted parameter set P, based on its mean μ β and standard deviation σ β L was calculated p :

[0073]

[0074] Loss function L opt Including style loss L g Color loss L c and prior loss L p The various losses are summed with different weights, where w1 and w2 are the weights of the corresponding losses, respectively:

[0075] L opt =L g +w1L p +w2L c

[0076] For the repetition count of warp and weft yarns, height field noise factor, and color noise factor, the optimization strategy is to perform a random perturbation every 5 optimizations. The repetition count of warp and weft yarns increases or decreases by 10 within 100 optimizations, increases or decreases by 5 between 100 and 150 optimizations, and increases or decreases by 2 after 150 optimizations. The height field noise factor and color noise factor increase or decrease by 1 with each random perturbation. After 200 optimization iterations, the final optimization parameter P is obtained. opt .

[0077] The method of this invention can be used to capture and reconstruct the appearance of fabrics such as twill and satin, and the results are as follows: Figure 7 As shown. By Figure 7It can be seen that the method of the present application can realistically reproduce the appearance of fabric, achieve similar simulation effects on yarn details, and achieve similar highlights on silk fabric.

Claims

1. A method for capturing the appearance of a fabric from a single image, characterized in that, Includes the following steps: Step 1: Set up a fabric capture scene, capture fabric images, and perform preprocessing operations such as cropping and resolution adjustment. Step 2: Edit the fabric yarn weaving pattern and export the corresponding normal and direction maps; Step 3: Using the forward shading model, randomly sample other parameters of the model based on the texture exported in Step 2 to generate a training dataset; Step four: Construct the network model and train it on the dataset generated in step three to obtain the trained network model. Specifically, the pre-trained VGG-19 neural network is used to extract the Gram matrix of the image, which is then connected to a two-layer, 256-node fully connected neural network to form the training network. VGG does not participate in weight updates during training. For an input image M, the image mean is subtracted pixel by pixel, and then divided by the image standard deviation (std) to obtain the preprocessed image. : Image Inputting these parameters into the training network yields the output parameters. The loss is calculated using the mean absolute value error loss function: Train the network until the loss plateaus and no longer decreases, thus obtaining the final network model; Step 5: Input the measured fabric image into the trained network model to obtain the predicted parameter set, and optimize the parameters using the inverse process framework; The specific implementation process is as follows: the captured image M is input into the network model trained in step four to obtain the prediction parameter set P, and the image is rendered using the forward coloring model. A two-dimensional normal distribution function is used to create an attenuation mask to simulate the light attenuation effect of a mobile phone flash. The attenuation mask is then applied to the image. Multiplication yields an image ; Put image M and The inputs are fed into a pre-trained VGG network to obtain their respective Gram matrices, and the style loss is calculated using the mean absolute value error loss function. : Image M and The style loss was calculated by downsampling to 8×16 resolution and using the mean absolute error loss function. : The height field scaling factor in the predicted parameter set P According to its mean and standard deviation Calculated : loss function Including style loss Color loss and prior loss The losses are added together with different weights. and These are the weights corresponding to the loss: For the repetition count of warp and weft yarns, height field noise factor, and color noise factor, the optimization strategy is to perform a random perturbation every 5 optimizations. The repetition count of warp and weft yarns increases or decreases by 10 within 100 optimizations, increases or decreases by 5 between 100 and 150 optimizations, and increases or decreases by 2 after 150 optimizations. The height field noise factor and color noise factor increase or decrease by 1 with each random perturbation. After 200 optimization iterations, the final optimized parameters are obtained. .

2. The method for capturing the appearance of a fabric from a single image according to claim 1, characterized in that, The specific implementation process of step one is as follows: (1.1) Place a cylinder with a radius of 3.5cm and a length of 10cm on a flat surface, and wrap the fabric sample around the cylinder; place a mobile phone in front of the cylinder as a camera, with the cylinder at the center of the camera's field of view; place another mobile phone in front of the cylinder and use its flash as a point light source; place a gray card vertically behind the cylinder, and use the camera to photograph the gray card to adjust the exposure and white balance parameters, and fix the shooting parameters; (1.2) Take a picture of the fabric using the fixed shooting parameters in step (1.1), convert the captured image into a PNG file, crop the edges of the image to keep the middle part, adjust the resolution to 300×800, and obtain the final image.

3. The method for capturing the appearance of a fabric from a single image according to claim 1, characterized in that, The specific implementation process of step two is as follows: Configure the yarn weaving method in the fabric weaving pattern file, specifying the yarn type, center position, length and width, twist angle, and bending angle. Model the yarn as a curved and twisted cylinder, and calculate the normal map of the yarn based on these attributes. and orientation map The calculation formulas are as follows: Where u and v represent the x-coordinate and y-coordinate of the yarn, respectively. Indicates the yarn twist angle, function This is the component of the initially vertically upward vector that rotates along the x or y axis.

4. The method for capturing the appearance of a fabric from a single image according to claim 1, characterized in that: Step three specifically involves assessing the roughness of the yarn. Height field scaling factor Highlight color Diffuse color Lambertian item weight Height field noise factor Color noise factor Number of repetitions of warp and weft yarns Perform random sampling to obtain a set of parameters P; write a program to establish the same simulation scene as the fabric measurement described in step one, use the forward coloring model, and use the parameter set P to render an image M. This image and the rendering parameter set are used as a sample of the dataset. At least 2000 samples were generated for each of the different normal maps and orientation maps to form the training dataset; the rendering formula for the forward shading model is: in For the formula of rendering shading points, For specular reflection, For diffuse reflection, Let be the angle of incidence of the colored point. Let the angle of emission be the point of color. For the macroscopic surface normal of the shading point, For the microscopic surface normal of the colored point, Let be the normal distribution function. For shadow-occlusion functions, for and The half-range vector, The microflake distribution matrix; based on the color noise factor The generated Burmester noise is multiplied by the specular color, based on the height field noise factor. Generation of Berlin noise and height field scaling factor Multiplication, height field scaling factor right Scaling, which in turn adds noise to the rendered image.

Images