Uncalibrated photometric three-dimensional surface reconstruction method based on multi-head attention nerve field

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to an uncalibrated luminosity three-dimensional surface reconstruction method based on a multi-head attention nerve field, which comprises the following steps of: firstly, inputting an input image into a residual error-based illumination estimation model to estimate a corresponding illumination direction and illumination intensity; meanwhile, a multi-head attention nerve field is constructed to obtain more space related information, and the space change weight, the diffuse reflection rate and the surface depth of each pixel coordinate point are obtained; the specular reflectance of the anisotropic material is obtained through a general reflectance model, a shadow area is processed by constructing an adaptive shadow rendering model, and a corresponding shadow map is output through the model; and finally, reconstructing the image based on the rendering equation, and jointly optimizing the estimation of the shadow map, the depth map, the surface normal map, the reflectivity and the illumination condition in an unsupervised manner. Through fitting adjustment, the accuracy of surface normal estimation and illumination estimation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field. Background Art

[0002] Uncalibrated photometric stereo aims to reconstruct the surface of an object from a set of images taken under different lighting directions. Due to its ability to recover pixel-level surface information, it is widely used in many fields such as industrial defect detection and aerospace engineering. However, it is difficult to recover the surface shape when the lighting conditions and material are unknown. Traditional uncalibrated photometric stereo methods rely on the assumption of Lambertian surfaces. However, when the surface is integrable, there is a three-parameter generalized bas-relief ambiguity in shape and lighting direction. To overcome this ambiguity, early works introduced specular reflectivity to alleviate this ambiguity, but these studies usually assumed that the albedo is uniformly distributed, which leads to instability of the solution.

[0003] In recent years, deep learning-based neural inverse rendering methods have further alleviated the ambiguity of generalized bas-relief by introducing a progressive isotropic specular reflection model. Although such models perform well on isotropic materials, their applicability to general (e.g., anisotropic) materials is limited. In addition, shadow cues play an important role in shape recovery, which solves the problems caused by the interaction between shape and light in uncalibrated photometric stereo by utilizing global shape-lighting information. However, these methods lack explicit continuous treatment of shadows and fail to fully utilize shadow cues, resulting in inaccurate normal estimates in shadow areas. Summary of the invention

[0004] The purpose of the present invention is to provide an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field, and to overcome the technical problem of low accuracy of the existing uncalibrated photometric stereo method based on deep learning in the three-dimensional reconstruction process by introducing an adaptive shadow rendering model and a multi-head attention neural field model.

[0005] To achieve the above object, the present invention provides an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field, comprising the following steps:

[0006] Step 1: Photograph the object to be reconstructed from different lighting directions to obtain a set of RGB images;

[0007] Step 2: Obtain the illumination direction and illumination intensity based on the residual illumination estimation model;

[0008] Step 3: Based on the multi-head attention neural field, the spatial variation weight, diffuse reflectance and surface depth are obtained, and the gradient of the depth map is calculated to obtain the surface normal map;

[0009] Step 4: Obtain the specular reflectivity of various anisotropic materials based on the general reflectivity model;

[0010] Step 5: Obtain a shadow map based on an adaptive shadow rendering model;

[0011] Step 6: Reconstruct the image based on the lighting direction, lighting intensity, diffuse reflectance, specular reflectance, surface normal map and shadow map;

[0012] Step 7: Train the network in an unsupervised manner, calculate the reconstruction loss between the reconstructed image and the input image, and optimize the network through the loss function;

[0013] Step 8: Use the trained surface normal map to reconstruct the surface 3D shape.

[0014] Optionally, the residual-based illumination estimation model in step 2 is composed of seven convolutional layers and three fully connected layers, where residual connections are performed between the second and third layers of the network.

[0015] Optionally, the multi-head attention neural field in step 3 consists of two parts: a multi-head attention neural depth field and a multi-head attention neural material field. The multi-head attention neural depth field consists of a multi-head attention module and a neural depth field. The multi-head attention module obtains more spatial information by introducing multiple independent attention heads. The neural depth field consists of 8 fully connected layers with 256 channels, and the output of the first fully connected layer is used as the input of the multi-head attention module; the multi-head attention neural material field has a similar structure to the neural depth field, but adds 3 fully connected layers with 128 channels at the end, and also uses the output of the first fully connected layer as the input of the multi-head attention module.

[0016] Optionally, the general reflectivity model in step 4 is expressed as a weighted sum of anisotropic spherical Gaussian basis points, and the calculation formula is expressed as where c k (α) represents the spatial variation weight, N is the ASG cardinality set according to experience, h is the half-angle unit vector between the viewing angle v and the illumination direction l, and represents the lobe width in the x and y directions, which are learnable parameters. x and y are the points p on the surface. i The tangent vector and binormal vector of the tangent plane at .

[0017] Optionally, the calculation formula for shadow rendering in step 5 is as follows:

[0018]

[0019] Where Sigmoid(·) represents a step function; γ controls the softness or hardness of the shadow; Represents the depth value of the surface point, Represents the depth value of the sampling point with the same coordinates as the surface point; δ is a learnable parameter that makes slight adjustments to the depth value of the object surface during rendering; β controls the soft and hard boundaries of the shadow.

[0020] Optionally, in step 6, the information of lighting direction, lighting intensity, spatial variation weight, diffuse reflectance, depth map, normal map, specular reflectance, and shadow map is integrated through the formula Perform image reconstruction.

[0021] Optionally, in step 7, the model is iterated for a total of 2000 rounds. In the first 500 rounds, a smoothness constraint is used; in the next 1500 rounds, a contour loss function is used to calculate the cosine similarity between the estimated and fitted contour normals.

[0022] The present invention provides an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field. It does not require the use of instruments and professionals for illumination calibration. It directly obtains the illumination direction and illumination intensity through a residual-based illumination estimation model. Then, it obtains the spatial variation weight, diffuse reflectance and depth map through the multi-head attention neural field, and calculates the depth map to obtain a normal map. Then, it obtains a more accurate shadow map through adaptive shadow rendering, and finally trains in an unsupervised manner and performs joint optimization through a loss function. After fitting adjustment, the present invention improves the accuracy of illumination estimation and surface normal estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a schematic diagram of the steps of an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field of the present invention.

[0025] Figure 2 Schematic diagram of the network architecture of the residual-based illumination estimation model of the present invention.

[0026] Figure 3 Schematic diagram of the multi-head attention neural field network architecture of the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0028] See also Figure 1 The present invention provides an uncalibrated photometric stereo surface reconstruction method based on a multi-head attention neural field, comprising the following steps:

[0029] S1: photograph the object to be reconstructed from different lighting directions to obtain a set of RGB images;

[0030] S2: Obtain the illumination direction and illumination intensity based on the residual illumination estimation model;

[0031] S3: Based on the multi-head attention neural field, spatial variation weights, diffuse reflectance and surface depth are obtained, and the gradient of the depth map is calculated to obtain the surface normal map;

[0032] S4: Obtain the specular reflectance of anisotropic materials based on the general reflectance model;

[0033] S5: Obtain shadow map based on adaptive shadow rendering model;

[0034] S6: Image reconstruction based on lighting direction, lighting intensity, diffuse reflectance, specular reflectance, surface normal map and shadow map;

[0035] S7: Train the network in an unsupervised manner, calculate the reconstruction loss between the reconstructed image and the input image, and optimize the network through the loss function;

[0036] S8: Reconstruct the 3D shape of the surface using the trained surface normal map.

[0037] The following is a further explanation based on the specific implementation steps:

[0038] See also Figure 2 The residual-based illumination estimation model in step S2 consists of seven convolutional layers and three fully connected layers. The number of convolution kernels in the convolutional layers are 64, 128, 128, 128, 128, 256, and 256, respectively, and each layer uses LReLU as the activation function. The fully connected layers have 64 channels and use ReLU as the activation function. The second and third layers of the network are residually connected, so that the features of the second layer can directly act on the subsequent layers. The convolutional layers and the fully connected layers perform feature fusion through adaptive average pooling.

[0039] See also Figure 3The multi-head attention neural field in step S3 consists of a multi-head attention neural depth field and a multi-head attention neural material field. Both use a 10-level Fourier basis position encoding strategy to encode the input pixel coordinates (x, y). The encoding formula is expressed as (γ(x),γ(y))=(sin(2 0 π(x,y)),cos(2 0 π(x,y)),…,sin(2 19 π(x,y)),cos(2 19 π(x,y))), and the generated (γ(x),γ(y)) is used as the input of the multi-head attention neural field. The multi-head attention neural depth field consists of two parts: the multi-head attention module and the neural depth field. The multi-head attention module introduces multiple independent attention heads, and each head performs the attention mechanism in different subspaces. It can capture different features in the input sequence and obtain more spatial information. The neural depth field consists of 8 fully connected layers, each with 256 channels. Except for the last layer, each layer is followed by a ReLU activation function. Feature fusion is performed in each jump connection layer, and the output of the first fully connected layer is used as the input of the multi-head attention module. The network finally outputs the depth value w of the corresponding point; the multi-head attention neural material field has a similar structure to the neural depth field, but adds 3 fully connected layers with 128 channels at the end. The output of the first fully connected layer is also used as the input of the multi-head attention module. The network finally outputs the diffuse reflectance ρ of the surface point. d and spatially varying weight c.

[0040] The general reflectivity model in step S4 is expressed as the weighted sum of various anisotropic spherical Gaussian basis points, and the calculation formula is expressed as where c k (α) represents the spatial variation weight, which is used to balance different ASG bases to simulate spatially varying specular reflectance. N is the ASG base set empirically. h is the half-angle unit vector between the viewing angle v and the illumination direction l. and represents the lobe width in the x and y directions, which are learnable parameters. x and y are the points p on the surface. i The tangent vector and binormal vector of the tangent plane at are calculated by the formula x=v-(v·n)n, y=x×n.

[0041] The calculation formula for shadow rendering in step S5 is as follows:

[0042]

[0043] Sigmoid(·) represents a step function. When s is close to 0, it means that the point is in the shadow area, otherwise it is not in the shadow area. γ controls the hardness of the shadow. When γ is close to 0, it means a hard shadow, otherwise it is a soft shadow. N p It indicates the number of sampling points adaptively sampled on a ray from a surface point along the direction of illumination. Represents the depth value of the surface point, Represents the depth value of the sampling point with the same coordinates as the surface point; δ is a learnable parameter that makes slight adjustments to the depth value of the object surface during the rendering process to reduce artifacts generated when rendering shadows; β controls the soft and hard boundaries of the shadow. When β>0, the significance of the shadow area is reduced, and vice versa, the significance of the shadow area is increased.

[0044] Step S6 obtains the lighting direction and intensity through the lighting estimation residual network, the spatial variation weights, diffuse reflectance and surface depth obtained through the multi-head attention neural field, the normal map obtained through depth map calculation, the mirror reflectance obtained through the general reflectance model, and the shadow map generated by the adaptive shadow rendering model. Finally, this information is integrated to reconstruct the image through the rendering equation.

[0045] In step S7, the model is iterated for a total of 2000 rounds, and the Adam optimization algorithm is used to update the model parameters, and the learning rate is set to 0.002. In the first 500 rounds, the smoothness constraint is used to guide the network to develop in the correct optimization direction; in the next 1500 rounds, the contour loss function is used to calculate the cosine similarity between the estimated and fitted contour normals to further reduce the GBR ambiguity.

[0046] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. An uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field, characterized in that: The following steps are involved: Step 1: Photograph the object to be reconstructed from different lighting directions to obtain a set of RGB images; Step 2: Obtain the illumination direction and illumination intensity based on the residual illumination estimation model; Step 3: Based on the multi-head attention neural field, the spatial variation weight, diffuse reflectance and surface depth are obtained, and the gradient of the depth map is calculated to obtain the surface normal map; Step 4: Obtain the specular reflectivity of various anisotropic materials based on the general reflectivity model; Step 5: Obtain a shadow map based on an adaptive shadow rendering model; Step 6: Reconstruct the image based on the lighting direction, lighting intensity, diffuse reflectance, specular reflectance, surface normal map and shadow map; Step 7: Train the network in an unsupervised manner, calculate the reconstruction loss between the reconstructed image and the input image, and optimize the network through the loss function; Step 8: Use the trained surface normal map to reconstruct the surface 3D shape.

2. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 1, characterized in that: The residual-based illumination estimation model in step 2 consists of seven convolutional layers and three fully connected layers, where residual connections are made between the second and third layers of the network.

3. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 2, characterized in that: The multi-head attention neural field in step 3 consists of two parts: the multi-head attention neural depth field and the multi-head attention neural material field. The multi-head attention neural depth field consists of a multi-head attention module and a neural depth field. The multi-head attention module obtains more spatial information by introducing multiple independent attention heads. The neural depth field consists of 8 fully connected layers with 256 channels, and the output of the first fully connected layer is used as the input of the multi-head attention module; the multi-head attention neural material field has a similar structure to the neural depth field, but adds 3 fully connected layers with 128 channels at the end, and also uses the output of the first fully connected layer as the input of the multi-head attention module.

4. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 3, characterized in that: The general reflectivity model in step 4 is expressed as a weighted sum of anisotropic spherical Gaussian basis points, and the calculation formula is expressed as where c k (α) represents the spatial variation weight, N is the ASG cardinality set according to experience, h is the half-angle unit vector between the viewing angle v and the illumination direction l, and represents the lobe width in the x and y directions, which are learnable parameters. x and y are the points p on the surface. i The tangent vector and binormal vector of the tangent plane at .

5. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 4, characterized in that: The calculation formula for shadow rendering in step 5 is as follows: Where Sigmoid(·) represents a step function; γ controls the softness or hardness of the shadow; Represents the depth value of the surface point, Represents the depth value of the sampling point with the same coordinates as the surface point; δ is a learnable parameter that makes slight adjustments to the depth value of the object surface during rendering; β controls the soft and hard boundaries of the shadow.

6. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 5, characterized in that: Image reconstruction refers to the comprehensive information of illumination direction, illumination intensity, spatial variation weight, diffuse reflectance, depth map, normal map, specular reflectance, and shadow map through the formula m=e·(ρ s +ρ d )·max(n T l,0)·s to reconstruct the image.

7. The uncalibrated photometric stereo surface reconstruction method based on multi-head attention neural field as claimed in claim 6, characterized in that: The training network refers to a total of 2000 iterations of the model as a whole. In the first 500 rounds, the smoothness constraint is used; in the next 1500 rounds, the contour loss function is used to calculate the cosine similarity between the estimated and fitted contour normals.

Images