High-precision three-dimensional reconstruction method for complex scene

Abstract

The invention discloses a high-precision three-dimensional reconstruction method for a complex scene, and the method comprises the steps: obtaining the position and orientation of a camera from a multi-view image through the pose estimation of the polarization camera, and obtaining a normal vector of a scene surface through an MLP network; according to the method, a scene surface normal vector is used for correcting an ambiguous polarization normal vector calculated by using a polarization image, so that a prior normal vector is obtained, and the prior normal vector is input into a pre-trained improved NeuS network to provide additional constraints, so that the geometric blurring problem of a non-texture region and a mirror reflection region is relieved. And compared with depth priori, the network based on the priori normal vector can generate a smooth geometric shape, and can also avoid the problem of depth scale blur among multiple views, thereby obtaining more accurate three-dimensional information. According to the method, high-precision reconstruction can also be carried out on an object with shielding and an object with a thin structure, and better geometric reconstruction can also be carried out on a scene with sudden depth change.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a high-precision stereoscopic reconstruction method for complex scenes. Background Art

[0002] Reconstructing 3D scenes from multiple input images is an important and challenging task in many real-world applications such as robotic navigation, virtual reality, and path planning. Recovering the 3D shape of real-world scenes is a fundamental problem in computer vision. The goal of inverse rendering is to recover scene parameters from a set of associated images. Inverse rendering methods traditionally rely on multi-view geometry, photometry, and structured illumination for 3D reconstruction. Due to the ill-posed nature of inverse rendering, these methods often require simplifying assumptions about the scene, such as textured surfaces, Lambertian reflectance, direct lighting, and simple geometry. Methods for generalized scene settings include incorporating scene priors, iterative scene optimization using differentiable rendering, and exploiting different properties of light, such as polarization, time of flight, and spectrum.

[0003] The emergence of neural implicit network representations in recent years has aroused people's interest in neural inverse rendering. Neural implicit representations use coordinate-based neural networks to represent visual signals such as images, videos, and 3D objects. These representations are powerful because the resolution of the underlying signal is only limited by the capacity of the network, not the discretization of the signal. The interest of the vision community largely stems from the neural radiance field (NeRF), which shows that modeling radiance using implicit representations can achieve high-quality novel view synthesis. However, NeRF encounters difficulties when it comes to textureless or specular surfaces, because the lack of texture leads to ambiguity in establishing correspondences, and the presence of specular reflections violates the photo consistency assumption. In real scenes, there are often textureless and highlight areas, resulting in inaccurate reconstruction results of this method. As an incoherent imaging technology, polarization 3D imaging technology has no essential requirement for modulated light sources, is not affected by propagation paths and coherence errors, is less sensitive to ambient light, has high spatial resolution, and can image in real time. It has very great practical application prospects in 3D imaging of complex targets. However, polarization 3D imaging technology has singularity problems when interpreting the polarization field of complex target surfaces, multi-value problems when using the polarization field to solve the normal vector field, and the reconstruction process is sensitive to polarization field errors. This is one of the main reasons why polarization 3D imaging technology has not been widely used in complex target scenes.

[0004] Currently, the more common methods for 3D reconstruction of large scenes include monocular depth estimation methods and neural radiation field-based 3D reconstruction methods using images from multiple perspectives. However, monocular depth estimation has the problem of inaccurate estimation in complex scenes and some places with occlusions and shadows; while the neural implicit 3D reconstruction method using multiple perspectives has the problem of inaccurate reconstruction of blurred geometric surfaces in areas without texture and mirror reflection. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a high-precision stereoscopic reconstruction method for complex scenes. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a high-precision stereoscopic reconstruction method for complex scenes, the method comprising:

[0007] Use a polarization camera to collect multi-view images and multi-angle polarization images of the target scene;

[0008] Process the multi-angle polarization images to obtain polarization information and the polarization degree of the target scene surface;

[0009] Based on the polarization information and the polarization degree of the target scene surface, obtaining the azimuth angle and the incident angle of the incident light on the target scene surface;

[0010] Based on the azimuth angle and the incident angle, a polarization normal vector is obtained;

[0011] Based on the multi-view image, the spatial position and the viewing direction of the polarization camera are obtained; based on the spatial position and the viewing direction, the signed distance function SDF and the color value of the target scene are obtained using the MLP network; based on the signed distance function SDF, the scene surface normal vector is obtained; the scene surface normal vector is used as a guide vector to correct the polarization normal vector to obtain a disambiguated surface normal vector;

[0012] The disambiguated surface normal vector is used as the prior normal vector, and the pre-trained improved NeuS network is used to perform volume rendering on the position and orientation of the polarization camera, the signed distance function SDF, the color value and the prior normal vector to obtain the predicted color value and normal vector; based on the difference between the predicted normal vector and the prior normal vector and the difference between the predicted color value and the color value of the target scene, the three-dimensional information of the target scene is obtained.

[0013] In one embodiment of the present invention, a polarization camera is used to collect multi-view images and multi-angle polarization images of a target scene, including:

[0014] Use the FLIR polarization camera to shoot the target scene and obtain video frames;

[0015] Extract multi-view images from the video frame, obtain multi-angle image data of the scene, and combine the polarizer on the polarization camera to obtain multi-angle polarization images I0, I 45 ,I 90 ,I 135 .

[0016] In one embodiment of the present invention, the multi-angle polarization image is processed to obtain polarization information and the polarization degree of the target scene surface, including:

[0017] The multi-angle polarization image is processed to obtain polarization information; the polarization information is composed of Stokes vectors; the expression of the Stokes vector is as follows:

[0018]

[0019] Among them, I 0° Indicates the light intensity passing through the 0° polarizer, I 90° Indicates the light intensity passing through the 90° polarizer, I +45° It indicates the intensity of light passing through the polarizer at 45° to the horizontal positive direction, I -45° It represents the light intensity obtained by the linear polarizer at a positive 135° angle to the horizontal. RH Represents the right-handed circularly polarized light component, I LH Represents left-handed circularly polarized light component.

[0020] Based on the Stokes vector, the polarization degree of the target scene surface is obtained using the first formula; the expression of the first formula is as follows:

[0021]

[0022] In one embodiment of the present invention, obtaining the azimuth angle and the incident angle of the incident light on the target scene surface based on the polarization information and the polarization degree of the target scene surface includes:

[0023] Based on the polarization information and the polarization degree of the target scene surface, using a second formula, obtaining the azimuth angle of the incident light on the target scene surface;

[0024] The second formula is as follows:

[0025]

[0026] in, represents the azimuth, S0 and S1 represent the Stokes vector;

[0027] Based on the relationship between the degree of polarization and the incident angle, the incident angle is obtained;

[0028] The relationship between the degree of polarization and the angle of incidence is as follows:

[0029] For specular reflection:

[0030]

[0031] For diffuse reflection:

[0032]

[0033] Wherein, P represents the degree of polarization, n represents the refractive index of the object surface, and θ represents the incident angle.

[0034] In one embodiment of the present invention, obtaining a polarization normal vector based on the azimuth angle and the incident angle includes:

[0035] Based on the azimuth and the angle of incidence, the polarization normal vector is obtained using the third formula:

[0036] The third formula is as follows:

[0037]

[0038] Among them, n x express The component in the x-axis direction, n y Represents the component in the y-axis direction, n z represents the component in the z-axis direction, represents the azimuth, and θ represents the angle of incidence.

[0039] In one embodiment of the present invention, obtaining the spatial position and the viewing direction of the polarization camera based on the multi-view images includes:

[0040] The multi-view image is processed using the COLMAP algorithm to obtain the position (x, y, z) and orientation of the polarization camera corresponding to the multi-view image.

[0041] The fourth formula is used to calculate the position (x, y, z) and orientation of the polarization camera. Perform position encoding to obtain the corresponding spatial position γ(x, y, z) and view direction

[0042] The fourth formula is as follows:

[0043] γ(η)=[sin(20πη),cos(20πη),…,sin((2L-1)πη),cos((2L-1)πη)];

[0044] Among them, γ(η)=γ(x,y,z) or L represents the preset parameters. When processing the spatial position, η = x, y, z. When processing the view direction,

[0045] In one embodiment of the present invention, based on the spatial position and the view direction, the signed distance function SDF and the color value of the target scene are obtained using an MLP network, including:

[0046] Mapping the spatial position to a signed distance of a surface to obtain a signed distance function SDF of the target scene;

[0047] The colors of the spatial position and the viewing direction are encoded to obtain a color value of the target scene.

[0048] In one embodiment of the present invention, obtaining a scene surface normal vector based on the signed distance function SDF includes:

[0049] The fifth formula is used to calculate the gradient of the signed distance function SDF to obtain the scene surface normal vector

[0050] The fifth formula is as follows:

[0051]

[0052] in, represents the gradient operation, and f(p) represents the signed distance function SDF.

[0053] In one embodiment of the present invention, the scene surface normal vector is used as a guide vector to correct the polarization normal vector to obtain a disambiguated surface normal vector, including:

[0054] The polarization normal vector is corrected based on the minimization of the energy loss function to obtain the disambiguated surface normal vector;

[0055] The minimized energy loss function is as follows:

[0056]

[0057] Among them, u = (x, y), v = (x + 1, y) and w = (x, y + 1) represent pixel points, E pol (n polar (u)) represents the polarization energy loss function, which is obtained based on the scene surface normal vector and the polarization normal vector. smooth (n polar (u),n polar (v),n polar (w)) represents the smoothing term energy loss function, n polar (u) represents the polarization normal vector at pixel u, n polar (v) represents the polarization normal vector at pixel v, n polar(w) represents the polarization normal vector at pixel w, and λ1 represents the preset weight.

[0058] In one embodiment of the present invention, the loss function of the improved NeuS network is as follows:

[0059] L all =L color +λ nomal L nomal +λ reg L reg ;

[0060] Among them, L color represents the color loss function, L nomal Denotes the normal vector loss function, L reg represents the process function loss, λ nomal and λ reg Indicates the corresponding weight.

[0061] Beneficial effects of the present invention:

[0062] In the solution provided by the present invention, the position and orientation of the camera are obtained from the multi-view image through polarization camera pose estimation, and the scene surface normal vector is obtained through the MLP network; the scene surface normal vector is used to correct the ambiguous polarization normal vector obtained by using the polarization image, so as to obtain the prior normal vector, and the prior normal vector is input into the pre-trained improved NeuS network to provide additional constraints to alleviate the geometric ambiguity problem of textureless areas and mirror reflection areas. Compared with the depth prior, the network based on the prior normal vector can produce smooth geometric shapes and avoid the problem of depth scale ambiguity between multiple views, so as to obtain more accurate three-dimensional information. The present invention can also perform high-precision reconstruction of objects with occlusion and objects with thin structures, and can also perform better geometric reconstruction of scenes with sudden depth changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of the steps of a high-precision stereoscopic reconstruction method for a complex scene provided by an embodiment of the present invention;

[0064] Figure 2 A schematic diagram of a process flow of a high-precision stereoscopic reconstruction method for a complex scene provided by an embodiment of the present invention;

[0065] Figure 3 A surface coordinate diagram of a target scene of a high-precision stereoscopic reconstruction method for a complex scene provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0067] In the prior art, Neural Radiance Fields (NeRF) is a computer vision technology used to generate high-quality 3D reconstruction models. It uses deep learning technology to extract the geometric shape and texture information of objects from images from multiple perspectives, and then uses this information to generate a continuous 3D radiation field, which can present a highly realistic 3D model at any angle and distance. NeRF technology combines the input images from multiple perspectives and camera parameters to establish a multi-layer perceptron (MLP) model to represent the color and density of each point in the scene. The implementation structure of NeRF includes two main components: encoder and decoder.

[0068] The encoder is responsible for extracting the spatial position and perspective features of each point in the scene from the input multiple perspective images and camera parameters. NeRF proposes a position encoding method that can map the input data from a low-dimensional space to a high-dimensional space and extract more complex feature representations.

[0069] The decoder is usually composed of a multi-layer perceptron, which is responsible for generating a continuous three-dimensional radiation field from the features extracted by the encoder. Specifically, the decoder accepts the spatial position and viewing angle features of each point from the encoder as input and outputs the color and density value of the point. Each MLP layer in the decoder can map the input data to another high-dimensional space and extract more complex feature representations.

[0070] During the training phase, NeRF uses a set of 2D images and corresponding camera parameters as input and uses the rendering equation to convert the input data into a 3D scene. The rendering equation is used to calculate the intersection points of light emitted from the camera position and direction with the objects in the scene and determine the color and density values ​​of each point. NeRF then uses a neural network to approximate the rendering equation to minimize the difference between the generated scene and the real image.

[0071] However, most neural implicit network-based 3D reconstruction methods fail to reconstruct scenes with large texture-free areas such as white walls, floors, and reflective scenes, because these areas do not contain enough visual features required for pixel-level optimization. And although NeRF's view-dependent appearance rendering seems reasonable at first glance, a closer inspection of specular highlights reveals fake glossy artifacts that fade in and out between rendered views, rather than moving smoothly across the surface in a physically reasonable way. This is because NeRF can only accurately render the appearance of scene points from a specific viewing direction observed in the training images, and interpolates the glossy appearance poorly from new viewpoints; NeRF tends to "fake" specular reflections using isotropic properties inside objects rather than view-dependent radiation emitted by surface points, resulting in objects with translucent or "foggy" shells.

[0072] In the prior art, the multi-view stereo method (MVSNet) based on deep learning takes a reference image and multiple source images as input to predict a depth map for the reference image, rather than the entire 3D scene. In the network, feature extraction is first performed on the 2D image to obtain a feature map, and then a 3D cost volume is constructed based on the camera frustum of the reference view through a differentiable homography transformation. The cost volume is then regularized using 3D convolution and regressed to obtain an initial depth map; the initial depth map is optimized through the reference image to obtain the final depth map.

[0073] However, the multi-view stereo method based on deep learning first estimates the depth map of the image separately, and then uses additional filtering and fusion depth to reconstruct the scene. However, due to the inconsistency caused by the individual estimation of the depth map, this method often has problems such as incompleteness, surface noise and scale ambiguity, and it is still difficult to handle textureless and repeated texture areas, resulting in insufficient accuracy of the reconstruction results. In addition, this method requires 3D geometric information (true value) for training, which is often difficult to obtain in real scenes. If synthetic data is used, the trained network will not be able to generalize well.

[0074] In order to solve the above problems, the embodiment of the present invention provides a high-precision stereoscopic reconstruction method for complex scenes, such as Figure 1 As shown, it may include:

[0075] S1, using a polarization camera to collect multi-view images and multi-angle polarization images of the target scene;

[0076] S2, processing the multi-angle polarization image to obtain polarization information and the polarization degree of the target scene surface;

[0077] S3, obtaining the azimuth and incident angle of the incident light on the surface of the target scene based on the polarization information and the polarization degree of the surface of the target scene;

[0078] S4, obtaining a polarization normal vector based on the azimuth angle and the incident angle;

[0079] S5, based on the multi-view images, the spatial position and view direction of the polarization camera are obtained; based on the spatial position and view direction, the signed distance function SDF and color value of the target scene are obtained using the MLP network; based on the signed distance function SDF, the scene surface normal vector is obtained; the scene surface normal vector is used as a guide vector to correct the polarization normal vector to obtain the disambiguated surface normal vector;

[0080] S6, using the disambiguated surface normal vector as the prior normal vector, performing volume rendering based on the position and orientation of the polarization camera, the signed distance function SDF, the color value and the prior normal vector, and obtaining predicted color values ​​and normal vectors; based on the difference between the predicted normal vector and the prior normal vector and the difference between the predicted color value and the color value of the target scene, obtaining the three-dimensional information of the target scene.

[0081] The embodiment of the present invention provides a high-precision stereo reconstruction method for complex scenes. The position and orientation of the camera are obtained as input through camera pose estimation, the scene surface normal vector is obtained through the MLP network, the scene surface normal vector is used to correct the polarization normal vector obtained using the polarization image to obtain the prior normal vector, and the prior normal vector is input into the pre-trained improved NeuS network to provide additional constraints to reduce the geometric ambiguity problem of textureless areas and mirror reflection areas. In addition, compared with the depth prior, the network based on the prior normal vector can produce smooth geometric shapes, and can also avoid the problem of depth scale ambiguity between multiple views, and obtain more accurate three-dimensional information.

[0082] For a detailed flowchart, see Figure 2 ,from Figure 2 It can be seen that after step S1 uses a polarization camera to acquire a multi-view image and a multi-angle polarization image of the target scene, it can be divided into two parallel processing steps, and the multi-view image and the multi-angle polarization image are processed accordingly. Specifically, the multi-angle polarization image is processed based on the Stokes vector using steps S2-S4 to obtain a polarization normal vector; the multi-view image is processed using the corresponding steps in step S5 to obtain a scene surface normal vector, and the polarization normal vector is corrected based on the scene surface normal vector to obtain a disambiguated surface normal vector as a rough normal vector prior; and then corresponding processing is performed using step S6 to obtain the three-dimensional information of the target scene. It can be understood that since obtaining the scene surface normal vector in S2 and S5 can be performed in parallel, the step numbers should not be understood as limitations on the steps of the embodiments of the present invention.

[0083] For ease of understanding, the following introduces the steps of a high-precision stereoscopic reconstruction method for a complex scene provided by an embodiment of the present invention.

[0084] For S1, this may include:

[0085] Use the FLIR polarization camera to shoot the target scene and obtain video frames;

[0086] Extract multi-view images from the video frame, obtain multi-angle image data of the scene, and combine the polarizer on the polarization camera to obtain multi-angle polarization images I0, I 45 ,I 90 ,I 135.

[0087] Specifically, the polarizer on the polarization camera is a four-directional (0°, 45°, 90° and 135°) polarizer on the camera sensor, and the polarizer can be used to output the intensity and polarization angle of each pixel, thereby obtaining multi-angle polarization images I0, I1 of the target scene at four angles. 45 ,I 90 ,I 135 .

[0088] For S2, this may include:

[0089] The multi-angle polarization image is processed to obtain polarization information; the polarization information is composed of Stokes vectors; the expression of Stokes vectors is as follows:

[0090]

[0091] Among them, I 0° Indicates the light intensity passing through the 0° polarizer, I 90° Indicates the light intensity passing through the 90° polarizer, I +45° It indicates the intensity of light passing through the polarizer at 45° to the horizontal positive direction, I -45° It represents the light intensity obtained by the linear polarizer at a positive 135° angle to the horizontal. RH Represents the right-handed circularly polarized light component, I LH Represents left-handed circularly polarized light component.

[0092] Based on the Stokes vector, the polarization degree of the target scene surface is obtained using the first formula; the expression of the first formula is as follows:

[0093]

[0094] Specifically, two or more photons with different polarization states are used to describe the polarization of light. The polarization state and its sum can be described by the Stoke vector.

[0095] It can be understood that the degree of polarization DOP of any optical signal can be directly determined using the Stokes vector, and the degree of polarization represents the ratio of polarization components.

[0096]

[0097] If the light beam is linearly polarized, the elliptical polarization and circular polarization components are zero, and the first formula can be obtained. P is generally called the linear polarization degree, which characterizes the linear polarization degree of the light beam. It can be understood that for a light beam that is linearly polarized, the polarization degree DOP at this time can be represented by P to represent the linear polarization degree.

[0098] For S3, this can include:

[0099] S31, based on the polarization information and the polarization degree of the target scene surface, using the second formula, obtain the azimuth of the incident light on the target scene surface; specifically, the surface coordinate diagram of the target scene, see Figure 3 .

[0100] The second formula is as follows:

[0101]

[0102] in, represents the azimuth, S0 and S1 represent the Stokes vector.

[0103] S32, obtaining the incident angle based on a relationship between the degree of polarization and the incident angle;

[0104] The relationship between the degree of polarization and the angle of incidence is as follows:

[0105] For specular reflection:

[0106]

[0107] For diffuse reflection:

[0108]

[0109] Wherein, P represents the degree of polarization, n represents the refractive index of the object surface, and θ represents the incident angle.

[0110] Alternatively, the refractive index n of the object surface can be taken as 1.5. If the pixel is diffusely dominated, the phase angle is uniquely determined by the polarization degree, and the azimuth angle is restricted to two possibilities by the phase angle, resulting in two possible normal vector directions. If the pixel is specularly dominated, the polarization degree restricts the viewing angle to two possibilities, and the azimuth angle is also restricted to two, giving a total of four possible normal vector directions.

[0111] For S4, this may include:

[0112] Based on the azimuth and incident angle, the polarization normal vector is obtained using the third formula

[0113] The third formula is as follows:

[0114]

[0115] Among them, n x express The component in the x-axis direction, n y Represents the component in the y-axis direction, n z represents the component in the z-axis direction, represents the azimuth, and θ represents the angle of incidence.

[0116] Since there are multiple polarizer rotation angles corresponding to a certain light intensity, there will be a 180° multi-value problem when calculating the azimuth angle by light intensity, resulting in the inability to accurately obtain the polarization normal vector. Therefore, before reconstruction, we need to correct the polarization normal vector.

[0117] For S5, this may include:

[0118] S51, based on the multi-view images, obtaining the spatial position and viewing direction of the polarization camera may include:

[0119] The multi-view images are processed using the COLMAP algorithm to obtain the position (x, y, z) and orientation of the polarization camera corresponding to the multi-view images.

[0120] The fourth formula is used to calculate the position (x, y, z) and orientation of the polarization camera respectively. Perform position encoding to obtain the corresponding spatial position γ(x, y, z) and view direction

[0121] The fourth formula is as follows:

[0122] γ(η)=[sin(20πη),cos(20πη),…,sin((2L-1)πη),cos((2L-1)πη)];

[0123] Among them, γ(η)=γ(x,y,z) or L represents the preset parameters. When processing the spatial position, η = x, y, z. When processing the view direction,

[0124] Specifically, for the COLMAP algorithm, the ground truth pose is used to reconstruct the point cloud, and then the mesh is obtained by using the filtered Poisson surface reconstruction to obtain the 3D coordinates and camera orientation of the camera corresponding to the image, and the position encoding is used to map it to high frequency. The preset parameter L can be set according to the needs. Optionally, the preset parameter L can be set to 10.

[0125] S52, based on the spatial position and the viewing direction, using the MLP network to obtain the signed distance function SDF and the color value of the target scene, may include:

[0126] Map the spatial position to the signed distance of the surface to obtain the signed distance function SDF of the target scene;

[0127] The color of the spatial position and view direction is encoded to obtain the color value of the target scene.

[0128] Specifically, the scene to be reconstructed can be encoded by two multi-layer perceptrons MPL, where the MLP encoding f can be composed of 8 hidden layers, each hidden layer size can be 256, and the activation function uses the Softplus function with β=100 to map the spatial position γ(x, y, z) to the signed distance of the surface, thereby obtaining the signed distance function SDF of the target scene, which processes the three-dimensional data into one-dimensional data (R 3 →R). The MLP of encoder c can consist of 6 hidden layers, each of which can be of size 256, for spatial position γ(x,y,z) and view direction The color of the target scene is encoded to obtain the color value of the target scene, which processes the three-dimensional data and the two-dimensional data into three-dimensional data (R 3 ×S 2 →R 3 ).

[0129] S53, obtaining a scene surface normal vector based on a signed distance function SDF, may include:

[0130] The fifth formula is used to calculate the gradient of the signed distance function SDF to obtain the scene surface normal vector

[0131] The fifth formula is as follows:

[0132]

[0133] in, represents the gradient operation, and f(p) represents the signed distance function SDF.

[0134] The obtained scene surface normal vector It can be used for subsequent correction of the polarization normal vector.

[0135] S54, using the scene surface normal vector as a guide vector to correct the polarization normal vector to obtain a disambiguated surface normal vector, may include:

[0136] The polarization normal vector is corrected based on the minimization of the energy loss function to obtain the disambiguated surface normal vector;

[0137] The energy loss function to be minimized is as follows:

[0138]

[0139] Among them, u=(x,y), v=(x+1,y), w=(x,y+1) represent pixel points, E pol (n polar (u)) represents the polarization energy loss function, which is obtained based on the scene surface normal vector and polarization normal vector. smooth(n polar (u),n polar (v),n polar (w)) represents the smoothing term energy loss function, n polar (u) represents the polarization normal vector at pixel u, n polar (v) represents the polarization normal vector at pixel v, n polar (w) represents the polarization normal vector at pixel point w, and λ1 represents the preset weight.

[0140] It can be understood that the scene surface normal vector, such as the constraint in S4, limits the surface normal vector at the pixel to 6 possible directions. A high-order graphics model can be selected for integrability-based disambiguation, and the disambiguated surface normal vector can be obtained by minimizing the energy loss function. The value of the preset weight λ1 can be determined by the computer itself during the calculation process to minimize the energy loss function.

[0141] Specifically, in minimizing the energy loss function, the expression of the polarization energy loss function can be The polarization energy loss function can be expressed as the polarization normal vector n polar (u) and the scene surface normal vector The angle difference between them.

[0142]

[0143] Where f(u) = exp(-n polar (u)·n), f(u) represents the polarization normal vector n polar (u) and the scene surface normal vector The cosine value between .

[0144] According to the previous analysis, n polar (u) There can be at most 6 solutions, D represents the 2 solutions corresponding to diffuse reflection, J represents the 4 solutions corresponding to specular reflection, and L can be the initial specular reflection mask.

[0145] The expression of the smoothing term energy loss function is as follows:

[0146]

[0147] in, Represents a set of pixel triplets (u,v,w).

[0148] The smoothing term energy loss function can express the integrability constraint of the scene surface normal vector. For an integrable surface, the mixed second-order partial derivatives on the gradient field should be equal, that is, p represents the gradient in the horizontal direction, and q represents the gradient in the vertical direction. However, in reality, due to noise and discrete pixel networks, the gradient field cannot have zero curl, so it is necessary to find the normal vector with the minimum curl.

[0149]

[0150] ψ(n polar (u), n polar (v), n polar (w))=p(w)-p(u)-(q(v)-q(u)).

[0151] It can be understood that the disambiguated surface normal vector at this time is the unambiguous n obtained by minimizing the energy function. polar (u). The embodiment of the present invention uses high-order belief propagation to minimize energy loss to obtain the disambiguated surface normal vector n′, but the disambiguated surface normal vector still has noise and is affected by low-frequency deviations.

[0152] For S6, the disambiguated surface normal vector is used as the prior normal vector, and the pre-trained improved NeuS network is used to perform volume rendering on the position and orientation of the polarization camera, the signed distance function SDF, the color value and the prior normal vector to obtain the predicted color value and normal vector; based on the difference between the predicted normal vector and the prior normal vector and the difference between the predicted color value and the color value of the target scene, the three-dimensional information of the target scene is obtained.

[0153] Specifically, for each pixel, a set of points are sampled along the corresponding emission light, and the sampling point p i =o+d i e, where o represents the center of the polarization camera, e represents the direction of the emitted light, and the color can be the accumulation of the colors of the sampling points along this direction. The specific formula is as follows:

[0154]

[0155] Among them, T i represents the discrete cumulative transmittance corresponding to the i-th sampling point, α i represents the discrete opacity corresponding to the i-th sampling point, α i It can be related to the i-th sampling point p i The corresponding signed distance function f(p i (t i )) related functions.

[0156] The normal vector of a viewpoint surface can also be predicted by volume rendering accumulation. Specifically,

[0157]

[0158] Among them, n i ′ represents the normal vector of the i-th sampling point.

[0159] In the training phase of the improved NeuS network, in the space {c k ,o k ,e k} randomly samples a batch of pixels, and uses the difference between the predicted normal vector and the prior normal vector, the difference between the predicted color value and the color value of the target scene, and the difference between the corresponding references as the loss function of the improved NeuS network. Based on the loss function, the corresponding weights are learned to obtain accurate 3D information of the target scene. k Represents the color value of the pixel, o k represents the center position of the polarization camera, e k represents the direction of the polarization camera, the number of sampling points along each ray is h, and the number of sampling batches is m.

[0160] The embodiment of the present invention uses polarization information to separate specular and diffuse reflections, solves the normal vector and uses it as a priori input in the improved NeuS network after disambiguation, which can well reconstruct the specular reflection and textureless parts in the scene and obtain high-precision three-dimensional scene morphology.

[0161] Specifically, the loss function of the improved NeuS network is as follows:

[0162] L all =L color +λ nomal L nomal +λ reg L reg ;

[0163] Among them, L color represents the color loss function, L nomal Denotes the normal vector loss function, L reg represents the process function loss, λ nomal and λ reg Indicates the corresponding weight.

[0164] Specifically, Represents the predicted color value, C k Represents the color value of the target scene, n′ represents the prior normal vector, represents the predicted normal vector, represents the gradient operation, h represents the number of sampling points, m represents the number of sampling batches, i represents the i-th sampling point, k represents the k-th pixel point, f(p k,i ) represents the signed distance function corresponding to k,i.

[0165] The high-precision stereoscopic reconstruction method for complex scenes provided by the embodiment of the present invention introduces polarization information to perform three-dimensional reconstruction of complex large scenes, and can accurately reconstruct texture-free areas and specular reflection areas in the scene, such as walls, floors, furniture, etc., effectively overcoming the technical defects of multi-view three-dimensional reconstruction methods; it also has the ability to reconstruct the fine details of occluded, complex, and thin-structured objects, overcoming the scale ambiguity problem of reconstruction methods based on depth estimation.

[0166] The key idea of ​​the embodiment of the present invention is to use the difference between polarization information for specular reflection and diffuse reflection, the robustness of polarization information in texture-free areas, and the dependence of polarization on surface normal vectors, and use the normal vector prior after disambiguation as a globally consistent geometric constraint to guide the optimization process. Specifically, the position and orientation of the camera are first obtained from the multi-view image through polarization camera pose estimation, and the scene surface normal vector is obtained through the MLP network; the scene surface normal vector is used to correct the ambiguous polarization normal vector obtained using the polarization image, so as to obtain the prior normal vector, and the prior normal vector is input into the pre-trained improved NeuS network to provide additional constraints to alleviate the geometric ambiguity problem of texture-free areas and specular reflection areas. And compared with the depth prior, the network based on the prior normal vector can produce smooth geometric shapes, and can also avoid the problem of depth scale ambiguity between multiple views, and obtain more accurate three-dimensional information. The present invention can also reconstruct objects with occlusion and objects with thin structures with high precision, and can also perform better geometric reconstruction for scenes with sudden depth changes.

[0167] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A high-precision stereo reconstruction method for complex scenes, characterized in that: include: Use a polarization camera to collect multi-view images and multi-angle polarization images of the target scene; Process the multi-angle polarization images to obtain polarization information and the polarization degree of the target scene surface; Based on the polarization information and the polarization degree of the target scene surface, obtaining the azimuth angle and the incident angle of the incident light on the target scene surface; Based on the azimuth angle and the incident angle, a polarization normal vector is obtained; Based on the multi-view images, obtaining a spatial position and a viewing direction of a polarization camera; Based on the spatial position and the viewing direction, a signed distance function SDF and a color value of the target scene are obtained using an MLP network; based on the signed distance function SDF, a scene surface normal vector is obtained; using the scene surface normal vector as a guide vector, a polarization normal vector is corrected, and a disambiguated surface normal vector is obtained; The disambiguated surface normal vector is used as the prior normal vector, and the pre-trained improved NeuS network is used to perform volume rendering on the position and orientation of the polarization camera, the signed distance function SDF, the color value and the prior normal vector to obtain the predicted color value and normal vector; based on the difference between the predicted normal vector and the prior normal vector and the difference between the predicted color value and the color value of the target scene, the three-dimensional information of the target scene is obtained.

2. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: The method of collecting multi-view images and multi-angle polarization images of a target scene by using a polarization camera includes: Use the FLIR polarization camera to shoot the target scene and obtain video frames; Extract multi-view images from the video frame, obtain multi-angle image data of the scene, and combine the polarizer on the polarization camera to obtain multi-angle polarization images I0, I 45 ,I 90 ,I 135 .

3. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: The processing of the multi-angle polarization image to obtain polarization information and the polarization degree of the target scene surface includes: The multi-angle polarization image is processed to obtain polarization information; the polarization information is composed of Stokes vectors; the expression of the Stokes vector is as follows: Among them, I 0° Indicates the light intensity passing through the 0° polarizer, I 90° Indicates the light intensity passing through the 90° polarizer, I +45° It indicates the intensity of light passing through the polarizer at 45° to the horizontal positive direction, I -45° It represents the light intensity obtained by the linear polarizer at a positive 135° angle to the horizontal. RH Represents the right-handed circularly polarized light component, I LH Represents left-handed circularly polarized light component. Based on the Stokes vector, the polarization degree of the target scene surface is obtained using the first formula; the expression of the first formula is as follows:

4. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: Based on the polarization information and the polarization degree of the target scene surface, the azimuth angle and the incident angle of the incident light on the target scene surface are obtained, including: Based on the polarization information and the polarization degree of the target scene surface, using a second formula, obtaining the azimuth angle of the incident light on the target scene surface; The second formula is as follows: in, represents the azimuth, S0 and S1 represent the Stokes vector; Based on the relationship between the degree of polarization and the incident angle, the incident angle is obtained; The relationship between the degree of polarization and the angle of incidence is as follows: For specular reflection: For diffuse reflection: Wherein, P represents the degree of polarization, n represents the refractive index of the object surface, and θ represents the incident angle.

5. The high-precision stereo reconstruction method for complex scenes according to claim 1, characterized in that: Based on the azimuth angle and the incident angle, a polarization normal vector is obtained, including: Based on the azimuth and the angle of incidence, the polarization normal vector is obtained using the third formula: The third formula is as follows: Among them, n x express The component in the x-axis direction, n y Represents the component in the y-axis direction, n z represents the component in the z-axis direction, represents the azimuth, and θ represents the incident angle.

6. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: Based on the multi-view images, obtaining a spatial position and a viewing direction of a polarization camera includes: The multi-view image is processed using the COLMAP algorithm to obtain the position (x, y, z) and orientation of the polarization camera corresponding to the multi-view image. The fourth formula is used to calculate the position (x, y, z) and orientation of the polarization camera. Perform position encoding to obtain the corresponding spatial position γ(x, y, z) and view direction The fourth formula is as follows: γ(η)=[sin(20πη),cos(20πη),…,sin((2L-1)πη),cos((2L-1)πη)]; Among them, γ(η)=γ(x,y,z) or L represents the preset parameters. When processing the spatial position, η = x, y, z. When processing the view direction, 7. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: Based on the spatial position and the viewing direction, the MLP network is used to obtain the signed distance function SDF and the color value of the target scene, including: Mapping the spatial position to a signed distance of a surface to obtain a signed distance function SDF of the target scene; The colors of the spatial position and the viewing direction are encoded to obtain a color value of the target scene.

8. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: Based on the signed distance function SDF, the scene surface normal vector is obtained, including: The fifth formula is used to calculate the gradient of the signed distance function SDF to obtain the scene surface normal vector The fifth formula is as follows: in, represents the gradient operation, and f(p) represents the signed distance function SDF.

9. The high-precision stereo reconstruction method of a complex scene according to claim 1, characterized in that: The scene surface normal vector is used as a guide vector to correct the polarization normal vector to obtain a disambiguated surface normal vector, including: The polarization normal vector is corrected based on the minimization of the energy loss function to obtain the disambiguated surface normal vector; The minimized energy loss function is as follows: Among them, u = (x, y), v = (x + 1, y) and w = (x, y + 1) represent pixel points, E pol (n polar (u)) represents the polarization energy loss function, which is obtained based on the scene surface normal vector and the polarization normal vector. smooth (n polar (u),n polar (v),n polar (w)) represents the smoothing term energy loss function, n polar (u) represents the polarization normal vector at pixel u, n polar (v) represents the polarization normal vector at pixel v, n polar (w) represents the polarization normal vector at pixel w, and λ1 represents the preset weight.

10. The high-precision stereoscopic reconstruction method of a complex scene according to claim 1, characterized in that: The loss function of the improved NeuS network is as follows: L all =L color +λ nomal L nomal +λ reg L reg ; Among them, L color represents the color loss function, L nomal Denotes the normal vector loss function, L reg represents the process function loss, λ nomal and λ reg Indicates the corresponding weight.

Images