High-light reflection scene object three-dimensional surface reconstruction method based on adaptive plane and jump sampling

Abstract

The invention discloses a highlight reflection scene object three-dimensional surface reconstruction method based on an adaptive plane and jump sampling, which distinguishes reflection interference from a target object through a pixel-by-pixel auxiliary plane, and introduces a Laplace gradient to reduce the influence of external reflection interference. In addition, the influence of reflection interference is reduced by adopting a jump sampling strategy, and the multi-view consistency and the surface fidelity are further improved. Finally, rejection loss is adopted, gradient differences of a target object and a reflection part are compared, the model is helped to more accurately distinguish the target object and the reflection part during initialization, and a more accurate model surface is reconstructed under highlight reflection interference.

Description

Technical Field

[0001] The present invention relates to the field of neural radiation field three-dimensional reconstruction and deep learning technology, and in particular to a three-dimensional surface reconstruction method for high-light reflection scene objects based on adaptive plane and jump sampling. Background Art

[0002] Reconstructing the 3D surface of an object from multi-view images is a challenging task in computer vision and graphics. Most current 3D reconstruction works are performed in low or no reflection scenarios. However, in many everyday situations, capturing images through transparent glass is unavoidable, in which case the images are often affected by reflection artifacts. These artifacts reduce the consistency of multiple views and introduce additional noise, posing a significant challenge to high-quality 3D surface reconstruction.

[0003] NeuS uses the signed distance function (SDF) to describe the multi-view surface reconstruction task of the object surface and can generate a detailed 3D surface. However, in the case of high-reflectivity scenes (HSR), it is difficult for NeuS to distinguish the interference part from the target object, and it tends to reconstruct the reflection interference on the target object to meet the requirements of photometric consistency loss. A single pixel in the HSR image contains information from multiple objects, and NeuS uses a single ray to point to the pixel point, resulting in ambiguity in the depth and SDF surface value of the point, thus affecting the quality of reconstruction. 3D reconstruction in HSR scenes faces two main problems: (1) accurate estimation of the shape of the target object, and (2) incorrect rendering due to inconsistency in multiple views.

[0004] To solve these problems, a simple and direct solution is to remove reflections by preprocessing the training set to obtain clean training objects, and then perform 3D reconstruction. However, most current image reflection removal methods require real data and other image cues for supervised training, and have poor generalization performance. NeRFReN models the reflections and projections in the scene to a certain extent, and uses bidirectional deep smoothing to help the network correctly model the reflections and projections. However, its modeling target is the entire scene, not the target object after removing interference. Ref-NeuS uses the multi-view color variance corresponding to the surface point as the reflection score to guide the network to correctly model the reflection part. However, when there is similar interference in the corresponding positions of multiple views, the reflection score will fail. NeuS-HSR sets an orthogonal mirror and divides the scene into two parts, in front of the mirror and behind the mirror. To a certain extent, it achieves the goal of removing HSR, but it does not consider the factor of multiple perspectives, and the effect is poor for scenes where the target object and the reflected ghost are highly overlapped.

[0005] In order to obtain a more accurate 3D model in the HSR scenario, this paper considers a more flexible decomposition strategy than NeuS-HSR to enhance the appearance of the target object. To achieve this goal, the following assumptions are made:

[0006] Assumption 1: The scene affected by HSR can be divided into the target object and the reflection part. The two are unrelated, and the reflection part can be obtained by a reflector.

[0007] Hypothesis 2: The pixel correlation of the reflection part decreases with the increase of pixel spacing, while the target object part is less affected. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and provide a method for 3D surface reconstruction of high-light reflective scene objects based on adaptive planes and jump sampling. The auxiliary plane design based on NeuS-HSR enhances its flexibility by modifying the reflection formula, and introduces Laplace gradient prior and repulsion loss to enhance the appearance of the target object. At the same time, the normalized cross-correlation (NCC) loss of Geo-NeuS is introduced to enhance the consistency of the surface, and using assumption 2, jump sampling is used to reduce HSR interference and reconstruct a more refined surface.

[0009] Technical solution: A method for reconstructing a three-dimensional surface of a high-light reflection scene object based on adaptive plane and jump sampling of the present invention comprises the following steps:

[0010] Step 1: Construct sampling light;

[0011] For each pixel on the input perspective image C(r), i sampling points x are obtained by sampling from the camera position o along the line of sight direction v at depth t. i , aggregate all sampling points to generate a sampling ray r = o + tv;

[0012] Step 2: Set the sampling point x in step 1 i Input to Geometric Network F s The training output is obtained by obtaining the surface normal n of the target object in the image and the sampling point weight w t and the signed distance function value SDF of the corresponding point; the geometric network is an 8-layer residual MLP network;

[0013] At the same time, the sight direction v is input to the mirror module F r The output is the mirror normal n r and the corresponding weight w r , and then use the mirror normal n r For the sampling point x iPerform linear transformation to obtain the reflected light point p r ;

[0014] Step 3: Use the camera's intrinsic parameter matrix K, extrinsic rotation matrix R and extrinsic translation vector t to transform the sampling point x in step 1 i Mapping back to the Laplace sharpened map G i The corresponding prior m is obtained from LoG , the prior m LoG With the sampling point x in step 1 i Input the target network F together c Output the rendered target object image C t ;

[0015] At the same time, the reflected light point p in step 2 r With zero tensor Input the reflection network to render the reflected object image C r , the reflected object image C r and the target object image C t Perform linear combination to get the final rendered image

[0016] Step 4: Use the signed distance function value SDF generated in step 2 to determine the position of the implicit surface s of the object, map it back to the source grayscale image (i.e. convert the RGB image into a grayscale image and perform Laplace sharpening on it), perform hierarchical jump sampling on it, compare the similarity of each view image, and finally use the photometric consistency loss To constrain; the specific method is as follows:

[0017] Step 4.1, determine the implicit surface s position of the target object;

[0018] Since the implicit surface s is a geometric body composed of points p with zero SDF values, the interpolation method is used to find the implicit surface s. If the SDF value changes from positive to negative / from negative to positive in a small area, it means that the implicit surface s passes through the area. Then the implicit surface s is mapped to the grayscale image I in the current line of sight direction. r and the source grayscale image I in other viewing directions s , and get the corresponding projection area p ref and Where N p Indicates the number of source grayscale images, and the size of the region is 11×11;

[0019] Then the projection area p ref The image points in Through the homogeneous matrix H and p n The corresponding points in Related:

[0020]

[0021]

[0022] Among them, t r and t s Represent the reference image I r Heyuan Grayscale Image I s The translation vector of r and R s represents the rotation matrix, K r and K s Represents their internal parameter matrix; the normal n is obtained through the geometric network F s At the surface point x s Calculated by automatic differentiation, where d = -n T x s ;

[0023] Step 4.2: The grayscale image I of the current sight direction r Divide into L layers, the pixel interval of each layer increases and the grayscale information entropy of each layer of pixels is calculated:

[0024]

[0025] Where N b Indicates the number of intervals of the grayscale histogram. Indicates skip sampling of the reference region p in the pyramid layer l ref Probability distribution of the grayscale values ​​of the inner pixels;

[0026] Step 4.3: To promote the reasonable allocation of multi-level regions, the final weight w ∈ (l) is defined as:

[0027]

[0028] Exploiting photometric consistency loss To enhance consistency between multiple views:

[0029]

[0030]

[0031] cov(·) and var(·) denote the covariance and variance, respectively, K denotes the best K NCC scores, and N denotes the number of rendered pixels in one training iteration;

[0032] Step 5: Use the perspective image C(r) input in step 1 to render the final rendered image in step 3 Perform supervision and compare the target object image C in step 3 t and the reflected object image C r The gradient similarity of To constrain.

[0033] Furthermore, the detailed process of constructing the sampling light in step 1 is as follows:

[0034] For each pixel (i, j) in the view image C(r), transform it to the normalized device coordinate system, where the focal length of the camera in the horizontal and vertical directions is f x and f y , the optical center (c x ,c y ) and pixel coordinates (i,j) are converted to normalized coordinates (x,y):

[0035]

[0036] In the normalized device coordinate system, the light direction is represented by a vector (x, y, 1), which means that the light direction is from the optical center (0, 0, 0) to (x, y, 1);

[0037] The resulting light direction is transformed from the camera coordinate system to the world coordinate system using the camera's rotation matrix R and translation vector t; the line of sight direction v is expressed in the world coordinate system as:

[0038] v=R[x,y,1] T

[0039] The starting point o=t of the ray is represented in the world coordinate system as the position of the camera.

[0040] Furthermore, in step 2, the reflected light point p is obtained by linear transformation. r The specific method is:

[0041] First, use the mirror module F r Map the view direction v to the volume density σ r , mirror normal n r and the reflected light source point p l , expressed as:

[0042] {σ r ,p l ,n r =F r (v)

[0043] Then, the line of sight v passes through the corresponding mirror normal n r Transformed into reflected light r mir :

[0044] r mir =v-2(v☉n r )☉n r

[0045] Since the reconstruction process involves three parts: the target object, the mirror surface, and the external interference, and the external interference of each pixel is represented by the corresponding reflected light source point p l Produced, so by reflecting the light r mir , camera position o and mirror normal n r It can be mapped to the mirror position p h ,Right now:

[0046] p h =p l +hr mir

[0047]

[0048] Where ⊙ represents the Hadamard product, 0<h<z s , z s Indicates the surface depth of the target object;

[0049] Finally, with the mirror position p h As the starting point, with the reflected light r mir is the ray direction, and the sampling ray point set of the reflected part is obtained by linearly mapping the depth t of the target object ray to obtain the reflected ray point p r :

[0050] p r =p h +flip(t)r mir

[0051] Where flip(·) represents a tensor flip operation.

[0052] Furthermore, in step 3, the priori m LoG From the sampling point x in step 1 i Map back to the Laplace sharpened image G through the intrinsic parameter matrix K, the extrinsic rotation matrix R and the extrinsic translation vector t i Income:

[0053] m LoG =interp(G i , K(Rx i +t)).

[0054] Furthermore, the training process in step 5 is divided into two stages:

[0055] The first stage is the preheating stage, which initializes the geometric contour of the model surface;

[0056] In the second stage, photometric consistency loss is introduced Through multi-view Figure 1 Refine the model surface by using consistency;

[0057] The final training loss function The definition is as follows:

[0058]

[0059] Color Loss Indicates the actual pixel color C b Final render color The difference between the two, b represents the color loss of a specific rendered pixel The expression is as follows:

[0060]

[0061] and is a regularization term that regularizes the gradient of the SDF network:

[0062]

[0063] Assume that the actual pixel color C of the supervised image b The edge information in the image C is obtained by reflecting the object. r and the target object image C t The two are separable and non-overlapping; the C generated by each iteration t and C r Combine them into small images T and R respectively, and perform N s Downsample and use Sigmoid to smooth the gradient, using Minimize the gradient correlation between the target object and the HSR interference at multiple resolutions, thereby obtaining rejection The expression is as follows:

[0064]

[0065]

[0066] in represents the gradient of the image, λ T and λ R There are two hyperparameters used to control the gradient difference between images. In most scenarios, λ T and λ R Defined as 1.

[0067] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0068] 1. The present invention proposes a flexible auxiliary plane to separate the target object and the high light interference, and introduces high-frequency image gradient information to assist in the surface reconstruction of the target object.

[0069] 2. The present invention adopts jump sampling to enhance the consistency of multiple perspectives and improve the fineness of the reconstructed surface.

[0070] 3. The design of the present invention introduces a rejection loss and utilizes the gradient difference between the target object and the interference to help correctly initialize the model surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the overall three-dimensional reconstruction structure of the present invention.

[0072] Figure 2 It is a structural diagram of a three-dimensional model scene in an embodiment of the present invention.

[0073] Figure 3 The figure is a structural diagram and effect comparison of the skip sampling scheme in the embodiment of the present invention.

[0074] Figure 4 It is a Laplace sharpening prior map in an embodiment of the present invention.

[0075] Figure 5 It is a grid comparison diagram of the technical solution of the present invention and the prior art on a synthetic data set. DETAILED DESCRIPTION

[0077] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0078] like Figure 1 As shown, the method for reconstructing the three-dimensional surface of a high-light reflection scene object based on adaptive plane and jump sampling of the present invention comprises the following steps:

[0079] Step 1: Construct sampling light;

[0080] For each pixel on the input perspective image C(r), i sampling points x are obtained by sampling from the camera position o along the line of sight direction v at depth t. i , aggregate all sampling points to generate a sampling ray r = o + tv;

[0081] Step 2: Set the sampling point x in step 1 i Input to Geometric Network F s The training output is obtained by obtaining the surface normal n of the target object in the image and the sampling point weight w t and the signed distance function value SDF of the corresponding point; the geometric network is an 8-layer residual MLP network;

[0082] At the same time, the sight direction v is input to the mirror module F r The output is the mirror normal n r and the corresponding weight w r , and then use the mirror normal n r For the sampling point x i Perform linear transformation to obtain the reflected light point p r ;

[0083] Step 3: Use the camera's intrinsic parameter matrix K, extrinsic rotation matrix R and extrinsic translation vector t to transform the sampling point x in step 1 i Mapping back to the Laplace sharpened map G i The corresponding prior m is obtained from LoG , the prior m LoG With the sampling point x in step 1 i Input the target network F together c Output the rendered target object image C t ;

[0084] At the same time, the reflected light point p in step 2 r With zero tensor Input the reflection network to render the reflected object image C r , the reflected object image C r and the target object image C t Perform linear combination to get the final rendered image This embodiment uses the same rendering network to render the target color and the reflection color (corresponding to the target network and the reflection network), that is, the spatial position of the sampling point is used to determine the difference between the two colors;

[0085] Step 4: Use the signed distance function value SDF generated in step 2 to determine the position of the implicit surface s of the object, map it back to the source grayscale image, perform hierarchical jump sampling on it, compare the similarity of each view image, and finally use the photometric consistency loss To constrain; the specific method is as follows:

[0086] Step 4.1, determine the implicit surface s position of the target object;

[0087] Since the implicit surface s is a geometric body composed of points p with zero SDF values, the interpolation method is used to find the implicit surface s. If the SDF value changes from positive to negative / from negative to positive in a small area, it means that the implicit surface s passes through the area. Then the implicit surface s is mapped to the grayscale image I in the current line of sight direction. r and the source grayscale image I in other viewing directions s , and get the corresponding projection area p ref and Where Np Indicates the number of source grayscale images, and the size of the region is 11×11;

[0088] Then the projection area p ref The image points in Through the homogeneous matrix H and p n The corresponding points in Related:

[0089]

[0090]

[0091] Among them, t r and t s Represent the reference image I r and source image I s The translation vector of r and R s represents the rotation matrix, K r and K s Represents their internal parameter matrix; the normal n is obtained through the geometric network F s At the surface point x s Calculated by automatic differentiation, where d = -n T x s ;

[0092] Step 4.2: The grayscale image I of the current sight direction r Divide into L layers, the pixel interval of each layer increases and the grayscale information entropy of each layer of pixels is calculated, such as Figure 4 As shown:

[0093]

[0094] Where N b Indicates the number of intervals of the grayscale histogram. Indicates skip sampling of the reference region p in the pyramid layer l ref Probability distribution of the grayscale values ​​of the inner pixels;

[0095] Step 4.3: To promote the reasonable allocation of multi-level regions, the final weight w ∈ (l) is defined as:

[0096]

[0097] use To enhance consistency between multiple views:

[0098]

[0099]

[0100] cov(·) and var(·) denote the covariance and variance, respectively, K denotes the best K NCC scores, and N denotes the number of rendered pixels in one training iteration;

[0101] Step 5: Use the perspective image C(r) input in step 1 to render the final rendered image in step 3 Perform supervision and compare the target object image C in step 3 t and the reflected object image C r The gradient similarity of To constrain.

[0102] In this embodiment, step 2 divides the reconstructed model space into three parts: target object, mirror surface and external interference. Figure 2 As shown, the three are based on implicit surface and virtual mirror position p h The reflection sampling ray p r The interpretability of HSR scenes is enhanced. The consistency between multiple views and the color differences of different objects prompt the neural network to autonomously separate the target objects from the interference parts, thereby achieving the aggregation of target objects and the elimination of inconsistent interference.

[0103] In this embodiment, the Laplace sharpening gradient is introduced in step 3 because highlight reflections are caused by light directly reflecting or refracting on the surface of the object and then entering the camera lens, resulting in areas with abnormally high brightness. When highlight reflections appear in the image, they are usually due to the rapid transition of brightness from the shadow area to the highlight area, which leads to a sharp change in the local gradient, which is displayed as a low value domain or edge (such as Figure 3 shown).

[0104] In step 4 of this embodiment, during the hierarchical jump sampling, the current view image is taken as a reference, and the images of other view images are processed in the same way. The similarities between them under the same conditions are compared to constrain the entire network.

[0105] In step 5 of this embodiment, the training process in step 5 is further divided into two stages:

[0106] The first stage is the warm-up stage, which initializes the geometric profile of the model surface;

[0107] In the second stage, photometric consistency loss is introduced Through multi-view Figure 1 Refine the model surface by using consistency;

[0108] The final training loss function The definition is as follows:

[0109]

[0110] Color loss Indicates the actual pixel color C i Final render color The difference between:

[0111]

[0112] and is a regularization term that regularizes the gradient of the SDF network:

[0113]

[0114] Assume that the actual pixel color C of the supervised image i (corresponding to the perspective image C(r), where C(r) takes the sampling light r as the standard color, C i Consider a specific pixel in the image) where the edge information is obtained by reflecting the object image C r and the target object image C t The two are separable and non-overlapping; the C generated by each iteration t and C r Combine them into small images T and R respectively, and perform N s Downsample and use Sigmoid to smooth the gradient, using Minimize the gradient correlation between the target object and the HSR interference at multiple resolutions, thereby obtaining rejection The expression is as follows:

[0115]

[0116]

[0117] in represents the gradient of the image, λ T and λ R There are two hyperparameters used to control the gradient difference between images. In most scenarios, λ T and λ R Defined as 1.

[0118] Example

[0119] This example applies the technical solution of the present invention to a specific data set for verification. The data set uses the publicly released data set of NeuS-HSR.

[0120] The dataset consists of two parts: 10 DTU high-light interference scenes generated by a single-image reflection synthesis method and 6 real high-light scenes from the Internet. Each scene in the DTU synthetic dataset contains 49 to 64 images with a resolution of 1200×1600, as well as the corresponding camera parameters. The images in the DTU synthetic dataset show different degrees of high-light reflection interference. The real scene consists of multiple images with a resolution of 1920×1080, and the camera parameters are estimated using COLMAP.

[0121] The training network of the present invention runs under the PyTorch deep learning framework, using two MLPs to model the geometric network F s and rendering network F c (including target network and reflection network), each MLP consists of 8 linear layers and contains a skip connection in the fourth layer. r A 3-layer MLP is used to predict volume density, and a 2-layer MLP is used to capture reflection-related properties. N = 512 rays are sampled per iteration, and the sampling points of each ray are sampled according to the stratified sampling strategy of NeuS. It takes about 9 hours of computation on a single NVIDIA RTX 4090 GPU. After training, the object mesh is extracted using the marching cubes algorithm.

[0122] The comparative analysis results of the technical solution of the present invention and other models are shown in Table 1.

[0123] Table 1

[0124] Scene ID UNISURF VolSDF NeuS NeuS-HSR The present invention Scan24 2.92 3.89 5.30 2.07 1.01 Scan37 4.26 2.91 2.29 1.89 0.83 Scan40 3.36 2.44 2.02 2.17 1.11 Scan55 2.11 3.95 1.73 1.25 0.81 Scan63 2.73 - 2.75 1.94 0.96 Scan65 1.57 - 0.93 1.15 0.85 Scan69 5.00 - 4.15 3.54 0.86 Scan83 1.81 - 2.55 1.42 1.40 Scan97 3.85 - 4.62 2.82 1.12 Scan105 2.01 - 1.53 1.31 1.15 Mean 2.96 / 2.79 1.96 1.01

[0125] In order to evaluate the quality of reconstruction, this embodiment uses the chamfer distance as a measurement indicator. The smaller the chamfer distance value, the better the performance and the more accurate the reconstructed model surface.

[0126] Table 1 shows the quantitative comparison results with other existing technologies in 10 scenes of the DTU synthetic dataset. Specifically, the technical solution of the present invention shows superior performance in the reconstruction of 3D object surfaces in high-light reflection scenes, and obtains more accurate results compared with other methods. When comparing the quality of visual reconstruction, the present invention shows high robustness and accuracy, especially in complex HSR scenes. These scenes usually have highly reflective surfaces and complex geometries, making accurate reconstruction more difficult. Figure 5 As shown, the present invention effectively reduces reflection interference, clearly reconstructs the geometric details of the correct target objects in Scan24 and porcelain scenes, and the reconstructed object surfaces are smoother and flatter.

Claims

1. A method for reconstructing three-dimensional surfaces of high-light reflection scene objects based on adaptive plane and jump sampling, characterized in that: The following steps are involved: Step 1: Construct sampling light; For each pixel on the input perspective image C(r), i sampling points x are obtained by sampling from the camera position o along the line of sight direction v at depth t. i , aggregate all sampling points to generate a sampling ray r = o + tv; Step 2: Set the sampling point x in step 1 i Input to Geometric Network F s The training output is obtained by obtaining the surface normal n of the target object in the image and the sampling point weight w t and the signed distance function value SDF of the corresponding point; the geometric network is an 8-layer residual MLP network; At the same time, the sight direction v is input to the mirror module F r The output is the mirror normal n r and the corresponding weight w r , and then use the mirror normal n r For the sampling point x i Perform linear transformation to obtain the reflected light point p r ; Step 3: Use the camera's intrinsic parameter matrix K, extrinsic rotation matrix R and extrinsic translation vector t to transform the sampling point x in step 1 i Mapping back to the Laplace sharpened map G i The corresponding prior m is obtained from LoG , the prior m LoG With the sampling point x in step 1 i Input the target network F together c Output the rendered target object image C t ; At the same time, the reflected light point p in step 2 r With zero tensor Input the reflection network to render the reflected object image C r , the reflected object image C r and the target object image C t Perform linear combination to get the final rendered image Step 4: Use the signed distance function value SDF generated in step 2 to determine the position of the implicit surface s of the object, map it back to the source grayscale image, perform hierarchical jump sampling on it, compare the similarity of each view image, and finally use the photometric consistency loss To constrain; the specific method is as follows: Step 4.1, determine the implicit surface s position of the target object; Since the implicit surface s is a geometric body composed of points p with zero SDF values, the interpolation method is used to find the implicit surface s. If the SDF value changes from positive to negative / from negative to positive in a small area, it means that the implicit surface s passes through the area. Then the implicit surface s is mapped to the grayscale image I in the current line of sight direction. r and the source grayscale image I in other viewing directions s , and get the corresponding projection area p ref and Where N p Indicates the number of source grayscale images, and the size of the region is 11×11; Then the projection area p ref The image points in Through the homogeneous matrix H and the corresponding points in pn Related: Among them, t r and t s Represent the reference image I r and source image I s The translation vector of r and R s represents the rotation matrix, K r and K s Represents their internal parameter matrix; the normal n is obtained through the geometric network F s At the surface point x s Calculated by automatic differentiation, where d = -n T x s ; Step 4.2: The grayscale image I of the current sight direction r Divide into L layers, the pixel interval of each layer increases and the grayscale information entropy of each layer of pixels is calculated: Where N b Indicates the number of intervals of the grayscale histogram. Indicates skip sampling of the reference region p in the pyramid layer l ref Probability distribution of the grayscale values ​​of the inner pixels; Step 4.3: To promote the reasonable allocation of multi-level regions, the final weight w ∈ (l) is defined as: Exploiting photometric consistency loss To enhance consistency between multiple views: cov(·) and var(·) denote the covariance and variance, respectively, K denotes the best K NCC scores, and N denotes the number of rendered pixels in one training iteration; Step 5: Use the perspective image C(r) input in step 1 to render the final rendered image in step 3 Perform supervision and compare the target object image C in step 3 t and the reflected object image C r The gradient similarity of To constrain.

2. The method for reconstructing three-dimensional surface of high-reflection scene objects based on adaptive plane and jump sampling according to claim 1, characterized in that: The detailed process of constructing the sampling light in step 1 is as follows: For each pixel (i, j) in the view image C(r), transform it to the normalized device coordinate system, where the focal length of the camera in the horizontal and vertical directions is f x and f y , the optical center (c x , c y ) and pixel coordinates (i, j) are converted to normalized coordinates (x, y): In the normalized device coordinate system, the light direction is represented by a vector (x, y, 1), which means that the light direction is from the optical center (0, 0, 0) to (x, y, 1); The resulting light direction is transformed from the camera coordinate system to the world coordinate system using the camera's rotation matrix R and translation vector t; the line of sight direction v is expressed in the world coordinate system as: v=R[x,y,1] T The starting point o=t of the ray is represented in the world coordinate system as the position of the camera.

3. The method for reconstructing three-dimensional surface of high-light reflection scene objects based on adaptive plane and jump sampling according to claim 1, characterized in that: In step 2, the reflected light point p is obtained by linear transformation. r The specific method is: First, use the mirror module F r Map the view direction v to the volume density σ r , mirror normal n r and the reflected light source point p l , expressed as: {s} r ,p l ,n r }=F r (v) Then, the line of sight v passes through the corresponding mirror normal n r Transformed into reflected light r mir : r mir =v-2(vv⊙n r )⊙n r Since the reconstruction process involves three parts: target object, mirror and external interference, and the external interference of each pixel is represented by the corresponding reflected light source point p l Produced, so by reflecting the light r mir , camera position o and mirror normal n r It can be mapped to the mirror position p h ,Right now: p h =p l +hr mir where ⊙ represents the Hadamard product, 0 <h<z s , z s Indicates the surface depth of the target object; Finally, with the mirror position p h As the starting point, with the reflected light r mir is the ray direction, and the sampling ray point set of the reflected part is obtained by linearly mapping the depth t of the target object ray to obtain the reflected ray point p r : p r =P h +flip(t)r mir Where flip(·) represents a tensor flip operation.

4. The method for reconstructing three-dimensional surface of high-reflection scene objects based on adaptive plane and jump sampling according to claim 1, characterized in that: In step 3, the prior m LoG From the sampling point x in step 1 i Map back to the Laplace sharpened image G through the intrinsic parameter matrix K, the extrinsic rotation matrix R and the extrinsic translation vector t i Income: m L o G =interp(G i ,K(Rx i +t))。 5. The method for reconstructing three-dimensional surface of high-light reflection scene objects based on adaptive plane and jump sampling according to claim 1, characterized in that: The training process in step 5 is divided into two stages: The first stage is the preheating stage, which initializes the geometric contour of the model surface; In the second stage, photometric consistency loss is introduced Refine model surfaces through multi-view consistency; The final training loss function The definition is as follows: Color Loss Indicates the actual pixel color Cb and the final rendered color The difference between the two, b represents the color loss of a specific rendered pixel The expression is as follows: and is a regularization term that regularizes the gradient of the SDF network: Assume that the actual pixel color C of the supervised image b The edge information in the image C is obtained by reflecting the object. r and the target object image C t The two are separable and non-overlapping; the C generated by each iteration t and C r Combine them into small images T and R respectively, and perform N s Downsample and use Sigmoid to smooth the gradient, using Minimize the gradient correlation between the target object and the HSR interference at multiple resolutions to obtain the rejection loss The expression is as follows: in represents the gradient of the image, λ T and λ R There are two hyperparameters used to control the gradient difference between images. In most scenarios, λ T and λ R Defined as 1.

Images