DES-NeRF-based post-transparent-surface scene reconstruction method and device

Abstract

The invention discloses a DES-NeRF-based post-transparent-surface scene reconstruction method and device, and the method comprises the steps: inputting the camera information and real depth maps corresponding to a plurality of visible light images of a target scene into a neural radiation field after the camera information and real depth maps corresponding to the visible light images are obtained; the neural radiation field training is supervised by using a pre-constructed loss function, and after the number of times of iteratively updating the network parameters of the neural radiation field reaches a preset number of times, the neural radiation field outputs the volume density and color of each sampling point in the target scene; the volume density and color of each sampling point in the target scene are converted to obtain a reconstructed target scene, and the pre-constructed loss function is constructed based on absolute depth error loss and depth edge perception loss. Through the method provided by the invention, the reduction degree of the reconstructed target scene can be improved, and the performance of the reconstructed target scene at the edge details is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional modeling, and specifically relates to a transparent surface back-scene reconstruction method and device based on DES-NeRF (depth & edge loss supervision Neural Radiance Fields). Background Art

[0002] The anti-aliased neural radiation field adopts a cone sampling strategy to synthesize high-quality anti-aliased images. In order to further optimize the coordinate processing of large scenes, the anti-aliased neural radiation field adopts a spherical parameterization strategy. This strategy enables the neural radiation field to calculate large-scale scenes by shrinking the coordinates of unbounded scenes into a bounded range. In addition, the anti-aliased neural radiation field introduces a multi-scale sampling strategy, which effectively balances anti-aliasing and image clarity, and encodes and indexes the training data through hash coding technology, which greatly reduces the data access and processing time during training, so that the anti-aliased neural radiation field has achieved significant improvements in image quality and training speed.

[0003] Existing reconstruction methods based on anti-aliasing neural radiance fields generally include NeRFReN (Neural Radiance Fields with Reflections), NeRS (Neural Reflectance Surfaces) and NeRRF (Neural Refractive-Reflective Fields). Among them, NeRFReN is a new view synthesis method for processing complex reflective scenes. By decomposing the scene into two parts, transmission and reflection, and modeling them with different neural radiance fields respectively, the depth estimation and view synthesis of the reflective area are more accurately processed, and the depth smoothness and bidirectional depth consistency constraints are used to guide the decomposition process, which solves the under-constraint problem of scene decomposition, thereby solving the limitations of NeRF (neural radiance field) in processing complex reflections. NeRS decomposes the view-dependent appearance into ambient lighting, diffuse color (albedo) and specular "glossiness" by learning neural bidirectional surface reflectance functions (BRDFs), thereby better capturing the surface reflection characteristics. It initializes the target as a cuboid or sphere, and fits the geometric shape by shrinking the watertight structure. NeRRF uses refraction-reflection fields to model the complex light path changes of transparent and specular objects, including refraction and reflection effects. This approach enables NeRRF to handle complex optical phenomena through unified modeling through Fresnel terms. NeRRF first uses Deep Marching Tetrahedra (DMTet) and progressive coding to reconstruct the geometry of non-Lambertian objects. This method allows accurate 3D shapes to be recovered from the outlines of objects. NeRRF is able to learn ambient lighting from reflective or refracting surfaces. In order to improve anti-aliasing effects and reduce high-frequency noise, NeRRF proposes a virtual cone-based supersampling technique that averages surface normals by sampling within the virtual cone, thereby improving rendering quality.

[0004] However, all of the above reconstruction methods have some shortcomings. NeRFReN requires manual annotation of transflective areas, which is a time-consuming process and cannot export a complete scene model. This means that although it can generate high-quality visual effects, it cannot be further used for physical simulation, virtual reality, or other applications that require precise geometric information. NeRS has limited ability to reconstruct complex topological structures, which makes it difficult to deal with scenes with rich geometric details and complex surface characteristics. NeRRF may encounter some challenges when relying solely on contours as supervision for neural radiation field reconstruction. Although contour information can provide the approximate shape of an object, it is not enough to capture the complex geometric structure and surface details of the object. This may lead to inaccurate reconstructed geometry, especially when dealing with non-vertex objects. In addition, NeRRF may also encounter difficulties when dealing with objects with complex lighting and occlusion relationships. This is because the contour information itself is not sufficient to fully describe the three-dimensional structure of the object, especially in the case of lighting changes or occlusion, the contour may become blurred. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method and device for reconstructing a transparent surface scene based on DES-NeRF.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a method for reconstructing a scene behind a transparent surface based on DES-NeRF, the method comprising: obtaining camera information and a true depth map corresponding to a plurality of visible light images; the plurality of visible light images are a plurality of images taken by a camera of a target scene at a plurality of viewing angles;

[0008] The camera information and the real depth map corresponding to each visible light image are input into the neural radiation field, and the pre-constructed loss function is used to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the loss function is constructed based on the absolute depth error loss and the depth edge perception loss; the absolute depth error loss is calculated based on the real depth value of each pixel point in the real depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the real depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene;

[0009] The volume density and color of each sampling point in the target scene are converted to obtain a reconstructed target scene.

[0010] Optionally, obtain camera information and true depth maps corresponding to multiple visible light images, including:

[0011] Acquire a visible light image sequence of a target scene; the visible light image sequence includes a plurality of visible light images of the target scene taken by a camera at multiple viewing angles;

[0012] Processing the visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image;

[0013] A true depth map is calculated based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

[0014] Optionally, converting the volume density and color of each sampling point in the target scene to obtain a reconstructed target scene includes:

[0015] The volume density and color of each sampling point in the target scene are converted using a truncated distance signed function to obtain a reconstructed target scene.

[0016] Optionally, the loss function is:

[0017] L=αL rec +βL abs +γL edge ;

[0018] Where, L represents the total loss; L rec Indicates that L abs and L edge Other losses other than abs represents the absolute depth error loss; L edge represents the depth edge perception loss; α, β, and γ are all preset weights.

[0019] Optionally, the absolute depth error loss is calculated as follows:

[0020]

[0021] in, Represents the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image; Represents the true depth value of each pixel in the true depth map corresponding to the i-th visible light image.

[0022] Optionally, the depth edge perception loss is calculated as follows:

[0023]

[0024] Where N represents the total number of visible light images; represents the predicted depth map corresponding to the i-th visible light image; represents the true depth map corresponding to the i-th visible light image; G(·) is the Sobel operator.

[0025] In a second aspect, the present invention provides a transparent surface scene reconstruction device based on DES-NeRF, the transparent surface scene reconstruction device comprising:

[0026] An acquisition module, used to acquire camera information and real depth maps corresponding to multiple visible light images; the multiple visible light images are multiple images taken by the camera of the target scene at multiple viewing angles;

[0027] A neural radiation field training module, used to input the camera information and true depth map corresponding to each visible light image into the neural radiation field, and use a pre-constructed loss function to supervise the neural radiation field training, after the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the loss function is constructed based on absolute depth error loss and depth edge perception loss; the absolute depth error loss is calculated based on the true depth value of each pixel point in the true depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the true depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene;

[0028] The target scene reconstruction module is used to transform the volume density and color of each sampling point in the target scene to obtain a reconstructed target scene.

[0029] Optionally, the acquisition module is specifically used to:

[0030] Acquire a visible light image sequence of a target scene; the visible light image sequence includes multiple visible light images of the target scene taken by a camera at multiple viewing angles; process the visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image; and calculate a true depth map based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

[0031] Optionally, the target scene reconstruction module is specifically used to:

[0032] The volume density and color of each sampling point in the target scene are converted using a truncated distance signed function to obtain a reconstructed target scene.

[0033] Optionally, the loss function is:

[0034] L=αL rec +βL abs +γL edge ;

[0035] Where, L represents the total loss; L rec Indicates that L abs and L edge Other losses other than abs represents the absolute depth error loss; L edge represents the depth edge perception loss; α, β, and γ are all preset weights.

[0036] Optionally, the absolute depth error loss is calculated as follows:

[0037]

[0038] in, Represents the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image; Represents the true depth value of each pixel in the true depth map corresponding to the i-th visible light image.

[0039] Optionally, the depth edge perception loss is calculated as follows:

[0040]

[0041] Where N represents the total number of visible light images; represents the predicted depth map corresponding to the i-th visible light image; represents the true depth map corresponding to the i-th visible light image; G(·) is the Sobel operator.

[0042] In a third aspect, the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0043] Memory, used to store computer programs;

[0044] The processor is used to implement the method steps described in any of the above-mentioned DES-NeRF-based transparent surface back scene reconstruction methods when executing the program stored in the memory.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any of the above-mentioned DES-NeRF-based transparent surface post-scene reconstruction methods are implemented.

[0046] The present invention provides a method for reconstructing a transparent surface back scene based on DES-NeRF. After acquiring the camera information and true depth map corresponding to a plurality of visible light images, the camera information and true depth map corresponding to each visible light image are input into a neural radiation field, and a pre-constructed loss function is used to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene, and then the volume density and color of each sampling point in the target scene are converted to obtain the reconstructed target scene. Among them, the loss function is constructed based on the absolute depth error loss and the depth edge perception loss. Since the absolute depth error loss is calculated based on the true depth value of each pixel in the true depth map corresponding to each visible light image and the predicted depth value of each pixel in the predicted depth map corresponding to each visible light image, the absolute depth error loss can be used to supervise the depth error of each pixel during the training process to improve the light prediction ability of the neural radiation field and improve the restoration degree of the reconstructed target scene. The depth edge perception loss is calculated based on the real depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator. The Sobel operator is an edge detection operator. Therefore, the depth edge perception loss can optimize the performance of the reconstructed target scene in the edge details, making the reconstructed target scene more accurate in details.

[0047] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a transparent surface back scene reconstruction method based on DES-NeRF provided by an embodiment of the present invention;

[0049] Figure 2 is a comparison diagram of the effects of reconstructing a target scene by an existing reconstruction method and a transparent surface back scene reconstruction method provided by an embodiment of the present invention;

[0050] Figure 3 is a structural schematic diagram of a transparent surface back scene reconstruction device based on DES-NeRF provided by an embodiment of the present invention;

[0051] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0053] In order to solve the problems of low reconstruction restoration and fineness in the existing transparent surface back scene reconstruction method, the embodiment of the present invention provides a transparent surface back scene reconstruction method based on DES-NeRF, see Figure 1 , Figure 1 : is a flow chart of a transparent surface back scene reconstruction method based on DES-NeRF provided by an embodiment of the present invention, which specifically includes the following steps:

[0054] Step S101, obtaining camera information and real depth maps corresponding to a plurality of visible light images; the plurality of visible light images are images captured by a camera of a target scene at multiple viewing angles.

[0055] In the embodiment of the present invention, the multiple visible light images are multiple images of the target scene taken by the camera at multiple viewing angles, specifically, multiple visible light images of the target scene to be reconstructed taken by the camera at different positions and different viewing angles.

[0056] By restoring multiple visible light images, the camera information when the camera took the visible light image and the three-dimensional point cloud of the target scene can be obtained from the multiple visible light images. The camera information may include the camera's rotation matrix, the camera's translation vector, the camera's focal length, distortion parameters, principal point coordinates, etc.

[0057] In the embodiment of the present invention, the multiple visible light images can be restored by using Structure-from-Motion (SfM), which is a technology that can restore camera parameters and scene 3D point clouds from multiple images.

[0058] In an embodiment of the present invention, each pixel in a visible light image can be indexed to a three-dimensional point in a three-dimensional point cloud through coordinates. For example, a pixel in visible light image 1 and a pixel in visible light image 2 can be indexed to the same three-dimensional point in a three-dimensional point cloud through coordinates. That is to say, a three-dimensional point in a three-dimensional point cloud corresponds to a pixel in multiple visible light images. Therefore, the depth information of each visible light image can be calculated based on multiple three-dimensional points in the three-dimensional point cloud and camera information corresponding to each visible light image. Among them, the depth information of each visible light image includes a depth value corresponding to each pixel in each visible light image. The real depth value corresponding to each pixel in all pixels can be obtained by weighted summing and calculating the average value of the depth value corresponding to each pixel in each visible light image, and then for each visible light image, the real depth value corresponding to each pixel of the visible light image is arranged according to the pixel order of the visible light image to obtain the real depth map corresponding to the visible light image.

[0059] Step S102, input the camera information and the true depth map corresponding to each visible light image into the neural radiation field, and use the pre-constructed loss function to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number of times, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the loss function is constructed based on the absolute depth error loss and the depth edge perception loss; the absolute depth error loss is calculated based on the true depth value of each pixel point in the true depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the true depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene.

[0060] Neural Radiance Field (NeRF) is a neural network that can reconstruct 3D scenes from 2D images. The input of NeRF is visible light images from multiple perspectives, camera information corresponding to each visible light image, and real depth map. The output is a continuous 3D radiation field, which can be understood as the color and volume density of each point in the 3D target scene. The volume density can be understood as transparency.

[0061] In the embodiment of the present invention, the neural radiation field is specifically an anti-aliasing neural radiation field.

[0062] In order to further improve the restoration and accuracy of the color and volume density of each sampling point in the target scene output by the neural radiation field, it is proposed in an embodiment of the present invention to use a pre-constructed loss function to supervise the training of the neural radiation field, so that the neural radiation field can update its own network parameters in the direction of minimizing the total loss through training iterations until the number of updates reaches a preset number, and then the trained neural radiation field is used to output the volume density and color of each sampling point in the target scene.

[0063] In an embodiment of the present invention, the pre-constructed loss function is constructed based on the absolute depth error loss and the depth edge perception loss. The absolute depth error loss is calculated based on the true depth value of each pixel in the true depth map corresponding to each visible light image and the predicted depth value of each pixel in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the true depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image.

[0064] In an embodiment of the present invention, the predicted depth map corresponding to each visible light image is obtained by predicting the light for each sampling point in the target scene by the neural radiation field. Specifically, in the neural radiation field rendering process, the measured distance after the camera emits light to the position of each sampling point represents the depth predicted by the light. A sampling point corresponds to a pixel point in the two-dimensional image. By emitting light to each sampling point of the target scene with a fixed camera, the predicted depth value corresponding to each pixel point is obtained, and then the predicted depth map is obtained. Since the input of the neural radiation field contains the camera information corresponding to each visible light image, the predicted depth map corresponding to each visible light image can be obtained based on the camera information corresponding to each visible light image.

[0065] In an embodiment of the present invention, the number of pixels in the visible light image, the real depth map corresponding to each visible light image, and the predicted depth map corresponding to each visible light image are consistent and the position of the same pixel is the same. Therefore, for the real depth map and the predicted depth map corresponding to each visible light image, the absolute depth error loss corresponding to the pixel in each visible light image can be obtained by the real depth value and the predicted depth value corresponding to the pixel at the same coordinate point. According to the real depth map and the predicted depth map corresponding to each visible light image, the depth edge perception loss corresponding to each visible light image can be obtained.

[0066] Step S103: converting the volume density and color of each sampling point in the target scene to obtain a reconstructed target scene.

[0067] In the embodiment of the present invention, after obtaining the volume density and color of each sampling point in the target scene, the volume density and color information of each sampling point in the target scene are visualized to obtain a reconstructed target scene.

[0068] In an embodiment of the present invention, after obtaining the camera information and the real depth map corresponding to a plurality of visible light images, the camera information and the real depth map corresponding to each visible light image are input into the neural radiation field, and the pre-constructed loss function is used to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene, and then the volume density and color of each sampling point in the target scene are converted to obtain the reconstructed target scene. Among them, the pre-constructed loss function is constructed based on the absolute depth error loss and the depth edge perception loss. Since the absolute depth error loss is calculated based on the real depth value of each pixel in the real depth map corresponding to each visible light image and the predicted depth value of each pixel in the predicted depth map corresponding to each visible light image, the absolute depth error loss can be used to supervise the depth error of each pixel during the training process to improve the light prediction ability of the neural radiation field and improve the restoration degree of the reconstructed target scene. The depth edge perception loss is calculated based on the real depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator. The Sobel operator is an edge detection operator. Therefore, the depth edge perception loss can optimize the performance of the reconstructed target scene in the edge details, making the reconstructed target scene more accurate in details.

[0069] In one implementation, camera information and true depth maps corresponding to multiple visible light images are obtained, including: obtaining a visible light image sequence of a target scene; the visible light image sequence includes multiple visible light images of the target scene taken by a camera at multiple viewing angles; processing the visible light image sequence using a motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image; and calculating the true depth map based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

[0070] In an embodiment of the present invention, a visible light image sequence of a target scene is obtained. exist Including I1, I2, I3...I N . Where N is the total number of visible light images in the visible light image sequence. I1, I2, I3…I N It is the visible light image obtained by the camera at different positions and different viewing angles.

[0071] In an embodiment of the present invention, COLMAP can be used to implement the step of processing a visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image. COLMAP is a general SfM and MVS (Multi-View Stereo) pipeline, which is commonly used for three-dimensional reconstruction.

[0072] Specifically, After inputting COLMAP, COLMAP performs feature extraction, feature matching, and sparse reconstruction on the input visible light image and then outputs the required information, which includes the 3D point cloud of the target scene and the camera information corresponding to each visible light image. The 3D point cloud is obtained based on the 3D points corresponding to the pixels in each visible light image in the visible light image sequence. The camera information can include the camera's rotation matrix, the camera's translation vector, and the camera's extrinsic matrix.

[0073] Visible light image I i For example, the information output by COLMAP is as follows:

[0074] (a) Visible light image I i The corresponding camera rotation matrix is ​​denoted as R' i , R' i It is represented in quaternion form and has a size of 3×3.

[0075] (b) Visible light image I i The corresponding camera translation vector is denoted as t' i , size is 3×1.

[0076] (c) Visible light image I i The corresponding camera external parameter matrix is ​​denoted as K i , including the camera's focal length, distortion parameters, and principal point coordinates, with a size of 3×3.

[0077] (d) Visible light image I i The corresponding two-dimensional point, that is, the set of pixel points is recorded as The 3D points in the 3D point cloud correspond to the pixels in the visible light image one by one, p2D ik The corresponding three-dimensional point number can be recorded as S j Among them, p2D ik Represents the i-th visible light image, i.e., I i The kth pixel in p2D; where k = 1, 2, 3...L, L represents the total number of pixels in the i-th visible light image. ik By S j Can be indexed to its corresponding 3D point p3D j ;p2D ik The homogeneous coordinates in the image coordinate system are expressed as (u ik ,v ik ,1), its Euclidean coordinate is expressed as (u ik ,v ik ). Among them, u ik represents the horizontal coordinate of the kth pixel in the i-th visible light image, v ikRepresents the vertical coordinate of the k-th pixel in the i-th visible light image.

[0078] (e) Visible light image sequence The set of all observed 3D points, that is, the set of all 3D points in the 3D point cloud, is denoted as Its serial number is S j ,and Wherein, j=1,2,3...M, M represents the total number of 3D points in the 3D point cloud. The expression in the world coordinate system is Its homogeneous coordinates are expressed as Its Euclidean coordinates are expressed as in, Represents the horizontal coordinate of the j-th 3D point in the world coordinate system; Represents the ordinate of the j-th 3D point in the world coordinate system; Represents the vertical coordinate of the j-th 3D point in the world coordinate system.

[0079] In the embodiment of the present invention, the process of calculating the real depth map according to the camera information corresponding to the three-dimensional point cloud and each visible light image is as follows:

[0080] First, the original depth corresponding to each pixel in each visible light image can be calculated through the Euclidean coordinates of the three-dimensional point in the world coordinate system, the rotation matrix and translation vector of the camera.

[0081] Specifically, pixel p2D ik The original depth is:

[0082]

[0083] Among them, depth ik,ini Represents pixel p2D ik The original depth of represents the z component of the rotation matrix of the camera corresponding to the i-th visible light image; T represents the transpose of the matrix; t i Represents the translation vector of the camera corresponding to the i-th visible light image; Represents pixel p2D ik The corresponding 3D point p3D j Euclidean coordinates in the world coordinate system.

[0084] Calculate the near plane corresponding to the i-th visible light image i and far plane far i :

[0085]

[0086] Far planei It can be understood as the plane parallel to the i-th visible light image where the farthest 3D point in the 3D point cloud that can be observed from the camera's view cone corresponding to the i-th visible light image is located. i It can be understood that the plane parallel to the i-th visible light image is where the nearest three-dimensional point in the three-dimensional point cloud that can be observed from the viewing cone of the camera corresponding to the i-th visible light image is located.

[0087] According to the near plane i and far plane far i Calculate the depth boundary bds raw :

[0088]

[0089] According to bds raw Calculate the scale factor sc:

[0090]

[0091] Among them, factor is a custom scaling factor, which is used to scale the image to constrain the depth within a reasonable range.

[0092] According to bds raw and sc to calculate the near plane t of the cutoff ray n and the far plane cutoff t f :

[0093] t n =min(bds raw )*sc n *sc;

[0094] t f =max(bds raw )*sc f *sc;

[0095] Among them, sc n is the near plane scale factor, sc f is the far plane scale factor, usually sc n =0.9, sc f =1.0. This step belongs to the normalization operation.

[0096] Calculate the depth value depth of the kth pixel in the i-th visible light image ik for:

[0097] depth ik =depth ik,ini *sc;

[0098] For a two-dimensional point p2D ikThe corresponding 3D point p3D j , use homogeneous coordinates to transform the 3D point from the world coordinate system to the camera coordinate system:

[0099]

[0100] in, is a three-dimensional point p3D j Coordinates in the world coordinate system is a three-dimensional point p3D j Coordinates in the camera coordinate system

[0101] Convert the 3D point from the camera coordinate system to the image coordinate system and get the predicted value:

[0102]

[0103] Among them, K i is the camera intrinsic parameter matrix, is a three-dimensional point p3D j Coordinates in the camera coordinate system is the 3D point p3D observed by the camera corresponding to the i-th visible light image j The corresponding two-dimensional point p2D ik The predicted value of the homogeneous coordinates of

[0104] Then for I i The reprojection error e(p2D ik )for:

[0105]

[0106] in, for The Euclidean coordinates, (u ik ,v ik ) is p2D ik The Euclidean coordinates of .

[0107] Average projection error for:

[0108]

[0109] Where N is the total number of visible light images, e(p2D ik ) is I i The reprojection error of the k-th pixel in .

[0110] Then the two-dimensional point p2D ik The weight of the calculated depth kfor:

[0111]

[0112] in, is the average projection error.

[0113] Assume a two-dimensional point set There are H elements in total, corresponding to the same three-dimensional feature point p3D j , it can be understood that in H visible light images, there is a pixel in each visible light image that can be indexed to a three-dimensional point p through coordinates, that is, there are H pixels that can correspond to the three-dimensional point p, and each of the H pixels has a corresponding depth value and weight. The weights of these two-dimensional points, i.e., pixels, are The depth value is Unify the depth of these pixels into depth ij , that is, the depth values ​​corresponding to each pixel in multiple visible light images are weighted and summed and the average value is calculated to obtain the true depth value corresponding to each pixel:

[0114]

[0115] Among them, depth ij Represents the true depth value of each pixel. Through the two-dimensional pixel index of the i-th visible light image, the true depth value of all pixels in the i-th visible light image can be obtained.

[0116]

[0117] Arrange the true depth values ​​of all pixels in pixel order to obtain the true depth map corresponding to the i-th visible light image.

[0118] In the embodiment of the present invention, the two-dimensional point After scaling, the corresponding coordinates coord are:

[0119]

[0120] Among them, factor is the custom scaling factor, (u ik ,v ik ) is p2D ik The Euclidean coordinates of .

[0121] In one implementation, the volume density and color of each sampling point in the target scene are converted to obtain a reconstructed target scene, including:

[0122] The volume density and color of each sampling point in the target scene are transformed using a truncated distance signed function to obtain the reconstructed target scene.

[0123] In the embodiment of the present invention, after obtaining the volume density and color of each sampling point in the target scene, it is necessary to perform visualization processing on the information to obtain an observable three-dimensional model, that is, a reconstructed target scene.

[0124] Specifically, the truncated signed distance function can be used to visualize the reconstructed target scene. The truncated signed distance function (TSDF) is a data structure that represents the surface of an object. It is used for 3D reconstruction and surface extraction. The distance value is stored in a voxel grid. The truncated signed distance function can generate an accurate 3D scene representation based on the volume density and color of each sampling point in the target scene.

[0125] In one implementation, during antialiased neural radiance field rendering, the camera emits rays Arriving at the sampling point position in the target scene, where o is the origin of the light, and after the camera translation vector is scaled, o = t i / factor is calculated; is the direction of light, that is, the z component of the camera rotation matrix; To measure the distance, it represents the depth predicted by the light, and is indexed to the depth value of the two-dimensional point through coord, that is, the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image.

[0126] In this embodiment of the present invention, the loss function is:

[0127] L=αL rec +βL abs +γL edge ;

[0128] Where, L represents the total loss; L rec Indicates that L abs and L edge Other losses other than abs represents the absolute depth error loss; L edge represents the depth edge perception loss, and α, β, and γ are all preset weights.

[0129] In the embodiment of the present invention, other losses refer to other losses commonly used in neural radiation fields, such as color loss, etc. α, β, and γ are all preset custom weights, and the weights usually need to be reasonably set to achieve the best reconstruction effect.

[0130] In one implementation, the absolute depth error loss is calculated as:

[0131]

[0132] in, Represents the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image; Represents the true depth value of each pixel in the true depth map corresponding to the i-th visible light image. By performing L2 regularization on it, the absolute depth error loss is obtained to supervise the predicted depth value in the reconstruction process.

[0133] In one implementation, the depth edge-aware loss is calculated as:

[0134]

[0135] Where N represents the total number of visible light images; represents the predicted depth map corresponding to the i-th visible light image; Represents the true depth map corresponding to the i-th visible light image.

[0136] G(·) is a Sobel operator, specifically an edge detection operator that can detect the position and direction of edges in an image. G(·) is divided into two components: horizontal and vertical:

[0137]

[0138] Among them, G x Represents the horizontal component; G y Represents the vertical component.

[0139] Based on this, and The calculations are:

[0140]

[0141] In an embodiment of the present invention, the depth edge perception loss constructed using an edge detection operator, a true depth map, and a predicted depth map corresponding to each visible light image can perform higher-weighted supervision on edge areas during the training process, so that the reconstructed target scene is more accurate in details.

[0142] See also Figure 2 , Figure 2 : is a comparison diagram of the effects of reconstructing the target scene by the existing reconstruction method and the transparent surface back scene reconstruction method provided by the embodiment of the present invention. Figure 2The target scene is shown above, the left side below is the effect diagram of the target scene reconstructed by the existing reconstruction method, and the right side is the effect diagram of the target scene reconstructed by the DES-NeRF-based transparent surface scene reconstruction method provided by the embodiment of the present invention. By comparison, it can be seen that the DES-NeRF-based transparent surface scene reconstruction method provided by the embodiment of the present invention is more suitable for reconstructing scenes that pass through the projected part of the transparent surface compared to other neural radiation field derivative algorithms. Compared with the methods of distinguishing the reflected part such as NeRFReN, NeRS, and NeRRF, the depth value used in the embodiment of the present invention is calculated based on SfM, so it is easier to obtain and has higher physical authenticity, which solves the problem of blurring of the neural radiation field when reconstructing scenes with transparent surfaces from the root. In addition, the DES-NeRF-based transparent surface scene reconstruction method provided by the embodiment of the present invention constrains the surface depth of the network in the reconstruction process from the sampling point level, and the calculation result has a higher degree of authenticity and credibility; through edge detection based on the depth map level, the reconstructed target scene has a higher weight in the edge area, which improves the performance of the reconstructed target scene in the scene details.

[0143] Based on the same inventive concept, the embodiment of the present invention also provides a transparent surface scene reconstruction device based on DES-NeRF, see Figure 3 , Figure 3 : is a structural schematic diagram of a transparent surface scene reconstruction device based on DES-NeRF provided by an embodiment of the present invention, the transparent surface scene reconstruction device comprises:

[0144] The acquisition module 301 is used to acquire camera information and a real depth map corresponding to a plurality of visible light images; the plurality of visible light images are a plurality of images taken by a camera of a target scene at multiple viewing angles;

[0145] A neural radiation field training module 302 is used to input the camera information and the true depth map corresponding to each visible light image into the neural radiation field, and use a pre-constructed loss function to supervise the neural radiation field training. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number of times, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the pre-constructed loss function is constructed based on the absolute depth error loss and the depth edge perception loss; the absolute depth error loss is calculated based on the true depth value of each pixel point in the true depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the true depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene;

[0146] The target scene reconstruction module 303 is used to transform the volume density and color of each sampling point in the target scene to obtain a reconstructed target scene.

[0147] In an embodiment of the present invention, after obtaining the camera information and the real depth map corresponding to a plurality of visible light images, the camera information and the real depth map corresponding to each visible light image are input into the neural radiation field, and the pre-constructed loss function is used to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene, and then the volume density and color of each sampling point in the target scene are converted to obtain the reconstructed target scene. Among them, the pre-constructed loss function is constructed based on the absolute depth error loss and the depth edge perception loss. Since the absolute depth error loss is calculated based on the real depth value of each pixel in the real depth map corresponding to each visible light image and the predicted depth value of each pixel in the predicted depth map corresponding to each visible light image, the absolute depth error loss can be used to supervise the depth error of each pixel during the training process to improve the light prediction ability of the neural radiation field and improve the restoration degree of the reconstructed target scene. The depth edge perception loss is calculated based on the real depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator. The Sobel operator is an edge detection operator. Therefore, the depth edge perception loss can optimize the performance of the reconstructed target scene in the edge details, making the reconstructed target scene more accurate in details.

[0148] Optionally, the acquisition module is specifically used to:

[0149] Acquire a visible light image sequence of a target scene; the visible light image sequence includes multiple visible light images of the target scene taken by a camera at multiple viewing angles; process the visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image; and calculate a true depth map based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

[0150] Optionally, the target scene reconstruction module is specifically used to:

[0151] The volume density and color of each sampling point in the target scene are converted using a truncated distance signed function to obtain a reconstructed target scene.

[0152] Optionally, the loss function is:

[0153] L=αL rec +βL abs +γL edge ;

[0154] Where, L represents the total loss; L rec Indicates that L abs and L edge Other losses other than abs represents the absolute depth error loss; L edge represents the depth edge perception loss; α, β, and γ are all preset weights.

[0155] Optionally, the absolute depth error loss is calculated as follows:

[0156]

[0157] in, Represents the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image; Represents the true depth value of each pixel in the true depth map corresponding to the i-th visible light image.

[0158] Optionally, the depth edge perception loss is calculated as follows:

[0159]

[0160] Where N represents the total number of visible light images; represents the predicted depth map corresponding to the i-th visible light image; represents the true depth map corresponding to the i-th visible light image; G(·) is the Sobel operator.

[0161] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0162] Memory 403, used for storing computer programs;

[0163] The processor 401 is used to implement the method steps described in any of the above-mentioned DES-NeRF-based transparent surface back scene reconstruction methods when executing the program stored in the memory 403.

[0164] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not represent that there is only one bus or one type of bus.

[0165] The communication interface is used for communication between the above electronic device and other devices.

[0166] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] The present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the method steps described in any of the above-mentioned DES-NeRF-based transparent surface back scene reconstruction methods are implemented.

[0169] Optionally, the computer-readable storage medium may be a non-volatile memory (NVM), such as at least one disk memory.

[0170] Optionally, the computer-readable memory may also be at least one storage device located away from the aforementioned processor.

[0171] In another embodiment of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method steps described in any of the above-mentioned DES-NeRF-based transparent surface post-scene reconstruction methods.

[0172] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0173] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0174] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0175] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0176] As for the device / electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] It should be noted that the device, electronic device and storage medium of the embodiments of the present invention are respectively the device, electronic device and storage medium that apply the above-mentioned method for reconstructing a transparent surface scene based on DES-NeRF. All embodiments of the above-mentioned method for reconstructing a transparent surface scene based on DES-NeRF are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0178] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A transparent surface scene reconstruction method based on DES-NeRF, characterized in that: The transparent surface back scene reconstruction method comprises: Obtaining camera information and real depth maps corresponding to a plurality of visible light images; the plurality of visible light images are a plurality of images taken by a camera of a target scene at multiple viewing angles; The camera information and the real depth map corresponding to each visible light image are input into the neural radiation field, and the pre-constructed loss function is used to supervise the training of the neural radiation field. After the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the loss function is constructed based on the absolute depth error loss and the depth edge perception loss; the absolute depth error loss is calculated based on the real depth value of each pixel point in the real depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the real depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene; The volume density and color of each sampling point in the target scene are converted to obtain a reconstructed target scene.

2. The method for reconstructing a scene behind a transparent surface according to claim 1, characterized in that: Get the camera information and real depth map corresponding to multiple visible light images, including: Acquire a visible light image sequence of a target scene; the visible light image sequence includes a plurality of visible light images of the target scene taken by a camera at multiple viewing angles; Processing the visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image; A true depth map is calculated based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

3. The method for reconstructing a scene behind a transparent surface according to claim 1, characterized in that: The volume density and color of each sampling point in the target scene are converted to obtain a reconstructed target scene, including: The volume density and color of each sampling point in the target scene are converted using a truncated distance signed function to obtain a reconstructed target scene.

4. The method for reconstructing a scene behind a transparent surface according to claim 1, characterized in that: The loss function is: L=αL rec +βL abs +γL edge ; Where, L represents the total loss; L rec Indicates that L abs and L edge Other losses other than abs represents the absolute depth error loss; L edge represents the depth edge perception loss; α, β, and γ are all preset weights.

5. The method for reconstructing scenes behind transparent surfaces according to claim 4, characterized in that: The absolute depth error loss is calculated as: in, Represents the predicted depth value of each pixel in the predicted depth map corresponding to the i-th visible light image; Represents the true depth value of each pixel in the true depth map corresponding to the i-th visible light image.

6. The method for reconstructing a scene behind a transparent surface according to claim 4, characterized in that: The depth edge perception loss is calculated as: Where N represents the total number of visible light images; represents the predicted depth map corresponding to the i-th visible light image; represents the true depth map corresponding to the i-th visible light image; G(·) is the Sobel operator.

7. A transparent surface scene reconstruction device based on DES-NeRF, characterized in that: The transparent surface back scene reconstruction device comprises: An acquisition module, used to acquire camera information and real depth maps corresponding to multiple visible light images; the multiple visible light images are multiple images taken by the camera of the target scene at multiple viewing angles; A neural radiation field training module, used to input the camera information and true depth map corresponding to each visible light image into the neural radiation field, and use a pre-constructed loss function to supervise the neural radiation field training, after the number of iterative updates of the network parameters of the neural radiation field reaches a preset number, the neural radiation field outputs the volume density and color of each sampling point in the target scene; wherein the loss function is constructed based on absolute depth error loss and depth edge perception loss; the absolute depth error loss is calculated based on the true depth value of each pixel point in the true depth map corresponding to each visible light image and the predicted depth value of each pixel point in the predicted depth map corresponding to each visible light image; the depth edge perception loss is calculated based on the true depth map corresponding to each visible light image and the predicted depth map corresponding to each visible light image using the Sobel operator; the predicted depth map corresponding to each visible light image is obtained by the neural radiation field performing light prediction on each sampling point in the target scene; The target scene reconstruction module is used to transform the volume density and color of each sampling point in the target scene to obtain a reconstructed target scene.

8. The transparent surface back scene reconstruction device according to claim 7, characterized in that: The acquisition module is specifically used for: Acquire a visible light image sequence of a target scene; the visible light image sequence includes multiple visible light images of the target scene taken by a camera at multiple viewing angles; process the visible light image sequence using motion recovery structure to generate a three-dimensional point cloud of the target scene and camera information corresponding to each visible light image; and calculate a true depth map based on the three-dimensional point cloud and the camera information corresponding to each visible light image.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the transparent surface back scene reconstruction method described in any one of claims 1-6 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for reconstructing a scene behind a transparent surface according to any one of claims 1 to 6 is implemented.

Images