Method, device and medium for three-dimensional reconstruction of reflection scenes based on neural radiation fields

Abstract

The present invention discloses a method, device and medium for three-dimensional reconstruction of a reflective scene based on a neural radiation field, which belongs to the field of image processing. The method comprises: step 1, collecting a scene RGB image and a depth map; step 2, constructing a back-end map mapping: encoding the spatial coordinates and the direction of light, mapping them into high-frequency features, inputting the trained geometry, high light, and low light color decoders to obtain an implicit representation of the scene; sampling light through perspective and surface perception, accumulating the predicted values ​​of the intersection of the sampled light and any voxel grid to render the scene depth and color; step 3, maintaining front-end camera tracking: fixing the voxel grid to store feature embedding and various decoder parameters, iteratively optimizing the depth camera posture; step 4, randomly selecting pixels from a key frame pixel database to perform global bundle adjustment, jointly optimizing the depth camera posture and the scene implicit representation to perform dense map mapping and camera tracking, and realizing scene three-dimensional reconstruction. The method can reconstruct a reflective scene in real time with high precision.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and medium for three-dimensional reconstruction of a reflection scene based on a neural radiation field. Background Art

[0002] Traditional simultaneous localization and mapping (SLAM) methods use optical flow or feature point methods combined with nonlinear optimization to solve the camera's pose and use an explicit display scene representation to update the global map. Most optical flow-based methods rely on a key assumption, namely, illumination invariance and photometric consistency. This means that the brightness and color of objects in the scene should be consistent at different viewing angles. These assumptions are often not true in real life, especially outdoors and in some special application scenarios, such as chemical laboratories. In chemical laboratory environments, highly reflective materials commonly used in laboratories, such as metal utensils, metal surface instruments, glassware, etc., will cause reflection and refraction of light, increasing the complexity of visual processing. In addition, for feature point methods, the feature points and descriptors generated by the reflective surface at different viewing angles may be different, because the reflective surface reflects other light sources or objects in the environment, forming a virtual image that does not belong to the real structure of the object in the reflective scene. These extracted false feature points and descriptors will increase the probability of false matching.

[0003] In view of this, the present invention is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device and medium for three-dimensional reconstruction of reflective scenes based on neural radiation fields, which explains the perspective correlation by introducing neural radiation fields in simultaneous positioning and map construction, and decomposes the highlight and low-light parts in the scene in combination with color level representation, thereby realizing real-time three-dimensional reconstruction in reflective scenes, and well solving the above-mentioned technical problems existing in the prior art.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for three-dimensional reconstruction of a reflection scene based on a neural radiation field, comprising:

[0007] Step 1, data acquisition: Use a depth camera to collect RGB images and depth maps of the reflection scene to be reconstructed from various angles;

[0008] Step 2, using the neural radiation field to build the back-end map mapping: encode the spatial coordinates and light direction of the three-dimensional space corresponding to the reconstructed reflection scene, map the low-frequency signal to high-frequency features, use the obtained high-frequency features as input, and obtain the implicit representation of the reconstructed reflection scene through the trained geometric decoder, highlight color decoder and low-light color decoder. The implicit representation of the reconstructed reflection scene includes: the color of the highlight part , low light part color , signed distance function value SDF and fixed tone mapping coefficients , use the highlight color Color with low light The linear combination of is used as the color representation of the reflected scene to be reconstructed;

[0009] Combining perspective perception sampling and surface perception sampling to perform light sampling from coarse to fine, the depth and color of the reflection scene to be reconstructed are rendered by accumulating the predicted values ​​of the intersection points of the sampled light and any voxel grid in the voxel grid divided into three-dimensional space corresponding to the reflection scene to be reconstructed;

[0010] Step 3, maintain the front-end camera tracking: fix the feature embedding stored in the voxel grid of the three-dimensional space divided corresponding to the reconstructed reflection scene and the parameters of the trained geometry decoder, the parameters of the highlight color decoder, and the parameters of the low-light color decoder, and iteratively optimize the depth camera pose to minimize the error;

[0011] Step 4, global bundle adjustment: the key frame pixels of the RGB image of the reflection scene to be reconstructed collected in step 1 are stored in a key frame pixel database, a predetermined number of pixels are randomly selected from the key frame pixel database for global bundle adjustment, and the depth camera pose and the implicit representation of the reflection scene to be reconstructed are jointly optimized to perform dense map mapping and camera tracking to obtain a high-precision 3D reconstruction result of the reflection scene to be reconstructed.

[0012] A processing device, comprising:

[0013] at least one memory for storing one or more programs;

[0014] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0015] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0016] Compared with the prior art, the method, device and medium for three-dimensional reconstruction of a reflection scene based on a neural radiation field provided by the present invention have the following beneficial effects:

[0017] Through data collection, building back-end map mapping, maintaining front-end camera tracking and global bundle adjustment, and using a multi-decoder framework that considers both perspective correlation and color level representation when building back-end map mapping, it is not only applicable to high-light reflection scenes in the real world, but also has achieved good results on synthetic datasets Replica and real datasets ScanNet. By combining perspective-aware sampling and surface-aware sampling, better utilization of close-range visual cues is achieved, deepening the model's deep understanding of the scene. This method can meet certain real-time requirements and has been successfully verified in a chemical laboratory scene with obvious reflection phenomena. The accuracy of the reconstructed model can meet the robot arm trajectory planning task of the chemical robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a method for three-dimensional reconstruction of a reflection scene based on a neural radiation field provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0021] First, the terms that may be used in this article are explained as follows:

[0022] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0023] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0024] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0025] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0026] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0027] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a method for three-dimensional reconstruction of a reflection scene based on a neural radiation field, comprising:

[0029] Step 1, data acquisition: Use a depth camera to collect RGB images and depth maps of the reflection scene to be reconstructed from various angles;

[0030] Step 2, using the neural radiation field to build the back-end map mapping: encode the spatial coordinates and light direction of the three-dimensional space corresponding to the reconstructed reflection scene, map the low-frequency signal to high-frequency features, use the obtained high-frequency features as input, and obtain the implicit representation of the reconstructed reflection scene through the trained geometric decoder, highlight color decoder and low-light color decoder. The implicit representation of the reconstructed reflection scene includes: the color of the highlight part , low light part color , signed distance function value SDF and fixed tone mapping coefficients , use the highlight color Color with low light The linear combination of is used as the color representation of the reflected scene to be reconstructed;

[0031] Combining perspective perception sampling and surface perception sampling to perform light sampling from coarse to fine, the depth and color of the reflection scene to be reconstructed are rendered by accumulating the predicted values ​​of the intersection points of the sampled light and any voxel grid in the voxel grid divided into three-dimensional space corresponding to the reflection scene to be reconstructed;

[0032] Step 3, maintain the front-end camera tracking: fix the feature embedding stored in the voxel grid of the three-dimensional space divided corresponding to the reconstructed reflection scene and the parameters of the trained geometry decoder, the parameters of the highlight color decoder, and the parameters of the low-light color decoder, and iteratively optimize the depth camera pose to minimize the error;

[0033] Step 4, global bundle adjustment: the key frame pixels of the RGB image of the reflection scene to be reconstructed collected in step 1 are stored in a key frame pixel database, a predetermined number of pixels are randomly selected from the key frame pixel database for global bundle adjustment, and the depth camera pose and the implicit representation of the reflection scene to be reconstructed are jointly optimized to perform dense map mapping and camera tracking to obtain a high-precision 3D reconstruction result of the reflection scene to be reconstructed.

[0034] Preferably, in step 2 of the above method, the back-end map mapping is constructed using the neural radiation field in the following manner, including:

[0035] Step 21, multi-resolution hash coding is performed on the spatial coordinates of the three-dimensional space corresponding to the reconstructed reflection scene, the three-dimensional space is divided into L voxel grids of different resolution levels, and the feature vector after multi-resolution hash coding is stored in a hash table with the voxel grid vertex as the index. , find the voxel grid where the input coordinates are located, and obtain the feature vectors of each resolution level through trilinear interpolation , for the feature vectors at each resolution level Splice to get multi-resolution hash code vector ;

[0036] Using a set of spherical harmonics To the direction of light Encoding to obtain spherical harmonics encoding , are two angles representing the direction of light in the common spherical coordinate system, It is the azimuth of the light, usually measured on the xy plane, which represents the angle between the projection of the light on the xy plane and the positive direction of the x-axis. Its value range is [0,2π]. is the polar angle of the light, which represents the angle between the light and the positive direction of the z-axis, and its value is [0,π];

[0037] The resulting multi-resolution hash code vector and spherical harmonics encoding As a high frequency feature;

[0038] Step 22, input the high-frequency features obtained in step 21 into the trained geometry decoder, highlight color decoder and low-light color decoder to obtain an implicit representation of the reflection scene to be reconstructed, that is:

[0039] Signed distance function value SDF: ;

[0040] Highlight color: ;

[0041] Low light color: ;

[0042] in, , , They are a geometry decoder, a low-light color decoder, and a highlight color decoder with trainable parameters; is a trilinear interpolation function; is the spatial coordinate of the three-dimensional space corresponding to the reflection scene to be reconstructed A multi-resolution hash code vector of ; is the geometric eigenvector; is the direction of light Spherical harmonics encoding of ; Color decoder for low light The learned fixed tone mapping coefficients are in the range [0,1] ; is the predicted signed distance function value;

[0043] Step 23: Change the highlight color obtained in step 22 With fixed tone mapping coefficients After multiplication and scaling, and with the low light part color and 1 minus the fixed tone mapping coefficient The product of the difference is added to obtain the final color representation of the reflected scene to be reconstructed :

[0044] ;

[0045] Step 24, rendering the depth and color of the reflection scene to be reconstructed by accumulating the predicted value of the signed distance function value and the predicted value of the final color representation of the intersection point between the sampling light and the arbitrary voxel grid, wherein the predicted value of the signed distance function value and the predicted value of the final color representation are obtained by steps 22 and 23;

[0046] Given the intrinsic parameters of the depth camera and the current depth camera pose, calculate the origin of the depth camera With light direction , = , sampling along the direction of the light Sampling points , Indicates the sampling depth of the i-th sampling point, i ranges from 1 to the total number of sampling points N, and the depth of each sampling point is and color prediction values Perform differentiable rendering, the probability of light termination at each sampling point It is expressed as:

[0047] ;

[0048] in, is the truncated signed distance predicted by The weight obtained by conversion; for function; is the preset cutoff distance; The closer to the cutoff distance, the farther away from the surface, the larger the score value, and the weight The smaller;

[0049] Color predictions for rendering and predicted depth value Respectively expressed as:

[0050] ;

[0051] ;

[0052] in, The final color representation predicted for the i-th sampling point ; is the sampling depth of the i-th sampling point; i is the serial number of the sampling point; is the total number of sampling points;

[0053] Step 25, by combining perspective perception sampling and surface perception sampling to obtain sampling points in color and depth rendering, firstly, perspective perception sampling is performed to complete the rough sampling, which is logarithmically increased on the basis of uniform sampling, as follows:

[0054] ;

[0055] in, is the index value of the sampling point; Depth sampled for perspective perception; and They represent the maximum depth and minimum depth of the reflection scene to be reconstructed respectively; is the total number of sampling points;

[0056] Then, the signed distance function (SDF) values ​​of these sampling points are calculated to determine the surface of the object in the reconstructed reflection scene, and the sampling of near-surface points is further increased in the voxel grid containing these surface intersections to complete the surface perception sampling.

[0057] Preferably, in step 3 of the above method, the camera tracking of the front end is maintained in the following manner, including:

[0058] Step 31: Combine the true color value of the RGB image collected in step 1, the true depth value of the depth map, and the color of the highlight part obtained in step 2 , low light part color The loss function in the process of optimizing the depth camera pose is constructed based on the predicted value of the signed distance function value SDF, including: RGB loss, depth loss, free space loss and signed distance function value SDF loss;

[0059] For the low-light color and the high-light color in the scene to be reconstructed, the most suitable order is found through experimental verification, and the spherical harmonic function is directly used to construct the true value of the two levels of color: the low-light color and the high-light color. for:

[0060] C ' c , r = [ Y 0 0 c , r , Y 1 - 1 c , r , Y 1 0 c , r , Y 1 1 c , r , … , Y l l c , r ] ;

[0061] in, is a spherical harmonic function, for [ - l , l ] integer in range; The color of the low light part or the color of the high light part; is the light direction;

[0062] The low light part color and highlight color The corresponding color levels are set to 2 and 4, respectively, for the low-light part color and highlight color Do The error obtained for each level of color loss is:

[0063] Final color loss: ;

[0064] Partial color loss in low light: ;

[0065] Color loss in highlights: ;

[0066] Where N is the total number of sampling points; n represents the index of the two-dimensional pixel on the two-dimensional image plane corresponding to the RGB image; is the true color value of the RGB image collected by the depth camera in step 1; Represents the predicted value of the final color representation obtained by combining the low-light color and the high-light color; , They are respectively the true color value of the low light part and the true color value of the high light part directly constructed by spherical harmonic functions; , They are the predicted values ​​of the color of the low-light part and the predicted values ​​of the color of the high-light part respectively;

[0067] Compare the rendering depth with the observed value Error gets depth loss for:

[0068] L d = 1 R d ∑ r ∈ R d d ˆ r - D [ u , v ] 2 ;

[0069] in, is the set of rays within the range of valid depth measurements; Indicates the direction of light; , is the two-dimensional pixel position on the two-dimensional image plane corresponding to the RGB image; D [ u , v ] is the true depth value corresponding to the two-dimensional pixel position; Represents the depth value along the direction of the light;

[0070] For sampling points far away from the surface of the reflected scene to be reconstructed, that is, The sampling point of the forced signed distance function value SDF is predicted as the truncated distance tr, and the free space loss is obtained. for:

[0071] ;

[0072] in, refers to free space, is the set of points on the ray between the depth camera and the measured surface cutoff distance, called the point set. Point set The number of midpoints, for Points of concentration, for The predicted signed distance function value for this point;

[0073] Finally, the geometry decoder, highlight color decoder, and low-light color decoder learn the accurate surface representation within the surface truncation region to obtain the signed distance function value SDF loss for:

[0074] L sdf = 1 R d ∑ r ∈ R d 1 S r tr ∑ q ∈ S r tr s q - ( D [ u , v ] - d i ) 2 ;

[0075] in, The set of points within the cutoff distance is called the point set. Point set The number of midpoints, for Points of concentration, for The predicted signed distance function value for this point; Indicates the sampling depth of the i-th sampling point, where i ranges from 1 to the total number of sampling points N;

[0076] for The distance between the sampling point and the observed depth value of the sampling point is used as an approximation of the true signed distance function value SDF value for supervision;

[0077] Final total loss for:

[0078] ;

[0079] in, ; ; ; ; ; ;

[0080] Step 32, during the camera tracking process, the voxel grid feature embedding obtained in step 21 and the parameters of the geometry decoder, the parameters of the low-light color decoder, and the parameters of the highlight color decoder obtained in step 22 are fixed, and the depth camera pose is iteratively optimized to minimize the error. Complete the initialization of the depth camera pose for each frame and the constant speed motion model for:

[0081] ;

[0082] in, , , They represent the depth camera pose at the current moment, the depth camera pose at the previous frame moment, and the depth camera pose at the previous two frames moment, respectively. for The inverse matrix of

[0083] Sample a sparse set of RGB images from step 1 The pixels are tracked and the frame pose is updated via back-propagation in each iteration.

[0084] Preferably, in step 4 of the above method, a predetermined number of pixels are randomly selected from the key frame pixel database for global bundle adjustment in the following manner, including:

[0085] Step 41, maintaining a key frame pixel database by selecting key frames at even intervals and storing 10% of the pixels of the key frames as a pixel subset;

[0086] Step 42, the joint optimization of global bundle adjustment is run in an alternating manner, and the initialization stage randomly selects a predetermined number of pixels from the key frame pixel data, and completes the implicit representation of the preliminary reflection scene to be reconstructed by running 100 training iterations to the first frame;

[0087] Step 43, when processing subsequent frames, first optimize the depth camera pose of several consecutive frames, and use the depth camera pose, that is, the light direction, as input to the geometry decoder, the highlight color decoder, and the low-light color decoder. At this stage, the weights and biases in the geometry decoder, the highlight color decoder, and the low-light color decoder are fixed, and the depth camera pose is gradient descended by minimizing the error until the depth camera pose optimization is completed;

[0088] Step 44, after the depth camera pose optimization is completed, the implicit representation of the reflective scene to be reconstructed is optimized. In this stage, the optimized depth camera pose is used to further train the geometry decoder, highlight color decoder and low-light color decoder to make the representation of the reflective scene to be reconstructed more accurate. Through this joint optimization, a high-precision three-dimensional reconstruction result of the reflective scene to be reconstructed is gradually obtained.

[0089] The embodiment of the present invention further provides a processing device, including:

[0090] at least one memory for storing one or more programs;

[0091] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0092] The embodiments of the present invention further provide a readable storage medium storing a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0093] In summary, the three-dimensional reconstruction method of the embodiment of the present invention cooperates with each step of data collection, building back-end map mapping, maintaining front-end camera tracking and global bundle adjustment, and uses a multi-decoder framework that simultaneously considers perspective correlation and color level representation in building back-end map mapping. It is not only applicable to high-light reflection scenes in the real world, but also has achieved good results on synthetic data sets Replica and real data sets ScanNet. By combining perspective perception sampling and surface perception sampling, better utilization of close-range visual clues is achieved, and the model's deep understanding of the scene is deepened. This method can meet certain real-time requirements and has been successfully verified in a chemical laboratory scene with obvious reflection phenomena. The accuracy of the reconstructed model can meet the robot arm trajectory planning task of the chemical robot.

[0094] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0095] Example 1

[0096] like Figure 1 As shown, an embodiment of the present invention provides a method for three-dimensional reconstruction of a reflection scene based on a neural radiation field, which can realize real-time and high-precision three-dimensional reconstruction of a reflection scene, including the following steps:

[0097] Step 1, data acquisition: Use a depth camera to collect RGB images and depth maps of the scene from various angles;

[0098] Step 2, using the neural radiation field to build the backend map mapping, includes the following steps:

[0099] Step 21, multi-resolution hash coding is performed on the spatial coordinates of the three-dimensional space corresponding to the reconstructed reflection scene, the three-dimensional space is divided into L voxel grids of different resolution levels, and the feature vector after multi-resolution hash coding is stored in a hash table with the voxel grid vertex as the index. , find the voxel grid where the input coordinates are located, and obtain the feature vectors of each resolution level by trilinear interpolation of the voxel grid , the feature vectors of each resolution level are concatenated to obtain a multi-resolution hash code vector ;

[0100] Using a set of spherical harmonics Encoding the light direction results in spherical harmonic encoding , are two angles representing the direction of light in the common spherical coordinate system, It is the azimuth of the light, usually measured on the xy plane, which represents the angle between the projection of the light on the xy plane and the positive direction of the x-axis. Its value range is [0,2π]. is the polar angle of the light, which represents the angle between the light and the positive direction of the z-axis, and its value is [0,π];

[0101] The resulting multi-resolution hash code vector and spherical harmonics encoding As a high frequency feature;

[0102] Step 22, input the high-frequency features obtained in step 21 into the trained geometry decoder, highlight color decoder and low-light color decoder to obtain an implicit representation of the reflection scene to be reconstructed:

[0103] Signed distance function value SDF: ;

[0104] Highlight color: ;

[0105] Low light color: ;

[0106] in, , , They are a geometric decoder, a low-light color decoder, and a high-light color decoder with trainable parameters respectively; is a trilinear interpolation function; is the spatial coordinate ; is the spatial coordinate of the three-dimensional space corresponding to the reflection scene to be reconstructed The multi-resolution hash code vector of is the geometric eigenvector; is the direction of light The spherical harmonics encoding of Color decoder for low light The learned fixed tone mapping coefficients are in the range [0,1] ; is the predicted signed distance function value;

[0107] Step 23: Change the highlight color obtained in step 22 After After multiplication and scaling, and with the low light part color and 1 minus the fixed tone mapping coefficient The product of the difference is added to obtain the final color representation of the reflected scene to be reconstructed :

[0108] ;

[0109] Step 24, rendering the depth and color of the reflection scene to be reconstructed by accumulating the predicted value of the signed distance function value and the predicted value of the final color representation of the intersection point between the sampling light and the arbitrary voxel grid, wherein the predicted value of the signed distance function value and the predicted value of the final color representation are obtained by steps 22 and 23;

[0110] Given the intrinsic parameters of the depth camera and the current depth camera pose, the camera origin can be calculated and light direction , = , further sampling along the direction of the light Points , and the depth of each sampling point and color prediction values Perform differentiable rendering, the probability of light termination at each sampling point It can be expressed as:

[0111] ;

[0112] in, is the truncated signed distance predicted by The weight obtained by conversion; for function; is the preset cutoff distance; The closer to the cutoff distance, the farther away from the surface, the larger the score value, and the weight The smaller.

[0113] Color predictions for rendering and predicted depth value It can be expressed as:

[0114] ;

[0115] ;

[0116] in, The final color predicted for each point , is the sampling depth of each point;

[0117] Step 25, the sampling points in color and depth rendering are obtained by combining perspective-aware sampling and surface-aware sampling. First, perspective-aware sampling is performed to complete the coarse sampling, which is in the form of logarithmic increase based on uniform sampling:

[0118] ;

[0119] in, is the index value of the sampling point; Depth sampled for perspective perception; and They represent the maximum depth and minimum depth of the reflection scene to be reconstructed respectively; is the total number of sampling points;

[0120] Then, the SDF values ​​of these points are calculated to determine the surface of the object in the reconstructed reflection scene, and the sampling of near-surface points is further increased in the voxel grid containing these surface intersection points to complete the surface perception sampling.

[0121] Step 3, maintaining the front-end camera tracking, includes the following steps:

[0122] Step 31, first combine the color true value of the RGB image collected in step 1 and the depth true value of the depth map and the color of the highlight part obtained in step 2 , low light part color The loss function in the process of optimizing the depth camera pose is constructed based on the predicted value of the signed distance function value SDF, including: RGB loss, depth loss, free space loss and signed distance function value SDF loss;

[0123] For the low-light and high-light colors in the scene to be reconstructed, the most suitable order is found through experimental verification, and the spherical harmonic function is directly used to construct the true value of the two levels of color. for:

[0124] C ' c , r = [ Y 0 0 c , r , Y 1 - 1 c , r , Y 1 0 c , r , Y 1 1 c , r , … , Y l l c , r ] ;

[0125] in, is a spherical harmonic function, for [ - l , l ] integer in range; The color of the low light part or the color of the high light part; is the light direction;

[0126] The low light part color and highlight color The corresponding color orders are set to 2 and 4 respectively, so as to ensure that the points unrelated to the view are as sensitive to the lighting information as possible, while the points related to the view are as sensitive to the lighting information as possible. and highlight color Do The error gets the color loss:

[0127] Final color loss: ;

[0128] Partial color loss in low light: ;

[0129] Color loss in highlights: ;

[0130] Where N is the total number of sampling points; n represents the index of the two-dimensional pixel on the two-dimensional image plane corresponding to the RGB image; is the true color value of the RGB image collected by the depth camera in step 1; Represents the predicted value of the final color representation obtained by combining the low-light color and the high-light color; , They are respectively the true color value of the low light part and the true color value of the high light part directly constructed by spherical harmonic functions; , They are the predicted values ​​of the color of the low-light part and the predicted values ​​of the color of the high-light part respectively;

[0131] Make a comparison between the rendering depth and the observation depth value The error gets the depth loss:

[0132] L d = 1 R d ∑ r ∈ R d d ˆ r - D [ u , v ] 2 ;

[0133] in, is the set of rays within the range of valid depth measurements; Indicates the direction of light; , is the two-dimensional pixel position on the two-dimensional image plane corresponding to the RGB image; D [ u , v ] is the true depth value corresponding to the two-dimensional pixel position; Represents the depth value along the direction of the light;

[0134] For points far away from the surface of the object in the reconstructed reflection scene (i.e. ), forcing the signed distance function value SDF to be predicted as the cutoff distance tr, resulting in free space loss:

[0135] ;

[0136] in, is the set of points on the ray between the depth camera and the measured surface cutoff distance; A collection of points The point in is the predicted signed distance function value;

[0137] Finally, the SDF loss is obtained by making the geometry decoder, highlight color decoder, and low-light color decoder learn accurate surface representation within the surface truncation region:

[0138] L sdf = 1 R d ∑ r ∈ R d 1 S r tr ∑ p ∈ S r tr s p - ( D [ u , v ] - d i ) 2 ;

[0139] in, is the set of points in the truncation region, A collection of points The point in is the predicted signed distance function value; Indicates the sampling depth of the i-th sampling point, where i ranges from 1 to the total number of sampling points N;

[0140] Specifically, for The distance between the sampling point and its observed depth value is used as an approximation of the true SDF value for supervision.

[0141] Final total loss for:

[0142] ;

[0143] in, , , , , , .

[0144] Step 32, during the camera tracking process, the voxel feature embedding obtained in step 21 and the parameters of the geometry decoder, the parameters of the highlight color decoder, and the parameters of the low-light color decoder obtained in step 22 are fixed, and the depth camera pose is iteratively optimized to minimize the error. For the initialization of each frame pose, a constant speed motion model is used. get:

[0145] ;

[0146] in, , , They represent the depth camera pose at the current moment, the depth camera pose at the previous frame moment, and the depth camera pose at the previous two frames moment, respectively. for The inverse matrix of

[0147] Sample a sparse set of RGB images from the input step 1 pixels to track, and the frame pose is updated via back-propagation in each iteration.

[0148] Step 4, global bundle adjustment: store the key frame pixels of the RGB image of the reflection scene to be reconstructed collected in step 1 through a key frame pixel database, randomly select a predetermined number of pixels from the key frame pixel database for global bundle adjustment, jointly optimize the depth camera pose and the implicit representation of the reflection scene to be reconstructed to perform dense map mapping and camera tracking, and obtain a high-precision 3D reconstruction result of the reflection scene to be reconstructed. The specific steps are as follows:

[0149] Step 41, maintaining a key frame pixel database by selecting key frames at even intervals and storing a 10% pixel subset of the key frames;

[0150] Step 42, the joint optimization of the global bundle adjustment is run in an alternating manner. In the initialization phase, a certain number of pixels are randomly selected from the key frame pixel data, and a preliminary representation of the scene is completed by running several training iterations on the first frame;

[0151] Step 43, when processing subsequent frames, first optimize the camera pose of several consecutive frames. The camera pose, i.e., the light direction, is used as input to the geometry decoder, the highlight color decoder, and the low-light color decoder. At this stage, the weights and biases in the geometry decoder, the highlight color decoder, and the low-light color decoder are fixed, and the depth camera pose is gradient descended by minimizing the error;

[0152] Step 44, once the depth camera pose optimization is completed, the implicit representation of the reflection scene to be reconstructed is optimized. In this stage, the optimized depth camera pose is used to further train the geometry decoder, the highlight color decoder and the low-light color decoder, so that the implicit representation of the reflection scene to be reconstructed is more accurate. Through this joint optimization, the accuracy of the 3D reconstruction of the reflection scene can be gradually improved.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0154] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for three-dimensional reconstruction of a reflection scene based on a neural radiation field, characterized in that: include: Step 1, data acquisition: Use a depth camera to collect RGB images and depth maps of the reflection scene to be reconstructed from various angles; Step 2, using the neural radiation field to build the back-end map mapping: encode the spatial coordinates and light direction of the three-dimensional space corresponding to the reconstructed reflection scene, map the low-frequency signal to high-frequency features, use the obtained high-frequency features as input, and obtain the implicit representation of the reconstructed reflection scene through the trained geometric decoder, highlight color decoder and low-light color decoder. The implicit representation of the reconstructed reflection scene includes: the color of the highlight part , low light part color , signed distance function value SDF and fixed tone mapping coefficients , use the highlight color Color with low light The linear combination of is used as the color representation of the reflected scene to be reconstructed; Combining perspective perception sampling and surface perception sampling to perform light sampling from coarse to fine, the depth and color of the reflection scene to be reconstructed are rendered by accumulating the predicted values ​​of the intersection points of the sampled light and any voxel grid in the voxel grid divided into three-dimensional space corresponding to the reflection scene to be reconstructed; Step 3, maintain the front-end camera tracking: fix the feature embedding stored in the voxel grid of the three-dimensional space divided corresponding to the reconstructed reflection scene and the parameters of the trained geometry decoder, the parameters of the highlight color decoder, and the parameters of the low-light color decoder, and iteratively optimize the depth camera pose to minimize the error; Step 4, global bundle adjustment: the key frame pixels of the RGB image of the reflection scene to be reconstructed collected in step 1 are stored in a key frame pixel database, a predetermined number of pixels are randomly selected from the key frame pixel database for global bundle adjustment, and the depth camera pose and the implicit representation of the reflection scene to be reconstructed are jointly optimized to perform dense map mapping and camera tracking to obtain a high-precision 3D reconstruction result of the reflection scene to be reconstructed.

2. The method for three-dimensional reconstruction of a reflection scene based on a neural radiation field according to claim 1, characterized in that: In step 2, the neural radiation field is used to construct a back-end map mapping in the following manner, including: Step 21, multi-resolution hash coding is performed on the spatial coordinates of the three-dimensional space corresponding to the reconstructed reflection scene, the three-dimensional space is divided into L voxel grids of different resolution levels, and the feature vector after multi-resolution hash coding is stored in a hash table with the voxel grid vertex as the index. , find the voxel grid where the input coordinates are located, and obtain the feature vectors of each resolution level through trilinear interpolation And concatenate to get multi-resolution hash code vector ; Using a set of spherical harmonics To the direction of light Encoding to obtain spherical harmonics encoding , are two angles representing the direction of light in the common spherical coordinate system, It is the azimuth of the light, usually measured on the xy plane, which represents the angle between the projection of the light on the xy plane and the positive direction of the x-axis. Its value range is [0,2π]. is the polar angle of the light, which represents the angle between the light and the positive direction of the z-axis, and its value is [0,π]; The resulting multi-resolution hash code vector and spherical harmonics encoding As a high frequency feature; Step 22, input the high-frequency features obtained in step 21 into the trained geometry decoder, highlight color decoder and low-light color decoder to obtain an implicit representation of the reflection scene to be reconstructed, that is: Signed distance function value SDF: ; Highlight color: ; Low light color: ; in, , , They are a geometry decoder, a low-light color decoder, and a highlight color decoder with trainable parameters; is a trilinear interpolation function; is the spatial coordinate of the three-dimensional space corresponding to the reflection scene to be reconstructed A multi-resolution hash code vector of ; is the geometric eigenvector; is the direction of light Spherical harmonics encoding of ; Color decoder for low light The learned fixed tone mapping coefficients are in the range ; is the predicted signed distance function value; Step 23: Change the highlight color obtained in step 22 With fixed tone mapping coefficients After multiplication and scaling, and with the low light part color and 1 minus the fixed tone mapping coefficient The product of the difference is added to obtain the final color representation of the reflected scene to be reconstructed : ； Step 24, rendering the depth and color of the reflection scene to be reconstructed by accumulating the predicted value of the signed distance function value and the predicted value of the final color representation of the intersection point between the sampling light and the arbitrary voxel grid, wherein the predicted value of the signed distance function value and the predicted value of the final color representation are obtained by steps 22 and 23; Given the intrinsic parameters of the depth camera and the current depth camera pose, calculate the origin of the depth camera With light direction , = , sampling along the direction of the light Sampling points , Indicates the sampling depth of the i-th sampling point, i ranges from 1 to the total number of sampling points N, and the depth of each sampling point is and color prediction values Perform differentiable rendering, the probability of light termination at each sampling point It is expressed as: ； in, is the truncated signed distance predicted by The weight obtained by conversion; for function; is the preset cutoff distance; The closer to the cutoff distance, the farther away from the surface, the larger the score value, and the weight The smaller; Color predictions for rendering and predicted depth value Respectively expressed as: ； ； in, The final color representation predicted for the i-th sampling point ; is the sampling depth of the i-th sampling point; i is the serial number of the sampling point; is the total number of sampling points; Step 25, by combining perspective perception sampling and surface perception sampling to obtain sampling points in color and depth rendering, firstly, perspective perception sampling is performed to complete the rough sampling, which is logarithmically increased on the basis of uniform sampling, as follows: ； in, is the index value of the sampling point; Depth sampled for perspective perception; and They represent the maximum depth and minimum depth of the reflection scene to be reconstructed respectively; is the total number of sampling points; Then, the signed distance function (SDF) values ​​of these sampling points are calculated to determine the surface of the object in the reconstructed reflection scene, and the sampling of near-surface points is further increased in the voxel grid containing these surface intersections to complete the surface perception sampling.

3. The method for three-dimensional reconstruction of a reflection scene based on a neural radiation field according to claim 2, characterized in that: In step 3, the camera tracking of the front end is maintained in the following manner, including: Step 31: Combine the true color value of the RGB image collected in step 1, the true depth value of the depth map, and the color of the highlight part obtained in step 2 , low light part color The loss function in the process of optimizing the depth camera pose is constructed based on the predicted value of the signed distance function value SDF, including: RGB loss, depth loss, free space loss and signed distance function value SDF loss; For the low-light color and the high-light color in the scene to be reconstructed, the most suitable order is found through experimental verification, and the spherical harmonic function is directly used to construct the true value of the two levels of color: the low-light color and the high-light color. for: ； in, is a spherical harmonic function, for integer in range; The color of the low light part or the color of the high light part; is the light direction; The low light part color and highlight color The corresponding color levels are set to 2 and 4, respectively, for the low-light part color and highlight color Do The error obtained for each level of color loss is: Final color loss: ; Partial color loss in low light: ; Color loss in highlights: ; Where N is the total number of sampling points; n represents the index of the two-dimensional pixel on the two-dimensional image plane corresponding to the RGB image; is the true color value of the RGB image collected by the depth camera in step 1; Represents the predicted value of the final color representation obtained by combining the low-light color and the high-light color; , They are respectively the true color value of the low light part and the true color value of the high light part directly constructed by spherical harmonic functions; , They are the predicted values ​​of the color of the low-light part and the predicted values ​​of the color of the high-light part respectively; Make a comparison between the rendering depth and the observation depth value Error gets depth loss for: ； in, is the set of rays within the range of valid depth measurements; Indicates the direction of light; , is the two-dimensional pixel position on the two-dimensional image plane corresponding to the RGB image; is the true depth value corresponding to the two-dimensional pixel position; Represents the depth value along the direction of the light; For sampling points far away from the surface of the object in the reflection scene to be reconstructed, that is, The sampling point of the forced signed distance function value SDF is predicted as the truncated distance tr, and the free space loss is obtained. for: ； in, refers to free space, is the set of points on the ray between the depth camera and the measured surface cutoff distance, called the point set. Point set The number of midpoints, for Points of concentration, for The predicted signed distance function value for this point; Finally, the geometry decoder, highlight color decoder, and low-light color decoder learn the accurate surface representation within the surface truncation region to obtain the signed distance function value SDF loss for: ； in, The set of points in the truncation area is called the point set; Point set The number of midpoints, for Points of concentration, for The predicted signed distance function value for this point; Indicates the sampling depth of the i-th sampling point, where i ranges from 1 to the total number of sampling points N; for The distance between the sampling point and the observed depth value of the sampling point is used as an approximation of the true signed distance function value SDF value for supervision; Final total loss for: ； in, ; ; ; ; ; ; Step 32, during the camera tracking process, the voxel grid feature embedding obtained in step 21 and the parameters of the geometry decoder, the parameters of the low-light color decoder, and the parameters of the highlight color decoder obtained in step 22 are fixed, and the depth camera pose is iteratively optimized to minimize the error. Complete the initialization of the depth camera pose for each frame and the constant speed motion model for: ； in, , , They represent the depth camera pose at the current moment, the depth camera pose at the previous frame moment, and the depth camera pose at the previous two frames moment, respectively. for The inverse matrix of Sample a sparse set of RGB images from step 1 The pixels are tracked and the frame pose is updated via back-propagation in each iteration.

4. The method for three-dimensional reconstruction of a reflection scene based on a neural radiation field according to any one of claims 1 to 3, characterized in that: In the step 4, a predetermined number of pixels are randomly selected from the key frame pixel database for global bundle adjustment in the following manner, including: Step 41, maintaining a key frame pixel database by selecting key frames at even intervals and storing 10% of the pixels of the key frames as a pixel subset; Step 42, the joint optimization of global bundle adjustment is run in an alternating manner, and the initialization stage randomly selects a predetermined number of pixels from the key frame pixel data, and completes the implicit representation of the preliminary reflection scene to be reconstructed by running 100 training iterations to the first frame; Step 43, when processing subsequent frames, first optimize the depth camera pose of several consecutive frames, and use the depth camera pose, that is, the light direction, as input to the geometry decoder, the highlight color decoder, and the low-light color decoder. At this stage, the weights and biases in the geometry decoder, the highlight color decoder, and the low-light color decoder are fixed, and the depth camera pose is gradient descended by minimizing the error until the depth camera pose optimization is completed; Step 44, after the depth camera pose optimization is completed, the implicit representation of the reflective scene to be reconstructed is optimized. In this stage, the optimized depth camera pose is used to further train the geometry decoder, highlight color decoder and low-light color decoder to make the representation of the reflective scene to be reconstructed more accurate. Through this joint optimization, a high-precision three-dimensional reconstruction result of the reflective scene to be reconstructed is gradually obtained.

5. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method described in any one of claims 1 to 4.

6. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 can be implemented.

Images