Neural scene representation-based iToF image three-dimensional reconstruction method and system
By combining guided sampling and neuroillumination modeling with traditional volume rendering technology, the error problem of iToF cameras in depth measurement is solved, and a high-precision three-dimensional reconstruction effect is achieved.
Patent Information
- Application Number
- CN202510452658.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-07-25
AI Technical Summary
The indirect time of flight (iToF) cameras in the prior art are affected by problems such as multipath interference, low signal-to-noise ratio and deep wrapping during depth measurement, and the applicability of deep learning methods in different iToF camera systems is limited, making it difficult to deal with changes in lighting conditions and deep aliasing problems.
The phase measurement value, ray center and ray direction of the iToF image are combined for guiding sampling, and the initial amplitude of the ray is predicted using the neural illumination function, and combined with the symbol distance function through traditional volume rendering technology, optimize the neural scene characterization, and perform three-dimensional reconstruction by minimizing the loss function.
It improves the depth measurement accuracy and robustness of iToF data, solves the problems of depth blur, phase winding and noise, and realizes high-quality three-dimensional reconstruction.
Smart Images
Figure CN120374851A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and particularly relates to an iToF image three-dimensional reconstruction method and system based on neural scene representation. Background Art
[0002] An indirect time-of-flight (iToF) camera captures depth information by measuring the phase difference between the transmitted and received signals, and is widely used for 3D perception because of its high cost-effectiveness and easy deployment. However, depth measurements are usually affected by errors caused by problems such as multipath interference (MPI), low signal-to-noise ratio (SNR), and depth wrapping.
[0003] To address these problems, many deep learning-based methods have been proposed, which are trained on simulated datasets and attempt to generalize to real data. However, these data-driven methods rely heavily on training data, limiting their applicability in different iToF camera systems and making it difficult to generalize to scenarios outside the training conditions.
[0004] In recent years, neural radiance fields (NeRF) and related technologies have received extensive attention as powerful tools for three-dimensional reconstruction from multi-view RGB images. NeRF can capture complex texture details and precise geometries by simulating the interaction of light with the three-dimensional structure of the scene and combining volume rendering techniques.
[0005] Although NeRF performs well in three-dimensional reconstruction, it has certain limitations when dealing with input data with changing lighting conditions. This is because NeRF assumes that the lighting conditions remain unchanged, while the iToF camera re-illuminates the entire scene when imaging from different positions, making the reconstruction process more complex. These limitations can lead to ambiguity in the scene because the observed radiance values may vary due to changes in lighting intensity at the same spatial position and viewing direction. In addition, the depth measurement of the iToF camera is based on phase shift, and its measurement range is limited by the modulation frequency, which can cause depth aliasing when the actual depth exceeds the measurement range. Therefore, an iToF image three-dimensional reconstruction method based on neural scene representation is proposed. Summary of the Invention
[0006] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide an iToF image three-dimensional reconstruction method and system based on neural scene representation, which solves the problems in the prior art.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] An iToF image three-dimensional reconstruction method based on neural scene representation, comprising the following steps:
[0009] Guided sampling is performed by combining the phase measurement values of the iToF image, the optical center of the light ray, and the light ray direction to obtain phase sampling points;
[0010] Using the neural illumination function, predict the initial amplitude of the light ray based on the light ray direction and the learned implicit encoding;
[0011] Based on the phase sampling points and the initial amplitude of the light ray, by combining the traditional volume rendering technique with the signed distance function, model the unique imaging mechanism of the iToF camera to obtain the rendered amplitude component, rendered depth, and rendered phase measurement value;
[0012] Optimize the neural scene representation by minimizing the loss function between the given iToF image data and the rendered data.
[0013] Further, the phase sampling point p i has the following expression:
[0014]
[0015] where c is the speed of light, f is the modulation frequency of the light ray, o is the optical center of the light ray, v is the light ray direction, and θ i represents the phase.
[0016] Further, the initial amplitude A0 of the light ray r is defined as:
[0017] A0(r) = F θ (v c , E j )
[0018] where F θ is the neural illumination function, V c is the direction vector in the camera coordinate system; E j is a learnable implicit encoding corresponding to the original iToF imaging M j .
[0019] Further, the rendered amplitude component is:
[0020]
[0021] where N is the total number of sampling points on a ray, is the rendered amplitude component, T i is the cumulative transmittance to the sampling point p i , R i is the reflection attenuation coefficient, A0 is the initial amplitude of the light ray, t i represents the distance from the camera to the sampling point p i on the light ray, α i is the opacity of the sampling point p i ; Φs (x) is a logical function that controls the smoothness of the transition, and SDF(p i ) is the SDF value of the sampling point p i ; f i is the feature vector predicted by the geometric network , and is the reflection attenuation coefficient prediction network.
[0022] Further, the rendering depth is:
[0023]
[0024] Wherein, represents the rendering depth;
[0025] The phasor measurement value of the rendering is:
[0026]
[0027] Wherein, is the phasor measurement value of the rendering.
[0028] Further, the loss function is:
[0029]
[0030] Wherein, is the photometric loss of the amplitude, is the phase wrapping perception loss, is the optical path function for regularizing the signed distance function, λ p , λ w , λ r are the weights of the photometric loss of the amplitude, the weight of the phase wrapping perception loss, and the weight of the optical path function of the regularized signed distance function, respectively.
[0031] The iToF image three-dimensional reconstruction system based on neural scene representation includes:
[0032] Guided sampling module: By combining the phase measurement value of the iToF image, the optical center of the light ray, and the light ray direction for guided sampling, phase sampling points are obtained;
[0033] Initial amplitude prediction module: Using the neural illumination function, the initial amplitude of the light ray is predicted according to the light ray direction and the learned implicit encoding;
[0034] Modeling module: Based on the phase sampling points and the initial amplitude of the light ray, by combining the traditional volume rendering technology with the signed distance function, the unique imaging mechanism of the iToF camera is modeled to obtain the rendered amplitude component, the rendering depth, and the rendered phase measurement value;
[0035] And, an optimization module: Optimize the neural scene representation by minimizing the loss function between the given iToF image data and the rendering data.
[0036] A computer storage medium stores a readable program that, when run, can execute the above-mentioned iToF image three-dimensional reconstruction method based on neural scene representation.
[0037] An electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete communication with each other through the communication bus.
[0038] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above-mentioned iToF image three-dimensional reconstruction method based on neural scene representation.
[0039] 10. A computer program product includes computer instructions that direct a computing device to execute the operations corresponding to the iToF image three-dimensional reconstruction method based on neural scene representation as described in any one of claims 1-6.
[0040] Advantages of the present invention:
[0041] The method of the present invention combines a custom rendering scheme, neural light modeling, an improved sampling strategy, and a wrapped-aware loss function for depth blur and phase wrapping problems to address the inherent depth blur, phase wrapping, and noise problems in iToF data. These innovations together improve the accuracy and efficiency of the framework and enable high-quality three-dimensional reconstruction. Description of the Drawings
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.
[0043] Figure 1 It is a flowchart of the iToF image three-dimensional reconstruction method of the present invention. Detailed Embodiments
[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.
[0045] Example 1
[0046] As Figure 1 shown, the 3D reconstruction method of iToF images based on neural scene representation includes the following steps:
[0047] S1. By combining the phase measurement value θ of the iToF image, the optical center o of the light ray, and the light ray direction v, guided sampling is performed to obtain the phase sampling point p i ;
[0048] In iToF imaging, the range of the phase measurement value is from 0 to 2π, corresponding to the distance range related to the modulation frequency f of the light ray. However, due to the periodicity of phase measurement (as shown in the formula where τ0 is the reference delay time, I 90 , I 180 , I 270 , and I0 are the measurement images obtained from the reference signal at different phases respectively), the observed phase θ may not be the true phase, and its true value may be in a periodic space:
[0049] θ true = θ + 2πk, where k ∈ {0, 1, 2...}. To effectively utilize the phase prior information in iToF data and solve the noise problem caused by phase aliasing, considering possible phase aliasing, uniform sampling is performed within the adjusted phase range.
[0050] Specifically, the specific steps of phase-guided sampling are as follows:
[0051] S11. Determine the phase space;
[0052] For the phase measurement value θ, determine that its possible true phase value is in the periodic space:
[0053] θ true = θ + 2πk, k ∈ {0, 1, 2...};
[0054] where θ true is the correct phase value.
[0055] S12. Calculate the periodic phase interval;
[0056] Define an interval [θ + 2πk - kε, θ + 2πk + kε] around each possible periodic phase space θ + 2πk; where ε is the tolerance value used to cope with measurement noise. The specific intervals include: the first period: [θ - ε, θ + ε], the second period: [2π + θ - ε, 2π + θ + ε], the third period: [4π + θ - ε, 4π + θ + ε], and so on.
[0057] S13. Calculate the phase sampling point p corresponding to the phasei ;
[0058] For the ray r, the optical center of the ray is o and the ray direction is v. Uniform sampling is performed within the range and for the phase θ i (from the above phase interval), the corresponding phase sampling point p is calculated i :
[0059]
[0060] where c is the speed of light and f is the modulation frequency of the ray.
[0061] This method can effectively alleviate the problem of inaccurate geometric representation caused by phase ambiguity and noise, ensuring higher robustness and accuracy in scene depth and structure modeling.
[0062] S2. Using the neural lighting function, predict the initial amplitude A0 of the ray r according to the ray direction and the learned implicit encoding;
[0063] The radiative integral formula under active illumination conditions can be extended to support the form of the signed distance function (SDF):
[0064]
[0065] where N is the total number of sampling points on a ray, is the amplitude component for rendering, is the cumulative transmittance to the sampling point p i , R i is the reflection attenuation coefficient, A0 is the initial amplitude of the ray, t i represents the distance from the camera to the sampling point p i on the ray, and α i is the opacity of the sampling point p i .
[0066] In an ideal case, the emitted infrared light signal is modeled as a point light source, i.e., its energy distribution is uniform in all directions; thus, A0 is regarded as a constant. However, in an actual iToF imaging device, the light source is usually a surface light source, which results in inconsistent signal amplitudes in different directions. In addition, the amplitude of the signal is also affected by the incident angle and the surface normal vector, and these factors will further affect the energy distribution in the reflection direction. To solve these problems, a neural lighting function (Neural Lighting Function) is proposed in this embodiment to implicitly model the amplitude distribution of the emitted signal through a neural network;
[0067] Specifically, for iToF imaging M j (M jRefers to the light ray r corresponding to a pixel in the original iToF measurement results obtained from a specific perspective j, including phase and amplitude information. The initial amplitude of the light ray r is defined as:
[0068] A0(r) = F θ (v c , E j )
[0069] where F θ is the neural illumination function, v c is the direction vector in the camera coordinate system, which can be calculated from the direction v through the internal parameters of the camera; E j is a learnable implicit encoding corresponding to the original iToF imaging M j ; this encoding captures the characteristics of the light source and the environmental conditions during the iToF imaging process, enabling the neural network to adaptively estimate the initial energy distribution of the light source for each scene.
[0070] S3. Based on the phase sampling points p i obtained from S1, and the initial amplitude A0 obtained from S2, by combining the traditional volume rendering technique with the signed distance function (SDF), the unique imaging mechanism of the iToF camera is modeled to obtain the rendered amplitude component, rendered depth, and rendered phase measurement values;
[0071] The specific rendering process is as follows:
[0072] S31. In the classical NeRF framework, the radiative integral formula is defined as follows:
[0073]
[0074] where represents the radiative integral from m to n segments along the ray, c(s) is the radiance at position s, is the medium absorption coefficient at position S, and s = 0 corresponds to a pixel position on the camera plane; in the standard NeRF framework, c(s) is usually assumed to be constant for a certain point in the same direction, which means that the radiance of each point is independent of the light source position. Considering the active imaging condition where the light source and the signal receiver are at the same position, c(s) can be further expressed as:
[0075]
[0076] where A0 is the initial amplitude of the light ray, 1 / s 2 describes the geometric attenuation, considers the attenuation caused by the medium, and R(s) represents the energy loss due to surface reflection. By substituting the above formula into the radiative integral formula, the radiative integral under active illumination conditions is obtained:
[0077]
[0078] Among them, is the radiation integral under active illumination conditions for the m-to-n segment along the ray.
[0079] It is further discretized into:
[0080]
[0081] This formula represents the amplitude information of the received signal; among them, is the rendered amplitude component, σ i , σ j represents the transmittance, δ i , δ j represents the distance between adjacent sampling points.
[0082] S32, this formula can be extended to support the form of the signed distance function (SDF):
[0083]
[0084] Among them, is the cumulative transmittance to the sampling point p i , α i , α j is the opacity, and the opacity α i based on the SDF value is defined as:
[0085]
[0086] Among them, Φ s (x) = (1 + e -sx ) -1 is a logical function that controls the transition smoothness. Substituting SDF(p i ) into it, the opacity of each sampling point p i can be obtained. This formula is based on the signed distance representation and models the attenuation of the signal amplitude during spatial propagation and reflection, rendering the amplitude information of the iToF imaging data.
[0087] Furthermore, given a ray r(t) = o + tv, a set of points is sampled on the ray For each sampling point p i obtained in S1, its signed distance function value SDF(p i ) and reflection attenuation coefficient R i are predicted by the neural network:
[0088]
[0089] Among them, fi is a geometric network The predicted eigenvector, SDF(p i ) provides scene geometry information, R i models the reflection attenuation coefficient, is the reflection attenuation coefficient prediction network. Substitute R i , α i and A0 obtained from S2 into the formula:
[0090]
[0091] The rendered amplitude component can be obtained
[0092] S33, and the predicted value of the rendered phase is derived through rendering depth estimation:
[0093]
[0094] where f is the modulation frequency of the light ray and C is the speed of light;
[0095] S4, optimizes the neural scene representation by minimizing the loss function between the given iToF image data and the rendered data;
[0096] Optimizes the neural scene representation by minimizing the loss function, which is based on a set of given indirect time-of-flight imaging data.
[0097]
[0098] This loss function consists of the photometric loss of the amplitude the phase-wrapped perception loss the optical path function for regularizing the signed distance function (SDF) and the weight λ of the related loss function p , λ w , λ r components.
[0099] The photometric loss of the amplitude:
[0100]
[0101] where is the batch of camera rays, is the rendered amplitude component, and A is the reference amplitude.
[0102] The depth loss of phase-wrapped perception:
[0103]
[0104] where Denote the rendering depth. D is the reference depth obtained from the phase measurement value, given by D(r) = c·θ(r) / 4πf. The parameter d = c / 2fd defines the depth range. The round(·) function performs the rounding operation. This loss function aims to measure the difference between the rendering depth and the reference depth D.
[0105] Regularization loss:
[0106]
[0107] where denotes the gradient of the signed distance function (SDF) at the sampling point p i . This helps to ensure that the SDF represents a valid distance function, improving the stability and accuracy of the model during training.
[0108] Based on a similar inventive concept, an embodiment of the present invention also provides a computer storage medium storing a readable program, which can execute the above-mentioned 3D reconstruction method of iToF images based on neural scene representation when the program runs.
[0109] Based on a similar inventive concept, an embodiment of the present invention provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete mutual communication through the communication bus;
[0110] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above-mentioned 3D reconstruction method of iToF images based on neural scene representation.
[0111] Based on a similar inventive concept, an embodiment of the present invention also provides a computer program product including computer instructions, and the computer instructions direct a computing device to execute the operations corresponding to the above-mentioned 3D reconstruction method of iToF images based on neural scene representation.
[0112] Embodiment 2
[0113] Based on the 3D reconstruction method of iToF images based on neural scene representation proposed in Embodiment 1, in this embodiment, a 3D reconstruction system of iToF images based on neural scene representation is proposed, including:
[0114] Guided sampling module: By combining the phase measurement value of the iToF image, the optical center of the light, and the light direction for guided sampling, phase sampling points are obtained;
[0115] Initial amplitude prediction module: Using the neural illumination function, predict the initial amplitude of the light according to the light direction and the learned implicit encoding;
[0116] Modeling module: Based on the phase sampling points and the initial amplitude of the light, by combining the traditional volume rendering technology with the signed distance function, model the unique imaging mechanism of the iToF camera to obtain the rendered amplitude component, rendered depth, and rendered phase measurement value;
[0117] And, an optimization module: Optimize the neural scene representation by minimizing the loss function between the given iToF image data and the rendered data.
[0118] The method of the present invention can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.
[0119] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.
Claims
1. A three-dimensional reconstruction method for iToF images based on neural scene representation, characterized in that, Including the following steps: Performing guided sampling by combining the phase measurement values of the iToF image, the optical center of the light ray, and the light ray direction to obtain phase sampling points; Using a neural lighting function to predict the initial amplitude of the light ray according to the light ray direction and the learned implicit encoding; Based on the phase sampling points and the initial amplitude of the light ray, by combining the traditional volume rendering technique with the signed distance function, modeling the unique imaging mechanism of the iToF camera to obtain the rendered amplitude component, rendered depth, and rendered phase measurement values; Optimizing the neural scene representation by minimizing the loss function between the given iToF image data and the rendered data.
2. The iToF image three-dimensional reconstruction method based on neural scene representation according to claim 1, wherein, Phase sampling point p i The expression is as follows: where c is the speed of light, f is the modulation frequency of the light ray, o is the optical center of the light ray, V is the light ray direction, and θ i represents the phase.
3. The iToF image three-dimensional reconstruction method based on neural scene representation according to claim 1, wherein The initial amplitude A0 of the light ray r is defined as: A0(r) = F θ (v c ,E j ) Among them, F θ is the neural illumination function, and v c is the direction vector in the camera coordinate system; E j is a learnable implicit encoding corresponding to the original iToF imaging M j .
4. The three-dimensional reconstruction method of iToF images based on neural scene representation according to claim 2, wherein The rendered amplitude component is: where N is the total number of sampling points on a ray, is the amplitude component for rendering, T i is the cumulative transmittance to the sampling point p i , R i is the reflection attenuation coefficient, A0 is the initial amplitude of the ray, t i represents the distance from the camera to the sampling point p i on the ray, α i is the opacity of the sampling point p i ; Φ s (x) is a logical function that controls the transition smoothness, SDF(p i ) is the SDF value of the sampling point p i ; f i is the feature vector predicted by the geometric network , is the reflection attenuation coefficient prediction network.
5. The iToF image three-dimensional reconstruction method based on neural scene representation according to claim 4, wherein, The rendered depth is: Among them, represents the rendering depth; The rendered phasor measurement value is: Among them, is the rendered phasor measurement value.
6. The iToF image three-dimensional reconstruction method based on neural scene representation according to claim 1, wherein The loss function is as follows: Among them, is the photometric loss of the amplitude, is the phase-wrapped perception loss, is the optical path function for regularizing the signed distance function, λ p , λ w , λ r are the weights of the photometric loss of the amplitude, the weight of the phase-wrapped perception loss, and the weight of the optical path function of the regularized signed distance function, respectively.
7. An iToF image three-dimensional reconstruction system based on neural scene representation, characterized in that, Including: Guided sampling module: Performing guided sampling by combining the phase measurement values of the iToF image, the optical center of the light ray, and the light ray direction to obtain phase sampling points; Initial amplitude prediction module: Using a neural lighting function to predict the initial amplitude of the light ray according to the light ray direction and the learned implicit encoding; Modeling module: Based on the phase sampling points and the initial amplitude of the light ray, by combining the traditional volume rendering technique with the signed distance function, modeling the unique imaging mechanism of the iToF camera to obtain the rendered amplitude component, rendered depth, and rendered phase measurement values; And an optimization module: Optimizing the neural scene representation by minimizing the loss function between the given iToF image data and the rendered data.
8. A computer storage medium stores a readable program, characterized in that, When the program runs, it can execute the 3D reconstruction method of the iToF image based on neural scene representation according to any one of claims 1-6.
9. An electronic device, characterized in that, Including: A processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the 3D reconstruction method of the iToF image based on neural scene representation according to any one of claims 1-6.
10. A computer program product comprising computer instructions, characterized in that, The computer instruction instructs the computing device to execute the operations corresponding to the 3D reconstruction method of the iToF image based on neural scene representation according to any one of claims 1-6.