Method for generating three-dimensional hologram based on neural radiating field prototype distribution calibration

Abstract

This invention discloses a method for generating three-dimensional holograms based on the calibration of neural radiation field prototype distribution. First, a three-dimensional radiation field model of the neural radiation field is constructed, and the target scene is reconstructed in three dimensions. The transmittance weight profile of each ray along the depth direction is extracted and resampled to a uniform depth range to obtain a pixel-wise continuous weight distribution. Then, based on the continuous weight distribution, the true value hologram of the neural radiation field scene is calculated. The binary amplitude mask of each layer in the layer-based angular spectral method is replaced with continuous layer weights, and the pixel-wise weight distribution is aligned with the center and clustered to obtain the axial distribution prototype. Next, the mixing coefficients of each prototype are determined based on the local features of the monocular depth estimation image to generate a calibration distribution, thus regenerating the true value hologram for large-scale training data. Finally, the true value hologram is used to train a convolutional neural network, achieving end-to-end real-time generation of a three-dimensional hologram from a single two-dimensional RGB image, and the inference process does not depend on the neural radiation field model.

Description

Technical Field

[0001] This invention relates to a computer-generated hologram method based on deep learning, belonging to the interdisciplinary field of computational holography and artificial intelligence, specifically involving a three-dimensional hologram generation method based on neural radiation field (NeRF) prototype distribution calibration. Technical Background

[0002] Holographic display is considered the most ideal 3D display technology, capable of providing complete depth perception cues and achieving a true 3D viewing experience without dizziness. Computer-generated holograms (CGH) are the core of holographic display. By numerically simulating the diffraction and propagation process of light, the wavefront information of a 3D scene is encoded into a 2D hologram, which is then loaded onto a spatial light modulator (SLM) to reconstruct a 3D image. However, traditional CGH calculations require complete 3D scene data, and the diffraction calculation time complexity is extremely high, making it difficult to meet the requirements of real-time display.

[0003] In recent years, deep learning-based CGH generation methods have made groundbreaking progress. Among them, the end-to-end generation framework from 2D images to 3D holograms is the most practical: it uses a monocular depth estimation (MDE) network (such as MiDaS) to predict the depth map from a single natural image, and then uses the layer-based angular spectral method (Layer-based ASM) to calculate the ground truth hologram (True Value Hologram) to train a convolutional neural network (CNN). After training, the CNN can directly generate holograms from 2D images in real time.

[0004] However, existing methods have two fundamental bottlenecks: First, the quality of GT holograms is heavily dependent on the depth estimation accuracy of MDE networks. Monocular depth estimation methods such as MiDaS have systematic errors at object boundaries, occluded areas, and depth discontinuities, which are directly transmitted to the diffraction pattern of GT holograms. Specifically, this manifests as: (1) blurred depth boundaries, where MDE produces smooth transitions rather than sharp jumps at object edges, resulting in incorrect diffraction fringes at the boundaries of the hologram; (2) single-value depth assumption, where all existing methods compress the 3D structure of the scene into a single-value depth map, with each pixel corresponding to only one depth value. For areas with multiple layers of light contribution, such as semi-transparent objects, hair strands, and leaf edges, the single-value depth map forces the selection of a single depth value and loses information from the other layers; (3) inability to characterize volume effects, where the energy of light is continuously distributed along the depth axis in scenes such as smoke and dust, rendering the single-value depth map completely ineffective.

[0005] Second, existing research has attempted to incorporate NeRF into the hologram generation process. However, NeRF requires the separate acquisition and training of multi-view images for each scene, which is costly and limited in data scale, making it difficult to cover the needs of large-scale and diverse training data. Directly using a small amount of NeRF scene data for mixed training offers limited improvement.

[0006] Therefore, there is an urgent need for a method that can both utilize the high-quality continuous volume information of NeRF and extend this information to large-scale arbitrary natural images. Summary of the Invention

[0007] The purpose of this invention is to address the dual problems in existing methods where the quality of real-value data is limited by the accuracy of monocular depth estimation and the scale of neural radiation field (NeRF) data, and to propose a three-dimensional hologram generation method based on neural radiation field prototype distribution calibration.

[0008] To achieve the above objectives, the present invention adopts the following technical solution, which includes the following steps:

[0009] Step 1: Construct a NeRF three-dimensional radiation field model, acquire multi-view images of the target scene, and use the neural radiation field method to perform three-dimensional reconstruction to obtain the continuous volume density field and radiation color field of the scene.

[0010] Step 2: Extract the transmittance weight profile of each ray along the depth direction from the NeRF volume rendering process, resample it to a uniform depth range, and obtain the weight histogram and color histogram at each pixel;

[0011] Step 3: Calculate the GT hologram of the NeRF scene based on the continuous weight distribution, replace the binary amplitude mask of each layer in the traditional layer-based angular spectrum method with continuous weight values, and propagate and coherently superimpose the angular spectrum for all depth layers respectively;

[0012] Step 4: After center-aligning and clustering the pixel-by-pixel weight distribution of the NeRF data, a set of axial distribution prototypes are obtained; each pixel is assigned to multiple prototypes according to the weights using the local features of the monocular depth estimation image to generate a calibration distribution;

[0013] Step 5: Using the calibration distribution obtained in Step 4, regenerate continuous layered weighted ground truth (GT) holograms for all large-scale monocular depth estimation data, replacing the traditional binary mask GT; then use the regenerated GT holograms to train the CNN hologram generation network. After training, the CNN generates holograms from a single 2D RGB image in real-time end-to-end inference, and the inference process does not depend on the NeRF model.

[0014] Preferably, in step one, the NeRF model is trained using the Nerfacto method of the NerfStudio framework, with a training step count of no less than 200,000; the camera pose parameters of each acquired image are estimated using COLMAP software; the depth map is exported from the NeRF training perspective and mapped to the depth range of the holographic coordinate system, which is mapped to the holographic coordinate system using formula (1):

[0015] (1)

[0016] in , , and This represents the extreme value of scene depth. RGB images are read from the original captured image to avoid color distortion during rendering.

[0017] Preferably, in step two, during the forward propagation of NeRF volume rendering, the weights of N sampling points for each ray are extracted, as defined by formula (2):

[0018] (2)

[0019] in Given the sampling interval, the cumulative transmittance is: .

[0020] Resample irregular samples to The uniform depth interval, the first The aggregate weights and colors for each interval are as follows:

[0021] (3)

[0022] (4)

[0023] Preferably, in step three, in the traditional angular spectrum method based on layers, the complex amplitude field of the i-th layer is controlled by a binary mask generated from the depth map:

[0024] (5)

[0025] in For the indicator function, only if the pixel depth value falls within the first... The weight is 1 when the layer is selected, and 0 otherwise. This invention replaces the binary mask with continuous weight values:

[0026] (6)

[0027] right The spectral propagation and superposition of the layers are separate:

[0028] (7)

[0029] The ASM transfer function is derived from the attached formula: Define, take This is a GT phase hologram. Parameters: pixel pitch , , .

[0030] Preferably, in step four, the pixel-wise weight distribution of the NeRF data obtained in step two is... First, calculate the weighted average depth for each pixel:

[0031] (8)

[0032] Then, shift the weight distribution along the depth axis to the zero center, preserving only the distribution shape:

[0033] (9)

[0034] Perform center-aligned distribution Clustering (taking M = 8 to 16 clusters) yields M axially distributed prototypes. These correspond to typical depth distribution patterns such as narrow monophasic, wide monophasic, forward skewed, backward skewed, weak biphasic, and obvious biphasic.

[0035] For each pixel in the monocular depth estimation dataset Extracting local feature vectors ,in This is a monocular depth estimate. For depth gradient magnitude, The magnitude of the image gradient. This represents the local depth variance. Using a regression model fitted from the NeRF data, we can... Mixing coefficients mapped to M prototypes:

[0036] (10)

[0037] The calibration distribution for this pixel is a weighted blend of all prototypes, shifted to the monocular depth center of that pixel:

[0038] (11)

[0039] Preferably, in step five, the calibration distribution obtained in step four is used to replace the traditional binary mask to regenerate a continuous layered weighted GT hologram for all 10,000 monocular depth estimation data. The complex amplitude field of the i-th layer is defined as:

[0040] (12)

[0041] The calibration distribution value replaces the traditional method. .right The angular spectra of each layer are propagated and superimposed, and the generation method is consistent with formula (7) in step three.

[0042] Then, the regenerated ground truth holograms are used to train the CNN. The CNN uses the U-Net architecture, and the total loss is:

[0043] (13)

[0044] in This represents the hologram amplitude error. The propagation loss is:

[0045] (14)

[0046] The CNN-predicted hologram and the ground truth hologram are propagated to M=3 focal planes (near, middle, and far) using ASM, and the mean square error of the reconstructed intensity is compared at each focal plane. Typical parameters: , Adam optimizer, learning rate The training process lasts for 30 epochs. During inference, the CNN receives a single H×W RGB image and outputs a holographic phase. Loaded into SLM Laser illumination reconstructs 3D images. Inference does not rely on NeRF or monocular depth estimation networks, meeting real-time display requirements.

[0047] The advantages of this invention are:

[0048] 1. This invention utilizes the mathematical duality between NeRF volume rendering and holographic wave propagation to directly extend the continuous volume density field to wave optics, skipping the lossy intermediate representation of single-value depth maps, and can correctly handle complex scenes such as semi-transparent objects, multi-layer occlusion, and volume scattering.

[0049] 2. This invention proposes a prototype distribution extension mechanism to extend the continuous volume knowledge of NeRF from a small number of scenes to all large-scale training data, which solves the scalability bottleneck of the limited scale of NeRF data and eliminates the need to perform NeRF reconstruction for every training image.

[0050] 3. Prototype clustering is performed on the center-aligned distribution, capturing only the shape of the distribution rather than the depth location itself, giving the prototypes position-independent generalization ability; the soft assignment mechanism allows a pixel to belong to a mixture of multiple prototypes at the same time, which is more accurate than hard classification.

[0051] 4. The inference stage does not rely on NeRF at all. CNN only needs a single two-dimensional image to generate a hologram in real time, maintaining the same inference efficiency as existing methods. Attached Figure Description

[0052] Figure 1 This is the training process for the CNN hologram generation network of the present invention;

[0053] Figure 2 This invention relates to an optical holographic reconstruction system.

[0054] Figure 3 The volume rendering weights and colors obtained in the embodiments of the present invention;

[0055] Figure 4 The distribution shape preserved for center alignment in this invention;

[0056] Figure 5 For the purposes of this invention embodiment The prototype distribution map obtained by clustering;

[0057] Figure 6 The calibration distribution is obtained during the application of the method of the present invention;

[0058] Figure 7 This invention utilizes GT holograms obtained from calibration distributions to reconstruct closer distances during the inference stage;

[0059] Figure 8 This involves reconstructing the effect of objects at greater distances during the inference phase; the image examples are from an open-source dataset.

[0060] Figure reference numerals: 1-Green single longitudinal mode laser; 2-Beam expander; 3-Collimator; 4-Beam splitter; 5-Phase type spatial light modulator (LCOS); 6-Computer; 7-Sensor digital camera (CCD); 8-4f system (two convex lenses, adjustable aperture). Detailed Implementation

[0061] Please see the appendix Figures 1 to 8 The embodiments of the present invention will be described in further detail with reference to the accompanying drawings. However, these embodiments are not intended to limit the present invention. Any similar structures or variations thereof that adopt the present invention should be included in the scope of protection of the present invention.

[0062] like Figure 1 As shown, during the training phase, a single RGB image is first input into the CNN hologram generation network, which outputs the amplitude and phase of the predicted hologram. Subsequently, a two-level supervision mechanism is constructed: firstly, in the hologram domain, the predicted hologram is directly compared with the ground truth hologram generated from NeRF geometric information, forming a data term loss. Secondly, the predicted hologram and the GT hologram are propagated to multiple focal planes using the angular spectral method (ASM), and their reconstruction results are compared to form the propagation loss. The two loss mechanisms jointly constrain the network to learn a holographic representation that is both close to the truth in terms of holographic representation and conforms to the 3D scene structure in terms of propagation and reconstruction behavior; the images are from an open-source dataset.

[0063] like Figure 2As shown, a holographic display system is composed of a computer (6), a green single-mode laser (1), a phase-type spatial light modulator (LCOS) (5), a 4f system (two convex lenses and an adjustable aperture) (8), a beam expander (2), a collimating lens (3), a beam splitter (4), and an industrial charge-coupled device (CCD) image sensor (7). In the optical reconstruction system, the experimental setup uses a 532 nm wavelength single-mode laser source as illumination. The laser light is shaped by wavefront shaping components (beam expander and collimator) to form parallel light covering the effective working surface of the LCOS. After refraction by the beam splitter, the parallel light illuminates the SLM panel with a modulated light field and enters the 4f system. This 4f system consists of two convex lenses and an adjustable aperture, which effectively suppresses zero-order diffraction noise and higher-order harmonic components. Three-dimensional reconstruction of the target light field is achieved through diffraction field propagation, and the optically reconstructed image is acquired by the CCD. The SLM used in the experimental system is [size missing]. The panel size is 0.65 inches, and the resolution of the loaded hologram is 1536×768;

[0064] The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration of the present invention is specifically implemented as follows:

[0065] Step 1: Construct a three-dimensional NeRF radiation field model;

[0066] Step 2: Extract the transmittance weight profile for each ray and resample it to a uniform depth range. Figure 3 To extract the volume rendering weights and colors of each ray along the depth direction from the NeRF volume rendering process;

[0067] Step 3: Calculate the continuous volume GT hologram of the NeRF scene based on the continuous weight distribution;

[0068] Step 4: Cluster the weight distribution to obtain a prototype, then extend the prototype to monocular depth estimation data. (See also...) Figure 6 Local feature vectors are extracted from each pixel in the monocular depth estimation dataset and mapped to the mixing coefficients of each prototype distribution. The prototype distributions are then weighted and combined and shifted to the depth center of the pixel to obtain the calibration distribution.

[0069] Step 5: Regenerate the ground truth holograms for all large-scale data using the calibration distribution; then train the CNN and generate holograms through end-to-end real-time inference.

[0070] The specific implementation method of step one above is as follows:

[0071] The NeRF model was trained using the Nerfacto method within the NerfStudio framework, with a training step count of no less than 200,000. Camera pose parameters for each acquired image were estimated using COLMAP software. Depth maps were derived from the NeRF training perspective and mapped to the depth range in the holographic coordinate system using formula (1):

[0072] (1)

[0073] in , , and This represents the extreme value of scene depth. RGB images are read from the original captured image to avoid color distortion during rendering.

[0074] The specific implementation method of step two above is as follows:

[0075] In the forward propagation of NeRF volume rendering, the weights of N sampling points are extracted for each ray, as defined by formula (2):

[0076] (2)

[0077] in Given the sampling interval, the cumulative transmittance is: .

[0078] Resample irregular samples to The uniform depth interval, the first The aggregate weights and colors for each interval are as follows:

[0079] (3)

[0080] (4)

[0081] The specific implementation method of step three above is as follows:

[0082] In step three, in the traditional angular spectrum method based on layers, the complex amplitude field of the i-th layer is controlled by a binary mask generated from the depth map:

[0083] (5)

[0084] in For the indicator function, only if the pixel depth value falls within the first... The weight is 1 when the layer is selected, and 0 otherwise. This invention replaces the binary mask with continuous weight values:

[0085] (6)

[0086] right The spectral propagation and superposition of the layers are separate:

[0087] (7)

[0088] The ASM transfer function is derived from the attached formula: Define, take This is a GT phase hologram. Parameters: pixel pitch , , .

[0089] The specific implementation method of step four above: pixel-wise weight distribution of the NeRF data obtained in step two. First, calculate the weighted average depth for each pixel:

[0090] (8)

[0091] Then, shift the weight distribution along the depth axis to the zero center, preserving only the distribution shape:

[0092] (9)

[0093] Figure 4 To center-align, remove only the distribution shape at the depth position. Then, perform the center-aligned distribution... Clustering (taking M = 8 to 16 clusters) yields M axially distributed prototypes. These correspond to typical depth distribution patterns such as narrow monophasic, wide monophasic, forward skew, backward skew, weak bimodal, and obvious bimodal. Please refer to [link / reference]. Figure 5 .

[0094] For each pixel in the monocular depth estimation dataset Extracting local feature vectors ,in This is a monocular depth estimate. For depth gradient magnitude, The magnitude of the image gradient. This represents the local depth variance. Using a regression model fitted from the NeRF data, we can... Mixing coefficients mapped to M prototypes:

[0095] (10)

[0096] The calibration distribution for this pixel is a weighted blend of all prototypes, shifted to the monocular depth center of that pixel:

[0097] (11)

[0098] The specific implementation method of step five above is as follows: The calibration distribution obtained in step four is used to replace the traditional binary mask, and a continuous layered weighted GT hologram is regenerated for all 10,000 monocular depth estimation data. The complex amplitude field of the i-th layer is defined as:

[0099] (12)

[0100] The calibration distribution value replaces the traditional method. .right The angular spectra of each layer are propagated and superimposed, and the generation method is consistent with formula (7) in step three.

[0101] Then, the regenerated ground truth holograms are used to train the CNN. The CNN uses the U-Net architecture, and the total loss is:

[0102] (13)

[0103] in This represents the hologram amplitude error. The propagation loss is:

[0104] (14)

[0105] The CNN-predicted hologram and the ground truth hologram are propagated to M=3 focal planes (near, middle, and far) using the angular spectral method (ASM), and the mean square error of the reconstructed intensity is compared at each focal plane. Typical parameters: , Adam optimizer, learning rate Train for 30 epochs. See also Figure 7 and Figure 8 During inference, the CNN receives a single H×W RGB image and outputs a holographic phase. Loaded into SLM Laser illumination reconstructs 3D images. Inference does not rely on NeRF or monocular depth estimation networks, meeting real-time display requirements.

[0106] Not limited to this, any variations or substitutions conceived without inventive effort should be included within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope defined in the claims.

Claims

1. A method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration, characterized in that, Includes the following steps: Step 1: Construct a three-dimensional radiation field model of the neural radiation field, perform three-dimensional reconstruction of the target scene, and obtain a continuous volume density field and radiation color field; Step 2: Extract the transmittance weight profile of each ray along the depth direction from the neural radiation field volume rendering process, and resample it to a uniform depth range to obtain the weight histogram at each pixel; Step 3: Calculate the true value hologram of the neural radiation field scene based on the continuous weight distribution, replace the binary amplitude mask of each layer in the layer-based angular spectrum method with continuous weight values, and perform angular spectrum propagation and coherent superposition on all depth layers respectively; Step 4: After center-aligning and clustering the pixel-by-pixel weight distribution of the neural radiation field data, a set of axial distribution prototypes are obtained; using the local features of the monocular depth estimation image, each pixel is assigned to multiple prototypes according to the weight to generate a calibration distribution. Step 5: Using the calibration distribution obtained in Step 4, regenerate the true value hologram of continuous layer weights for the large-scale monocular depth estimation data; Step Six: Train a convolutional neural network hologram generation network using the ground truth holograms regenerated in Step Five; After training, the convolutional neural network generates holograms from a single 2D RGB image in real time from end to end, and the reasoning process does not depend on the neural radiation field model.

2. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step one, the neural radiation field model is trained using the Nerfacto method of the NerfStudio framework, with a training step count of no less than 200,000 steps; camera pose parameters are estimated using COLMAP software; the depth map is exported from the training viewpoint and mapped to the depth range of the holographic coordinate system using formula (1): (1) in , , and To represent the extreme value of scene depth, RGB images are read from the original captured image to avoid color distortion during rendering.

3. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step two, during the forward propagation of NeRF volume rendering, the weights of N sampling points are extracted for each ray, as defined by formula (2): (2) in Given the sampling interval, the cumulative transmittance is: .

4. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 3, characterized in that, Resample irregular samples to The uniform depth interval, the first The aggregate weights and colors of each interval are as follows: (3) (4)。 5. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step three, in the traditional angular spectrum method based on layers, the complex amplitude field of the i-th layer is controlled by a binary mask generated from the depth map: (5) in For the indicator function, only if the pixel depth value falls within the first... The value is 1 if the layer is selected, otherwise it is 0.

6. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 5, characterized in that, This invention replaces the binary mask with continuous weight values: (6) right The spectral propagation and superposition of the layers are separate: (7) The transfer function of the angular spectrum method is given by the attached formula: Define, take For GT phase hologram, parameters: pixel pitch , , .

7. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step four, the pixel-wise weight distribution of the neural radiation field data obtained in step two is described. First, calculate the weighted average depth for each pixel: (8) Then, shift the weight distribution along the depth axis to the zero center, preserving only the distribution shape: (9) Perform center-aligned distribution Clustering, taking M=8 to 16 clusters, yields M axial distribution prototypes. These correspond to typical depth distribution patterns such as narrow monophasic, wide monophasic, forward skewed, backward skewed, weak biphasic, and obvious biphasic.

8. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 7, characterized in that, For each pixel in the monocular depth estimation dataset Extracting local feature vectors ,in This is a monocular depth estimate. For depth gradient magnitude, The magnitude of the image gradient. For local depth variance; using a regression model fitted from neural radiation field data, Mixing coefficients mapped to M prototypes: (10) The calibration distribution for this pixel is a weighted blend of all prototypes, shifted to the monocular depth center of that pixel: (11)。 9. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step five, the calibration distribution obtained in step four is used to replace the traditional binary mask to regenerate the true hologram for all 10,000 monocular depth estimation data. The complex amplitude field of the i-th layer is defined as: (12)。 10. The method for generating three-dimensional holograms based on neural radiation field prototype distribution calibration according to claim 1, characterized in that, In step six, a convolutional neural network is trained using the real-value hologram regenerated in step five. The convolutional neural network adopts the U-Net architecture, and the total loss is: (13) in This refers to the hologram amplitude error. The propagation loss is: (14) The predicted hologram and the true hologram from the convolutional neural network are propagated to the near, middle, and far focal planes using the angular spectrum method, and the mean square error of the reconstructed intensity is compared at each focal plane.

Images