Unsupervised deep learning method for realizing three-dimensional holographic display

Abstract

The invention relates to the technical field of computer-generated holographic three-dimensional display and deep learning, in particular to an unsupervised deep learning method for realizing three-dimensional holographic display. The method comprises the steps of generating a depth map corresponding to a two-dimensional image; the depth image and the two-dimensional image are spliced in the channel dimension to serve as input of a hologram encoder, and a double-U-Net cascade neural network architecture serves as the hologram encoder; angular spectrum diffraction back propagation is carried out through the generated pure phase hologram to realize three-dimensional scene discretization reconstruction, a reconstructed image with a specified depth is obtained, a depth map is uniformly quantized to obtain a plurality of binary masks, loss calculation is carried out on the reconstructed image and a target image superposed with the corresponding depth binary masks, network parameter optimization is carried out, and a target image with the depth corresponding to the target image is obtained. And when the training of the double U-Net cascade neural network architecture is converged, the training stage is ended. According to the method, high-quality three-dimensional hologram reconstruction is realized through layered angular spectrum propagation, and the method has relatively high precision and detail reduction capability.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer-generated holography three-dimensional display and deep learning, and particularly to an unsupervised deep learning method for realizing three-dimensional holographic display. Background Art

[0002] Compared with two-dimensional display, three-dimensional scenes have greater information content and higher degrees of freedom. In computer graphics, in order to represent three-dimensional scenes more precisely, they usually need to be discretized into different types of computational primitives. According to the different selected primitives, the algorithms based on physical simulation can be divided into the point element method, the surface element method, and the tomography method. Among them, the point element method and the surface element method need to process a large number of discrete primitives, and have the defects of low computational efficiency and large memory consumption. In contrast, the tomography method has fewer discrete primitives and can extend the algorithm of two-dimensional computer-generated hologram to three-dimensional computer-generated hologram, so it has received extensive attention from researchers. However, the existing two-dimensional to three-dimensional holographic display deep learning algorithms based on the tomography method are limited by the supervised learning framework, and usually need to use prior information such as depth maps and binocular views to train the network, and these prior information often need to rely on professional equipment or complex 3D rendering techniques to obtain. The acquisition of prior information is difficult and the production cost is high, which seriously restricts the popularization and development of three-dimensional holographic display technology. Therefore, it is necessary to study a three-dimensional holographic display algorithm based on unsupervised deep learning without prior information to solve the problems of high computational complexity and strong data dependence in the prior art. Summary of the Invention

[0003] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide an unsupervised deep learning method for realizing three-dimensional holographic display, which solves the problems of high computational complexity and strong data dependence.

[0004] To achieve the above object and other related objects, the present invention provides an unsupervised deep learning method for realizing three-dimensional holographic display, including:

[0005] S1. Depth prediction stage: Establish a mapping relationship between the amplitude information of a two-dimensional image and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map corresponding to the two-dimensional image, and uniformly quantize the depth map to obtain a number of binary masks;

[0006] S2. Hologram encoding stage: Concatenate the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder, and use a dual U-Net cascaded neural network architecture as the hologram encoder. Among them, the first sub-network is a real-valued U-Net for performing the task of predicting the phase information of the target plane, and the second sub-network is a complex-valued U-Net for performing the task of predicting and generating a pure-phase hologram of the spatial light modulator plane;

[0007] S3. Hologram decoding stage: The 3D scene is discretized and reconstructed by performing angular spectrum diffraction back propagation on the generated POH to obtain a reconstructed image of a specified depth. The depth map is uniformly quantized to obtain several binary masks. The loss of the reconstructed image and the target image superimposed with the corresponding depth binary mask is calculated, and the network parameters are optimized. When the dual U-Net cascade neural network architecture training converges, the training stage ends.

[0008] In one embodiment of the present invention, it further includes:

[0009] S4. Numerical simulation test: The test sample is input into the trained dual U-Net cascade neural network architecture, and the POH is numerically reconstructed at different diffraction distances based on the depth information. The peak signal-to-noise ratio and structural similarity of the reconstructed image are calculated with the target image superimposed with the corresponding depth binary mask to evaluate the quality of the reconstructed image;

[0010] S5. Optical experimental test: Build a holographic display optical system and conduct optical experiments, capture the reconstructed image through an industrial camera, and evaluate the quality of the reconstructed image.

[0011] In one embodiment of the present invention, the depth prediction stage in step S1 includes:

[0012] A pre-trained monocular depth estimation model is used to establish the mapping relationship between the amplitude information of the two-dimensional image and the three-dimensional depth, and a depth map D corresponding to the two-dimensional image A is generated. The depth map is uniformly quantized to obtain several binary masks. The binary mask is expressed as:

[0013]

[0014] Where d represents the pixel value of the depth map, D i Indicates the quantization interval.

[0015] In one embodiment of the present invention, the hologram encoding stage in step S2 includes:

[0016] The depth map and the two-dimensional image are spliced in the channel dimension as the input I of the hologram encoder. A dual U-Net cascade neural network architecture is used as the hologram encoder, where the first sub-network is a real-valued U-Net, which is used to perform the phase information of the target plane. The prediction task is expressed as:

[0017] I=Cat(A,D),

[0018]

[0019] Among them, f U-Net1(·) represents Sub-network 1 of the dual U-Net cascaded architecture. This network consists of an initial convolutional block, 4 downsampling blocks, 4 upsampling blocks, and an output layer. Each sampling block contains a spatial-frequency domain dual-branch convolution to achieve the fusion of frequency domain features and spatial domain features. The spatial branch of the spatial-frequency domain dual-branch convolution extracts local spatial features through a double-layer 3×3 convolution. The frequency domain branch performs a fast Fourier transform on the input, decomposes the real or imaginary part, performs frequency domain filtering through a 1×1 convolution, and after inverse fast Fourier transform reduction, adds it to the spatial features to achieve the fusion of frequency domain to spatial domain features;

[0020] Sub-network 1 of the dual U-Net cascaded architecture executes the phase information of the target plane After being superimposed with the target amplitude, the complex amplitude distribution of the target plane is obtained. The complex amplitude of the target plane is propagated to the SLM plane through the angular spectrum diffraction method to obtain the complex amplitude distribution of the SLM plane. The mathematical representation of the angular spectrum diffraction method is as follows:

[0021]

[0022] where, λ represents the wavelength, f x and f y respectively represent the spatial frequencies along the x-axis and y-axis directions, and z represents the diffraction distance;

[0023] The second sub-network of the hologram encoder is a complex-valued U-Net. Using the complex amplitude distribution of the SLM plane as the input, it predicts and generates the POH of the SLM plane. This process is expressed as:

[0024]

[0025] where, f U-Net2 [·] represents Sub-network 2 of the dual U-Net cascaded architecture, f ASM (·) represents angular spectrum diffraction propagation, represents POH. Sub-network 2 of the dual U-Net cascaded architecture consists of 4 downsampling blocks, 4 upsampling blocks, and a skip connection module. Each sampling block includes a complex-valued convolutional residual block and a complex-valued convolutional layer. The complex-valued convolutional layer is expressed as:

[0026] W*x = (w r *x r -w i *x i ) + i(w r *x i +w i *x r ),

[0027] Among them, W represents the input complex-valued matrix, x represents the convolution kernel, and the activation function used by the complex-valued U-Net is the complex-valued Relu function (CRelu). The CRelu activation function performs activation operations on the real and imaginary parts respectively, which is expressed as:

[0028] CRelu(W) = Relu(W r ) + iRelu(W i ).

[0029] In an embodiment of the present invention, the hologram decoding stage in step S3 includes:

[0030] Performing angular spectrum diffraction backpropagation through the generated POH to achieve discrete reconstruction of the three-dimensional scene, obtaining a reconstructed image at a specified depth, and the calculation process is expressed as:

[0031]

[0032] Among them, L i (x, y) is the complex amplitude distribution of the i-th layer, and z i is the propagation distance of the i-th layer from the hologram plane. By performing backpropagation calculation of hierarchical angular spectrum diffraction on the POH, a reconstructed image at a specified depth is obtained Among them, z represents the diffraction distance, represents the reconstructed image of the i-th layer, and f LBASM (·, -z) represents the process of hierarchical angular spectrum diffraction backpropagation. The reconstructed image and the target image superimposed with the binary mask corresponding to the corresponding depth are used for loss calculation to optimize the network parameters;

[0033] The reconstructed image and the target image superimposed with the binary mask corresponding to the corresponding depth are used for loss calculation to optimize the network parameters. During the training process, the difference between the reconstructed amplitude and the target amplitude is evaluated through the MSE loss function:

[0034]

[0035] Among them, N represents the total number of pixels.

[0036] The present invention also provides an unsupervised deep learning system for realizing three-dimensional holographic display, including:

[0037] A depth prediction module for establishing a mapping relationship between the two-dimensional image amplitude information and the three-dimensional depth through a pre-trained monocular depth estimation model, generating a depth map corresponding to the two-dimensional image, and uniformly quantizing the depth map to obtain a number of binary masks;

[0038] A hologram encoding module is used to splice the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder. A dual U-Net cascaded neural network architecture is adopted as the hologram encoder. Among them, the first sub-network is a real-valued U-Net for performing the phase information prediction task of the target plane, and the second sub-network is a complex-valued U-Net for performing the pure phase hologram prediction and generation task of the spatial light modulator plane;

[0039] A hologram decoding module is used to perform angular spectrum diffraction backpropagation through the generated POH to achieve three-dimensional scene discretized reconstruction, obtain a reconstructed image at a specified depth, uniformly quantize the depth map to obtain a number of binary masks, calculate the loss between the reconstructed image and the target image superimposed with the corresponding depth binary mask, and optimize the network parameters. When the training of the dual U-Net cascaded neural network architecture converges, the training phase ends.

[0040] As described above, an unsupervised deep learning method for realizing three-dimensional holographic display of the present invention has the following beneficial effects:

[0041] (1) The unsupervised deep learning method for realizing three-dimensional holographic display of the present invention can directly realize three-dimensional holographic display based on a single-frame two-dimensional image through an unsupervised deep learning framework, significantly reducing the dependence of three-dimensional holographic display on prior information, and solving problems such as high data acquisition cost, low calculation efficiency, and large memory consumption in the prior art. Through hierarchical angular spectrum propagation, the present invention can achieve high-quality three-dimensional hologram reconstruction, with high accuracy and detail restoration ability.

[0042] (2) The unsupervised deep learning method for realizing three-dimensional holographic display of the present invention integrates a pre-trained depth prediction model and a hierarchical angular spectrum diffraction reconstruction mechanism, eliminating the dependence of model training on prior information such as depth maps and binocular views, significantly reducing the training cost. The dual U-Net cascaded network has excellent feature extraction ability and learning ability, which helps to generate high-quality POH and improve the quality of three-dimensional holographic display. Description of the Drawings

[0043] Figure 1 It is a system framework diagram of an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application.

[0044] Figure 2 It is a program flow chart of an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application.

[0045] Figure 3 It is a network structure diagram of sub-network 1 of the dual U-Net cascaded architecture of an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application.

[0046] Figure 4 It is a schematic diagram of the network structure of sub-network 2 of the dual U-Net cascaded architecture for an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application. Specific implementation manners

[0047] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0048] This part will describe the specific embodiments of the present invention in detail. The preferred embodiments of the present invention are shown in the drawings. The role of the drawings is to supplement the description in the text part of the specification, enabling people to intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the protection scope of the present invention.

[0049] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, so it cannot be understood as a limitation on the present invention; the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0050] It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the drawings, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0051] Please refer to Figure 1 、 Figure 2 , Figure 1 It is a system framework diagram of an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application. Figure 2This is a program flowchart of an unsupervised deep learning method for realizing three-dimensional holographic display provided by an embodiment of the present application. The present invention provides an unsupervised deep learning method for realizing three-dimensional holographic display, designs a cascaded double U-Net neural network architecture driven by a hierarchical angular spectrum diffraction physical model, and realizes the generation of a three-dimensional pure phase hologram (Phase-Only Hologram, POH) based on a single-frame two-dimensional image. Moreover, this method does not rely on prior information such as depth maps or binocular views, and specifically includes:

[0052] Step S1, depth prediction stage: Establish a mapping relationship between the amplitude information of a two-dimensional image and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map corresponding to the two-dimensional image, and uniformly quantize the depth map to obtain a number of binary masks.

[0053] Step S2, hologram encoding stage: Concatenate the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder. Use a cascaded double U-Net neural network architecture as the hologram encoder. Among them, the first sub-network is a real-valued U-Net for performing the phase information prediction task of the target plane, and the second sub-network is a complex-valued U-Net for performing the pure phase hologram prediction and generation task of the spatial light modulator (SLM) plane.

[0054] Step S3, hologram decoding stage: Realize the discrete reconstruction of the three-dimensional scene through the angular spectrum diffraction backpropagation of the generated POH to obtain a reconstructed image at a specified depth. Uniformly quantize the depth map to obtain a number of binary masks, calculate the loss between the reconstructed image and the target image superimposed with the corresponding depth binary mask, and optimize the network parameters. When the cascaded double U-Net neural network architecture training converges, the training stage ends.

[0055] Step S4, numerical simulation test: Input the test samples into the trained cascaded double U-Net neural network architecture, numerically reconstruct the POH at different diffraction distances according to the depth information, and calculate the peak signal-to-noise ratio (Peak Signal-to Noise Ratio, PSNR) and structural similarity (Structural Similarity, SSIM) between the reconstructed image and the target image superimposed with the corresponding depth binary mask to evaluate the quality of the reconstructed image.

[0056] Step S5, optical experiment test: Build a holographic display optical system and conduct an optical experiment, collect the reconstructed image through an industrial camera, and evaluate the quality of the reconstructed image.

[0057] In one embodiment of the present invention, the dual U-Net cascaded neural network architecture includes two U-Net networks with different structures. The first sub-network is a real-valued U-Net integrating Fourier convolutional blocks, which is used to predict the phase information of the target plane. The second sub-network is a complex-valued U-Net, which is used for the prediction and generation of POH.

[0058] In one embodiment of the present invention, the real-valued U-Net processes the input image through spatial-frequency domain dual-branch convolution. Among them, the spatial branch extracts local spatial features through double-layer convolution, and the frequency domain branch extracts frequency domain features through fast Fourier transform (FFT), and then performs inverse Fourier transform to add the frequency domain features and spatial features to achieve the fusion of frequency domain features and spatial domain features.

[0059] In one embodiment of the present invention, the complex-valued U-Net uses complex-valued convolutional residual blocks. The complex-valued convolutional layer directly processes complex-valued data to improve the hologram encoding ability of the network, and uses the complex-valued Relu function (CRelu) as the activation function to perform activation operations on the real and imaginary parts respectively.

[0060] In one embodiment of the present invention, the complex-valued U-Net adopts a complex-valued sub-pixel convolution strategy during the upsampling process, replacing the invalid zero parameters in the traditional transposed convolution with learnable parameters, enhancing the context information of the network, and thus improving the detail restoration ability of the hologram.

[0061] In one embodiment of the present invention, a hierarchical diffraction reconstruction mechanism is adopted to realize the unsupervised training of the neural network. The three-dimensional scene is discretized into two-dimensional images with different depths. During the hierarchical angular spectrum backpropagation process, by randomly selecting the diffraction distance, the hierarchical angular spectrum propagation calculation of POH is performed to obtain the reconstructed image at the specified depth. The loss is calculated between the reconstructed image and the target image superimposed with the binary mask corresponding to the depth to optimize the network parameters.

[0062] Step S1, depth prediction stage: Construct a mapping relationship between two-dimensional amplitude information and three-dimensional depth through a pre-trained monocular depth estimation model. Specifically, the amplitude A of the two-dimensional target image is used as the feature input. After extracting multi-level visual features by the depth neural network, the corresponding depth map D is output;

[0063] Step S2, hologram encoding stage: Use a dual U-Net cascaded architecture to realize the prediction of the hologram. The hologram encoder consists of two U-Nets with different structures. First, the amplitude of the target image and the depth map obtained in the depth prediction stage are concatenated in the channel dimension as the input feature I of the sub-network 1 of the dual U-Net cascaded architecture. This network learns the joint features of amplitude and depth through a Fourier-inspired encoder-decoder structure and outputs the phase information of the target plane This process can be expressed as:

[0064] I = Cat(A, D),

[0065]

[0066] where f U-Net1 (·) represents Sub-network 1 of the dual U-Net cascaded architecture, and its network structure is as Figure 3 shown.

[0067] After the phase information of the target plane is superimposed with the target amplitude, the complex amplitude distribution of the target plane is obtained. Subsequently, the complex amplitude of the target plane is propagated to the SLM plane through the angular spectrum diffraction method to obtain the complex amplitude distribution of the SLM plane. The mathematical representation of the angular spectrum diffraction method is as follows:

[0068]

[0069] where λ represents the wavelength, f x and f y represent the spatial frequencies along the x-axis and y-axis directions respectively, and z represents the diffraction distance.

[0070] Sub-network 2 of the dual U-Net cascaded architecture takes the complex amplitude of the SLM plane as the input and predicts the corresponding POH. This process can be expressed as:

[0071]

[0072] where f U-Net2 [·] represents Sub-network 2 of the dual U-Net cascaded architecture, f ASM (·) represents the angular spectrum diffraction propagation, represents POH. Sub-network 2 is a complex-valued U-Net structure that directly processes complex-valued data using complex-valued convolutional layers to enhance the hologram encoding ability of the network. The calculation of the complex-valued convolutional layer can be expressed as:

[0073] W * x = (w r * x r - w i * x i ) + i(w r * x i + w i * x r ),

[0074] where W represents the input complex-valued matrix, x represents the convolutional kernel, and the activation function used by the complex-valued U-Net is the complex-valued Relu function (CRelu). The CRelu activation function performs activation operations on the real and imaginary parts respectively and can be expressed as:

[0075] CRelu(W) = Relu(Wr ) + iRelu(W i ),

[0076] The network structure of sub - network 2 of the double U - Net cascaded architecture is as Figure 4 shown;

[0077] Step S3, hologram decoding stage: A hierarchical diffraction reconstruction mechanism is adopted to achieve unsupervised training of the neural network. The calculation process of hierarchical angular spectrum diffraction can be expressed as:

[0078]

[0079] where, L i (x, y) is the complex amplitude distribution of the i - th layer; z i is the propagation distance of the i - th layer from the hologram plane. In the process of hierarchical angular spectrum back - propagation, by randomly selecting the diffraction distance, the hierarchical angular spectrum propagation calculation of the POH is performed to obtain the reconstructed image at the specified depth. This process can be expressed as:

[0080]

[0081] where, z represents the diffraction distance, represents the reconstructed image of the i - th layer, f LBASM (·, - z) represents the process of hierarchical angular spectrum diffraction back - propagation.

[0082] The depth map generated in step S11 is uniformly quantized into a number of binary masks, and the binary mask can be expressed as:

[0083]

[0084] where, d represents the pixel value of the depth map, D i represents the quantization interval, and the loss is calculated between the reconstructed image and the target image superimposed with the corresponding depth binary mask to optimize the network parameters.

[0085] In an embodiment of the present invention, the network training uses the DF2K dataset as the training dataset, uses the Adam optimizer, the learning rate is 0.001, the training period is set to 30 epochs, and the difference between the reconstructed amplitude and the target amplitude is evaluated through the MSE loss function:

[0086]

[0087] where, N represents the total number of pixels.

[0088] Testing part:

[0089] After the neural network training converges, numerical simulation tests and optical experiment tests are performed on the performance of the network.

[0090] Step S4, numerical simulation test: Select 100 two-dimensional images from the validation dataset of the DF2K dataset and input them into the trained neural network model to obtain the POH corresponding to the two-dimensional images. Numerically reconstruct the POH at different diffraction distances according to the depth information, and calculate the PSNR and SSIM between the reconstructed image and the target image corresponding to the depth:

[0091]

[0092]

[0093] where MAX represents the maximum possible value of image pixels, MSE represents the mean square error, x and y respectively represent two images or image regions to be compared, u x and u y respectively represent the local means of images x and y, and respectively represent the local variances of images x and y, σ xy represents the local covariance of images x and y, and C1 and C2 are constants introduced to avoid the denominator being zero. Evaluate the quality of the reconstructed image through PSNR and SSIM. The larger the PSNR value, the higher the quality of the reconstructed image and the smaller the noise; the closer the SSIM value is to 1, the more similar the two images are.

[0094] Step S5, optical experiment test: Build a holographic display optical system and conduct an optical experiment. The SLM used in this system is Holoeye LETO, with a resolution of 1920×1080 pixels and a pixel size of 6.4 μm. Load the POH predicted by the neural network onto the phase-type SLM. The laser beam with a wavelength of 532 nm is expanded and collimated and then irradiated onto the SLM. After the light wave is modulated and reflected by the SLM, it is transmitted to the 4-f system through a beam splitter prism, and the reconstructed images of each depth of the target object are reproduced at different distances and captured by the CCD.

[0095] Similar to the principle of an unsupervised deep learning method for realizing three-dimensional holographic display of the present invention, the present invention also provides an unsupervised deep learning system for realizing three-dimensional holographic display, including:

[0096] A depth prediction module, configured to establish a mapping relationship between the amplitude information of a two-dimensional image and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map corresponding to the two-dimensional image, and uniformly quantize the depth map to obtain a plurality of binary masks;

[0097] A hologram encoding module is used to stitch the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder, and adopts a dual U-Net cascade neural network architecture as the hologram encoder, wherein the first sub-network is a real-valued U-Net, which is used to perform the phase information prediction task of the target plane, and the second sub-network is a complex-valued U-Net, which is used to perform the pure phase hologram prediction and generation task of the spatial light modulator plane;

[0098] The hologram decoding module is used to discretize the 3D scene using the generated POH by backpropagation of angular spectrum diffraction to obtain a reconstructed image at a specified depth. The depth map is uniformly quantized to obtain a number of binary masks. Loss calculations are performed on the reconstructed image and the target image superimposed with the corresponding depth binary masks. Network parameter optimization is performed. The training phase ends when the dual U-Net cascade neural network architecture converges.

[0099] In summary, the unsupervised deep learning method for achieving 3D holographic display in the present invention, through an unsupervised deep learning framework, can achieve 3D holographic display directly from a single 2D image frame, significantly reducing the reliance on prior information for 3D holographic display and addressing the existing issues of high data acquisition costs, low computational efficiency, and high memory consumption. Through layered angular spectrum propagation, the present invention enables high-quality 3D holographic reconstruction with high accuracy and detail restoration capabilities.

[0100] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are intended solely to illustrate the principles of the present invention and are not to be construed in any way as limiting the scope of protection of the present invention. Based on the explanations herein, those skilled in the art will be able to devise other specific embodiments of the present invention without inventive effort, and such equivalent variations or substitutions are intended to be encompassed within the scope of the claims of this application.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised deep learning method for realizing three-dimensional holographic display, characterized in that, Including: S1. Depth prediction stage: Establish a mapping relationship between the two-dimensional image amplitude information and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map corresponding to the two-dimensional image, and uniformly quantize the depth map to obtain a number of binary masks; S2. Hologram encoding stage: Concatenate the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder. Use a double U-Net cascaded neural network architecture as the hologram encoder. Among them, the first sub-network is a real-valued U-Net for performing the phase information prediction task of the target plane, and the second sub-network is a complex-valued U-Net for performing the pure phase hologram prediction and generation task of the spatial light modulator plane; S3. Hologram decoding stage: Realize the discrete reconstruction of the three-dimensional scene through angular spectrum diffraction backpropagation of the generated POH, obtain the reconstructed image at the specified depth, uniformly quantize the depth map to obtain a number of binary masks, calculate the loss between the reconstructed image and the target image superimposed with the corresponding depth binary mask, and optimize the network parameters. When the double U-Net cascaded neural network architecture training converges, the training stage ends.

2. The unsupervised deep learning method for realizing three-dimensional holographic display according to claim 1, wherein Also including: S4. Numerical simulation test: Input the test sample into the trained double U-Net cascaded neural network architecture, numerically reconstruct the POH at different diffraction distances according to the depth information, and calculate the peak signal-to-noise ratio and structural similarity between the reconstructed image and the target image superimposed with the corresponding depth binary mask to evaluate the quality of the reconstructed image; S5. Optical experiment test: Build a holographic display optical system and conduct optical experiments, collect the reconstructed image through an industrial camera, and evaluate the quality of the reconstructed image.

3. An unsupervised deep learning method for realizing three-dimensional holographic display according to claim 2, characterized in that, The depth prediction stage in step S1 includes: Establish a mapping relationship between the two-dimensional image amplitude information and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map D corresponding to the two-dimensional image A, uniformly quantize the depth map to obtain a number of binary masks, and the binary mask is expressed as: where d represents the pixel value of the depth map, and D i represents the quantization interval.

4. An unsupervised deep learning method for realizing three-dimensional holographic display according to claim 3, characterized in that, The hologram encoding stage in step S2 includes: The depth map and the two-dimensional image are concatenated in the channel dimension as the input I of the hologram encoder. A dual U-Net cascaded neural network architecture is used as the hologram encoder. Among them, the first sub-network is a real-valued U-Net for performing the phase information prediction task, expressed as: I = Cat(A, D), Among them, f U-Net1 (·) represents Sub-network 1 of the double U-Net cascaded architecture. This network consists of an initial convolutional block, 4 downsampling blocks, 4 upsampling blocks, and an output layer. Each sampling block contains a spatial-frequency domain dual-branch convolution to achieve the fusion of frequency domain features and spatial domain features. The spatial branch of the spatial-frequency domain dual-branch convolution extracts local spatial features through a double-layer 3×3 convolution. The frequency domain branch performs a fast Fourier transform on the input, decomposes the real or imaginary part, performs frequency domain filtering through a 1×1 convolution, and after restoration by an inverse fast Fourier transform, adds it to the spatial features to achieve the fusion of frequency domain to spatial domain features; The sub-network 1 of the double U-Net cascade architecture executes the phase information of the target plane After being superimposed with the target amplitude, the complex amplitude distribution of the target plane is obtained. By using the angular spectrum diffraction method, the complex amplitude of the target plane is propagated to the SLM plane, and the complex amplitude distribution of the SLM plane is obtained. The mathematical representation of the angular spectrum diffraction method is as follows: where λ represents the wavelength, f x and f y represent the spatial frequencies in the x-axis and y-axis directions respectively, and z represents the diffraction distance; The second sub-network of the hologram encoder is a complex-valued U-Net, taking the complex amplitude distribution of the SLM plane as the input, predicting and generating the POH of the SLM plane, and this process is expressed as: Among them, f U-Net2 [·] represents Sub-network 2 of the dual U-Net cascaded architecture, and f ASM (·) represents angular spectrum diffraction propagation, represents POH. Sub-network 2 of the dual U-Net cascaded architecture consists of 4 downsampling blocks, 4 upsampling blocks and a skip connection module. Each sampling block includes a complex-valued convolutional residual block and a complex-valued convolutional layer, and the complex-valued convolutional layer is expressed as: W*x=(w r *x r -w i *x i )+i(w r *x i +w i *x r ), Among them, W represents the input complex-valued matrix, x represents the convolution kernel, and the activation function used by the complex-valued U-Net is the complex-valued Relu function (CRelu). The CRelu activation function activates the real part and the imaginary part respectively, and is expressed as: CRelu(W) = Relu(W r ) + iRelu(W i )。 5. An unsupervised deep learning method for realizing three-dimensional holographic display according to claim 4, characterized in that, The hologram decoding stage in step S3 includes: Realize the discrete reconstruction of the three-dimensional scene through angular spectrum diffraction backpropagation of the generated POH, obtain the reconstructed image at the specified depth, and the calculation process is expressed as: Among them, L i (x, y) is the complex amplitude distribution of the i-th layer, and z i is the propagation distance of the i-th layer from the hologram plane. By performing the backward propagation calculation of the angular spectrum diffraction of the POH layer by layer, the reconstructed image at the specified depth is obtained Among them, z represents the diffraction distance, represents the reconstructed image of the i-th layer, and f LBASM (·, -z) represents the backward propagation process of the angular spectrum diffraction layer by layer. The loss is calculated between the reconstructed image and the target image superimposed with the binary mask corresponding to the depth, and the optimization of the network parameters is realized; Calculate the loss between the reconstructed image and the target image superimposed with the corresponding depth binary mask to realize the optimization of the network parameters. During the training process, evaluate the difference between the reconstructed amplitude and the target amplitude through the MSE loss function: Among them, N represents the total number of pixels.

6. An unsupervised deep learning system for realizing three-dimensional holographic display, characterized in that, Including: A depth prediction module, which is used to establish a mapping relationship between the two-dimensional image amplitude information and the three-dimensional depth through a pre-trained monocular depth estimation model, generate a depth map corresponding to the two-dimensional image, and uniformly quantize the depth map to obtain a number of binary masks; A hologram encoding module, which is used to splice the depth map and the two-dimensional image in the channel dimension as the input of the hologram encoder. A dual U-Net cascaded neural network architecture is used as the hologram encoder. Among them, the first sub-network is a real-valued U-Net, which is used to perform the phase information prediction task of the target plane, and the second sub-network is a complex-valued U-Net, which is used to perform the pure phase hologram prediction and generation task of the spatial light modulator plane; A hologram decoding module, which is used to realize the discrete reconstruction of the three-dimensional scene through angular spectrum diffraction backpropagation of the generated POH, obtain a reconstructed image at a specified depth, uniformly quantize the depth map to obtain a number of binary masks, calculate the loss between the reconstructed image and the target image superimposed with the corresponding depth binary mask, and optimize the network parameters. When the training of the dual U-Net cascaded neural network architecture converges, the training stage ends.

Images