Deep learning-based panoramic holographic image inter-frame motion compensation method

Abstract

The invention designs an all-round holographic image inter-frame motion compensation method based on deep learning, and belongs to the field of holographic imaging and computer-generated hologram rapid generation methods, and the method comprises the steps: setting an interval between visual angles, and shooting an intensity image and a depth image of a three-dimensional scene under a sparse horizontal visual angle; making a data set containing sparse and dense horizontal view angle information, training a DeepStereo network model, and then inputting an intensity map and a depth map under the sparse view angle into the trained model; for each sparse view angle, predicting a hologram in an adjacent dense view angle range, and finally generating a complete panoramic holographic image; and loading to a panoramic holographic display system to realize a high-fidelity naked-eye dynamic panoramic three-dimensional display effect. According to the method, inter-frame motion compensation of the panoramic holographic image is quickly and efficiently completed by using the neural network model, the complexity of data acquisition and a processing end is remarkably reduced, and high-fidelity dynamic panoramic three-dimensional reconstruction is realized by using the parallel computing capability and the iterative optimization effect of the network, so that the real-time performance of the panoramic holographic image is improved. And the watching experience and the application value of the naked eye surround-view three-dimensional display are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of holographic imaging and methods for rapid generation of computational holograms, and particularly to a deep learning-based method for inter-frame motion compensation of panoramic holographic images. Background Technology

[0002] Holographic displays are widely recognized as the ideal ultimate solution for 3D displays because they can completely reproduce all depth cues required by the human visual system. Among these, computational holography, which replaces complex optical recording processes with digital means and enables the holographic reproduction of virtual objects, has become the mainstream development direction in this field. However, limited by the spatial bandwidth product of commercial spatial light modulators, existing systems suffer from narrow field of view and insufficient motion parallax in reconstructed images. To overcome this bottleneck, 360-degree holographic display technology has emerged, employing a combination of spatial division multiplexing and time division multiplexing to form a ring-shaped viewing area by seamlessly stitching together multiple viewing windows. This technology requires calculating holograms for hundreds of horizontal viewpoints to ensure the continuity of motion parallax when the observer's viewpoint changes. These holograms, serving as the base frames for panoramic holographic video, need to be continuously displayed on digital micromirror devices at ultra-high refresh rates. However, the massive computational load severely restricts the realization of 360-degree dynamic holographic displays, which has become a core technical challenge that urgently needs to be solved.

[0003] To date, various fast algorithms for hologram computation have been proposed, mainly including point source methods, line element methods, surface element methods, tomography methods, and computational methods based on graphics processing units (GPUs) and deep learning. However, most of these methods focus on accelerating the computation of single holograms, meaning that generating surround-view holographic videos containing massive numbers of frames still requires frame-by-frame computation. As the number of frames increases, the computation time grows linearly, becoming a major bottleneck restricting the development of surround-view dynamic display applications. To overcome this problem, some researchers have drawn inspiration from traditional image motion compensation schemes and proposed motion compensation theories for holographic images. For example, they utilize the translation invariance of lookup tables to compensate for translational motion in holograms, and utilize the rotational properties of spherical holograms to compensate for three-dimensional rotational motion in holograms. However, these methods are limited by two factors: firstly, the inherent properties of diffraction—any movement of an object will cause an overall change in the hologram; and secondly, the inherent limitations of motion compensation theory—the occlusion relationships between object points change with movement, resulting in inherent missing information in motion compensation. Therefore, current motion compensation schemes for surround-view holographic images suffer from low compensation efficiency and poor image quality. It is therefore crucial to provide a method for efficiently achieving high-fidelity inter-frame motion compensation in surround-view holographic images. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a deep learning-based inter-frame motion compensation method for panoramic holographic images, in order to solve the problems of poor image quality and low compensation efficiency caused by the inherent theoretical defects of existing motion compensation schemes.

[0005] This invention provides a deep learning-based method for inter-frame motion compensation in surround-view holographic images, the method comprising the following steps:

[0006] Set the viewing interval and capture intensity and depth maps of the 3D scene under sparse horizontal viewing angles;

[0007] Create a dataset containing sparse and dense horizontal viewpoint information, train a DeepStereo network model, and then input the intensity map and depth map under the sparse viewpoint into the trained model.

[0008] For each sparse viewpoint, predict the hologram within its adjacent dense viewpoint range, and finally generate a complete surround view hologram;

[0009] Loaded onto a surround-view holographic display system, it achieves a high-fidelity naked-eye dynamic surround-view 3D display effect.

[0010] Furthermore, the three-dimensional scene includes a real three-dimensional scene and a virtual three-dimensional scene.

[0011] Furthermore, for real 3D scenes, intensity maps and depth maps can be obtained using any of the passive ranging sensing technologies such as binocular stereo vision and monocular depth estimation, or active ranging sensing technologies such as TOF cameras, structured light scanning, and laser scanning; for virtual 3D scenes, intensity maps and depth maps can be obtained using any of the professional modeling and rendering software such as 3ds Max and Blender.

[0012] Furthermore, the DeepStereo network model training process includes dataset creation and model training.

[0013] Furthermore, for dataset creation, intensity and depth maps of the 3D scene need to be collected separately under sparse and dense viewpoints. Specifically, the information from the sparse viewpoint is normalized to create the original dataset; the information from the dense viewpoint is layered based on the depth map, and the focusing depth of the imaging device is adjusted to obtain focused images at different layer depths, which are then normalized to create the target dataset.

[0014] Furthermore, for model training, the original dataset is used as input, and the DeepStereo network is used to predict holograms under dense viewpoints. Then, the predicted reconstructed images of the holograms at different depths are calculated through angular spectral diffraction propagation and compared with the target dataset to construct a loss function. After that, the parameters of the DeepStereo network are updated. Through multiple iterations, the similarity between the predicted reconstructed image and the target dataset reaches a predetermined value.

[0015] Furthermore, the predicted hologram type includes any one of amplitude holograms, phase holograms, and complex amplitude holograms.

[0016] Furthermore, the loss function can be any one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), or Root Mean Square Error (RMS).

[0017] Furthermore, the surround-view holographic display system structure includes any one of time-division multiplexing structure, space-division multiplexing structure, and spatiotemporal multiplexing structure; the surround-view holographic display system type includes any one of monochrome surround-view holographic display system and color surround-view holographic display system.

[0018] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0019] 1. The neural network used in this invention only requires input of information about the 3D scene under sparse viewpoints to predict holograms under dense viewpoints. Throughout the entire holographic image generation process, complex camera arrays or turntables are not needed to capture the 3D scene under dense viewpoints, reducing the complexity of the surround-view holographic 3D display system at the data acquisition and processing end. This facilitates the system's portability, miniaturization, and integration, enabling its application in numerous scenarios.

[0020] 2. The convolutional neural network used in this invention has powerful parallel computing capabilities, which greatly improves the generation efficiency of holographic images and enables panoramic dynamic holographic three-dimensional display.

[0021] 3. In the training process of the neural network, this invention incorporates an angular spectrum diffraction propagation layer to supervise the network. Angular spectrum diffraction is used to reconstruct the reproduction effect of the predicted hologram at different depths, and the results are compared with the target dataset in the dataset. The network parameters are then optimized and updated using the obtained loss function, thereby improving the accuracy of the network and the quality of the predicted hologram, ultimately achieving high-fidelity inter-frame motion compensation.

[0022] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts:

[0024] Figure 1 This is a flowchart of the deep learning-based inter-frame motion compensation method for surround-view holographic images according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of information acquisition under a sparse viewpoint in a three-dimensional scene according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of information acquisition under dense perspective in a three-dimensional scene according to an embodiment of the present invention;

[0027] Figure 4 This is a flowchart illustrating the dataset creation process for the DeepStereo network model in this embodiment of the invention.

[0028] Figure 5 This is a flowchart illustrating the training process of the DeepStereo network model according to an embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram of the surround-view holographic image generation process based on the DeepStereo network model in an embodiment of the present invention; Detailed Implementation

[0030] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0031] Example 1

[0032] One embodiment of the present invention discloses a deep learning-based method for inter-frame motion compensation in surround-view holographic images. For example... Figure 1 As shown, the method includes the following steps:

[0033] Step S1: Set the viewing angle interval and capture the intensity map and depth map of the 3D scene under a sparse horizontal viewing angle;

[0034] Step S2: Create a dataset containing sparse and dense horizontal viewpoint information, train the DeepStereo network model, and then input the intensity map and depth map from S1 into the trained model.

[0035] Step S3: For each sparse viewpoint, predict the hologram within its adjacent dense viewpoint range, and finally generate a complete surround view holographic image;

[0036] Step S4: Load the image onto the surround-view holographic display system to achieve a high-fidelity naked-eye dynamic surround-view 3D display effect.

[0037] Specifically, in step S1, the three-dimensional scene includes a real three-dimensional scene and a virtual three-dimensional scene. For the real three-dimensional scene, the intensity map and depth map can be obtained using any one of the passive ranging sensing technologies such as binocular stereo vision and monocular depth estimation, or active ranging sensing technologies such as TOF cameras, structured light scanning, and laser scanning. For the virtual three-dimensional scene, the intensity map and depth map can be obtained using any one of the professional modeling and rendering software such as 3ds Max and Blender.

[0038] For example, such as Figure 2 As shown in Figure 11, for a real 3D scene 11, m binocular RGB-Depth cameras are used, as shown in Figures 12, 13, and 14. They are arranged in a ring above the 3D scene with a preset tilt angle and the horizontal viewing angle interval between the cameras is θ. The intensity information and depth information of the 3D scene under the sparse horizontal viewing angle mθ are captured and collected in sequence.

[0039] Specifically, in step S2, the DeepStereo network model training process includes dataset creation and model training. During dataset creation, it is necessary to collect intensity maps and depth maps of the 3D scene under both sparse and dense viewpoints.

[0040] For example, Figure 3 As shown in Figures 12, 13, and 14, m×n binocular RGB-Depth cameras are used, arranged in a ring above the 3D scene at a preset tilt angle, with a horizontal viewing angle interval of θ between the cameras. p Capture and collect intensity and depth information of a 3D scene from m×n dense horizontal viewpoints, where n = θ / θ p .

[0041] Furthermore, information from a sparse perspective is used to create the original dataset, while information from a dense perspective is used to create the target dataset.

[0042] For example, such as Figure 4As shown, information from sparse perspectives is normalized to create the original dataset; information from dense perspectives is layered based on the depth map, and then the focusing depth of the shooting device is adjusted to obtain focused images at different layer depths, which are then normalized to create the target dataset.

[0043] Specifically, during model training, the original dataset is used as input, and the holograms predicted by the network under dense perspectives are simulated and reproduced. By comparing with the target dataset, the network parameters are iterated and optimized.

[0044] For example, such as Figure 5 As shown, the intensity map and depth map under the sparse horizontal view are input, and the DeepStereo network is used to predict the hologram under the dense view. Then, the predicted reconstruction images of the hologram at different depths are calculated by angular spectral diffraction propagation, and compared with the standard images at the corresponding depths to construct a loss function. Then, the parameters of the DeepStereo network are updated. Through multiple iterations, the similarity between the predicted reconstruction image and the target dataset reaches a predetermined value.

[0045] Furthermore, the predicted hologram type includes any one of amplitude holograms, phase holograms, and complex amplitude holograms.

[0046] Furthermore, the loss function can be any one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), or Root Mean Square Error (RMS).

[0047] Specifically, in step S3, the DeepStereo network model is used to predict the holograms within the adjacent view range of each sparse level, and then they are integrated to generate a complete surround-view holographic image.

[0048] For example, such as Figure 6 As shown, the input consists of m sparse horizontal viewpoints for 3D information. For the 3D scene information under the sparse viewpoint (mθ), the network predicts the hologram within the range of the dense viewpoint [(m-1 / 2)*θ, (m+1 / 2)*θ]. Through the above method, m×n holograms under the dense horizontal viewpoints can be generated. They are then merged according to the viewpoint order to generate a complete panoramic holographic image.

[0049] Specifically, in step S4, the surround holographic display system structure includes any one of time-division multiplexing structure, space-division multiplexing structure, and spatiotemporal multiplexing structure; the surround holographic display system type includes any one of monochrome surround holographic display system and color surround holographic display system.

[0050] Compared with existing technologies, the beneficial effects of the deep learning-based inter-frame motion compensation method for surround-view holographic images provided by this invention are as follows:

[0051] 1. The neural network used in this invention only requires input of information about the 3D scene under sparse viewpoints to predict holograms under dense viewpoints. Throughout the entire holographic image generation process, complex camera arrays or turntables are not needed to capture the 3D scene under dense viewpoints, reducing the complexity of the surround-view holographic 3D display system at the data acquisition and processing end. This facilitates the system's portability, miniaturization, and integration, enabling its application in numerous scenarios.

[0052] 2. The convolutional neural network used in this invention has powerful parallel computing capabilities, which greatly improves the generation efficiency of holographic images and enables panoramic dynamic holographic three-dimensional display.

[0053] 3. In the training process of the neural network, this invention incorporates an angular spectrum diffraction propagation layer to supervise the network. Angular spectrum diffraction is used to reconstruct the reproduction effect of the predicted hologram at different depths, and the results are compared with the target dataset in the dataset. The network parameters are then optimized and updated using the obtained loss function, thereby improving the accuracy of the network and the quality of the predicted hologram, ultimately achieving high-fidelity inter-frame motion compensation.

[0054] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0055] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-based method for inter-frame motion compensation in surround-view holographic images, characterized in that, The method includes the following steps: Set the viewing interval and capture intensity and depth maps of the 3D scene under sparse horizontal viewing angles; Create a dataset containing sparse and dense horizontal viewpoint information, train a DeepStereo network model, and then input the intensity map and depth map under the sparse viewpoint into the trained model. For each sparse viewpoint, predict the hologram within its adjacent dense viewpoint range, and finally generate a complete surround view hologram; Loaded onto a surround-view holographic display system, it achieves a high-fidelity naked-eye dynamic surround-view 3D display effect.

2. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 1, characterized in that, The three-dimensional scene includes both real three-dimensional scenes and virtual three-dimensional scenes.

3. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 2, characterized in that, For real 3D scenes, intensity maps and depth maps can be obtained using any of the following methods: passive ranging sensing technologies such as binocular stereo vision and monocular depth estimation; active ranging sensing technologies such as TOF cameras, structured light scanning, and laser scanning. For virtual 3D scenes, intensity maps and depth maps can be obtained using any of the following professional modeling and rendering software such as 3ds Max and Blender.

4. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 1, characterized in that, The DeepStereo network model training process includes dataset creation and model training.

5. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 4, characterized in that, To create the dataset, intensity and depth maps of the 3D scene need to be collected separately under sparse and dense viewpoints. For the information under sparse viewpoints, it is normalized to create the original dataset. For the information under dense viewpoints, it is layered according to the depth map, and then the focus depth of the shooting device is adjusted to obtain focused images at different layer depths. These images are then normalized to create the target dataset.

6. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 4, characterized in that, For model training, the original dataset is used as input, and the DeepStereo network is used to predict holograms under dense viewpoints. Then, the predicted reconstructed images of the holograms at different depths are calculated through angular spectral diffraction propagation and compared with the target dataset. A loss function is constructed, and then the parameters of the DeepStereo network are updated. Through multiple iterations, the similarity between the predicted reconstructed image and the target dataset reaches a predetermined value.

7. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 6, characterized in that, The predicted hologram type includes any one of amplitude holograms, phase holograms, and complex amplitude holograms.

8. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 6, characterized in that, The loss function can be any one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), or Root Mean Square Error (RMS).

9. The deep learning-based inter-frame motion compensation method for surround-view holographic images according to claim 1, characterized in that, The surround-view holographic display system structure includes any one of time-division multiplexing structure, space-division multiplexing structure, and spatiotemporal multiplexing structure; the surround-view holographic display system type includes any one of monochrome surround-view holographic display system and color surround-view holographic display system.

Images