Three-dimensional holographic image reconstruction method, system, device and storage medium

Abstract

The present application discloses a 3D holographic image reconstruction method, system, device and storage medium, which belongs to the field of optical display and artificial intelligence; the method comprises: acquiring a data set, constructing a 3D holographic reconstruction network model, and training and testing the 3D holographic reconstruction network model according to the data set; acquiring a target depth map and a target intensity map of a target scene, and splicing the target intensity map and the target depth map into a target tensor map; processing the unit target tensor map according to the 3D holographic reconstruction network model to obtain a 3D hologram of the unit target scene; processing the unit target 3D hologram according to a preset optical 3D reconstruction device to obtain a 3D reconstructed image of the unit target scene; processing the tensor map by an improved U-net subnetwork and an improved fully convolutional residual subnetwork in the 3D holographic reconstruction network model, so as to obtain a 3D holographic image at a high speed and accurately.

Description

Technical Field

[0001] The present application belongs to the field of optical display and artificial intelligence technology, and specifically relates to a three-dimensional holographic image reconstruction method, system, device and storage medium. Background Art

[0002] In recent years, three-dimensional display technology has flourished. Compared to two-dimensional displays, three-dimensional displays can provide richer information. Real-time three-dimensional reconstruction of the real-world environment, a key technology in the metaverse and the Internet of Things, holds broad application prospects. With the emergence of applications such as remote surgery, remote education, and remote conferencing, the demand for real-time acquisition and reconstruction of the real environment is increasing. Conventional methods of displaying two-dimensional images on flat screens suffer from issues such as a fixed local user perspective and a limited shared field of view, making them incapable of providing an immersive user experience.

[0003] Common 3D imaging technologies include light field imaging, time-of-flight imaging, and structured light imaging. 3D displays can be achieved through holographic 3D displays, binocular vision displays, volumetric 3D displays, and light field displays. Holographic 3D display technology utilizes wavefront information to reconstruct 3D scenes. Holograms contain amplitude and phase information of the 3D scene, enabling complete reconstruction of the scene's intensity and depth. Therefore, holographic 3D displays are considered the ultimate solution for 3D display.

[0004] However, one of the challenges facing three-dimensional holographic display is the lack of three-dimensional information content. Common three-dimensional content acquisition methods used for three-dimensional holographic reconstruction have problems such as high system complexity and long time consumption. Therefore, it is necessary to develop a real-time three-dimensional scene acquisition and three-dimensional holographic reconstruction method with a simple system and fast imaging speed. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a three-dimensional holographic image reconstruction method, system, device and storage medium, which can solve the technical problems of high system complexity and long time consumption in three-dimensional content acquisition methods.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional holographic image reconstruction method, the method comprising:

[0008] Acquire depth maps and intensity maps of multiple different scenes, and concatenate the depth maps and intensity maps corresponding to each scene to obtain a tensor map for each scene, so as to construct a data set based on the multiple tensor maps;

[0009] Constructing a three-dimensional holographic reconstruction network model, wherein the three-dimensional holographic reconstruction network model includes an improved U-net subnetwork, an angular spectrum algorithm, and an improved full convolution residual subnetwork;

[0010] Training the three-dimensional holographic reconstruction network model according to a portion of the data in the data set, and testing the trained three-dimensional holographic reconstruction network model according to the other portion of the data;

[0011] Acquire a target depth map and a target intensity map of a target scene, and splice the target intensity map and the target depth map into a target tensor map;

[0012] Processing the target tensor map according to the tested qualified three-dimensional holographic reconstruction network model to obtain a target three-dimensional hologram of the target scene;

[0013] The three-dimensional hologram of the target is processed according to a preset optical three-dimensional reconstruction device to obtain a three-dimensional reconstructed image of the target scene.

[0014] As an optional implementation of the first aspect of the present application, the three-dimensional holographic reconstruction network model processes the target tensor map to obtain a target three-dimensional hologram of the target scene, specifically:

[0015] Processing the target tensor graph according to the improved U-net subnetwork to obtain target phase amplitude features;

[0016] Processing the target phase and amplitude characteristics according to the angular spectrum algorithm to obtain holographic phase and amplitude characteristics;

[0017] The holographic phase amplitude features are processed according to the improved full convolution residual subnetwork to obtain the target three-dimensional hologram.

[0018] As an optional implementation manner of the first aspect of the present application, the improved U-net subnetwork has a four-layer structure, including four downsampling units with the same structure and channels increasing sequentially, and four upsampling units with the same structure corresponding to the number of channels of the downsampling units; the outputs of the first three downsampling units are respectively jump-connected to the inputs of the last three upsampling units;

[0019] The process of data processing by the downsampling unit is specifically as follows:

[0020] performing batch normalization and activation processing on the downsampled input features input to the downsampled unit in sequence according to the batch normalization layer and the activation layer to obtain a first activated feature;

[0021] Performing a 3×3 two-dimensional convolution process on the first activation feature according to the first convolution layer to obtain a first convolution feature;

[0022] performing batch normalization and activation processing on the first convolutional features in sequence according to a batch normalization layer and an activation layer to obtain a second activation feature;

[0023] Performing a 3×3 two-dimensional convolution process on the second activation feature according to the second convolution layer to obtain a second convolution feature;

[0024] Performing 1×1 two-dimensional convolution processing on the downsampled input features according to the third convolution layer to obtain a third convolution feature, and performing a residual connection on the second convolution feature and the third convolution feature to obtain an output feature of the downsampling unit.

[0025] As an optional implementation manner of the first aspect of the present application, the process of the upsampling unit processing data is specifically as follows:

[0026] performing batch normalization and activation processing on the upsampled input features input to the upsampling unit in sequence according to the batch normalization layer and the activation layer to obtain a third activated feature;

[0027] Performing a 3×3 two-dimensional transposed convolution process on the third activation feature according to the first transposed convolution layer to obtain a first transposed convolution feature;

[0028] performing batch normalization and activation processing on the first transposed convolutional feature in sequence according to a batch normalization layer and an activation layer to obtain a fourth activated feature;

[0029] Performing a 3×3 two-dimensional convolution process on the fourth activation feature according to the second convolution layer to obtain a fourth convolution feature;

[0030] The up-sampled input features are subjected to 2×2 two-dimensional transposed convolution processing according to the second transposed convolution layer to obtain second transposed convolution features, and the second transposed convolution features are residually connected with the fourth convolution features to obtain output features of the up-sampling unit.

[0031] As an optional implementation of the first aspect of the present application, the improved fully convolutional residual subnetwork includes multiple first convolution units and one second convolution unit with the same structure; the holographic phase amplitude feature is the input of the first of the first convolution units, and the holographic phase amplitude feature is jump-connected to the second convolution unit;

[0032] The data processing process of the first convolution unit is specifically as follows:

[0033] Performing a 3×3 two-dimensional convolution on the first input feature of the first convolution unit according to the second convolution layer to obtain a fifth convolution feature;

[0034] The fifth convolutional feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the fifth activation feature;

[0035] Performing a 3×3 two-dimensional convolution process on the fifth activation feature according to the second convolution layer to obtain a sixth convolution feature;

[0036] The sixth convolutional feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the sixth activated feature;

[0037] Perform a residual connection on the sixth activation feature and the first input feature to obtain a first output feature of the first convolutional unit.

[0038] As an optional implementation manner of the first aspect of the present application, the data processing process of the second convolution unit is specifically as follows:

[0039] Performing a 3×3 two-dimensional convolution on the second input feature of the second convolution unit according to the second convolution layer to obtain a seventh convolution feature;

[0040] Performing a residual connection between the seventh convolution feature and the target phase amplitude feature to obtain a first residual feature;

[0041] Performing a 3×3 two-dimensional convolution process on the first residual feature according to the second convolution layer to obtain an eighth convolution feature;

[0042] The eighth convolution feature is processed according to a Tanh activation function to obtain the target three-dimensional hologram.

[0043] As an optional implementation of the first aspect of the present application, the optical 3D reconstruction device processes the target 3D hologram to obtain a 3D reconstructed image of the target scene, specifically:

[0044] The target three-dimensional hologram is input into the spatial light modulator in the optical three-dimensional reconstruction device for loading, while the laser in the image reconstruction device continuously emits red laser light;

[0045] The red laser beam is expanded and then directed by a prism so that the expanded red laser beam is incident vertically into the effective area of ​​the spatial light modulator.

[0046] The spatial light modulator loaded with the target three-dimensional hologram modulates the incoming red laser light wave and reflects the modulated red laser light to the 4f system to eliminate the zero-order beam, thereby obtaining a three-dimensional reconstructed image of the target scene;

[0047] The CCD camera moves along the reconstructed optical axis to display the three-dimensional reconstructed image of the target scene.

[0048] In a second aspect, an embodiment of the present application provides a three-dimensional holographic image reconstruction system, the system comprising:

[0049] The first acquisition module acquires depth maps and intensity maps of multiple different scenes, and concatenates the depth map and intensity map corresponding to each scene to obtain a tensor map of each scene, so as to construct a data set based on the multiple tensor maps;

[0050] Construction module: constructing a 3D holographic reconstruction network model, which includes an improved U-net subnetwork, an angular spectrum algorithm, and an improved full convolution residual subnetwork;

[0051] Training module: training the 3D holographic reconstruction network model according to a portion of the data in the data set, and testing the trained 3D holographic reconstruction network model according to the other portion of the data;

[0052] The second acquisition module is configured to acquire a target depth map and a target intensity map of the target scene, and to splice the target intensity map and the target depth map into a target tensor map;

[0053] Processing module: processing the target tensor map according to the three-dimensional holographic reconstruction network model that has passed the test to obtain a target three-dimensional hologram of the target scene;

[0054] Imaging module: processes the target three-dimensional hologram according to a preset optical three-dimensional reconstruction device to obtain a three-dimensional reconstructed image of the target scene.

[0055] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0056] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0057] Compared with the prior art, the embodiments of the present invention have the following technical effects:

[0058] (1) Through the collaboration between the depth camera, the 3D holographic reconstruction network model, and the optical 3D reconstruction device, real-time 3D scene reconstruction can be achieved through a simple system structure;

[0059] (2) By processing the tensor graph through the improved U-net subnetwork and the improved fully convolutional residual subnetwork in the three-dimensional holographic reconstruction network model, a three-dimensional holographic image can be obtained quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1is a flow chart of a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0061] Figure 2 This is a diagram of an improved U-net subnetwork structure in a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0062] Figure 3 This is a structural diagram of an improved full convolution residual subnetwork in a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0063] Figure 4 is a structural diagram of a downsampling unit in a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0064] Figure 5 is a structural diagram of an upsampling unit in a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0065] Figure 6 is a structural diagram of a first convolution unit in a three-dimensional holographic image reconstruction method provided by some embodiments of the present application;

[0066] Figure 7 This is a structural diagram of the second convolution unit in a three-dimensional holographic image reconstruction method provided in some embodiments of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0068] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0069] The following, in conjunction with the accompanying drawings, describes in detail a three-dimensional holographic image reconstruction method, system, device, and storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0070] A three-dimensional holographic image reconstruction method comprises the following steps:

[0071] S100: Acquire depth maps and intensity maps of multiple different scenes, and splice the depth map and intensity map corresponding to each scene to obtain a tensor map of each scene, so as to construct a data set based on the multiple tensor maps;

[0072] It should be understood that the depth camera continuously captures continuous frames of RGB-D images of multiple real scenes, extracts the red channel and depth information in the RGB-D image as an intensity map and a depth map, and then splices the intensity map and depth map corresponding to each frame of the RGB-D image to obtain a tensor map corresponding to each frame of the RGB-D image. A continuous frame of RGB-D image is captured in each real scene, so each real scene corresponds to multiple continuous tensor maps; finally, a data set is constructed based on multiple continuous tensor maps corresponding to multiple real scenes. The depth map and intensity map in the data set are one-to-one corresponding, corresponding to the same frame of RGB-D image;

[0073] S200: Construct a 3D holographic reconstruction network model, which includes an improved U-net sub-network, an angular spectrum algorithm, and an improved fully convolutional residual sub-network;

[0074] S300: training a 3D holographic reconstruction network model based on a portion of the data in the data set, and testing the trained 3D holographic reconstruction network model based on the other portion of the data;

[0075] It should be understood that the data set is divided into a training set and a test set according to a preset 3:1 ratio, the three-dimensional holographic reconstruction network model is trained according to the test set, and then the three-dimensional holographic reconstruction network model after training is tested according to the test set; secondly, during the training process, the angular spectrum algorithm in the three-dimensional holographic reconstruction network model also plays a self-supervisory role in the three-dimensional holographic reconstruction network model process.

[0076] S400: Acquire a target depth map and a target intensity map of a target scene, and stitch the target intensity map and the target depth map into a target tensor map;

[0077] S500: processing the target tensor map according to the tested qualified 3D holographic reconstruction network model to obtain a target 3D hologram of the target scene;

[0078] Furthermore, in S500, the 3D holographic reconstruction network model processes the target tensor map to obtain a target 3D hologram of the target scene, specifically:

[0079] S510: Processing the target tensor graph according to the improved U-net sub-network to obtain target phase amplitude features;

[0080] S520: Processing the target phase and amplitude characteristics according to the angular spectrum algorithm to obtain holographic phase and amplitude characteristics;

[0081] S530: Processing the holographic phase amplitude features according to the improved fully convolutional residual subnetwork to obtain a target three-dimensional hologram;

[0082] Among them, the improved U-net subnetwork in S510 is a four-layer structure, including four downsampling units with the same structure and channels increasing in sequence, and four upsampling units with the same structure corresponding to the number of channels of the downsampling units; in the improved U-net subnetwork, the outputs of the first three downsampling units are jump-connected with the inputs of the last three upsampling units respectively; that is, the output of the first downsampling unit is jump-connected to the input of the last upsampling unit (the fourth upsampling unit), the output of the second downsampling unit is jump-connected to the input of the third upsampling unit, and the output of the third downsampling unit is jump-connected to the input of the second upsampling unit; the improved full convolution residual subnetwork in S530 includes multiple first convolution units with the same structure and one second convolution unit, the holographic phase amplitude feature is the input of the first convolution unit, and the holographic phase amplitude feature is jump-connected to the second convolution unit. In this embodiment, the number of first convolution units is 10.

[0083] It should be understood that the target tensor graph of size 1×2×1024×1024 is input into the improved U-net subnetwork for encoding processing. First, the target tensor graph is downsampled by the first downsampling unit to obtain the output features of the first downsampling unit. The output features of the first downsampling unit are then used as the input of the second downsampling unit. At the same time, the output features of the first downsampling unit are jump-connected to the input of the last upsampling unit. The processing process of the second downsampling unit and the third downsampling unit is similar. However, the output of the last downsampling unit (the fourth downsampling unit in this embodiment) is directly used as the input of the first upsampling unit. Therefore, the input of the first upsampling unit is only the output of the fourth upsampling unit; finally, the output features of the penultimate upsampling unit and the output features of the first downsampling unit are simultaneously used as the input of the last upsampling unit (the fourth upsampling unit), and are input into the fourth upsampling unit for processing to obtain the target phase and amplitude features; the target phase and amplitude features of the improved U-net subnetwork output tensor size is 1×2×1024×1024, and the target phase and amplitude features contain the amplitude and phase fields of the target scene with the target tensor diagram. The target phase and amplitude features are processed using the angular spectrum algorithm to decompose the complex amplitude distribution on the plane light field into multiple linear plane waves with different propagation directions. superposition; simulate the collected target amplitude and phase features into holographic plane amplitude and phase, thereby obtaining a holographic phase amplitude feature containing the holographic plane amplitude and phase field; then, input the holographic phase amplitude feature into the improved fully convolutional residual subnetwork to generate a three-dimensional hologram of size 1×1×1024×1024; the improved U-net subnetwork has a four-layer structure, including four downsampling units with the same structure and increasing channels in sequence, and four upsampling units with the same structure corresponding to the number of channels of the downsampling units; the process of processing the target tensor graph by the improved U-net subnetwork is the same as that of the traditional four-level U-net network, the difference is that this paper improves the structure of the encoding layer and the decoding layer Furthermore, the bridging layer in the traditional U-net network (used to connect the last encoder and the first decoder) is deleted; the improved full convolution residual subnetwork includes multiple first convolution units with the same structure and one second convolution unit. The holographic phase amplitude feature is the input of the first first convolution unit, and the holographic phase amplitude feature is jump-connected to the second convolution unit. The data processing process of the improved full convolution residual subnetwork is to process the holographic phase amplitude feature step by step through multiple first convolution units, and then input the output of the last first convolution unit and the holographic phase amplitude feature jump-connected to the second convolution unit into the second convolution unit for processing, thereby obtaining the target three-dimensional hologram.

[0084] Furthermore, the data processing process of the downsampling unit in S510 is specifically as follows:

[0085] S51011: performing batch normalization and activation processing on the downsampled input features of the input downsampling unit in sequence according to the batch normalization layer and the activation layer to obtain a first activation feature;

[0086] S51012: Performing a 3×3 two-dimensional convolution process on the first activation feature according to the first convolution layer to obtain a first convolution feature;

[0087] S51013: performing batch normalization and activation processing on the first convolution feature in sequence according to the batch normalization layer and the activation layer to obtain a second activation feature;

[0088] S51014: Performing a 3×3 two-dimensional convolution process on the second activation feature according to the second convolution layer to obtain a second convolution feature;

[0089] S51015: Perform 1×1 two-dimensional convolution processing on the downsampled input features according to the third convolution layer to obtain a third convolution feature, and perform a residual connection on the second convolution feature and the third convolution feature to obtain the output feature of the downsampling unit.

[0090] It should be understood that the downsampled input features of the input downsampling unit are batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the first activation feature; then, the first activation feature is subjected to a 3×3 two-dimensional convolution process according to the first convolution layer to obtain the first convolution feature; the first convolution layer is a two-dimensional convolution layer with a convolution kernel size of 3 and a stride of 2; then, the first convolution feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the second activation feature; secondly, the second activation feature is subjected to a 3×3 two-dimensional convolution process according to the second convolution layer to obtain the second convolution feature; the second convolution layer is a two-dimensional convolution layer with a convolution kernel size of 3 and a stride of 1. Convolution layer; finally, the downsampled input features are processed by 1×1 two-dimensional convolution according to the third convolution layer to obtain the third convolution feature. The third convolution layer is a two-dimensional convolution layer with a convolution kernel size of 1 and a stride of 2. Then, the second convolution feature and the third convolution feature are residually connected to obtain the output feature of the downsampling unit; the input of the downsampling block passes through a 1×1 convolution layer with a stride of 2 and is added to the output of the downsampling block to form a ResNet structure; after the downsampled input feature is input into the downsampling block, the number of output feature channels will increase and the height and width will become half of the input. The downsampling operation can expand the receptive field of the network, thereby perceiving more low-frequency information in the input features.

[0091] Furthermore, the process of data processing by the upsampling unit in S510 is specifically as follows:

[0092] S51021: performing batch normalization and activation processing on the upsampled input features of the input upsampling unit in sequence according to the batch normalization layer and the activation layer to obtain a third activated feature;

[0093] S51022: Performing a 3×3 two-dimensional transposed convolution process on the third activation feature according to the first transposed convolution layer to obtain a first transposed convolution feature;

[0094] S51023: performing batch normalization and activation processing on the first transposed convolutional feature in sequence according to the batch normalization layer and the activation layer to obtain a fourth activated feature;

[0095] S51024: Performing a 3×3 two-dimensional convolution process on the fourth activation feature according to the second convolution layer to obtain a fourth convolution feature;

[0096] S51025: Perform 2×2 two-dimensional transposed convolution processing on the upsampled input features according to the second transposed convolution layer to obtain second transposed convolution features, and perform residual connection on the second transposed convolution features and the fourth convolution features to obtain output features of the upsampling unit.

[0097] It should be understood that the upsampled input features of the input upsampling unit are batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the third activation feature; then, the third activation feature is subjected to 3×3 two-dimensional transposed convolution processing according to the first transposed convolution layer to obtain the first transposed convolution feature; the first transposed convolution layer is a two-dimensional transposed convolution layer with a convolution kernel size of 3 and a step size of 2; after that, the first transposed convolution feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the fourth activation feature; then, the fourth activation feature is subjected to 3×3 two-dimensional convolution processing according to the second convolution layer to obtain the fourth convolution feature; the second convolution layer is the same as that in the upsampling unit, and is also a two-dimensional convolution layer with a convolution kernel size of 3 and a step size of 1; finally, according to the second The transposed convolution layer performs a 2×2 two-dimensional transposed convolution on the upsampled input features to obtain the second transposed convolution features. The second transposed convolution layer is a two-dimensional transposed convolution with a convolution kernel size of 2 and a stride of 1. The second transposed convolution features obtained are residually connected with the fourth convolution features to obtain the output features of the upsampling unit. The upsampling unit and the downsampling unit have similar structures. The transposed convolution layer replaces some convolution layers in the upsampling unit, thereby doubling the feature height and width while reducing the number of channels. At the same time, a jump connection structure is introduced between the upsampling unit and the downsampling unit. The jump connection structure fuses the feature information obtained by the downsampling unit and the upsampling unit. The jump connection can also avoid gradient disappearance and network degradation, thereby improving the training efficiency of the network model. The improved U-net subnetwork outputs a target phase amplitude feature of size 1×2×1024×1024, where channel 1 is amplitude information and channel 2 is phase information. Channel 2 is processed using the Tanh activation function to constrain the phase of the target phase amplitude feature to .

[0098] Furthermore, the data processing process of the first convolution unit in S530 is specifically as follows:

[0099] S53011: Performing a 3×3 two-dimensional convolution on the first input feature of the first convolution unit according to the second convolution layer to obtain a fifth convolution feature;

[0100] S53012: performing batch normalization and activation processing on the fifth convolutional feature in sequence according to the batch normalization layer and the activation layer to obtain the fifth activation feature;

[0101] S53013: Perform a 3×3 two-dimensional convolution on the fifth activation feature according to the second convolutional layer to obtain a sixth convolution feature;

[0102] S53014: performing batch normalization and activation processing on the sixth convolutional feature in sequence according to the batch normalization layer and the activation layer to obtain the sixth activated feature;

[0103] S53015: Perform a residual connection on the sixth activation feature and the first input feature to obtain the first output feature of the first convolutional unit.

[0104] It should be understood that, first, the first convolution unit performs a 3×3 two-dimensional convolution on the first input feature of the first convolution unit according to the second convolution layer to obtain the fifth convolution feature; the second convolution layer is the same as that in the upsampling unit and the downsampling unit, and is a two-dimensional convolution layer with a convolution kernel size of 3 and a step length of 1; then, the fifth convolution feature is batch normalized and activated according to the batch normalization layer and the activation layer to obtain the fifth activation feature; then, the fifth activation feature is subjected to a 3×3 two-dimensional convolution through a second convolution layer to obtain the sixth convolution feature; finally, the sixth convolution feature is subjected to a batch normalization layer and the activation layer. Batch normalization and activation processing are performed to obtain the sixth activation feature; then, the sixth activation feature is residually connected to the first input feature to obtain the first output feature of the first convolutional unit; the convolutional layers used in the first convolutional unit are all second convolutional layers, that is, 3×3 convolutional layers with a stride of 1, and the 3×3 convolutional layers with a stride of 1 will not change the height and width of the image tensor. Except for the first first convolutional unit with a channel number of 2, the number of channels of all remaining first convolutional units is 15, and the width and height of the first convolutional unit are both 1024. The input and output of the first convolutional block are added to form the ResNet structure.

[0105] Furthermore, the data processing process of the second convolution unit in S530 is specifically as follows:

[0106] S53021: Performing a 3×3 two-dimensional convolution on the second input feature of the second convolution unit according to the second convolution layer to obtain a seventh convolution feature;

[0107] S53022: Perform a residual connection on the seventh convolution feature and the target phase amplitude feature to obtain a first residual feature;

[0108] S53023: Performing a 3×3 two-dimensional convolution on the first residual feature according to the second convolution layer to obtain an eighth convolution feature;

[0109] S53024: Process the eighth convolution feature according to the Tanh activation function to obtain the target three-dimensional hologram.

[0110] It should be understood that, first, the second input feature of the second convolution unit is subjected to 3×3 two-dimensional convolution processing according to the second convolution layer to obtain the seventh convolution feature; the second convolution layer is the same as that in the first convolution unit, both of which are two-dimensional convolution layers with a convolution kernel size of 3 and a step size of 1; then, the seventh convolution feature is residually connected with the target phase and amplitude feature connected by jump to obtain the first residual feature; the first residual feature is then subjected to 3×3 two-dimensional convolution processing through a second convolution layer to obtain the eighth convolution feature; finally, the eighth convolution feature is processed according to the Tanh activation function to obtain the target three-dimensional hologram; jump connection is also introduced in the improved full convolution residual subnetwork, and the input of the improved full convolution residual subnetwork (target phase and amplitude feature) is connected to the second convolution unit through jump connection. The second convolution unit changes the number of channels of the input feature to 1, and uses the Tanh activation function to constrain the output target three-dimensional hologram to The improved fully convolutional residual subnetwork has a consistent and repetitive structure, without pooling or transposed convolution layers to shrink and expand the dimensions of intermediate tensors, making it easier to construct and preserve high-frequency features in the forward propagation of the network.

[0111] S600: Process the target 3D hologram according to the preset optical 3D reconstruction device to obtain a 3D reconstructed image of the target scene

[0112] Furthermore, in S600, the optical 3D reconstruction device processes the target 3D hologram to obtain a 3D reconstructed image of the target scene, specifically:

[0113] S610: Inputting the target three-dimensional hologram into the spatial light modulator in the image reconstruction device for loading, while the laser in the image reconstruction device continuously emits red laser light;

[0114] S620: After the red laser beam is expanded, the direction is adjusted by the prism so that the expanded red laser beam is vertically incident on the effective area of ​​the spatial light modulator.

[0115] S630: The spatial light modulator loaded with the target 3D hologram modulates the incoming red laser light wave and reflects the modulated red laser light to the 4f system to eliminate the zero-order beam, thereby obtaining a 3D reconstructed image of the target scene.

[0116] S640: Displaying the three-dimensional reconstructed image of the target scene according to the movement of the CCD camera along the reconstruction optical axis.

[0117] It should be understood that, first, the target three-dimensional hologram is input into the spatial light modulator in the optical three-dimensional reconstruction device for loading, and at the same time, the laser in the optical three-dimensional reconstruction device continuously emits red laser light. After the red laser light undergoes the beam expansion operation of the beam expansion system, its direction is adjusted by the prism so that the expanded red laser light vertically enters the effective area of ​​the spatial light modulator; then, the spatial light modulator loaded with the target three-dimensional hologram modulates the light wave of the incoming red laser light, and reflects the modulated red laser light to the 4f system (4f correlator or 4f optical processor) to eliminate the zero-order light beam, thereby obtaining a three-dimensional reconstructed image of the target scene; finally, the CCD camera moves along the reconstruction optical axis so that the reconstructed three-dimensional reconstructed image is displayed.

[0118] According to a three-dimensional holographic image reconstruction method of this embodiment, a depth camera is used to continuously capture image information of a real scene at high speed, and the intensity information and depth information of the real scene are quickly obtained; a three-dimensional holographic reconstruction network model is used to realize a three-dimensional holographic display visible to the naked eye; the advantage is that a simple system can be used to quickly and in real time complete the capture of the real three-dimensional scene and the reconstruction of the three-dimensional hologram; thereby helping to expand the application of three-dimensional holographic display technology in remote collaboration and entertainment, as well as education, advertising, rehabilitation and other fields.

[0119] It should be noted that the 3D holographic image reconstruction method provided in the embodiments of the present application can be executed by a 3D holographic image reconstruction system, or by a control module in the 3D holographic image reconstruction system for executing and loading a 3D holographic image reconstruction method. In the embodiments of the present application, the 3D holographic image reconstruction method provided in the embodiments of the present application is described by taking a 3D holographic image reconstruction system executing and loading a 3D holographic image reconstruction method as an example.

[0120] A three-dimensional holographic image reconstruction system, comprising:

[0121] The first acquisition module acquires depth maps and intensity maps of multiple different scenes, and concatenates the depth map and intensity map corresponding to each scene to obtain a tensor map of each scene, so as to construct a dataset based on the multiple tensor maps.

[0122] Construction module: Construct a 3D holographic reconstruction network model, which includes an improved U-net sub-network, an angular spectrum algorithm, and an improved fully convolutional residual sub-network;

[0123] Training module: trains the 3D holographic reconstruction network model based on part of the data set, and tests the trained 3D holographic reconstruction network model based on the other part of the data set;

[0124] The second acquisition module: obtains the target depth map and target intensity map of the target scene, and splices the target intensity map and target depth map into a target tensor map;

[0125] Processing module: processes the target tensor map according to the tested 3D holographic reconstruction network model to obtain the target 3D hologram of the target scene;

[0126] Imaging module: processes the target 3D hologram according to the preset optical 3D reconstruction device to obtain a 3D reconstructed image of the target scene.

[0127] A three-dimensional holographic image reconstruction system in an embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), while the non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), etc., which are not specifically limited in the embodiments of the present application.

[0128] A 3D holographic image reconstruction system in an embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0129] The three-dimensional holographic image reconstruction system provided in the embodiment of the present application can achieve Figures 1 to 7 To avoid repetition, the various processes and technical effects achieved by a three-dimensional holographic image reconstruction method in the method embodiment are not described here.

[0130] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned three-dimensional holographic image reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0131] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned three-dimensional holographic image reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0133] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0134] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0135] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A three-dimensional holographic image reconstruction method, characterized in that: The method comprises: Acquire depth maps and intensity maps of multiple different scenes, and concatenate the depth maps and intensity maps corresponding to each scene to obtain a tensor map for each scene, so as to construct a data set based on the multiple tensor maps; Constructing a three-dimensional holographic reconstruction network model, wherein the three-dimensional holographic reconstruction network model includes an improved U-net subnetwork, an angular spectrum algorithm, and an improved full convolution residual subnetwork; Training the three-dimensional holographic reconstruction network model according to a portion of the data in the data set, and testing the trained three-dimensional holographic reconstruction network model according to the other portion of the data; Acquire a target depth map and a target intensity map of a target scene, and splice the target intensity map and the target depth map into a target tensor map; Processing the target tensor map according to the tested qualified three-dimensional holographic reconstruction network model to obtain a target three-dimensional hologram of the target scene; Processing the three-dimensional hologram of the target according to a preset optical three-dimensional reconstruction device to obtain a three-dimensional reconstructed image of the target scene; The three-dimensional holographic reconstruction network model processes the target tensor map to obtain a target three-dimensional hologram of the target scene, specifically: Processing the target tensor graph according to the improved U-net subnetwork to obtain target phase amplitude features; Processing the target phase and amplitude characteristics according to the angular spectrum algorithm to obtain holographic phase and amplitude characteristics; The improved U-net subnetwork has a four-layer structure, including four downsampling units with the same structure and channels increasing in sequence, and four upsampling units with the same structure corresponding to the number of channels in the downsampling units; the outputs of the first three downsampling units are jump-connected to the inputs of the last three upsampling units respectively; The improved full convolution residual subnetwork includes multiple first convolution units and one second convolution unit with the same structure; the holographic phase amplitude feature is the input of the first first convolution unit, and the holographic phase amplitude feature is jump-connected to the second convolution unit; the first convolution unit processes the first input feature of the first convolution unit through two consecutive two-dimensional convolutions, batch normalization, and activation, and connects the last activated feature with the residual of the first input feature to obtain the first output feature of the first convolution unit; The data processing process of the second convolution unit is specifically as follows: Performing a 3×3 two-dimensional convolution on the second input feature of the second convolution unit according to the second convolution layer to obtain a seventh convolution feature; Performing a residual connection between the seventh convolution feature and the target phase amplitude feature to obtain a first residual feature; Performing a 3×3 two-dimensional convolution process on the first residual feature according to the second convolution layer to obtain an eighth convolution feature; The eighth convolution feature is processed according to a Tanh activation function to obtain the target three-dimensional hologram.

2. The three-dimensional holographic image reconstruction method according to claim 1, characterized in that: The process of data processing by the downsampling unit is specifically as follows: performing batch normalization and activation processing on the downsampled input features input to the downsampled unit in sequence according to the batch normalization layer and the activation layer to obtain a first activated feature; Performing a 3×3 two-dimensional convolution process on the first activation feature according to the first convolution layer to obtain a first convolution feature; performing batch normalization and activation processing on the first convolutional features in sequence according to a batch normalization layer and an activation layer to obtain a second activation feature; Performing a 3×3 two-dimensional convolution process on the second activation feature according to the second convolution layer to obtain a second convolution feature; Performing 1×1 two-dimensional convolution processing on the downsampled input features according to the third convolution layer to obtain a third convolution feature, and performing a residual connection on the second convolution feature and the third convolution feature to obtain an output feature of the downsampling unit.

3. The three-dimensional holographic image reconstruction method according to claim 2, characterized in that: The process of data processing by the upsampling unit is specifically as follows: performing batch normalization and activation processing on the upsampled input features input to the upsampling unit in sequence according to the batch normalization layer and the activation layer to obtain a third activated feature; Performing a 3×3 two-dimensional transposed convolution process on the third activation feature according to the first transposed convolution layer to obtain a first transposed convolution feature; performing batch normalization and activation processing on the first transposed convolutional feature in sequence according to a batch normalization layer and an activation layer to obtain a fourth activated feature; Performing a 3×3 two-dimensional convolution process on the fourth activation feature according to the second convolution layer to obtain a fourth convolution feature; The up-sampled input features are subjected to 2×2 two-dimensional transposed convolution processing according to the second transposed convolution layer to obtain second transposed convolution features, and the second transposed convolution features are residually connected with the fourth convolution features to obtain output features of the up-sampling unit.

4. The three-dimensional holographic image reconstruction method according to claim 3, characterized in that: The data processing process of the first convolution unit is specifically as follows: Performing a 3×3 two-dimensional convolution on the first input feature of the first convolution unit according to the second convolution layer to obtain a fifth convolution feature; The fifth convolutional feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the fifth activation feature; Performing a 3×3 two-dimensional convolution process on the fifth activation feature according to the second convolution layer to obtain a sixth convolution feature; The sixth convolutional feature is batch normalized and activated in sequence according to the batch normalization layer and the activation layer to obtain the sixth activated feature; Perform a residual connection on the sixth activation feature and the first input feature to obtain a first output feature of the first convolutional unit.

5. The three-dimensional holographic image reconstruction method according to claim 1, characterized in that: The optical 3D reconstruction device processes the target 3D hologram to obtain a 3D reconstructed image of the target scene, specifically: Inputting the target three-dimensional hologram into a spatial light modulator in the optical three-dimensional reconstruction device for loading, while the laser in the optical three-dimensional reconstruction device continuously emits red laser light; The red laser beam is expanded and then directed by a prism so that the expanded red laser beam is incident vertically into the effective area of ​​the spatial light modulator. The spatial light modulator loaded with the target three-dimensional hologram modulates the incoming red laser light wave and reflects the modulated red laser light to the 4f system to eliminate the zero-order beam, thereby obtaining a three-dimensional reconstructed image of the target scene; The CCD camera moves along the reconstruction optical axis to display the three-dimensional reconstructed image of the target scene.

6. A three-dimensional holographic image reconstruction system capable of implementing a three-dimensional holographic image reconstruction method according to any one of claims 1 to 5, characterized in that: The system comprises: The first acquisition module acquires depth maps and intensity maps of multiple different scenes, and concatenates the depth map and intensity map corresponding to each scene to obtain a tensor map of each scene, so as to construct a data set based on the multiple tensor maps; Construction module: constructing a 3D holographic reconstruction network model, which includes an improved U-net subnetwork, an angular spectrum algorithm, and an improved full convolution residual subnetwork; Training module: training the 3D holographic reconstruction network model according to a portion of the data in the data set, and testing the trained 3D holographic reconstruction network model according to the other portion of the data; The second acquisition module is configured to acquire a target depth map and a target intensity map of the target scene, and to splice the target intensity map and the target depth map into a target tensor map; Processing module: processing the target tensor map according to the three-dimensional holographic reconstruction network model that has passed the test to obtain a target three-dimensional hologram of the target scene; Imaging module: processes the target three-dimensional hologram according to a preset optical three-dimensional reconstruction device to obtain a three-dimensional reconstructed image of the target scene.

7. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of a three-dimensional holographic image reconstruction method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the three-dimensional holographic image reconstruction method according to any one of claims 1 to 5 are implemented.

Images