Holographic generation method and system based on generative artificial intelligence

Abstract

The invention discloses a holographic generation method and system based on generative artificial intelligence, and the method comprises the following steps: S1, obtaining image information and depth information of a target scene at different visual angles, and generating a structured tensor based on the image information and the depth information; s2, performing space-direction joint coding on the structured tensor generated in the step S1, and extracting a structural latent variable; s3, using the structure latent variable extracted in the step S2 as a control signal to generate a complex amplitude diagram of the target scene, the complex amplitude diagram including amplitude information and phase information; s4, based on a physical propagation model, reconstructing the complex amplitude graph generated in the step S3 into a multi-view image sequence; and S5, mapping and adapting the multi-view image sequence reconstructed in the step S4 to a terminal display device for output display. According to the invention, while the physical consistency is ensured, the high-quality complex amplitude image is generated by directly utilizing the actual light field data, and the multi-view real restoration is realized.

Description

Technical Field

[0001] This invention relates to the field of holographic image technology, and in particular to a holographic generation method and system based on generative artificial intelligence. Background Technology

[0002] Holographic image generation, as a crucial technology for 3D information reconstruction, has shown great promise in recent years in fields such as virtual reality, augmented reality, medical imaging, and interactive displays. Traditional holographic generation methods are mostly based on physical modeling, generating complex amplitude holograms by numerically simulating the propagation and interference of light waves in space. While theoretically accurate, these methods suffer from extremely high computational complexity, typically requiring high-performance hardware and failing to meet the real-time and interactive requirements of practical scenarios. With the development of deep learning, academia and industry have attempted to accelerate holographic generation using methods such as generative adversarial networks (GANs) and autoencoders. Under certain conditions, these methods can improve generation speed, but most can only output pure phase holograms, with limited support for complex amplitude holograms. Furthermore, they are inadequate in phase continuity and amplitude reconstruction, leading to significant discrepancies between the generated images and the actual optical propagation results. Simultaneously, these methods often rely on large-scale labeled data and use only simple 2D images or a fixed number of viewpoints as input, failing to fully express the spatial-directional joint information of the real 3D scene. This results in problems such as missing details, blurred edges, or inconsistencies between viewpoints in the generated images. Other methods compress input data through sparse coding or light field to hologram conversion. While this improves data storage and processing, it sacrifices image quality and optical reconfigurability and cannot guarantee stable consistency under multi-view conditions.

[0003] Therefore, existing technologies generally lack a complete system solution that can directly utilize actual light field data to generate high-quality complex amplitude images and achieve realistic multi-view reproduction while ensuring physical consistency.

[0004] The above background information is provided only to aid in understanding the concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a hologram generation method and system based on generative artificial intelligence. This method can directly generate high-quality complex amplitude images using actual light field data while ensuring physical consistency and achieving realistic multi-view reconstruction. This results in the efficient generation of high-quality complex amplitude holograms with strong physical consistency and realistic multi-view reconstruction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention discloses a holographic generation method based on generative artificial intelligence, comprising the following steps:

[0008] S1: Obtain image information and depth information of the target scene from different perspectives, and generate a structured tensor based on the image information and the depth information;

[0009] S2: Perform spatial-directional joint encoding on the structured tensor generated in step S1 to extract structural latent variables;

[0010] S3: Using the structural latent variables extracted in step S2 as control signals, generate a complex amplitude diagram of the target scene, wherein the complex amplitude diagram includes amplitude information and phase information;

[0011] S4: Based on the physical propagation model, reconstruct the complex amplitude map generated in step S3 into a multi-view image sequence;

[0012] S5: Map and adapt the multi-view image sequence reconstructed in step S4 to the terminal display device for output display.

[0013] Preferably, generating a structured tensor based on the image information and the depth information in S1 includes: normalizing the pixel values ​​of the image information and the depth values ​​of the depth information respectively, and then generating a structured tensor containing spatial position and viewpoint direction based on the normalized image information and the depth information.

[0014] Preferably, the structural latent variables extracted in S2 are represented as follows:

[0015]

[0016] In the formula, Indicates pixel position The structural latent variable w after aggregating all perspectives i This represents the weight of the i-th viewpoint. Indicates pixel position The local structure encoding of the i-th viewpoint, where N represents the total number of viewpoints;

[0017] The weight of the i-th viewpoint is expressed as:

[0018]

[0019] In the formula, Let ρ be the angular difference between the i-th viewpoint and the main viewpoint, σ be an adjustable attenuation control parameter, and ρ be the angle difference between the i-th viewpoint and the main viewpoint. i is the entropy index of image information within the i-th viewpoint.

[0020] Preferably, the entropy index of image information within each viewpoint is calculated using the grayscale channel entropy of the image from each viewpoint. The entropy index of image information within the i-th viewpoint is expressed as:

[0021]

[0022] Where, k c This represents the number of channels in an image. Indicates the kth c The pixel probability distribution of an image with each channel.

[0023] Preferably, in S3, a complex amplitude generation model with a diffusion model architecture is used to generate the complex amplitude map of the target scene, wherein phase continuity constraints and energy balance constraints are introduced in the process of generating the complex amplitude map of the target scene.

[0024] Preferably, the phase continuity constraint is implemented by using the gradient direction of the structural latent variable as a guiding constraint for phase change, and the formula for the phase continuity constraint is as follows:

[0025]

[0026] in, pixel position Phase diagram at time step t+1 The pixel positions in the structural control tensor for the compression of the structural latent variables. In the content of the corresponding phase channel, This represents the gradient direction in space, where η is the structure adjustment factor. pixel position Complex noise perturbation at time step t.

[0027] Preferably, the energy balance constraint includes constraining the overall energy range of the amplitude portion of the generated complex amplitude map of the target scene. When performing the above energy balance constraint, the following loss function is used:

[0028]

[0029] in, This represents the loss of the complex amplitude generation model during training, where H and W represent the image height and width, respectively. Indicates pixel position The amplitude energy density, κ is the target energy constant; β represents the angle between the phase change direction and the structural control direction, and is an adjustment parameter.

[0030] Preferably, S4 includes:

[0031] S41: Based on the physical propagation model, the Fourier diffraction propagation principle is used to simulate the wavefront propagation of the complex amplitude image at different viewpoints. The complex amplitude image is propagated from the main viewpoint to the offset viewpoint using the following formula, and spatial movement is achieved through frequency domain phase modulation:

[0032]

[0033] in, This represents the intensity image at the offset viewpoint. and These represent the two-dimensional Fourier transform and its inverse transform, respectively. Represents a complex amplitude image. For frequency domain coordinates, Let be the displacement vector of the k-th output viewpoint;

[0034] S42: Perform phase coherence correction between multiple views during reconstruction: calculate the phase fusion residual between adjacent views, and determine whether to perform interpolation compensation based on the residual. The following formula is used to verify the consistency of phase information between adjacent views before generating the multi-view image sequence:

[0035]

[0036] Among them, R k This represents the phase fusion residual score of the k-th output view. Indicates pixel position phase, Let be the displacement vector of the k-th output viewpoint.

[0037] Preferably, S5 includes:

[0038] S51: By calculating the cosine similarity between the viewpoint of the multi-view image sequence and the target viewpoint of the terminal display device, a viewpoint allocation function is set to allocate appropriate reconstructed image frames to the device viewpoint, wherein the viewpoint allocation function is:

[0039]

[0040] Where, ω k It is a weighting factor for image quality. This represents the angle of the k-th output viewpoint. This represents the j-th viewing angle supported by the terminal display device;

[0041] S52: Construct an exponential decay function centered on the main viewpoint to compress the frequency and frame rate of non-main viewpoint frames in the current frame sequence, so as to dynamically adjust the output priority and refresh rate of images from different viewpoints. The exponential decay function is as follows:

[0042]

[0043] Where, ρ k This represents the refresh weight of the k-th output viewpoint in the output frame sequence, where λ is the attenuation coefficient. Let Z represent the perspective of the main viewpoint, and Z be the normalization term that makes ∑ k ρ k =1;

[0044] S53: Before outputting the frame sequence through the terminal display device, a priority allocation function is introduced to adjust the frame refresh priority, wherein the priority allocation function is:

[0045]

[0046] Where, γ k σ is the interaction priority coefficient for the k-th output viewpoint. s The Sigmoid function is used for smoothing. This represents the movement trend vector constructed using a sliding window.

[0047] Secondly, this invention discloses a holographic generation system based on generative artificial intelligence, comprising:

[0048] A multi-view acquisition module is used to acquire image information and depth information of the target scene from different viewpoints, and generate a structured tensor based on the image information and the depth information;

[0049] The joint encoding module, connected to the multi-view acquisition module, is used to perform spatial-directional joint encoding on the structured tensor generated by the multi-view acquisition module to extract structural latent variables;

[0050] A complex amplitude generation module, connected to the joint coding module, is used to generate a complex amplitude map of the target scene using the structural latent variables extracted by the joint coding module as control signals. The complex amplitude map includes amplitude information and phase information.

[0051] The physical reconstruction module, connected to the complex amplitude generation module, is used to reconstruct the complex amplitude map generated by the complex amplitude generation module into a multi-view image sequence based on the physical propagation model.

[0052] The output docking module is connected to the physical reconstruction module and is used to map and adapt the multi-view image sequence reconstructed by the physical reconstruction module to the terminal display device for output display.

[0053] Thirdly, the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program is configured to be run by a processor to perform the holographic generation method based on generative artificial intelligence as described in the first aspect.

[0054] Compared with existing technologies, the beneficial effects of this invention are as follows: The hologram generation method and system based on generative artificial intelligence proposed in this invention first generates a structured tensor based on image information and depth information of the target scene from different perspectives. Then, the structured tensor is spatially and directionally jointly encoded to extract structural latent variables. Then, a complex amplitude map is generated based on the structural latent variables. A multi-view image sequence is reconstructed based on the complex amplitude map. Finally, the multi-view sequence is mapped to the terminal display device. This method and system form a complete closed loop from light field acquisition, latent variable modeling, complex amplitude generation to physical reconstruction and device display. It realizes efficient and physically accurate holographic generation and a real-world closed loop based on actual light field data. It has strong generalization ability and real-time performance. It can stably generate high-quality complex amplitude holographic images in complex application scenarios, significantly improving the practicality and deployability of holographic display.

[0055] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0056] Figure 1 This is a flowchart of the hologram generation method based on generative artificial intelligence in Embodiment 1 of the present invention. Detailed Implementation

[0057] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0058] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for both fixing and circuit / signal connectivity.

[0059] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0061] To address the problems of high computational cost, insufficient complex amplitude map generation, poor physical consistency, and poor multi-view consistency in existing hologram generation techniques, such as... Figure 1 As shown, Embodiment 1 of the present invention discloses a holographic generation method based on generative artificial intelligence, comprising the following steps:

[0062] S1: Obtain image and depth information of the target scene from different perspectives, and generate structured tensors based on the image and depth information;

[0063] The generation of structured tensors based on image and depth information includes: normalizing the pixel values ​​of the image information and the depth values ​​of the depth information respectively, and then generating a structured tensor containing spatial location and viewpoint direction based on the normalized image and depth information.

[0064] Specifically, step S1 mainly consists of two parts: the first part is the acquisition of multi-view light field data, and the second part is the organization of a structured tensor. This step aims to construct an input data tensor with spatial-angular information representation capabilities for the 3D real scene, providing necessary structural support for the subsequent complex amplitude map generation model. Considering the high dependence of hologram generation tasks on spatial consistency and angular coverage, the original input data must not only contain image content from different angles but also accurately capture the scene's changes in the depth dimension. Therefore, this step revolves around "dense acquisition of light field features," constructing a camera array system using multiple synchronous camera devices to acquire and organize image and depth information from different viewpoints, outputting a data tensor with a unified structure and regular format.

[0065] In a specific implementation, the camera array system uses eight depth cameras, arranged linearly and evenly on a 1.4-meter track. The horizontal spacing between each depth camera is approximately 20 centimeters, forming a multi-view acquisition system covering a left-right ±70° field of view. All depth cameras are controlled for unified time synchronization via ROS (Robot Operating System), and are configured with uniform exposure time and frame rate (30fps) to ensure data alignment in time.

[0066] During the acquisition process, each camera outputs a set of RGB images and a precisely aligned depth map. The image resolution is 640×480, and the depth data is a 16-bit grayscale image, with each pixel representing the distance information of that location in the camera coordinate system. The depth map is generated in real time by the infrared structured light module built into the depth camera, and the data stream is read using a software development kit.

[0067] After all the depth camera data streams are collected centrally at the control terminal, they are encapsulated into a structured tensor. This structured tensor has dimensions of N×H×W×4, where N=8 represents the number of viewpoints, H=480 and W=640 represent the image height and width, respectively, and the last channel contains three color channels (R, G, B) and one D channel (i.e., depth).

[0068] The original pixel values ​​of the acquired images range from 0 to 255. To improve the stability of subsequent model processing, a normalization process is performed, linearly mapping all pixel values ​​to the range of 0 to 1.

[0069]

[0070] in, This represents the normalized pixel values ​​of the RGB image. Represents the pixel coordinates of the image. This represents the viewpoint angle of the i-th camera (i.e., the viewpoint angle of the i-th camera). This is the original pixel value at that location.

[0071] The depth map has a large value range (commonly 400mm to 4000mm), and directly inputting it into the model would result in incomparability between different viewpoints. Therefore, a unified normalization method is used to process it into a dimensionless form:

[0072]

[0073] in, This represents the normalized depth value. D represents the depth value of a pixel in the original depth image. min and D max These represent the minimum and maximum values ​​among all depth images acquired in this batch, respectively.

[0074] For example, if a user places a cube 1 meter directly in front of a camera array, different cameras in the array will capture images containing different occlusion edges. Simultaneously, due to the cube's deformation, the depth map will show significant jumps along the contours of the 3D structure. In the resulting structured tensor X, the image texture and depth values ​​at the same pixel location change in an ordered manner from different viewpoints, providing a data foundation for subsequent encoding models to extract structural differences.

[0075] Throughout the entire data construction process, no image cropping, filtering, or dimensionality reduction operations were performed, preserving maximum information. Image data and depth data were correlated one-to-one and input into the subsequent model as tensors, rather than processed as an image sequence. This approach aims to retain the dual information of "spatial location + viewpoint orientation" in the structural dimension, achieving integrated modeling.

[0076] Each element in the final output structured tensor corresponds to an image coordinate. A collection perspective and the corresponding 4-channel vector For example:

[0077] X[i,240,320]=[0.52,0.38,0.41,0.67] (3)

[0078] [i,240,320] represents the RGB value and normalized depth value at the center point of the image from the i-th viewpoint.

[0079] The output is a structured tensor X with dimensions N×H×W×4, where N=8, H=480, and W=640. Each channel consists of a standard normalized RGB image and a depth map. This structured tensor will be directly used as the input to step S2, entering the structured encoding model without further format conversion or additional preprocessing.

[0080] S2: Perform spatial-directional joint encoding on the structured tensor generated in step S1 to extract structural latent variables.

[0081] Furthermore, a spatial-orientation joint encoder is used to encode the structured input tensor to extract structural latent variables that characterize the geometric structure and viewpoint relationships of the target scene. The encoding process introduces an attention fusion mechanism based on the geometric positional relationships between viewpoints and the content information of viewpoints. Furthermore, the weight calculation of the attention fusion mechanism based on the geometric positional relationships between viewpoints and the content information of viewpoints depends on the angle difference between viewpoints and the entropy index that reflects the content information of viewpoints.

[0082] Specifically, in this step, structural latent variables are extracted using a space-direction joint coding model; the core task of this step is to encode the structured tensor X obtained in step S1 into structural latent variables. This provides complete spatial-angular feature support for the subsequent generation of complex amplitude maps.

[0083] Since the goal of hologram generation in this embodiment of the invention is to obtain a realistically reconstructable complex amplitude map, rather than general image synthesis, the extracted structural latent variables must not only cover texture information but also accurately reflect spatial occlusion relationships, differences between viewpoints, and the joint distribution pattern between depth, direction, and image. This step specifically designs a joint encoding structure with angle-aware capabilities, which, through viewpoint selectivity and angle-guided attention mechanisms, preserves all the geometric and semantic information required for complex amplitude generation while compressing the input data.

[0084] The input for this step is the structured tensor output from step S1. Where N=8 represents the number of viewpoints, each viewpoint corresponds to a set of image data with a resolution of H×W, and each pixel contains normalized color values ​​for the R, G, and B channels. The normalized depth value of the channel. This structured tensor has already undergone time synchronization, spatial registration, and normalization during the acquisition phase, and does not require further preprocessing before being directly fed into the encoding network.

[0085] First, an independent encoding channel is constructed for each viewpoint. Each channel contains three 2D convolutional blocks for image-depth joint feature extraction. Each layer includes a 3×3 convolutional kernel (with 32, 64, and 64 channels respectively), a ReLU activation function, and batch normalization. The convolutional operation is applied to each viewpoint branch of the structured tensor X. Generate local structure encoding Used to represent the i-th viewpoint angle The image and depth fusion features below.

[0086] In traditional methods, these viewpoint features are often simply stacked or averaged, which leads to blurred occlusion boundaries, unresolved viewpoint conflicts, and ultimately distorted hologram reconstruction. To address this, this embodiment proposes an angle-guided attention fusion module and designs an angle-difference-sensitive fusion mechanism. This mechanism utilizes the geometric positional relationships between the various viewpoints in the camera array to calculate the i-th viewpoint. Compared to the first-person perspective angular offset Based on this, a Gaussian decay weighting function is designed:

[0087]

[0088] Among them, w i This represents the weight of the i-th viewpoint. Let ρ be the angular difference between the i-th viewpoint and the main viewpoint, σ be an adjustable attenuation control parameter, and ρ be the angle difference between the i-th viewpoint and the main viewpoint. iσ is the entropy index of image information within the i-th viewpoint, used to reflect whether that viewpoint has key structures. σ must be positive, typically ranging from 0.5 to 2 times the distance between viewpoints to the maximum angular difference; too small a value leads to over-concentration, while too large a value tends towards averaging. This formula balances geometric location and content distribution, allowing the encoder to prioritize image features with high information density and angles close to the main viewpoint, thereby improving the accuracy and robustness of the fusion process.

[0089] After obtaining the weights w of all perspectives i Then, a weighted fusion is performed to obtain the aggregated unified structure representation tensor:

[0090]

[0091] in, Indicates pixel position The structural latent variables, after aggregation from all perspectives, encompass comprehensive geometric semantics such as spatial location, directional response, depth variation, and edge overlap. These structural latent variables will be used in the next step, S3, to control the generation process of the complex amplitude map, including phase continuity and amplitude edge construction.

[0092] In implementation, this embodiment will use ρ i The estimate is obtained by calculating the gray-level channel entropy of the image from each viewpoint, that is, after converting the image to grayscale, calculating the pixel probability distribution of each image. Entropy after calculation:

[0093]

[0094] Where, k c This represents the number of channels in an image. Indicates the kth c The pixel probability distribution of the image in each channel is used to achieve unsupervised evaluation of image information. This step makes contribution weight judgments on viewpoint features without introducing additional supervision signals, demonstrating the ability of this step to automatically model the geometric focal region of the scene under unlabeled conditions.

[0095] For example, in a typical scenario, a portion of a target object is completely occluded from one viewpoint, but remains clearly visible in adjacent viewpoints. Traditional average fusion methods would blur the occluded area, while this approach, due to the image entropy ρ of the occluded viewpoint... i Lower, angle difference The larger the value, the higher its weight w. i The automatic reduction in weighting protects the accuracy of structural reconstruction. This angle-and-content weighting mechanism is an innovative solution addressing the tendency of structural errors to propagate in holographic generation tasks.

[0096] The output of this step is the structural latent variable. Each pixel location is assigned a 64-dimensional structural feature vector. This variable will serve as a conditional control vector in subsequent steps, guiding the fitting process of amplitude and phase for each pixel in the complex amplitude map generation model.

[0097] S3: Using the structural latent variables extracted in step S2 as control signals, generate a complex amplitude diagram of the target scene. The complex amplitude diagram includes amplitude information and phase information.

[0098] The complex amplitude generation model employs a diffusion model architecture to generate the complex amplitude map of the target scene. Phase continuity constraints and energy balance constraints are introduced during the generation process. The phase continuity constraint is implemented by using the gradient direction of the structural latent variables as a guiding constraint for phase changes. The energy balance constraint includes constraining the overall energy range of the generated hologram amplitude portion.

[0099] Specifically, this step generates a complex amplitude diagram containing amplitude and phase information based on a complex amplitude diagram generation model using structural latent variable conditions. The goal of this step is to transform the structural latent variables... Transform into an image with complex amplitude characteristics This provides a direct output for the final hologram generation. Unlike natural image generation tasks, the output here is a complex image, which must simultaneously possess reconfigurability (supporting optical propagation), controllability (guided by the geometric priors provided in the previous step S2), and physical consistency (satisfying wavefront continuity, coherent interference laws, etc.). The generation scheme designed in this step does not directly learn the complex amplitude map in a "black box," but rather constructs a complex diffusion map generation system that is structure-guided, process-controllable, and physically constrained, specifically serving the "structure-to-complex amplitude map" mapping task in the scenario of this embodiment.

[0100] Compared to existing generative models, the biggest difference lies in the fact that this embodiment not only optimizes pixel reconstruction errors or subjective visual scoring, but also specifically introduces a regularization mechanism applicable to complex amplitude maps, enabling the model to be reconstructable and viewpoint-switching in a physical holographic propagation system. Therefore, the model must not only "look like an image," but also "propagate correctly," which is the core principle of this step's design.

[0101] The input for this step is the structural latent variable output from step S2. The features of each channel of the structural latent variable are derived from features such as depth edges, viewpoint angle differences, and spatial consistency during the multi-view fusion process. During the training phase, The first four channels have been guided to learn and fit physical structural properties such as edge amplitude, spatial location, occlusion relationship, and angular direction response, respectively. During the inference phase, This will be used as a conditional input to control the subsequent generation process of complex amplitude images.

[0102] This embodiment designs a complex diffusion generation network, consisting of a structure condition encoding module and a complex amplitude map generation module. The structure condition encoding module first uses three 1×1 convolutional layers (with 32, 16, and 2 output channels respectively) to generate complex amplitude maps. Compressed into a two-channel structure control tensor This corresponds to the pixel-by-pixel control vectors for amplitude A and phase φ. During the training phase, the gradient of C is forced to the structural edges and depth transition regions, thereby concentrating the control signal at the locations where geometric changes are most pronounced.

[0103] The complex amplitude map generation module is built based on a diffusion mechanism and uses an improved UNet structure for stepwise sampling. At each diffusion time step t, the current complex amplitude map state is... The data is used as input, combined with the control tensor C and time step t, to generate a new complex amplitude map estimate. Unlike conventional diffusion, this embodiment introduces a phase structure guidance mechanism, using the gradient direction of the structure encoded in the previous step S2 as a phase offset constraint, as shown in the following formula:

[0104]

[0105] in, For the phase map update at the next time step, To control the contents of the corresponding phase channels in the tensor. Let η be the gradient direction in space, and η be the structure adjustment factor. This represents the complex noise perturbation for the current step. This phase constraint ensures that the phase change trend aligns with the object's edge direction, suppressing the phase jump problem commonly encountered in complex amplitude map generation.

[0106] To further improve the optical propagability of the generated graph, this embodiment proposes a complex domain energy balance regularization term, which is added to the loss during the training phase:

[0107]

[0108] in, The first term represents the loss of the complex amplitude generation model during training. This represents the amplitude energy density, where κ is the target energy constant (e.g., 0.6), used to force the overall energy of the amplitude map to remain within a certain range, preventing overexposure or complete darkness; in the second term, The angle between the phase change direction and the structural control direction (evaluated via dot product) is used to encourage the phase map to change synchronously with structural features. β is an adjustment parameter; for example, setting it to 0.1 can achieve a good balance between edge consistency and wavefront continuity.

[0109] This regularization avoids the problem that traditional loss functions can only limit pixel grayscale. It models the optical behavior of the entire image from the complex domain, providing both theoretical and engineering support for the goal of "generating reconfigurable holograms based on structure control" in this embodiment.

[0110] The entire diffusion model underwent 500 warm-up rounds during the training phase, with each round performing 50 time-step sampling processes. The initial input was a complex noise map. Model optimization employed the AdamW optimizer with an initial learning rate of 2×10⁻⁶. -4 The weights of the structure encoding network are frozen in the first 100 rounds, and then jointly trained with the generative network. This strategy ensures... The structural semantics are not destroyed during the generation stage, and it has a stable guiding effect on the generation of complex amplitude.

[0111] For example, in a set of portrait data, the hair area contains a large amount of detail and complex edges. Without structural control, the phase map often shows high-frequency noise in these areas. However, in this scheme, the phase channel control area in the structural encoding is precisely aligned with these details. The phase gradient constraint during training makes the wavefront smoothly fit the edges. The final reconstructed image displays a complete three-dimensional surface and sense of depth in physical simulation, which is a performance that is difficult to achieve in existing technologies.

[0112] The output of this step is a complex amplitude diagram. in This is an amplitude diagram. The phase diagram has dimensions of H×W, where j is the imaginary unit. This complex amplitude diagram exhibits physical propagation consistency and good continuity between the edge structure and the wavefront, and will be directly used as the input for multi-view optical reconstruction in step S4.

[0113] S4: Based on the physical propagation model, reconstruct the complex amplitude map generated in step S3 into a multi-view image sequence;

[0114] Specifically, based on the physical optical propagation model, the complex amplitude map is reconstructed into reconstructed images under multiple preset viewpoints, and phase coherence correction between multiple viewpoints is performed during the reconstruction process. Furthermore, based on the physical propagation model, the Fourier diffraction propagation principle is used to simulate the wavefront propagation of the complex amplitude map under different viewpoints. The phase coherence correction between multiple viewpoints includes: calculating the phase fusion residual between adjacent viewpoints, and determining whether to perform interpolation compensation based on the residual.

[0115] Specifically, this step involves multi-view image reconstruction based on a physical propagation model; the core task of this step is to reconstruct the complex amplitude image generated in step S3. As input, the two-dimensional reconstructed images from multiple viewing angles are calculated using a physical propagation model. This provides realistic multi-view images for the final display system. Unlike modeling or generation tasks, this step is an explicit physical process reconstruction operation. It no longer involves model training or feature extraction, but is based on holographic propagation theory to simulate the propagation of the light field from the complex amplitude map in free space to various viewpoints.

[0116] This step, situated in the transitional phase from generation to display within the overall process, serves as a crucial bridge connecting the AI ​​model output with the physically visualized image output. It is essential to ensure that: on the one hand, the complex amplitude diagram is fully utilized. Complex information, on the other hand, the reconstructed image It possesses true spatial consistency and angular visibility. Therefore, this embodiment designs a reconstruction mechanism that combines Fourier diffraction propagation and angle-guided interpolation, and introduces a calibration term based on phase interferogram to suppress image tearing or jumps that occur during angle switching when synthesizing images from multiple angles.

[0117] This step strictly uses the complex amplitude diagram output from step three. As input. The amplitude portion of the image. Texture structure is controlled by a diffusion generation model, phase component The phase guidance mechanism reflects the edge direction and structural continuity. The dimension is H×W, the data format is a complex matrix, and it is stored in a standard complex tensor, where H and W are consistent with the acquired image (480×640 in this system).

[0118] This step is based on the Fourier diffraction propagation principle in optics to reconstruct the image. The core characteristic of a hologram is that it is essentially a record of spatial wavefront information, and the reconstruction process is equivalent to simulating the evolution of the wavefront at a specific propagation distance and angle. This embodiment uses the Rayleigh-Sommerfeld approximation based on frequency domain transformation to simulate spatial propagation in the digital domain.

[0119] For each preset perspective In this embodiment, the corresponding propagation offset vector is first constructed. This offset defines the relative position of the viewpoint k with respect to the main viewpoint (main observation angle). Then, the simulation of complex wave propagation is performed using the following formula:

[0120]

[0121] in, and These represent the two-dimensional Fourier transform and its inverse transform, respectively. For frequency domain coordinates, Let be the displacement vector (in pixels) of the k-th output viewpoint. This formula propagates the complex amplitude map from the main viewpoint to the offset viewpoint, achieving spatial movement through frequency domain phase modulation. Finally, the intensity image at this viewpoint is obtained through modulus squaring.

[0122] In the specific implementation The value is calculated from the preset viewing angle and imaging distance. For example, with the imaging distance set to d = 20cm and the viewing angle range [-15°, +15°] evenly distributed among 6 observation points, θ k Let be the angle of the k-th output viewpoint. This displacement vector, after pixel-scale conversion, is used to calculate the frequency domain modulation factor.

[0123] In real-world scenarios, due to complex amplitude diagrams High-frequency interference structures may exist in some regions, causing image structure discontinuities or jumps during viewpoint switching. To address this, this embodiment designs a viewpoint fusion regularization mechanism based on phase coherence, which verifies the consistency of phase information from adjacent viewpoints before generating multi-view images. Specifically:

[0124]

[0125] Among them, R k This represents the phase fusion residual score for the k-th output viewpoint, used to assess whether there is a significant phase jump in the image from that viewpoint. If R... k If the value exceeds a set threshold (e.g., 0.5), it indicates that there may be an unstable propagation region at that viewpoint. The system will automatically insert intermediate viewpoint reconstruction frames to compensate for the image continuity of that viewpoint through interpolation.

[0126] For example, when capturing a holographic scene containing glass structures and metal edges, these areas will exhibit abrupt changes in the phase map. Directly reconstructing multiple viewpoints may result in image tearing in these areas. Through phase residual detection in this step, the system will automatically identify these areas and perform frame interpolation between viewpoints, ensuring that the final output viewpoint map will not exhibit flickering or discontinuity during panoramic scanning.

[0127] In engineering implementation, the reconstruction process uses the standard FFT library for forward / backward Fourier transforms, the view modulation term is implemented through element-wise complex multiplication, and the final output image is stored in [location missing]. In a tensor, the data format is an H×W real-valued image with a value range of [0,1]; it can be directly used for 8-bit grayscale image display.

[0128] The output is a multi-view image sequence. k represents the k-th output viewpoint, and M represents the total number of output viewpoints. Each image is a real-valued grayscale image of size H×W, representing the intensity reconstruction of the complex amplitude map at different viewpoints (observation angles). This image sequence will be directly used as input to the next step S5, entering the display and system interface module.

[0129] S5: Map and adapt the multi-view image sequence reconstructed in step S4 to the terminal display device for output display.

[0130] The mapping adaptation includes: allocating appropriate reconstructed image frames to the device's viewpoint based on the matching degree between the viewpoint sequence supported by the terminal device and the reconstructed image viewpoint, as well as image quality weights. Output display also includes: dynamically adjusting the output priority and refresh rate of images from different viewpoints based on the display device's cache limitations and predictions of user viewpoint interaction behavior.

[0131] Specifically, this step interfaces the image sequence output with the terminal display system. This step is the final stage from generation to practical application, and its core objective is to reconstruct the multi-view image sequence obtained in step S4. The image is mapped to a format that can be received by specific terminal devices, enabling the final display on holographic or virtual reality terminals. In traditional image generation or video stream processing, most solutions only consider static image output, ignoring the differences between different devices in terms of viewpoint rendering methods, frame buffer structures, and interaction latency. The innovation of this step lies in constructing a systematic output mechanism that integrates "viewpoint-device mapping, time-space compression, and three-dimensional linkage of user behavior feedback." Specifically, for the complex amplitude hologram generation scenario supported by this embodiment, a dynamic weight-driven viewpoint reordering scheme and an interactive view prediction function are proposed, achieving optimal image scheduling guided by the user's viewpoint while ensuring spatial consistency of the displayed content.

[0132] The input for this step is the viewpoint reconstructed image sequence output from step S4. Each of the images Corresponding complex amplitude diagram The k-th output view The reconstruction results are as follows. All images are H×W in dimension, single-channel grayscale images, and their values ​​are normalized to [0,1]. They are arranged in memory as tensors in the form of M×H×W.

[0133] First, to ensure that the output image conforms to the device display strategy, a terminal viewpoint mapping table M is established. map Record the view sequence supported by the terminal device. A viewpoint assignment function is defined by calculating the cosine similarity between the input image viewpoint and the device target viewpoint:

[0134]

[0135] Where, ω k This is a weighting factor for image quality, initially set to 1, but dynamically adjusted in subsequent steps based on indicators such as image sharpness and phase coherence. This mapping function implements a frame selection mechanism driven by both angle matching and image availability, resulting in a final output frame sequence. It is precisely aligned with the required viewing angle of the device, while ensuring optimal visual clarity.

[0136] Secondly, to address the limited display buffering issue in practical terminal devices, particularly the frame buffering limitations common in holographic projection terminals (such as spatial light modulators), this embodiment introduces a time-space dynamic compression mechanism. The core idea is to compress the current frame sequence... According to the observer's perspective Centered on the main view, an exponential decay function is constructed to compress the frequency and frame rate of non-prime view frames, thereby freeing up buffer space for high-frequency refresh of the main view and its neighboring regions.

[0137]

[0138] Where, ρ k Let λ represent the refresh weight of the k-th output viewpoint in the output sequence, λ be the decay coefficient, and Z be the normalization term such that ∑ k ρ k =1. This is determined by ρ during the actual output process. k By allocating refresh counts and cache lengths, limited device resources can be used for the most critical viewpoint areas, thereby enhancing user immersion and interactive experience. This strategy is particularly effective in holographic devices and VR terminals, avoiding the waste of resources due to excessive redundant frames and reducing image tearing caused by images far from the primary viewpoint.

[0139] In addition, to adapt to the interactive control system, a view prediction module was designed in this step. This module is based on the user's historical view jump behavior. Constructing a movement trend vector using a sliding window A set of viewpoints used to predict the most likely visits within a certain future time window. This embodiment introduces the following prediction priority allocation function before the output frame sequence:

[0140]

[0141] Where, γ k σ is the interaction priority coefficient for the k-th output viewpoint. sThe Sigmoid function is used for smoothing. This value adjusts the frame refresh priority, allowing the system to preload relevant frames before the user turns their head or changes their viewpoint, significantly reducing interaction latency and improving the subjective sense of smoothness.

[0142] At the implementation level, all output images are 8-bit grayscale converted and encapsulated according to the terminal protocol (such as HDMI, DisplayPort, custom communication protocols, etc.), and sent to the corresponding display buffer through the frame management thread in the main control system. The frame refresh cycle is connected to the system's global synchronization control module to achieve synchronous updates with interactive modules such as depth sensors and gyroscopes, meeting the real-time display requirements under dynamic viewing angles.

[0143] For example, in wearable holographic glasses for mobile devices, only 5 complete frames of images can be stored at a time due to power consumption limitations, but users may switch perspectives 10 times per second. Using the aforementioned prediction mechanism and exponential refresh strategy, the system can stably maintain image preloading of the current plus the predicted next perspective area, successfully compressing the image response time to within 30ms, ensuring a seamless latency experience in actual use.

[0144] The output is a sequence of image frames adapted to the terminal device. Each frame is a single-channel grayscale image of size H×W, and the refresh rate and frame sequence order are determined by ρ. k With γ k The decision is made. This sequence is transmitted in real time via the device communication interface for the final dynamic display of holographic images or virtual perspective content.

[0145] The holographic generation method based on generative artificial intelligence disclosed in Embodiment 1 of this invention constructs a multi-view light field acquisition and encoding mechanism to fuse images and depth information from different angles into a unified tensor. Then, a spatial-orientation joint encoder is used to extract latent variable representations, which can fully reflect the geometric structure, directional differences, and occlusion relationships of the scene. Subsequently, a conditional diffusion mechanism is introduced in the complex amplitude generation stage, using the latent variables as control signals to generate a complex image containing amplitude and phase. Specific phase continuity constraints and energy balance regularization design ensure that the generated result satisfies the laws of optical propagation. Furthermore, in the reconstruction stage, this invention employs a multi-view synthesis method based on a physical propagation model to restore the complex image to real images from different angles, and improves the consistency of multi-view output through phase coherence correction. Finally, this invention proposes a viewpoint mapping and frame scheduling strategy for interfacing with terminal devices, enabling efficient display of multi-view sequences on holographic display devices or virtual reality terminals. This method forms a complete closed loop from light field acquisition, latent variable modeling, complex amplitude generation to physical reconstruction and device display. It has strong generalization ability and real-time performance, and can stably generate high-quality complex amplitude holographic images in complex application scenarios, significantly improving the practicality and deployability of holographic display.

[0146] Embodiment 2 of the present invention discloses a holographic generation system based on generative artificial intelligence, comprising:

[0147] The multi-view acquisition module is used to acquire image information and depth information of the target scene from different perspectives, and generate a structured tensor based on the image information and the depth information. Furthermore, the specific steps and effects performed by the multi-view acquisition module are the same as those in step S1 of Embodiment 1, and will not be repeated here.

[0148] The joint encoding module, connected to the multi-view acquisition module, is used to perform spatial-directional joint encoding on the structured tensor generated by the multi-view acquisition module to extract structural latent variables. Furthermore, the specific steps and effects performed by the joint encoding module are the same as those in step S2 of Embodiment 1, and will not be repeated here.

[0149] The complex amplitude generation module, connected to the joint encoding module, is used to generate a complex amplitude diagram of the target scene using the structural latent variables extracted by the joint encoding module as control signals. The complex amplitude diagram includes amplitude information and phase information. Furthermore, the specific steps and effects performed by the complex amplitude generation module are the same as those in step S3 of Embodiment 1, and will not be repeated here.

[0150] The physical reconstruction module is connected to the complex amplitude generation module and is used to reconstruct the complex amplitude map generated by the complex amplitude generation module into a multi-view image sequence based on the physical propagation model. Furthermore, the specific steps and effects performed by the physical reconstruction module are the same as those in step S4 of Embodiment 1, and will not be repeated here.

[0151] The output docking module, connected to the physical reconstruction module, is used to map and adapt the multi-view image sequence reconstructed by the physical reconstruction module to the terminal display device for output display. Furthermore, the specific steps and effects performed by the output docking module are the same as those in step S5 of Embodiment 1, and will not be repeated here.

[0152] Embodiment 3 of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program is configured to be run by a processor to perform the steps of the holographic generation method based on generative artificial intelligence in Embodiment 1 above.

[0153] Optionally, the aforementioned computer-readable storage media may include, but are not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0154] The background section of this invention may include background information about the problems or circumstances surrounding the invention, rather than a description of prior art by others. Therefore, the content included in the background section is not an admission of prior art by the applicant.

[0155] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope defined by the appended claims.

Claims

1. A holographic generation method based on generative artificial intelligence, characterized in that, Includes the following steps: S1: Obtain image information and depth information of the target scene from different perspectives, and generate a structured tensor based on the image information and the depth information; S2: Perform spatial-directional joint encoding on the structured tensor generated in step S1 to extract structural latent variables; S3: Using the structural latent variables extracted in step S2 as control signals, generate a complex amplitude diagram of the target scene, wherein the complex amplitude diagram includes amplitude information and phase information; S4: Based on the physical propagation model, reconstruct the complex amplitude map generated in step S3 into a multi-view image sequence; S5: Map and adapt the multi-view image sequence reconstructed in step S4 to the terminal display device for output display.

2. The holographic generation method based on generative artificial intelligence according to claim 1, characterized in that, S1 generates a structured tensor based on the image information and the depth information by: normalizing the pixel values ​​of the image information and the depth values ​​of the depth information respectively, and then generating a structured tensor containing spatial position and viewpoint direction based on the normalized image information and the depth information.

3. The holographic generation method based on generative artificial intelligence according to claim 1, characterized in that, The structural latent variables extracted from S2 are represented as follows: In the formula, Indicates pixel position The structural latent variable w after aggregating all perspectives i This represents the weight of the i-th viewpoint. Indicates pixel position The local structure encoding of the i-th viewpoint, where N represents the total number of viewpoints; The weight of the i-th viewpoint is expressed as: In the formula, Let ρ be the angular difference between the i-th viewpoint and the main viewpoint, σ be an adjustable attenuation control parameter, and ρ be the angle difference between the i-th viewpoint and the main viewpoint. i is the entropy index of image information within the i-th viewpoint.

4. The holographic generation method based on generative artificial intelligence according to claim 3, characterized in that, The entropy index of image information content within each viewpoint is calculated using the grayscale channel entropy of the image from each viewpoint. The entropy index of image information content within the i-th viewpoint is expressed as: Where, k c This represents the number of channels in an image. Indicates the kth c The pixel probability distribution of an image with each channel.

5. The holographic generation method based on generative artificial intelligence according to claim 1, characterized in that, S3 uses a diffusion model architecture to generate a complex amplitude diagram of the target scene, wherein phase continuity constraints and energy balance constraints are introduced in the process of generating the complex amplitude diagram of the target scene.

6. The holographic generation method based on generative artificial intelligence according to claim 5, characterized in that, The phase continuity constraint is implemented by using the gradient direction of the structural latent variable as a guiding constraint for phase change. The formula for the phase continuity constraint is as follows: in, pixel position Phase diagram at time step t+1 The pixel positions in the structural control tensor for the compression of the structural latent variables. In the content of the corresponding phase channel, This represents the gradient direction in space, where η is the structure adjustment factor. pixel position Complex noise disturbance at time step t; Preferably, the energy balance constraint includes constraining the overall energy range of the amplitude portion of the generated complex amplitude map of the target scene. When performing the energy balance constraint, the following loss function is used: in, This represents the loss of the complex amplitude generation model during training, where H and W represent the image height and width, respectively. Indicates pixel position The amplitude energy density, κ is the target energy constant; β represents the angle between the phase change direction and the structural control direction, and is an adjustment parameter.

7. The holographic generation method based on generative artificial intelligence according to claim 1, characterized in that, S4 includes: S41: Based on the physical propagation model, the Fourier diffraction propagation principle is used to simulate the wavefront propagation of the complex amplitude image at different viewpoints. The complex amplitude image is propagated from the main viewpoint to the offset viewpoint using the following formula, and spatial movement is achieved through frequency domain phase modulation: in, This represents the intensity image at the offset viewpoint. and These represent the two-dimensional Fourier transform and its inverse transform, respectively. Represents a complex amplitude image. For frequency domain coordinates, Let be the displacement vector of the k-th output viewpoint; S42: Perform phase coherence correction between multiple views during reconstruction: calculate the phase fusion residual between adjacent views, and determine whether to perform interpolation compensation based on the residual. The following formula is used to verify the consistency of phase information between adjacent views before generating the multi-view image sequence: Among them, R k This represents the phase fusion residual score of the k-th output view. Indicates pixel position phase, Let be the displacement vector of the k-th output viewpoint.

8. The holographic generation method based on generative artificial intelligence according to claim 1, characterized in that, S5 includes: S51: By calculating the cosine similarity between the viewpoint of the multi-view image sequence and the target viewpoint of the terminal display device, a viewpoint allocation function is set to allocate appropriate reconstructed image frames to the device viewpoint, wherein the viewpoint allocation function is: Where, ω k It is a weighting factor for image quality. This represents the angle of the k-th output viewpoint. This represents the j-th viewing angle supported by the terminal display device; S52: Construct an exponential decay function centered on the main viewpoint to compress the frequency and frame rate of non-main viewpoint frames in the current frame sequence, so as to dynamically adjust the output priority and refresh rate of images from different viewpoints. The exponential decay function is as follows: Where, ρ k This represents the refresh weight of the k-th output viewpoint in the output frame sequence, where λ is the attenuation coefficient. Let Z represent the perspective of the main viewpoint, and Z be the normalization term that makes ∑ k ρ k =1; S53: Before outputting the frame sequence through the terminal display device, a priority allocation function is introduced to adjust the frame refresh priority, wherein the priority allocation function is: Where, γ k σ is the interaction priority coefficient for the k-th output viewpoint. s The Sigmoid function is used for smoothing. This represents the movement trend vector constructed using a sliding window.

9. A holographic generation system based on generative artificial intelligence, characterized in that, include: A multi-view acquisition module is used to acquire image information and depth information of the target scene from different viewpoints, and generate a structured tensor based on the image information and the depth information; The joint encoding module, connected to the multi-view acquisition module, is used to perform spatial-directional joint encoding on the structured tensor generated by the multi-view acquisition module to extract structural latent variables; A complex amplitude generation module, connected to the joint coding module, is used to generate a complex amplitude map of the target scene using the structural latent variables extracted by the joint coding module as control signals. The complex amplitude map includes amplitude information and phase information. The physical reconstruction module, connected to the complex amplitude generation module, is used to reconstruct the complex amplitude map generated by the complex amplitude generation module into a multi-view image sequence based on the physical propagation model. The output docking module is connected to the physical reconstruction module and is used to map and adapt the multi-view image sequence reconstructed by the physical reconstruction module to the terminal display device for output display.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to be run by a processor to perform the holographic generation method based on generative artificial intelligence as described in any one of claims 1 to 8.

Images