A holographic display method based on AIGC

Abstract

The application relates to the fields of optical display and artificial intelligence, in particular to an intelligent holographic display system, in particular to a three-dimensional display system using an artificial intelligence content generation model (AIGC) and computer holography, which can significantly improve system interaction performance. The system comprises a computer, a spatial light modulator, a laser, a lens and a support structure. According to the description of the object to be displayed by the observer, a three-dimensional digital scene is generated by a three-dimensional generation model loaded by the computer; then the three-dimensional scene is projected on different observation surfaces to obtain RGB-D images under different viewing angles; then corresponding computer holograms are generated based on the RGB-D images; finally, the holograms are loaded onto the spatial light modulator and illuminated by the laser, so that the display of the above-mentioned three-dimensional scene is realized. The application combines AIGC with holographic display, and uses the powerful pattern generation capability of AIGC, so that the display system can directly display the personalized customized three-dimensional scene based on the related description of the observer.

Description

Technical Field

[0001] The present invention relates to the fields of optical display and artificial intelligence, and in particular to an interactive intelligent holographic display method. Background Art

[0002] Holography uses light interference recording and diffraction to fully record and reproduce the wavefront information of an object, enabling 3D displays. Among numerous 3D display solutions, holographic display technology is the only one that can provide all three-dimensional visual information, achieving a realistic and natural display effect.

[0003] Early holographic display technology required the use of photosensitive materials and a complex interference process to record the object's wavefront information. The advent of computers made it possible to simulate and generate holograms. Existing computer-generated hologram (CGH) methods achieve holographic display through three main steps: 1) first modeling the 3D scene to be displayed; 2) using a phase retrieval algorithm to generate a hologram capable of displaying the 3D scene in space; and 3) loading the designed hologram onto a spatial light modulator and illuminating it with a light source to display the preset 3D scene. Research in the field of holographic display has primarily focused on the latter two steps: first, at the algorithmic level, studying how to quickly generate holograms and achieve high-quality 3D display effects; and second, at the hardware level, studying spatial light modulators with large size, high resolution, and high spatiotemporal bandwidth product to meet the application requirements of holographic display, which require large size, large field of view, and high refresh rate.

[0004] In recent years, deep learning has been applied to the field of holographic display. This method, trained on large amounts of data, develops a neural network model capable of rapidly generating holograms. Holograms with a resolution of 1920×1080 pixels can be generated at 60Hz on a single consumer-grade graphics processing unit. Furthermore, technological advancements have increased the refresh rate of spatial light modulators to the kHz level. Metasurface-based modulation schemes can manipulate light signals at subwavelength scales, enabling holographic displays with large fields of view. Clearly, advances in algorithms and hardware have made fast, high-quality holographic displays possible, and holographic display is expected to become a key technology in the metaverse concept.

[0005] However, the generation of 3D digital models, the first step in holographic display, has not received widespread attention. Existing methods generally rely on manual design or 3D imaging of real scenes to create 3D digital models. Most of these generated 3D models are merely projections of 3D objects at a specific viewing angle, not true 3D models, such as RGB-D images. Furthermore, the process of obtaining 3D digital models lacks human-computer interaction, making it impossible to intelligently display 3D scenes based on the needs of ordinary observers. Instead, specialized modeling tools must be used by professionals to create 3D digital models based on the observer's needs.

[0006] The recent rapid development of AI-generated content (AIGC) has brought new opportunities to numerous fields. Text-to-3D methods, such as DreamFusion and Magic3D, can automatically generate 3D models based on user descriptions. Existing holographic display methods have not yet effectively utilized AIGC. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a holographic display method based on AIGC, which can effectively solve the problems of difficulty in constructing three-dimensional digital models and lack of human-computer interaction in holographic display.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A holographic display method based on AIGC is characterized by comprising:

[0010] Step 1. Convert the three-dimensional virtual scene described by the observer into a three-dimensional digital model;

[0011] Step 2. Projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles and generating corresponding holograms;

[0012] Step 3. Based on the observer's needs, holograms at different viewing angles are loaded onto the spatial light modulator, and a laser beam is irradiated onto the spatial light modulator to display three-dimensional scenes at different viewing angles.

[0013] Furthermore, the step 1. converting the three-dimensional virtual scene described by the observer into a three-dimensional digital model specifically includes:

[0014] S1.1: The observer describes the 3D scene to be displayed, including the appearance, category, relative positional relationship between objects, color, texture, and / or theme of the objects, to obtain a 3D virtual scene description file. The 3D virtual scene description file is in text and / or voice.

[0015] S1.2: Convert the three-dimensional virtual scene description file into English text using translation software and speech recognition software;

[0016] S1.3: Generate a three-dimensional digital model of the English text using the three-dimensional digital modeling Text-to-3D method in AIGC.

[0017] Furthermore, the step 2. projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles and generating corresponding holograms specifically includes:

[0018] S2.1 divides the three-dimensional digital model O = f(x, y, z) into multiple viewing angles for projection, ensuring that the RGB-D image at each viewing angle can be displayed by the current hardware system;

[0019] S2.2 At a certain viewing angle RGB-D image under Denoted as:

[0020]

[0021] Where P(·) is the perspective projection sign operator, The angles between the center of an observation surface and the line connecting the circle and the x, y, and z coordinate axes are respectively. The range of the angle is 0-2π. The angle sampling interval is N≥1 determines the redundancy between RGB-D images obtained by projection from adjacent viewpoints, is the field of view of the holographic display, λ is the laser wavelength, and Δx is the pixel size of the display device.

[0022] S2.3 Generate a corresponding hologram based on the RGB-D image using the CGH method: use a pre-trained hologram generation model, take the RGB-D image as input, and output three holograms for displaying the depth and intensity information of the three channels R, G, and B respectively.

[0023] Furthermore, the observer requirement in step 3 is the viewing angle information of the three-dimensional object that the observer wants to view.

[0024] Furthermore, based on the observer's voice, text, action or gesture, the observer's needs are clarified, and the hologram corresponding to the object perspective that the observer wants to display is selected for loading into the spatial light modulator to meet the observer's display needs for three-dimensional objects at different perspectives.

[0025] Furthermore, the spatial light modulator in step 3 is a liquid crystal spatial light modulator or a programmable dynamic metasurface.

[0026] Furthermore, the laser in step 3 includes R, G, and B three-color lasers, which are respectively used to illuminate the holograms for displaying R, G, and B channels, thereby realizing the display of RGB-D images.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] By integrating holographic display with AIGC, this invention addresses the difficulties of generating 3D digital models and the lack of human-computer interaction in holographic displays. It also enriches the presentation of AIGC-generated content. The holographic display system designed based on the proposed method can directly display a virtual 3D scene based on the observer's description of the scene. Furthermore, the observer can select the viewing angle of the currently displayed 3D scene, realizing a new type of intelligent holographic display. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of obtaining a three-dimensional digital scene using AIGC in steps S1-S3 of the present invention.

[0030] Figure 2 It is a schematic diagram of projecting a three-dimensional digital scene to obtain RGB-D images at different viewing angles in step S4 of the present invention.

[0031] Figure 3 It is a schematic diagram of the system device for loading the hologram in step S7 of the present invention.

[0032] Figure 4 It is a flowchart for describing the steps of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below with reference to the accompanying drawings and in combination with preferred embodiments, but this should not limit the scope of protection of the present invention.

[0034] A holographic display method of AIGC specifically comprises the following steps:

[0035] S1: The observer describes the 3D scene to be displayed: The observer can subjectively imagine a 3D scene and then describe the scene. The description method includes text and voice. The description content includes the shape, category, relative position relationship between objects, color, texture, and theme of the objects.

[0036] S2: Recognizing the description as English text: uniformly converting the text and speech describing the three-dimensional virtual scene into English, and the conversion is completed by using text translation and speech recognition.

[0037] S3: Generating a 3D digital model based on the text using Text-to-3D: Using the Text-to-3D method for 3D digital modeling in the AIGC field, a 3D digital model is rapidly generated that matches the viewer's description. The Text-to-3D method is a large AIGC model with advantages in both speed and visual quality, and it also provides contextualization. Based on the differences between the generated 3D digital scene or the 3D scene displayed in step S7 and the viewer's fictitious 3D scene, the viewer can use a new description to enable the system to display a 3D scene that better meets their requirements.

[0038] Figure 1 This is a schematic diagram of steps S1-S3 of the present invention, illustrating the use of AIGC to generate a three-dimensional digital scene. As can be seen, through steps S1-S3, an observer can easily and automatically convert an imagined three-dimensional virtual scene into a three-dimensional digital scene. The three-dimensional scene described in step S1 is created by the observer. The observer's description of the displayed three-dimensional scene can be manually entered text or spoken speech.

[0039] S4: Project the generated 3D digital model at different viewing angles to obtain RGB-D images at different viewing angles: The field of view of the holographic display is twice the diffraction angle:

[0040]

[0041] Where λ is the laser wavelength and Δx is the pixel size of the display device.

[0042] The pixel size of existing spatial light modulators is in the micron range, and the field of view of holographic display under visible light illumination is limited to about 10 degrees. Therefore, the full picture of the generated three-dimensional digital scene cannot be displayed. In step S4, the three-dimensional digital scene is divided into RGB-D images under multiple viewing angles to ensure that the RGB-D image under each viewing angle can be displayed by the current hardware system. The three-dimensional scene is depicted by switching the display of RGB-D images of different viewing angles. Specifically, Figure 2 As shown, the three-dimensional digital scene O = f(x, y, z) is projected at a certain viewing angle, and the RGB-D image at that viewing angle is recorded as

[0043]

[0044] Where P(·) is the perspective projection sign operator, Perspective The RGB-D image obtained by down-projection is are the angles between the center of an observation surface and the line connecting the circle and the x, y, and z coordinate axes respectively. The range of the angle is 0-2π, and the angle sampling interval is N ≥ 1 determines the redundancy between RGB-D images obtained by projecting adjacent viewpoints.

[0045] The projection of the three-dimensional digital model on the viewing angle in step S4 is calculated to obtain the information distribution of the three-dimensional digital model at a certain viewing angle. The RGB-D image is the information distribution at the above-mentioned certain viewing angle.

[0046] S5: Based on the RGB-D image, a corresponding hologram is rapidly generated using the CGH method: a pre-trained hologram generation network model is used, taking the RGB-D image as input and rapidly outputting the generated hologram. The resulting hologram can be loaded onto a spatial light modulator for high-quality display of the RGB-D image.

[0047] The CGH method used in step S5 is deep learning. The CGH method based on deep learning takes into account both the speed of generating holograms and the holographic display effect. The data used to train the neural network include: a large number of RGB-D images and corresponding high-quality label holograms. The holograms include: Fresnel holograms and Fourier transform holograms. The RGB-D images correspond to three holograms for displaying three-dimensional information of R, G, and B channels respectively. In one embodiment of the present invention, the acquisition method of the RGB-D image includes: projection of the three-dimensional digital model pre-generated by Text-to-3D at various viewing angles, open source RGB-D images (such as MIT-CGH-4K), manually designed RGB-D images, and three-dimensional imaging results of actual objects. The high-quality label hologram can display a high-quality three-dimensional scene, that is, the corresponding RGB-D image.

[0048] S6: Loading holograms at different viewing angles onto the spatial light modulator based on the observer's needs: Due to the limited field of view, a hologram can only display the 3D information of a 3D object within a certain viewing angle range. By loading holograms at different viewing angles, the 3D information of a 3D object can be displayed at different viewing angles. Based on the observer's voice, text, movements, gestures, etc., the observer's needs are determined, and the hologram corresponding to the desired viewing angle of the object is selected and loaded into the spatial light modulator.

[0049] S7: Laser irradiation of spatial light modulator to realize the display of three-dimensional objects at different viewing angles: Figure 3 The holographic display device shown uses three R, G, and B laser beams to illuminate the three R, G, and B holograms loaded into the spatial light modulator in sequence, thereby realizing the display of the RGB-D scene in the target area.

[0050] S8: Repeat steps S1 to S7 for a new 3D scene to be displayed: Figure 4 As shown, when the observer wants to display a new three-dimensional scene or modify the current display scene, the above steps are repeated.

[0051] Furthermore, in one embodiment of the present invention, the method for obtaining the high-quality label hologram is: based on the current RGB-D image, using methods such as Gerchberg-Saxton, Wirtinger Holography, camera-in-the-loop (CITL) optimization, etc., to obtain a hologram with the best display effect for the current RGB-D image.

[0052] The neural network architectures used in deep learning-based CGH methods include convolution, residual connections, and the Transformer architecture. The neural network training methods used in deep learning-based CGH methods include data-driven supervised learning, physical model-driven self-supervised learning, and semi-supervised learning driven by a combination of physical models and data. These physical models include diffraction propagation physics models such as angular spectrum diffraction, Fresnel diffraction, and Fraunhofer diffraction.

[0053] The spatial light modulator used to generate the hologram in step S6 can be phase-controlled, amplitude-controlled, or simultaneously amplitude- and phase-controlled. Implementations of these spatial light modulators include liquid crystal spatial light modulators (LC-SLMs) and programmable dynamic metasurfaces. In step S6, the RGB-D image can be displayed at a specific viewing angle, depending on the viewer's needs.

[0054] In one embodiment of the present invention, the observer's demand expression method includes: the observer selecting body language, text description, and voice description.

[0055] The display system used in step S7 includes: a laser, a spatial light modulator, a beam splitter, a lens, and a support structure. The laser used in step S7 generates red, green, and blue laser light. Furthermore, in one embodiment of the present invention, the spatial light modulator used in step S7 sequentially loads holograms for displaying three-channel R, G, and B three-dimensional information. The spatial light modulator loaded with the hologram for displaying three-channel R, G, and B three-dimensional information is illuminated by R, G, and B laser light, respectively. Display of the RGB-D image can be achieved by time multiplexing and spatial multiplexing. The time multiplexing specifically includes: the hologram for displaying three-channel R, G, and B three-dimensional information is loaded into the same spatial position and illuminated sequentially by R, G, and B laser light. The spatial multiplexing specifically includes: the hologram for displaying three-channel R, G, and B three-dimensional information is loaded into different spatial positions and illuminated simultaneously by R, G, and B laser light.

[0056] A preferred embodiment of the present invention employs an AIGC-based holographic display method and device. Based on an observer's description of the displayed object, a computer-loaded 3D generative model generates a 3D digital scene. This 3D scene is then projected onto different viewing surfaces to produce RGB-D images from different viewing angles. Based on these RGB-D images, corresponding computational holograms are generated. Finally, these holograms are loaded onto a spatial light modulator and illuminated by a laser, thereby displaying the 3D scene. This invention combines AIGC with holographic display, leveraging AIGC's powerful pattern generation capabilities to enable the display system to display a customized 3D scene directly based on the observer's description of the scene.

[0057] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that several equivalent substitutions or obvious variations can be made without departing from the scope of the present invention, and that any equivalent performance or application should be considered to fall within the scope of protection of the present invention.

Claims

1. A holographic display method based on AIGC, characterized in that: include: Step 1. Convert the three-dimensional virtual scene described by the observer into a three-dimensional digital model; Step 2. Projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles and generating corresponding holograms; Step 3. Based on the observer's needs, holograms at different viewing angles are loaded onto a spatial light modulator. The laser beam emitted by the laser illuminates the spatial light modulator to display different viewing angles of the three-dimensional scene. The step 1. converting the three-dimensional virtual scene described by the observer into a three-dimensional digital model specifically includes: S1.1: The observer describes the 3D scene to be displayed, including the appearance, category, relative positional relationship between objects, color, texture, and / or theme of the objects, to obtain a 3D virtual scene description file. The 3D virtual scene description file is in text and / or voice. S1.2: Convert the three-dimensional virtual scene description file into English text using translation software and speech recognition software; S1.3: Generate a three-dimensional digital model of the English text using the three-dimensional digital modeling Text-to-3D method in AIGC.

2. The AIGC-based holographic display method according to claim 1, characterized in that: The step 2. projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles and generating corresponding holograms specifically includes: S2.1 divides the three-dimensional digital model O = f(x, y, z) into multiple viewing angles for projection, ensuring that the RGB-D image at each viewing angle can be displayed by the current hardware system; S2.2 At a certain viewing angle RGB-D image under Denoted as: Where P(·) is the perspective projection sign operator, The angles between the center of an observation surface and the line connecting the circle and the x, y, and z coordinate axes are respectively. The range of the angle is 0-2π. The angle sampling interval is N≥1 determines the redundancy between RGB-D images obtained by projection from adjacent viewpoints, is the field of view of the holographic display, λ is the laser wavelength, △x is the pixel size of the display device; S2.3 Generate a corresponding hologram based on the RGB-D image using the CGH method: use a pre-trained hologram generation model, take the RGB-D image as input, and output three holograms for displaying the depth and intensity information of the three channels R, G, and B respectively.

3. The AIGC-based holographic display method according to claim 1, characterized in that: The observer requirement in step 3 is the viewing angle information of the three-dimensional object that the observer wants to view.

4. The AIGC-based holographic display method according to claim 3, characterized in that: According to the observer's voice, text, action or gesture, the observer's needs are clarified, and the hologram corresponding to the object viewing angle that the observer expects to display is selected for loading into the spatial light modulator to meet the observer's display needs for three-dimensional objects at different viewing angles.

5. The AIGC-based holographic display method according to claim 1, characterized in that: The spatial light modulator in step 3 is a liquid crystal spatial light modulator or a programmable dynamic metasurface.

6. The AIGC-based holographic display method according to claim 1, characterized in that: In step 3, the laser includes R, G, and B three-color lasers, which are respectively used to illuminate the holograms for displaying R, G, and B channels, thereby realizing the display of RGB-D images.

Images