Holographic display method based on AIGC

Abstract

The invention relates to the field of optical display and artificial intelligence, in particular to an intelligent holographic display system, in particular to a three-dimensional display system using an artificial intelligence content generation model (AIGC) and computer-generated holography, and the interaction performance of the system can be remarkably improved. The system comprises a computer, a spatial light modulator, a laser, a lens and a supporting structure. Generating a three-dimensional digital scene by a three-dimensional generation model loaded by a computer according to the description of an observer on a to-be-displayed object; then projecting the three-dimensional scene on different observation surfaces to obtain RGB-D images under different visual angles; generating a corresponding computer-generated hologram based on the RGB-D images; and finally, the holograms are loaded to a spatial light modulator and are illuminated by a laser, so that the display of the three-dimensional scene is realized. According to the method, the AIGC and holographic display are combined, and the powerful mode generation capability of the AIGC is utilized, so that the display system can display the scene directly based on the relevant description of an observer on the personalized customized three-dimensional scene.

Description

Technical Field

[0001] The present invention relates to the fields of optical display and artificial intelligence, and in particular to an interactive intelligent holographic display method. Background Art

[0002] Holography can completely record and reproduce the wavefront information of an object by recording and reproducing light interference and diffraction, so it can be used to realize three-dimensional display. Among many three-dimensional display schemes, holographic display technology is the only one that can provide all three-dimensional visual information and achieve a real and natural display effect.

[0003] Early holographic display technology required the use of photosensitive materials and a complex interference process to record the wavefront information of the object. The emergence of computers made it possible to simulate and calculate the generation of holograms. The existing methods based on computer-generated holograms (CGH) mainly achieve holographic display through three steps: 1) first model the three-dimensional scene to be displayed; 2) use the phase recovery algorithm to generate a hologram that can display the three-dimensional scene in space; 3) load the designed hologram onto the spatial light modulator and illuminate it with a light source to realize the display of the preset three-dimensional scene. Research in the field of holographic display mainly focuses on the latter two steps: first, at the algorithm level, study how to quickly generate holograms and achieve high-quality three-dimensional display effects; second, at the hardware level, study large-size, high-resolution and time-space bandwidth product spatial light modulators to achieve the application requirements of holographic display for large size, large field of view and high refresh rate.

[0004] In recent years, deep learning has been used in the field of holographic display. This method obtains a neural network model that can quickly generate holograms by training on a large amount of data. It can generate holograms with a resolution of 1920×1080 pixels at 60Hz on a single consumer-grade graphics processing unit. On the other hand, due to technological advances, the refresh rate of spatial light modulators has also been increased to the KHz level; the modulation scheme based on metasurfaces can regulate light signals at a subwavelength scale, and can achieve holographic display with a large field of view. It can be seen that the advancement of algorithms and hardware has made it possible to achieve fast and high-quality holographic display, and holographic display is also expected to become one of the key technologies in the concept of the metaverse.

[0005] However, as the first step of holographic display, the generation of three-dimensional digital models has not received widespread attention. Existing methods generally obtain three-dimensional digital models through manual design or three-dimensional imaging of real scenes. Most of the generated three-dimensional models are only projections of three-dimensional objects at a certain perspective, not true three-dimensional models, such as RGB-D images. In addition, the process of obtaining three-dimensional digital models lacks human-computer interaction, and it is impossible to intelligently display three-dimensional scenes according to the needs of ordinary observers. Instead, professional modeling tools are required for professionals to build three-dimensional digital models according to the needs of observers.

[0006] Recently, the rapid development of AI Generated Content (AIGC) has brought new opportunities to many fields. Among them, Text-to-3D methods represented by DreamFusion and Magic3D can automatically generate 3D models based on user descriptions. Existing holographic display methods have not effectively utilized AIGC. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a holographic display method based on AIGC, which can effectively solve the problems of difficulty in constructing three-dimensional digital models and lack of human-computer interaction in holographic display.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A holographic display method based on AIGC is characterized by comprising:

[0010] Step 1. Convert the three-dimensional virtual scene described by the observer into a three-dimensional digital model;

[0011] Step 2. Projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles, and generating corresponding holograms;

[0012] Step 3. According to the needs of the observer, the holograms at different viewing angles are loaded onto the spatial light modulator, and the laser beam illuminates the spatial light modulator to realize the display of different viewing angles of the three-dimensional scene.

[0013] Furthermore, the step 1. converting the three-dimensional virtual scene described by the observer into a three-dimensional digital model specifically includes:

[0014] S1.1: The observer describes the 3D scene to be displayed, including the appearance, category, relative position relationship between objects, color, texture and / or theme of the objects, and obtains a 3D virtual scene description file, which is in the form of text and / or voice;

[0015] S1.2: converting the three-dimensional virtual scene description file into English text using translation software and speech recognition software;

[0016] S1.3: Generate a three-dimensional digital model of the English text using the three-dimensional digital modeling Text-to-3D method in AIGC.

[0017] Furthermore, the step 2. projects the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles, and generates corresponding holograms, specifically including:

[0018] S2.1 divides the three-dimensional digital model O = f(x, y, z) into multiple viewing angles for projection, ensuring that the RGB-D image at each viewing angle can be displayed by the current hardware system;

[0019] S2.2 At a certain viewing angle RGB-D image under Denoted as:

[0020]

[0021] Where P(·) is the perspective projection sign operator, The angles between the center of an observation surface and the line connecting the circle and the x, y, and z coordinate axes are respectively, and the value range of the angle is 0-2π. The angle sampling interval N≥1 determines the redundancy between RGB-D images obtained by projection from adjacent viewpoints. is the field of view of the holographic display, λ is the laser wavelength and Δx is the pixel size of the display device.

[0022] S2.3 Generate a corresponding hologram based on the RGB-D image using the CGH method: use a pre-trained hologram generation model, take the RGB-D image as input, and output three holograms for displaying the depth and intensity information of the three channels R, G, and B respectively.

[0023] Furthermore, the observer requirement in step 3 is the viewing angle information of the three-dimensional object that the observer wants to view.

[0024] Furthermore, based on the observer's voice, text, action or gesture, the observer's needs are clarified, and the hologram corresponding to the object viewing angle that the observer expects to display is selected for loading into the spatial light modulator to meet the observer's display needs for three-dimensional objects at different viewing angles.

[0025] Furthermore, the spatial light modulator in step 3 is a liquid crystal spatial light modulator or a programmable dynamic metasurface.

[0026] Furthermore, in step 3, the laser includes three-color lasers of R, G, and B, which are respectively used to illuminate the holograms for displaying the R, G, and B channels, thereby realizing the display of RGB-D images.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention solves the problem of difficulty in generating three-dimensional digital models and lack of human-computer interaction in holographic display by integrating holographic display with AIGC, and also enriches the presentation form of AIGC-generated content. The holographic display system designed based on the proposed method can directly display the virtual three-dimensional scene according to the observer's description of the virtual three-dimensional scene; in addition, the observer can also select the viewing angle of the currently displayed three-dimensional scene to realize a new type of intelligent holographic display. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of obtaining a three-dimensional digital scene using AIGC in steps S1-S3 of the present invention.

[0030] Figure 2 It is a schematic diagram of projecting a three-dimensional digital scene to obtain RGB-D images at different viewing angles in step S4 of the present invention.

[0031] Figure 3 It is a schematic diagram of a system device for loading a hologram in step S7 of the present invention.

[0032] Figure 4 It is a flowchart for describing the steps of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below with reference to the accompanying drawings and in combination with preferred embodiments, but this should not limit the protection scope of the present invention.

[0034] A holographic display method of AIGC specifically comprises the following steps:

[0035] S1: The observer describes the three-dimensional scene to be displayed: The observer can subjectively imagine a three-dimensional scene and then describe the scene. The description method includes text and voice. The description content includes the shape, category, relative position relationship between objects, color, texture, and theme of the objects.

[0036] S2: Recognize the description as English text: uniformly convert the text and speech describing the three-dimensional virtual scene into English, and the conversion is completed by using text translation and speech recognition.

[0037] S3: Generate a three-dimensional digital model by Text-to-3D according to the text: Use the Text-to-3D method related to three-dimensional digital modeling in the AIGC field to quickly generate a three-dimensional digital model that fits the description of the observer. The Text-to-3D method is an AIGC large model with advantages in both generation speed and visual effects, and has the function of connecting context. The observer can use a new description based on the difference between the generated three-dimensional digital scene or the three-dimensional scene displayed in step S7 and the observer's fictitious three-dimensional scene so that the system can display a three-dimensional scene that better meets the requirements.

[0038] Figure 1 This is a schematic diagram of using AIGC to obtain a three-dimensional digital scene in steps S1-S3 of the present invention. As can be seen from the figure, through steps S1-S3, the observer can easily convert the imagined three-dimensional virtual scene into a three-dimensional digital scene. The three-dimensional scene in step S1 is fictionalized by the observer. The observer's description of the three-dimensional scene to be displayed can be text manually input by the observer, or his or her spoken voice.

[0039] S4: Project the generated three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles: The viewing angle of the holographic display is twice the diffraction angle:

[0040]

[0041] Where λ is the laser wavelength and Δx is the pixel size of the display device.

[0042] The pixel size of existing spatial light modulators is in the micrometer range, and the field of view of holographic display under visible light illumination is limited to about 10 degrees. Therefore, the full picture of the generated three-dimensional digital scene cannot be displayed. In step S4, the three-dimensional digital scene is divided into RGB-D images under multiple viewing angles to ensure that the RGB-D image under each viewing angle can be displayed by the current hardware system, and the three-dimensional scene is depicted by switching the display of RGB-D images of different viewing angles. Specifically, Figure 2 As shown in the figure, the three-dimensional digital scene O = f(x, y, z) is projected at a certain viewing angle, and the RGB-D image at that viewing angle is recorded as

[0043]

[0044] Where P(·) is the perspective projection sign operator, For perspective The RGB-D image obtained by down-projection is are the angles between the center of a certain observation surface and the line connecting the circle and the x, y, and z coordinate axes respectively. The value range of the angle is 0-2π, and the angle sampling interval is N ≥ 1 determines the redundancy between RGB-D images obtained by projection from adjacent viewpoints.

[0045] The projection of the three-dimensional digital model on the viewing angle is calculated in step S4 to obtain the information distribution of the three-dimensional digital model at a certain viewing angle. The RGB-D image is the information distribution at the above-mentioned certain viewing angle.

[0046] S5: Based on the RGB-D image, a corresponding hologram is quickly generated using the CGH method: a pre-trained hologram generation network model is used, the RGB-D image is used as input, and the generated hologram is quickly output. The obtained hologram can be loaded onto a spatial light modulator to realize high-quality display of the RGB-D image.

[0047] The CGH method used in step S5 is deep learning, and the CGH method based on deep learning takes into account both the speed of generating holograms and the holographic display effect. The data used to train the neural network include: a large number of RGB-D images and corresponding high-quality label holograms. The holograms include: Fresnel holograms and Fourier transform holograms. The RGB-D image corresponds to three holograms for displaying three-dimensional information of R, G, and B channels respectively. In one embodiment of the present invention, the acquisition method of the RGB-D image includes: the projection of the three-dimensional digital model pre-generated by Text-to-3D at various viewing angles, open source RGB-D images (such as MIT-CGH-4K), manually designed RGB-D images, and three-dimensional imaging results of actual objects. The high-quality label hologram can display a high-quality three-dimensional scene, that is, the corresponding RGB-D image.

[0048] S6: Loading holograms at different viewing angles onto the spatial light modulator according to the observer's needs: Due to the limited field of view, a hologram can only display the three-dimensional information of a three-dimensional object within a certain viewing angle range. By loading holograms at different viewing angles, the three-dimensional information of a three-dimensional object at different viewing angles can be displayed. According to the observer's voice, text, action, gesture, etc., the observer's needs are clarified, and the hologram corresponding to the object viewing angle that the observer expects to display is selected for loading into the spatial light modulator.

[0049] S7: Laser irradiation of spatial light modulator to realize the display of three-dimensional objects at different viewing angles: Figure 3 The holographic display device shown in the figure uses three R, G, and B laser beams to illuminate the three R, G, and B holograms loaded into the spatial light modulator in sequence, thereby realizing the display of the RGB-D scene in the target area.

[0050] S8: Repeat steps S1 to S7 for a new 3D scene to be displayed. Figure 4 As shown, when the observer wants to display a new three-dimensional scene or modify the currently displayed scene, the above steps are repeated.

[0051] Furthermore, in one embodiment of the present invention, the method for obtaining the high-quality label hologram is: based on the current RGB-D image, using Gerchberg-Saxton, Wirtinger Holography, camera-in-the-loop (CITL) optimization and other methods, to obtain a hologram with the best display effect for the current RGB-D image.

[0052] The neural network architecture used in the CGH method based on deep learning includes: convolution, residual connection and Transformer architecture. The training methods of the neural network used in the CGH method based on deep learning are: data-driven supervised learning, physical model-driven self-supervised learning, physical model and data-driven semi-supervised learning, etc. The physical models include: angular spectrum diffraction, Fresnel diffraction, Fraunhofer diffraction and other diffraction propagation physical models.

[0053] The spatial light modulator used to load and generate the hologram in step S6 can be: phase type, amplitude type, amplitude and phase simultaneously controlled. The implementation of the spatial light modulator includes: liquid crystal spatial light modulator (LC-SLM), programmable dynamic metasurface. In step S6, the RGB-D image under a certain viewing angle can be displayed according to the needs of the observer.

[0054] In one embodiment of the present invention, the observer's demand expression method includes: the observer selects body language, text description, and voice description.

[0055] The display system device used in step S7 includes: a laser, a spatial light modulator, a beam splitter, a lens, and a supporting structure. The laser used in step S7 generates red, green, and blue lasers. Further, in one embodiment of the present invention, the spatial light modulator used in step S7 sequentially loads a hologram for displaying three-dimensional information of three channels of R, G, and B. The spatial light modulator loaded with the hologram for displaying three-dimensional information of three channels of R, G, and B is illuminated by three-color lasers of R, G, and B, respectively. The display of the RGB-D image can be realized by time multiplexing and space multiplexing. The time multiplexing is specifically: the hologram for displaying three-dimensional information of three channels of R, G, and B is loaded into the same spatial position, and is sequentially illuminated by three-color lasers of R, G, and B. The space multiplexing is specifically: the hologram for displaying three-dimensional information of three channels of R, G, and B is loaded into different spatial positions, and is illuminated by three-color lasers of R, G, and B at the same time.

[0056] A preferred embodiment of the present invention is a holographic display method and device based on AIGC. According to the observer's description of the object to be displayed, a three-dimensional digital scene is generated by a three-dimensional generation model loaded by a computer; then the three-dimensional scene is projected on different observation surfaces to obtain RGB-D images at different viewing angles; then corresponding computational holograms are generated based on these RGB-D images; finally, these holograms are loaded onto a spatial light modulator and illuminated by a laser, thereby realizing the display of the above three-dimensional scene. The present invention combines AIGC with holographic display and utilizes the powerful pattern generation capability of AIGC, so that the display system can directly display the scene based on the observer's description of the personalized three-dimensional scene.

[0057] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A holographic display method based on AIGC, characterized in that: include: Step 1. Convert the three-dimensional virtual scene described by the observer into a three-dimensional digital model; Step 2. Projecting the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles, and generating corresponding holograms; Step 3. According to the needs of the observer, the holograms at different viewing angles are loaded onto the spatial light modulator, and the laser beam illuminates the spatial light modulator to realize the display of different viewing angles of the three-dimensional scene.

2. The holographic display method based on AIGC according to claim 1, characterized in that: The step 1. converting the three-dimensional virtual scene described by the observer into a three-dimensional digital model specifically comprises: S1.1: The observer describes the 3D scene to be displayed, including the appearance, category, relative position relationship between objects, color, texture and / or theme of the objects, and obtains a 3D virtual scene description file, which is in the form of text and / or voice; S1.2: converting the three-dimensional virtual scene description file into English text using translation software and speech recognition software; S1.3: Generate a three-dimensional digital model of the English text using the three-dimensional digital modeling Text-to-3D method in AIGC.

3. The holographic display method based on AIGC according to claim 1, characterized in that: The step 2. projects the three-dimensional digital model at different viewing angles to obtain RGB-D images at different viewing angles, and generates corresponding holograms, specifically including: S2.1 divides the three-dimensional digital model O = f(x, y, z) into multiple viewing angles for projection, ensuring that the RGB-D image at each viewing angle can be displayed by the current hardware system; S2.2 At a certain viewing angle RGB-D image under Denoted as: Where P(·) is the perspective projection sign operator, The angles between the center of an observation surface and the line connecting the circle and the x, y, and z coordinate axes are respectively, and the value range of the angle is 0-2π. The angle sampling interval N≥1 determines the redundancy between RGB-D images obtained by projection from adjacent viewpoints. is the field of view of the holographic display, λ is the laser wavelength and Δx is the pixel size of the display device. S2.3 Generate a corresponding hologram based on the RGB-D image using the CGH method: use a pre-trained hologram generation model, take the RGB-D image as input, and output three holograms for displaying the depth and intensity information of the three channels R, G, and B respectively.

4. The holographic display method based on AIGC according to claim 1, characterized in that: The observer requirement in step 3 is the viewing angle information of the three-dimensional object that the observer wants to view.

5. The AIGC-based holographic display method according to claim 4, characterized in that: According to the observer's voice, text, action or gesture, the observer's needs are clarified, and the hologram corresponding to the object viewing angle that the observer expects to display is selected for loading into the spatial light modulator to meet the observer's display needs for three-dimensional objects at different viewing angles.

6. The AIGC-based holographic display method according to claim 1, characterized in that: The spatial light modulator in step 3 is a liquid crystal spatial light modulator or a programmable dynamic metasurface.

7. The holographic display method based on AIGC according to claim 1, characterized in that: In the step 3, the laser includes three-color lasers of R, G, and B, which are respectively used to illuminate the holograms for displaying the R, G, and B channels, thereby realizing the display of RGB-D images.

Images