Holographic imaging system and method for VR holographic theater

Abstract

The invention discloses a holographic imaging system and method for a VR holographic theater, and relates to the technical field of imaging. Depth information, texture information and light field parameters are acquired to construct an initial light field, a machine learning algorithm is adopted to generate a self-adaptive light field based on audience position data, and the light field parameters are adjusted in real time; a holographic image can be dynamically optimized according to the viewing angle of audiences, image drift and distortion caused by position change are reduced, a view cone mapping algorithm is introduced to optimize parallax, and the projection angle is dynamically adjusted in combination with structured light projection and an optical modulator to compensate optical offset of different viewing positions; a light beam path is adjusted and optimized by combining a refractor array and AI optics, and is projected to a holographic display medium, so that the light field distribution of a holographic image is more uniform, meanwhile, the illumination condition in a theater is detected through an ambient light sensing unit, and the brightness, contrast and shadow adaptation of the image are dynamically adjusted based on a detection result. And the visual consistency can be kept in different illumination environments.

Description

Technical Field

[0001] The present invention relates to the field of imaging technology, and in particular to a holographic imaging system and method for a VR holographic theater. Background Art

[0002] With the continuous development of VR technology and the diversification of social needs, VR holographic theaters have emerged. Through holographic screen imaging, realistic three-dimensional images can be seen and viewed freely from different angles.

[0003] The current holographic imaging methods mainly use light field projection or interference imaging at a specific angle to provide clear and stable three-dimensional images at a fixed viewing angle. When the audience moves or watches from a larger angle, the image will mostly be distorted, drift, or even disappear. The image shape seen by the audience in the edge area does not match the actual content, and the three-dimensional sense is weakened.

[0004] It can be seen that the current position of the holographic theater has a great impact on the viewing experience. The core factor causing this problem is the calculation method of light field projection. Traditional holographic projection is a preset projection angle. When the audience deviates from the designed viewing angle, the optical path calculation is insufficient and the imaging algorithm cannot perform real-time correction. To solve this problem, some traditional solutions increase the number of projection arrays so that the audience can obtain relatively consistent three-dimensional images within a certain range, or use head tracking technology to dynamically adjust the projection angle according to the audience's viewing angle. However, these methods have extremely high computational costs and are difficult to achieve complete synchronization in multi-person scenarios. Therefore, how to enable audiences in different positions to see more stable and realistic holographic images is a technical problem that needs to be overcome urgently. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a holographic imaging system and method for VR holographic theater to solve the problem that traditional solutions are difficult to adapt to different viewing angles in real time in scenarios where multiple people are watching and moving freely, resulting in image distortion, drift or disappearance, affecting the immersive experience.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a holographic imaging system for a VR holographic theater, comprising:

[0009] A three-dimensional data acquisition module, used to acquire three-dimensional data of the holographic image and construct an initial light field based on the three-dimensional data;

[0010] A light field calculation module receives the initial light field and generates an adaptive light field based on the audience position data through a machine learning algorithm;

[0011] An optical adjustment module, based on an adaptive light field, uses a cone mapping algorithm to optimize parallax, and combines structured light projection. It adjusts the projection angle through an optical modulator to generate an optimized light field;

[0012] An optical modulation module, according to the optimized light field, optimizes the light beam path based on a refractive mirror array and AI optical adjustment, and projects the light field onto a holographic display medium;

[0013] An interaction system receives the optically optimized light field, combines the audience's position, movement, and gaze tracking data, and adjusts the light field parameters in real time.

[0014] As a preferred solution of a holographic imaging system for a VR holographic theater according to the present invention, wherein: the optical modulation module includes an ambient light sensing unit for obtaining the lighting conditions of the theater;

[0015] The optical modulation module further combines the ambient light sensing unit to detect the lighting conditions of the theater, and adjusts the light field brightness, contrast, and shadow adaptation based on the detection results.

[0016] In a second aspect, the present invention provides a holographic imaging method for a VR holographic theater, including,

[0017] Step S1, obtaining three-dimensional data of a holographic image, and constructing an initial light field based on the three-dimensional data;

[0018] Step S2, inputting the initial light field into a light field calculation module, generating an adaptive light field based on the audience position data through a machine learning algorithm, and dynamically adjusting the light field parameters;

[0019] Step S3, inputting the adaptive light field into the optical adjustment module, using a cone mapping algorithm to optimize parallax, and combining structured light projection. Adjust the projection angle through an optical modulator to compensate for the viewing position offset and generate an optimized light field;

[0020] Step S4, inputting the optimized light field into the optical modulation module, optimizing the light beam path based on a refractive mirror array and AI optical adjustment, and projecting it onto a holographic display medium;

[0021] In step S4, combine the ambient light sensing unit to detect the lighting conditions of the theater, and further adjust the light field brightness, contrast, and shadow adaptation based on the detection results;

[0022] Step S5, inputting the optically optimized light field of step S4 into the interaction system, combining the audience's position, movement, and gaze tracking data, and adjusting the light field parameters in real time;

[0023] In step S5, perform multi-view synchronous calibration.

[0024] As a preferred embodiment of the holographic imaging method for a VR holographic theater according to the present invention, wherein: the three-dimensional data includes depth information, texture information, and light field parameters.

[0025] As a preferred embodiment of the holographic imaging method for a VR holographic theater according to the present invention, wherein: the step of constructing an initial light field based on the three-dimensional data is as follows:

[0026] Assume that the three-dimensional data of the holographic image consists of depth information, texture information, and light field parameters, and define the spatial coordinate system of the target scene as P(x, y, z):

[0027] P(x, y, z) = (D(x, y), T(x, y), L(x, y, θ, φ)),

[0028] wherein, P(x, y, z) represents the three-dimensional coordinate point in the scene, x, y, and z respectively represent the spatial coordinate components, D(x, y) represents the depth information, describing the distance from the scene point to the reference plane, T(x, y) represents the texture information, including color, illumination, and reflection characteristics, and L(x, y, θ, φ) represents the light field parameter, defined as the light intensity and propagation characteristics in the direction (θ, φ) starting from the point (x, y).

[0029] Calculate the light intensity distribution based on the light field rendering equation, and the light intensity distribution formula is:

[0030] I(x, y, θ, φ) = ∫ λ R(x, y, λ)L(x, y, θ, φ, λ)dλ,

[0031] wherein, I(x, y, θ, φ) represents the light field intensity at the point (x, y) in the direction (θ, φ), λ represents the wavelength of light, ∫ λ · represents the integration over the wavelength within the spectral range to obtain the final visible light intensity, R(x, y, λ) represents the reflectivity of the object surface to light of different wavelengths, and L(x, y, θ, φ, λ) represents the light field parameter varying with wavelength.

[0032] Calculate the phase information of the light field using the light wave propagation formula, and the formula is:

[0033] Ψ(x, y, θ, φ) = e ikD(x,y) ,

[0034] wherein, Ψ(x, y, θ, φ) represents the phase information of the light wave, e is the base of the natural logarithm, i represents the imaginary unit, and satisfies i 2 = -1,

[0035] is the wave number, describing the spatial frequency of the light wave, and D(x, y) represents the depth information;

[0036] Then the initial light field formula is as follows:

[0037] F(x,y,θ,φ) = I(x,y,θ,φ)·Ψ(x,y,θ,φ),

[0038] where F(x,y,θ,φ) is the complex expression of the initial light field intensity, combining the light field intensity and phase information.

[0039] As a preferred solution of the holographic imaging method for a VR holographic theater according to the present invention, wherein: the step of generating an adaptive light field based on the audience position data through a machine learning algorithm and dynamically adjusting the light field parameters is as follows:

[0040] Collect the audience position data and establish an observation coordinate system, defined as V(u,v,w):

[0041] V(u,v,w) = (X v ,Y v ,Z v ,α v ,β v ),

[0042] where V(u,v,w) represents the position state of the audience in the theater, u, v, w are the spatial position components in the observation coordinate system, X v ,Y v ,Z v represent the three-dimensional coordinates of the audience, and α v ,β v represent the viewing direction of the audience, defined as the horizontal angle and the vertical angle.

[0043] Based on the collected audience position information V(u,v,w), use the neural network M NN to calculate the adaptation adjustment parameters of the light field. The formula is as follows:

[0044] Θ(x,y,u,v,w) = M NN (V(u,v,w),L 0 (x,y,θ,φ)),

[0045] where Θ(x,y,u,v,w) represents the light field adaptive adjustment parameter, M NN is the neural network model for light field optimization, and L 0 (x,y,θ,φ) is the initial light field.

[0046] Use the light field adjustment parameter Θ(x,y,u,v,w) predicted by the neural network to perform real-time optimization on the light field. The optimization formula is as follows:

[0047] L 1 (x,y,θ,φ) = L 0(x, y, θ, φ) · Θ(x, y, u, v, w),

[0048] where L 1 (x, y, θ, φ) is the dynamically adjusted light field, and the initial light field L 0 (x, y, θ, φ) is adaptively adjusted through Θ(x, y, u, v, w);

[0049] To cope with the real-time changes in the viewing angle of the audience, the adjusted light field is optimized and compensated, and the compensation formula is:

[0050]

[0051] where C(x, y, θ, φ) represents the dynamic compensation term of the light field, calculating the influence of the change in the viewer's position V(u, v, w) on the light field,

[0052] Finally, the optimized adaptive light field is calculated:

[0053] L opt (x, y, θ, φ) = L 1 (x, y, θ, φ) + C(x, y, θ, φ),

[0054] where L opt (x, y, θ, φ) is the final adaptive light field.

[0055] As a preferred solution of the holographic imaging method for a VR holographic theater described in the present invention, wherein: the step of optimizing the parallax using the frustum mapping algorithm, combining structured light projection, adjusting the projection angle through an optical modulator, compensating for the viewing position offset, and generating an optimized light field is,

[0056] In the holographic theater, the viewing positions of each audience are different. Here, the frustum mapping parameters are calculated according to the audience's position, and the light field is optimized using the parallax compensation algorithm. Let the frustum mapping of the audience be C v (u, v, w):

[0057] C v (u, v, w) = (θ v , φ v , d v ),

[0058] where C v (u, v, w) represents the frustum mapping parameters of the audience, u, v, w are the spatial position components in the observation coordinate system, θ v is the horizontal viewing angle range of the audience, φ v is the vertical viewing angle range of the audience, d v is the distance from the audience to the display plane,

[0059] Calculate the parallax compensation factor based on the viewer's cone mapping parameters, with the formula:

[0060]

[0061] Among them, S(x, y, u, v, w) represents the parallax compensation factor, which is calculated using gradients:

[0062] Calculate the influence of the viewer's perspective change on the light field;

[0063] The optimized light field is expressed as:

[0064] L view (x, y, θ, φ) = L opt (x, y, θ, φ) + S(x, y, u, v, w),

[0065] Among them, L view (x, y, θ, φ) is the light field optimized by parallax;

[0066] To improve the uniformity of the light field, structured light projection is used here to optimize the light field. Define the structured light projection function P struct :

[0067] P struct (x, y, θ, φ) = L view (x, y, θ, φ) · G(x, y),

[0068] Among them, P struct (x, y, θ, φ) represents the light field optimized by structured light projection, and G(x, y) is the structured light modulation function, G(x, y) = cos(k x x + k y y + φ g ), where G(x, y) is the structured light modulation function, k x , k y are the spatial frequency components respectively, and φ g is the phase shift phase;

[0069] The light field optimized by structured light is further adjusted by an optical modulator to align the projection angle with the viewer's position. Define the optical modulation function T adj :

[0070] O mod (x, y, θ, φ) = P struct (x, y, θ, φ) · T adj (x, y, θ, φ),

[0071] Among them, O mod (x, y, θ, φ) is the light field adjusted by the optical modulator, and T adj(x, y, θ, φ) is the optical modulator adjustment function used to compensate for the viewing position offset. The optical modulation adopts phase control based on the liquid crystal spatial light modulator (SLM):

[0072] T adj (x, y, θ, φ) = e iΦ(x,y,θ,φ) ,

[0073] where T adj (x, y, θ, φ) is the optical modulator adjustment function, Φ(x, y, θ, φ) is the phase modulation distribution for controlling the beam direction, e is the base of the natural logarithm, and i is the imaginary unit satisfying i 2 = -1,

[0074] Finally, the optimized light field is calculated as:

[0075] L proj (x, y, θ, φ) = O mod (x, y, θ, φ),

[0076] where L proj (x, y, θ, φ) is the light field finally projected onto the holographic display medium.

[0077] As a preferred solution of the holographic imaging method for a VR holographic theater according to the present invention, wherein: the step of optimizing the beam path based on the refractive mirror array and AI optical adjustment and projecting it onto the holographic display medium is

[0078] Before the holographic light field is projected onto the display medium, the beam path needs to be optimized by the refractive mirror array. Let the refractive mirror array be M r (x, y), and the calculation formula for the beam refraction angle is defined as:

[0079]

[0080] where θ r (x, y) is the beam refraction angle after passing through the refractive mirror array, θ proj (x, y) is the beam incident angle before being projected onto the refractive mirror, n 1 , n 2 are the refractive indices of air and the refractive mirror material,

[0081] The corrected light field is:

[0082] L refract (x, y, θ r , φ) = L proj (, y, θ proj , φ) · T r (x, y),

[0083] where L refract(x, y, θ r , φ) is the light field after passing through the refractive mirror array, and T r (x, y) is the transmission coefficient of the refractive mirror, which describes the energy loss of the light beam.

[0084] To further optimize the light beam path, an AI optical adjustment model is adopted to adjust the light beam direction to make the light field more uniform. Let the AI optimization function be A opt , and calculate the light beam adjustment parameters:

[0085] Θ AI (x, y) = A opt (L refract (x, y, θ r , φ)),

[0086] where

[0087] Θ AI (x, y) is the light beam adjustment parameter calculated by AI optimization, and A opt is the AI optical adjustment model.

[0088] The optimized light field is:

[0089] L AI (x, y, θ r , φ) = L refract (x, y, θ r , φ) · Θ AI (x, y),

[0090] where L AI (x, y, θ r , φ) is the light field optimized by AI optical adjustment;

[0091] The finally optimized light field is projected onto the holographic display medium, and the projected light field is defined as:

[0092] L holo (x, y, θ r , φ) = L AI (x, y, θ r , φ) · T h (x, y),

[0093] where L holo (x, y, θ r , φ) is the light field finally projected onto the holographic display medium, and T h (x, y) is the transmission characteristic of the holographic medium.

[0094] As a preferred solution of the holographic imaging method for a VR holographic theater according to the present invention, in step S5, based on the interaction system, combining the audience position, motion and gaze tracking data, the light field parameters are adjusted in real time, and multi-view synchronous calibration is performed. Specifically:

[0095] The interaction system obtains the position, motion and gaze data of the audience in real time through the sensor array, including three-dimensional position, head orientation, gaze direction and fixation point position, for judging the current viewing angle and interaction state of the audience;

[0096] The neural network model is used to process the collected data to predict the dynamic adjustment parameters required by the light field. The input of the neural network is the interaction state of the audience and the current holographic projection light field, and the output is the real-time light field adjustment coefficient, which is used to correct the projection light field to match the current viewing angle and interaction behavior of the audience;

[0097] According to the adjustment parameters output by the neural network, the current holographic display light field is updated, and the updated content includes perspective adaptive adjustment, brightness and contrast compensation, and light field focus optimization;

[0098] The interaction system calculates the light field error under different audience perspectives and corrects it through a synchronous calibration algorithm. The synchronous calibration method includes error equalization optimization and global compensation algorithm, and finally generates an optimized synchronous light field.

[0099] As a preferred solution of the holographic imaging method for a VR holographic theater according to the present invention, the interaction system continuously monitors the behavior changes of the audience. If the audience position, head posture or gaze direction changes, the interaction system immediately calculates new light field adjustment parameters and updates the projection light field;

[0100] The error equalization optimization adjusts the light field consistency of different perspectives, and the global compensation algorithm corrects the light field distortion under multiple perspectives;

[0101] The sensor array includes an infrared camera, a depth camera and an eye movement tracking sensor.

[0102] The beneficial effects of the present invention are as follows: in the present invention, the depth information, texture information and light field parameters are obtained through the three-dimensional data acquisition module to construct the initial light field. The light field calculation module generates an adaptive light field based on the audience position data by using a machine learning algorithm and adjusts the light field parameters in real time, so that the holographic image can be dynamically optimized according to the perspective of the audience, reducing image drift and distortion caused by position changes; in addition, the frustum mapping algorithm is introduced to optimize the parallax, and combined with structured light projection and optical modulator to dynamically adjust the projection angle, compensating for the optical offset at different viewing positions, thereby improving the consistency of the image under each perspective.

[0103] In the present invention, an optical modulation module combines a refractive mirror array and AI optical adjustment to optimize the light beam path and project it onto a holographic display medium, making the light field distribution of the holographic image more uniform. At the same time, an ambient light sensing unit detects the lighting conditions in the theater, and based on the detection results, dynamically adjusts the brightness, contrast, and shadow adaptation of the image, enabling it to maintain visual consistency in different lighting environments. In addition, the interaction system combines the audience's position, movement, and line-of-sight tracking data, adjusts the light field parameters in real time, and performs multi-view synchronous calibration to ensure the stability and consistency of the holographic image when multiple people are watching.

[0104] The key point of the present invention is to improve the viewing angle adaptation ability of the holographic image, enabling it to be dynamically adjusted according to the viewing angles of different audiences, reducing problems such as image distortion, drift, or disappearance. At the same time, combined with ambient light adaptation and interaction optimization, the immersion of the holographic image is enhanced, enabling it to more realistically integrate into the actual application environment of the VR holographic theater. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0106] Figure 1 It is a schematic framework diagram of the holographic imaging system for the VR holographic theater of the present invention.

[0107] Figure 2 It is a schematic flow diagram of the holographic imaging method for the VR holographic theater of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0108] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present invention in conjunction with the drawings of the specification.

[0109] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0110] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation of the present invention. The phrase "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that mutually excludes other embodiments.

[0111] Example 1, referring to Figure 1 and Figure 2 , this example provides a holographic imaging system for a VR holographic theater, including:

[0112] A three-dimensional data acquisition module for acquiring three-dimensional data of a holographic image and constructing an initial light field based on the three-dimensional data;

[0113] A light field calculation module that receives the initial light field and generates an adaptive light field through a machine learning algorithm based on the audience position data;

[0114] An optical adjustment module that optimizes the parallax based on the adaptive light field using a frustum mapping algorithm, and combines structured light projection to adjust the projection angle through an optical modulator to generate an optimized light field;

[0115] An optical modulation module that optimizes the light beam path based on the optimized light field based on a refractive mirror array and AI optical adjustment, and projects the light field onto a holographic display medium;

[0116] The optical modulation module includes an ambient light perception unit for acquiring the theater lighting conditions;

[0117] The optical modulation module further combines the ambient light perception unit to detect the theater lighting conditions, and adjusts the light field brightness, contrast, and shadow adaptation based on the detection results;

[0118] An interaction system that receives the optically optimized light field, combines the audience position, motion, and gaze tracking data, and adjusts the light field parameters in real time.

[0119] This example also provides a holographic imaging method for the above-mentioned holographic imaging system for a VR holographic theater, including:

[0120] Step S1: Acquire three-dimensional data of a holographic image and construct an initial light field based on the three-dimensional data;

[0121] The three-dimensional data includes depth information, texture information, and light field parameters;

[0122] The step of constructing an initial light field based on the three-dimensional data is

[0123] Assume that the three-dimensional data of the holographic image is composed of depth information, texture information, and light field parameters, and define the spatial coordinate system of the target scene as P(x, y, z):

[0124] P(x, y, z) = (D(x, y), T(x, y), L(x, y, θ, φ)),

[0125] Among them, P(x, y, z) represents the three-dimensional coordinate points in the scene, where x, y, and z represent the spatial coordinate components respectively, D(x, y) represents the depth information, describing the distance from the scene point to the reference plane, T(x, y) represents the texture information, including color, lighting, and reflection characteristics, and L(x, y, θ, φ) represents the light field parameters, defined as the light intensity and propagation characteristics in the direction of (θ, φ) starting from the point (x, y).

[0126] Based on the light field rendering equation, the light intensity distribution is calculated, and the light intensity distribution formula is:

[0127] I(x, y, θ, φ) = ∫ λ R(x, y, λ)L(x, y, θ, φ, λ)dλ,

[0128] Among them, I(x, y, θ, φ) represents the light field intensity at the point (x, y) in the direction of (θ, φ), λ represents the wavelength of light, and ∫ λ · represents the integration over the wavelengths within the spectral range to obtain the final visible light intensity, R(x, y, λ) represents the reflectivity of the object surface to light of different wavelengths, and L(x, y, θ, φ, λ) represents the light field parameters varying with the wavelength.

[0129] The phase information of the light field is calculated using the light wave propagation formula, and the formula is:

[0130] Ψ(x, y, θ, φ) = e ikD(x,y) ,

[0131] Among them, Ψ(x, y, θ, φ) represents the phase information of the light wave, e is the base of the natural logarithm, i represents the imaginary unit, satisfying i 2 = -1,

[0132] is the wave number, describing the spatial frequency of the light wave, and D(x, y) represents the depth information;

[0133] Then the initial light field formula is:

[0134] F(x, y, θ, φ) = I(x, y, θ, φ)·Ψ(x, y, θ, φ),

[0135] Among them, F(x, y, θ, φ) is the complex expression of the initial light field intensity, combining the light field intensity and phase information;

[0136] Specifically, here a light field model is constructed from three-dimensional data; the spatial positions of the scene points are calculated using the depth information, and the color and reflection characteristics are determined by combining the texture information. The distribution of light rays in different directions is calculated through the light field parameters, and the final light field intensity is calculated through the light field rendering equation.

[0137] Step S2: Input the initial light field into the light field calculation module. Based on the audience position data, generate an adaptive light field through a machine learning algorithm and dynamically adjust the light field parameters.

[0138] The steps of generating an adaptive light field through a machine learning algorithm and dynamically adjusting the light field parameters based on the audience position data are as follows:

[0139] Collect the audience position data and establish an observation coordinate system, defined as V(u, v, w):

[0140] V(u, v, w) = (X v , Y v , Z v , α v , β v ),

[0141] where V(u, v, w) represents the position state of the audience in the theater, u, v, w are the spatial position components in the observation coordinate system, X v , Y v , Z v represent the three-dimensional coordinates of the audience, and α v , β v represent the viewing direction of the audience, defined as the horizontal angle and the vertical angle.

[0142] Based on the collected audience position information V(u, v, w), use the neural network M NN to calculate the adaptation adjustment parameters of the light field. The formula is:

[0143] Θ(x, y, u, v, w) = M NN (V(u, v, w), L 0 (x, y, θ, φ)),

[0144] where Θ(x, y, u, v, w) represents the light field adaptive adjustment parameters, M NN is the neural network model for light field optimization, and L 0 (x, y, θ, φ) is the initial light field.

[0145] Use the light field adjustment parameters Θ(x, y, u, v, w) predicted by the neural network to perform real-time optimization on the light field. The optimization formula is:

[0146] L 1 (x, y, θ, φ) = L 0 (x, y, θ, φ) · Θ(x, y, u, v, w),

[0147] where L 1 (x, y, θ, φ) is the light field after dynamic adjustment, and the initial light field L 0(x, y, θ, φ) Adaptive adjustment;

[0148] To cope with the real-time changes in the audience's perspective, optimize and compensate the adjusted light field, and the compensation formula is:

[0149]

[0150] Among them, C(x, y, θ, φ) represents the dynamic compensation term of the light field, which calculates the influence of the change in the audience's position V(u, v, w) on the light field.

[0151] Finally, the optimized adaptive light field is calculated:

[0152] L opt L(x, y, θ, φ) = L 1 (x, y, θ, φ) + C(x, y, θ, φ),

[0153] Among them, L opt (x, y, θ, φ) is the final adaptive light field;

[0154] Specifically, here the light field is dynamically adjusted based on the audience's position information to adapt to different viewing angles. Specifically: Define the audience's position information, calculate the light field adjustment parameters through a neural network and optimize the light field. Further, calculate the dynamic compensation term of the light field to correct the light field distortion caused by the change in the audience's position, and finally generate the optimized adaptive light field;

[0155] Step S3, input the adaptive light field into the optical adjustment module, optimize the parallax using the frustum mapping algorithm, and combine structured light projection. Adjust the projection angle through the optical modulator to compensate for the viewing position offset and generate the optimized light field;

[0156] The steps of optimizing the parallax using the frustum mapping algorithm, combining structured light projection, and adjusting the projection angle through the optical modulator to compensate for the viewing position offset and generate the optimized light field are as follows.

[0157] In the holographic theater, each audience has a different viewing position. Here, calculate the frustum mapping parameters according to the audience's position and optimize the light field using the parallax compensation algorithm. Let the frustum mapping of the audience be C v (u, v, w):

[0158] C v (u, v, w) = (θ v , φ v , d v ),

[0159] Among them, C v (u, v, w) represents the frustum mapping parameters of the audience, u, v, w are the spatial position components in the observation coordinate system, θ vis the horizontal viewing angle range of the viewer, φ v is the vertical viewing angle range of the viewer, d v is the distance from the viewer to the display plane

[0160] Based on the viewer's cone mapping parameters, calculate the parallax compensation factor, and the formula is:

[0161]

[0162] Among them, S(x,y,u,v,w) represents the parallax compensation factor, and gradient calculation is adopted:

[0163] Calculate the influence of the viewer's viewing angle change on the light field;

[0164] The optimized light field is expressed as:

[0165] L view (x,y,θ,φ) = L opt (x,y,θ,φ) + S(x,y,u,v,w),

[0166] Among them, L view (x,y,θ,φ) is the light field optimized by parallax;

[0167] To improve the uniformity of the light field, structured light projection is used to optimize the light field here, and the structured light projection function P struct is defined as:

[0168] P struct (x,y,θ,φ) = L view (x,y,θ,φ) · G(x,y),

[0169] Among them, P struct (x,y,θ,φ) represents the light field optimized by structured light projection, G(x,y) is the structured light modulation function, G(x,y) = cos(k x x + k y y + φ g ), among which, G(x,y) is the structured light modulation function, k x , k y are the spatial frequency components respectively, and φ g is the phase shift phase;

[0170] The light field optimized by structured light is further adjusted by an optical modulator to align the projection angle with the viewer's position, and the optical modulation function T adj is defined as:

[0171] O mod (x,y,θ,φ) = P struct (x,y,θ,φ) · T adj(x, y, θ, φ),

[0172] where O mod (x, y, θ, φ) is the optical field adjusted by the optical modulator, and T adj (x, y, θ, φ) is the adjustment function of the optical modulator for compensating the viewing position offset. The optical modulation adopts phase control based on the liquid crystal spatial light modulator SLM:

[0173] T adj (x, y, θ, φ) = e iΦ(x,y,θ,φ) ,

[0174] where T adj (x, y, θ, φ) is the adjustment function of the optical modulator, Φ(x, y, θ, φ) is the phase modulation distribution for controlling the beam direction, e is the base of the natural logarithm, and i is the imaginary unit satisfying i 2 = -1,

[0175] Finally, the optimized optical field is calculated:

[0176] L proj (x, y, θ, φ) = O mod (x, y, θ, φ),

[0177] where L proj (x, y, θ, φ) is the optical field finally projected onto the holographic display medium,

[0178] Specifically, here the optimal viewing angle of the audience is calculated through cone mapping, and the optical field is optimized based on the parallax compensation algorithm to align the optical field with different viewing positions; structured light projection is used to enhance the uniformity of the optical field, modulated by the phase shift grating function, and finally the projection angle is adjusted by the liquid crystal spatial light modulator SLM, and the adjustment function of the optical modulator is calculated using phase control to optimize the beam direction, so that the optical field projected onto the holographic medium can adapt to the perspectives of different audiences, improving the realism and immersion of the holographic image;

[0179] Step S4: Input the optimized optical field into the optical modulation module, optimize the beam path based on the refractive mirror array and AI optical adjustment, and project it onto the holographic display medium;

[0180] In step S4, the environmental light perception unit is combined to detect the lighting conditions in the theater, and the brightness, contrast, and shadow adaptation of the optical field are further adjusted based on the detection results;

[0181] The step of optimizing the beam path based on the refractive mirror array and AI optical adjustment and projecting it onto the holographic display medium is as follows:

[0182] Before the holographic optical field is projected onto the display medium, the beam path needs to be optimized through the refractive mirror array. Let the refractive mirror array be M r(x, y), the calculation formula for the refraction angle of the light beam is defined as:

[0183]

[0184] Among them, θ r (x, y) is the refraction angle of the light beam after passing through the refractive mirror array, and θ proj (x, y) is the incident angle of the light beam projected in front of the refractive mirror, and n 1 , n 2 are the refractive indices of air and the refractive mirror material.

[0185] The corrected light field is:

[0186] L refract (x, y, θ r , φ) = L proj (x, y, θ proj , φ) · T r (x, y),

[0187] Among them, L refract (x, y, θ r , φ) is the light field after passing through the refractive mirror array, and T r (x, y) is the transmission coefficient of the refractive mirror, which describes the energy loss of the light beam.

[0188] To further optimize the light beam path, an AI optical adjustment model is adopted to adjust the light beam direction to make the light field more uniform. Let the AI optimization function be A opt , and calculate the light beam adjustment parameters:

[0189] Θ AI (x, y) = A opt (L refract (x, y, θ r , φ)),

[0190] Among them,

[0191] Θ AI (x, y) is the light beam adjustment parameter calculated by AI optimization, and A opt is the AI optical adjustment model.

[0192] The optimized light field is:

[0193] L AI (x, y, θ r , φ) = L refract (x, y, θ r , φ) · Θ AI (x, y),

[0194] Among them, L AI (x, y, θ r, φ) is the light field optimized through AI optical adjustment;

[0195] The finally optimized light field is projected onto the holographic display medium, and the projected light field is defined as:

[0196] L holo (x, y, θ r , φ) = L AI (x, y, θ r , φ) · T h (x, y),

[0197] where L holo (x, y, θ r , φ) is the light field finally projected onto the holographic display medium, and T h (x, y) is the transmission characteristic of the holographic medium;

[0198] Specifically, here the beam direction is adjusted through a refractive mirror array to make it more suitable for the angle of holographic projection: the beam refraction angle is calculated according to the refraction law and the light field is corrected, then the AI optical adjustment model is used to calculate the beam optimization parameters and adjust the light field, and finally the optimized light field is projected onto the holographic display medium;

[0199] Step S5: Input the optically optimized light field in step S4 into the interaction system, and combine the viewer's position, actions, and gaze tracking data to adjust the light field parameters in real time;

[0200] In step S5, multi-view synchronous calibration is performed;

[0201] In step S5, based on the interaction system combined with the viewer's position, actions, and gaze tracking data, the light field parameters are adjusted in real time, and multi-view synchronous calibration is performed. Specifically:

[0202] The interaction system obtains the viewer's position, actions, and gaze data in real time through a sensor array, including three-dimensional position, head orientation, gaze direction, and fixation point position, for judging the viewer's current viewing angle and interaction state;

[0203] A neural network model is used to process the collected data to predict the dynamic adjustment parameters required for the light field. The input of the neural network is the viewer's interaction state and the current holographic projection light field, and the output is the real-time light field adjustment coefficient, which is used to correct the projection light field to match the viewer's current viewing angle and interaction behavior;

[0204] According to the adjustment parameters output by the neural network, the current holographic display light field is updated, and the update content includes perspective adaptive adjustment, brightness and contrast compensation, and light field focus optimization;

[0205] The interaction system calculates the light field error from different viewer perspectives and corrects it through a synchronous calibration algorithm. The synchronous calibration method includes error equalization optimization and a global compensation algorithm, and finally generates an optimized synchronous light field;

[0206] The interaction system continuously monitors the behavior changes of the viewer. If the viewer's position, head pose, or line-of-sight direction changes, the interaction system immediately calculates new light field adjustment parameters and updates the projection light field;

[0207] Error equalization optimization adjusts the light field consistency of different perspectives, and the global compensation algorithm corrects the light field distortion under multiple perspectives;

[0208] The sensor array includes an infrared camera, a depth camera, and an eye movement tracking sensor.

[0209] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A holographic imaging system for VR holographic theater, characterized in that: include, A three-dimensional data acquisition module, used to acquire three-dimensional data of the holographic image and construct an initial light field based on the three-dimensional data; A light field calculation module receives the initial light field and generates an adaptive light field based on the audience position data through a machine learning algorithm; The optical adjustment module uses the cone mapping algorithm to optimize the parallax based on the adaptive light field, and combines it with structured light projection to adjust the projection angle through the optical modulator to generate an optimized light field. The optical modulation module optimizes the beam path based on the optimized light field, the refractor array and AI optical adjustment, and projects the light field onto the holographic display medium; The interactive system receives the optically optimized light field and adjusts the light field parameters in real time based on the audience's position, movement and gaze tracking data.

2. A holographic imaging system for VR holographic theater according to claim 1, characterized in that: The optical modulation module includes an ambient light sensing unit for acquiring theater lighting conditions; The optical modulation module further detects the theater lighting conditions in conjunction with the ambient light sensing unit, and adjusts the light field brightness, contrast and shadow adaptation based on the detection results.

3. A holographic imaging method for a VR holographic theater, based on a holographic imaging system for a VR holographic theater as claimed in any one of claims 1 to 2, characterized in that: include: Step S1, acquiring three-dimensional data of a holographic image, and constructing an initial light field based on the three-dimensional data; Step S2, inputting the initial light field into the light field calculation module, generating an adaptive light field through a machine learning algorithm based on the audience position data, and dynamically adjusting the light field parameters; Step S3, inputting the adaptive light field into the optical adjustment module, optimizing the parallax using the cone mapping algorithm, and combining the structured light projection, adjusting the projection angle through the optical modulator, compensating for the viewing position offset, and generating the optimized light field; Step S4, inputting the optimized light field into the optical modulation module, optimizing the beam path based on the refractor array and AI optics, and projecting it onto the holographic display medium; In step S4, the theater lighting conditions are detected in combination with the ambient light sensing unit, and the light field brightness, contrast and shadow adaptation are further adjusted based on the detection results; Step S5, inputting the optically optimized light field of step S4 into the interactive system, and adjusting the light field parameters in real time in combination with the audience's position, action and sight tracking data; In step S5, multi-view synchronous calibration is performed.

4. A holographic imaging method for VR holographic theater according to claim 3, characterized in that: The three-dimensional data includes depth information, texture information and light field parameters.

5. A holographic imaging method for VR holographic theater according to claim 4, characterized in that: The step of constructing the initial light field based on the three-dimensional data is: Assume that the three-dimensional data of the holographic image consists of depth information, texture information and light field parameters, and define the spatial coordinate system of the target scene as P(x, y, z): P(x,y,z)=(D(x,y),T(x,y),L(x,y,θ,φ)), Among them, P(x,y,z) represents the three-dimensional coordinate point in the scene, x,y,z represent the spatial coordinate components respectively, D(x,y) represents the depth information, describing the distance from the scene point to the reference plane, T(x,y) represents the texture information, including color, lighting and reflection characteristics, L(x,y,θ,φ) represents the light field parameters, defined as the light intensity and propagation characteristics in the direction of angle (θ,φ) starting from the point (x,y), The light intensity distribution is calculated based on the light field rendering equation. The light intensity distribution formula is: I(x,y,θ,φ)=∫ λ R(x,y,λ)L(x,y,θ,φ,λ)dλ, Where I(x,y,θ,φ) represents the light field intensity along the direction (θ,φ) at the point (x,y), λ represents the wavelength of light, ∫ λ · represents the integration of wavelengths within the spectral range to obtain the final visible light intensity, R(x,y,λ) represents the reflectivity of the object surface to light of different wavelengths, L(x,y,θ,φ,λ) represents the light field parameters that vary with wavelength, The phase information of the light field is calculated using the light wave propagation formula: Ψ(x,y,θ,φ)=e ikD(x,y) , Among them, Ψ(x, y, θ, φ) represents the phase information of the light wave, e is the base of the natural logarithm, i represents the imaginary unit, and i satisfies 2 =-1, is the wave number, describing the spatial frequency of the light wave, and D(x,y) represents the depth information; Then the initial light field formula is: F(x,y,θ,φ)=I(x,y,θ,φ)·Ψ(x,y,θ,φ), Among them, F(x, y, θ, φ) is the complex expression of the initial light field intensity, combining the light field intensity and phase information.

6. A holographic imaging method for VR holographic theater according to claim 5, characterized in that: The steps of generating an adaptive light field through a machine learning algorithm based on audience position data and dynamically adjusting light field parameters are: Collect audience position data and establish an observation coordinate system, defined as V(u,v,w): V(u,v,w)=(X v ,Y v ,Z v ,α v ,β v ), Among them, V(u,v,w) represents the position of the audience in the theater, u,v,w are the spatial position components in the observation coordinate system, and X v ,Y v ,Z v represents the 3D coordinates of the audience, α v ,β v Indicates the viewing direction of the viewer, defined as horizontal and vertical angles, Based on the collected audience position information V(u,v,w), the neural network M NN Calculate the adaptation adjustment parameters of the light field, the formula is: Θ(x,y,u,v,w)=M NN (V(u,v,w),L0(x,y,θ,φ)), Among them, Θ(x,y,u,v,w) represents the light field adaptive adjustment parameters, M NN is the neural network model for light field optimization, L0(x,y,θ,φ) is the initial light field; Use the light field adjustment parameters Θ(x, y, u, v, w) predicted by the neural network to optimize the light field in real time. The optimization formula is: L1(x,y,θ,φ)=L0(x,y,θ,φ)·Θ(x,y,u,v,w), Among them, L1(x, y, θ, φ) is the dynamically adjusted light field, and the adaptive adjustment of the initial light field L0(x, y, θ, φ) is realized through θ(x, y, u, v, w); In order to cope with the real-time changes in the audience's perspective, the adjusted light field is optimized and compensated. The compensation formula is: Among them, C(x,y,θ,φ) represents the light field dynamic compensation term, which calculates the impact of the audience position change V(u,v,w) on the light field. The optimized adaptive light field is finally calculated: L opt (x,y,θ,φ)=L1(x,y,θ,φ)+C(x,y,θ,φ), Among them, L opt (x, y, θ, φ) is the final adaptive light field.

7. A holographic imaging method for VR holographic theater according to claim 6, characterized in that: The steps of optimizing the parallax by using the cone mapping algorithm, combining with structured light projection, adjusting the projection angle by an optical modulator, compensating for the viewing position offset, and generating the optimized light field are as follows: In the holographic theater, each audience member has a different viewing position. Here, the cone mapping parameters are calculated according to the audience position, and the parallax compensation algorithm is used to optimize the light field. Suppose the audience's cone mapping is, C v (u,v,w): C v (u,v,w)=(θ v ,f v ,d v ), Among them, C v (u,v,w) represents the viewer's cone mapping parameters, u,v,w are the spatial position components in the observation coordinate system, θ v is the audience's horizontal viewing angle, φ v is the vertical viewing angle of the audience, d v is the distance from the viewer to the display plane, Based on the viewer's cone mapping parameters, the parallax compensation factor is calculated as follows: Among them, S(x,y,u,v,w) represents the parallax compensation factor, which is calculated using gradient: Calculate the effect of changing viewer perspective on the light field; The optimized light field is expressed as: L view (x,y,θ,φ)=L opt (x,y,θ,φ)+S(x,y,u,v,w), Among them, L view (x, y, θ, φ) is the parallax-optimized light field; In order to improve the uniformity of the light field, structured light projection is used here to optimize the light field, and the structured light projection function P is defined struct : P struct (x,y,θ,φ)=L view (x,y,θ,φ)·G(x,y), Among them, P struct (x, y, θ, φ) represents the light field after structured light projection optimization, G(x, y) is the structured light modulation function, G(x, y) = cos(k x x+k y y+φ g ), where G(x,y) is the structured light modulation function, k x ,k y are the spatial frequency components, φ g is the phase shift phase; The light field after structured light optimization is further adjusted through an optical modulator to align the projection angle with the audience position, and the optical modulation function T is defined. adj : The mod (x,y,θ,φ)=P struct (x,y,θ,φ)·T adj (x,y,θ,φ), Among them, O mod (x, y, θ, φ) is the light field adjusted by the optical modulator, T adj (x, y, θ, φ) is the optical modulator adjustment function, which is used to compensate for the viewing position offset. The optical modulation adopts phase control based on liquid crystal spatial light modulator SLM: T adj (x,y,θ,φ)=e iΦ(x,y,θ,φ) , Among them, T adj (x, y, θ, φ) is the optical modulator adjustment function, Φ(x, y, θ, φ) is the phase modulation distribution, which controls the direction of the light beam, e is the base of the natural logarithm, and i is the imaginary unit, satisfying i 2 =-1, The optimized light field is finally calculated: L proj (x,y,θ,φ)=O mod (x,y,θ,φ), Among them, L proj (x, y, θ, φ) is the light field finally projected onto the holographic display medium.

8. A holographic imaging method for VR holographic theater according to claim 7, characterized in that: The steps of optimizing the beam path based on the refractor array and AI optical adjustment and projecting it onto the holographic display medium are: Before the holographic light field is projected onto the display medium, the beam path needs to be optimized through a refractor array. Suppose the refractor array is M r (x,y), the calculation formula of the beam refraction angle is defined as: Among them, θ r (x, y) is the refraction angle of the light beam after passing through the refractor array, θ proj (x, y) is the incident angle of the light beam projected onto the refractor, n1, n2 are the refractive indices of air and the refractor material, The corrected light field is: L refract (x,y,θ r ,φ)=L proj (x,y,θ proj ,φ)·T r (x,y), Among them, L refract (x,y,θ r ,φ) is the light field after passing through the refractor array, T r (x,y) is the transmission coefficient of the refractor, which describes the energy loss of the beam. In order to further optimize the beam path, the AI ​​optical adjustment model is used to adjust the beam direction to make the light field more uniform. The AI ​​optimization function is set as A opt , calculate the beam adjustment parameters: I AI (x,y)=A opt (L refract (x,y,θ r ,φ)), in, Θ AI (x, y) is the beam adjustment parameter calculated by AI optimization, A opt Adjust the model for AI optics, The optimized light field is: L AI (x,y,θ r ,φ)=L refract (x,y,θ r ,φ)·Θ AI (x,y), Among them, L AI (x,y,θ r ,φ) is the light field after AI optical adjustment and optimization; The final optimized light field is projected onto the holographic display medium, and the projected light field is defined as: L holo (x,y,θ r ,φ)=L AI (x,y,θ r ,φ)·T h (x,y), Among them, L holo (x,y,θ r ,φ) is the light field finally projected onto the holographic display medium, T h (x,y) is the transmission characteristic of the holographic medium.

9. A holographic imaging method for VR holographic theater according to claim 8, characterized in that: In step S5, based on the interactive system combined with the audience's position, action and sight tracking data, the light field parameters are adjusted in real time, and multi-view synchronous calibration is performed. Specifically: The interactive system uses a sensor array to obtain real-time data on the audience's position, movement, and sight line, including three-dimensional position, head orientation, sight line direction, and gaze point position, to determine the audience's current viewing angle and interactive status; A neural network model is used to process the collected data and predict the dynamic adjustment parameters required for the light field. The input of the neural network is the audience's interaction status and the current holographic projection light field, and the output is a real-time light field adjustment coefficient, which is used to correct the projection light field to match the audience's current observation angle and interaction behavior. According to the adjustment parameters output by the neural network, the current holographic display light field is updated, including adaptive adjustment of viewing angle, compensation of brightness and contrast, and optimization of light field focus; The interactive system calculates the light field error under different audience perspectives and corrects it through a synchronous calibration algorithm. The synchronous calibration method includes error equalization optimization and global compensation algorithm, and finally generates an optimized synchronous light field.

10. A holographic imaging method for VR holographic theater according to claim 9, characterized in that: The interactive system continuously monitors the audience's behavior changes. If the audience's position, head posture or line of sight direction changes, the interactive system immediately calculates new light field adjustment parameters and updates the projected light field. The error equalization optimization adjusts the consistency of the light field at different viewing angles, and the global compensation algorithm corrects the light field distortion at multiple viewing angles; The sensor array includes an infrared camera, a depth camera, and an eye-tracking sensor.

Images