Multi-dimensional holographic image real-time rendering method and system based on dynamic light field modulation

Abstract

The invention discloses a multi-dimensional holographic image real-time rendering method and system based on dynamic light field modulation, and relates to the technical field of holographic display, the method is applied to terminal equipment and a server, the terminal equipment determines a space coordinate range based on quantum dot super-structure surface characteristics by obtaining related parameters of a target holographic scene, and the real-time rendering of a holographic image is realized. A quantum dot excitation sequence is generated to regulate and control the micro-nano structure of the super-structure surface, a joint coding model is constructed, a target light field modulation instruction set is generated through optimization, corresponding multi-dimensional light field data is requested from a server, and a holographic scene is generated through processing, fusion and rendering; the server receives the request, calls original holographic data, constructs a coding model and a compression model, and sends multi-dimensional light field data subjected to entropy compression to the terminal through a multi-channel transmission protocol; according to the method, efficient real-time rendering of the multi-dimensional holographic image is achieved, the rendering efficiency and quality are improved, and the method is suitable for scenes with high requirements for real-time performance of the holographic image.

Description

Technical Field

[0001] The present invention relates to the field of holographic display technology, and more specifically, to a method and system for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation. Background Art

[0002] In today's digital age, with the rapid development of cutting-edge technologies such as virtual reality (VR), augmented reality (AR) and the metaverse, the demand for high-quality, immersive visual experience has exploded. As a technology that can achieve the ultimate visual experience for the human eye, the real-time rendering of holographic images has become a major bottleneck restricting the development of related industries. Traditional holographic display technology has difficulty achieving a good balance between display effects, update speed and device portability, and is unable to meet the market's requirements for real-time, high-resolution and high-fidelity holographic images.

[0003] Real-time rendering technology for multi-dimensional holographic images has become a key direction for solving this problem. By capturing and reconstructing multi-dimensional information such as the propagation direction, intensity, and phase of light in space, this technology can present users with natural and realistic three-dimensional visual effects. However, existing technologies often expose many problems when facing complex dynamic scenes: On the one hand, traditional rendering methods use a unified light field sampling and modulation mode, ignoring the characteristic differences of the scene in spatial, temporal and perspective dimensions, resulting in huge consumption of computing resources when rendering large-scale scenes, making it difficult to achieve real-time rendering; for example, when the spatial range of the scene changes or the perspective parameters are frequently adjusted, the system needs to re-sample and modulate the light field of the entire scene, which cannot be efficiently utilized. On the other hand, in the data transmission process, existing technologies usually adopt fixed encoding and transmission strategies. For light field data containing multi-dimensional rich information such as depth information and perspective information, compression distortion and data loss are prone to occur. In addition, the priority and accuracy of data transmission cannot be dynamically adjusted according to the real-time feedback of the terminal device, which affects the final presentation quality of the holographic image. In addition, some methods find it difficult to accurately restore the phase and amplitude information of the light field when processing dynamic scenes or objects of different materials, resulting in obvious defects in the generated holographic images in terms of edge details, transparent object performance, and light and shadow effects, which cannot meet the high-fidelity requirements of the human eye for real scenes.

[0004] Therefore, the existing dynamic light field modulation holographic image rendering technology is insufficient in local data utilization, transmission accuracy and detail restoration, which affects image quality and real-time performance. Summary of the Invention

[0005] In order to overcome the problems of existing dynamic light field modulation holographic image rendering technology in local data utilization, transmission accuracy and detail restoration, the present invention discloses a real-time rendering method and system for multi-dimensional holographic images based on dynamic light field modulation, which can effectively solve the above technical problems.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] A multi-dimensional holographic image real-time rendering method based on dynamic light field modulation is applied to a terminal device, and the method comprises:

[0008] Obtaining the target space coordinate range, target viewing angle parameter set, and target timestamp of the target holographic scene to be rendered, and obtaining the reference space coordinate range and reference light field characteristics of the previous rendered scene of the target holographic scene;

[0009] Based on the dynamic response characteristics of the quantum dot metasurface, determining a newly added spatial coordinate range and an overlapping spatial coordinate range according to the target spatial coordinate range and the reference spatial coordinate range, and generating a quantum dot excitation sequence corresponding to the newly added spatial coordinate range;

[0010] The metasurface micro-nanostructure is regulated by the quantum dot excitation sequence to generate an initial light field modulation pattern, and a NeRF-holographic joint encoding model is constructed in combination with the target viewing angle parameter set;

[0011] Inputting the initial light field modulation pattern and the NeRF-holographic joint encoding model parameters into the prediction compensation rendering engine, performing spatiotemporal continuity optimization in combination with the reference light field characteristics, and generating a target light field modulation instruction set;

[0012] Sending a dynamic light field data request including the target light field modulation instruction set and the target timestamp to the server, wherein the dynamic light field data request is used to request multidimensional light field data corresponding to the newly added spatial coordinate range, the target viewing angle parameter set, and the target timestamp; the multidimensional light field data is four-dimensional light field tensor data encoded by entropy compression, and is generated by preprocessing the multi-source heterogeneous holographic data stored in the target database;

[0013] The entropy-compressed multidimensional light field data returned by the server is received, spatiotemporal entropy decoding and light field reconstruction are performed through a prediction compensation rendering engine, and fusion rendering is performed in combination with the historical light field data corresponding to the overlapping spatial coordinate range to generate the target holographic scene.

[0014] Preferably, the reference light field features include multi-resolution spatiotemporal light field primitives and corresponding characteristic entropy values, and the target viewing angle parameter set includes spatial viewing angle coordinates, temporal phase parameters, and optical characteristic parameters of the observation device;

[0015] Before obtaining the target space coordinate range, target viewing angle parameter set and target timestamp of the target holographic scene to be rendered, the method further includes:

[0016] Performing spatiotemporal frequency analysis on the reference light field characteristics to construct a multi-scale light field prediction model;

[0017] generating pre-rendered light field data using the multi-scale light field prediction model according to the target timestamp;

[0018] The generating target light field modulation instruction set comprises:

[0019] Perform residual analysis on the pre-rendered light field data and the output of the NeRF-holographic joint encoding model to generate light field compensation parameters;

[0020] The target light field modulation instruction set is generated by combining the initial light field modulation mode and the light field compensation parameters.

[0021] Preferably, after receiving the entropy-compressed multi-dimensional light field data returned by the server, the method further comprises:

[0022] Based on the nonlinear optical response characteristics of quantum dot metasurfaces, a light field entropy decoding function is constructed;

[0023] Performing spatiotemporal entropy decoding on the multidimensional light field data using the light field entropy decoding function to restore four-dimensional light field tensor data;

[0024] Inputting the four-dimensional light field tensor data into a holographic reconstruction module, performing phase recovery and amplitude modulation in combination with the target light field modulation instruction set, and generating reconstructed light field data;

[0025] The performing fusion rendering in combination with the historical light field data corresponding to the overlapping spatial coordinate range includes:

[0026] Performing a spatiotemporal consistency check on the reconstructed light field data and the historical light field data to generate a light field fusion weight matrix;

[0027] Performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data;

[0028] The final rendered light field data is converted into a holographic image signal through a dynamic light field modulator, and the target holographic scene is generated by projection.

[0029] Preferably, performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data includes:

[0030] Constructing a spatiotemporal attention mechanism to generate spatiotemporal attention weights according to the target view parameter set and the target timestamp;

[0031] Combining the spatiotemporal attention weight and the light field fusion weight matrix, performing spatiotemporal attention fusion on the reconstructed light field data and the historical light field data to generate an initial fused light field;

[0032] The prediction compensation rendering engine is used to perform high-frequency detail enhancement and low-frequency noise suppression on the initial fused light field to generate the final rendered light field data.

[0033] Preferably, a multi-dimensional holographic image real-time rendering method based on dynamic light field modulation is applied to a server, and the method includes:

[0034] Receiving a dynamic light field data request sent by a terminal device, wherein the dynamic light field data request includes a target light field modulation instruction set, a newly added spatial coordinate range, a target viewing angle parameter set, and a target timestamp;

[0035] Retrieving original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from a target database, wherein the original holographic data includes a multi-view depth image sequence, light field intensity distribution data, and phase modulation information;

[0036] Decoding the target light field modulation instruction set through a quantum dot metasurface response model to generate light field modulation parameters;

[0037] Constructing a NeRF-holographic joint coding model based on the light field modulation parameters, performing feature extraction and encoding on the original holographic data, and generating a multi-dimensional light field feature tensor;

[0038] According to the target view parameter set and the target timestamp, the multi-dimensional light field feature tensor is optimized for spatiotemporal continuity by a prediction compensation rendering engine to generate four-dimensional light field tensor data;

[0039] performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model; performing entropy compression encoding on the four-dimensional light field tensor data based on the light field entropy compression model to generate entropy-compressed multidimensional light field data;

[0040] The entropy-compressed multi-dimensional light field data is sent to the terminal device through a multi-channel light field transmission protocol, so that the terminal device combines the historical light field data for fusion rendering to generate a target holographic scene.

[0041] Preferably, before receiving the dynamic light field data request sent by the terminal device, the method further includes:

[0042] Acquire multi-source heterogeneous holographic data sources, including image sequences acquired by multi-angle camera arrays, four-dimensional light field data acquired by light field cameras, and computer-generated virtual holographic data;

[0043] Performing spatiotemporal synchronization and registration on the multi-source heterogeneous holographic data sources to generate spatiotemporally aligned original holographic data;

[0044] Build a quantum dot metasurface database to store the optical response functions of metasurfaces under different excitation conditions;

[0045] The step of retrieving the original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from the target database includes:

[0046] According to the target light field modulation instruction set, matching a corresponding optical response function from the quantum dot metasurface database;

[0047] constructing a light field modulation filter based on the optical response function, and screening and retrieving matching original holographic data from the target database;

[0048] The target database is organized in a spatiotemporal octree structure, and each spatiotemporal node stores a light field feature descriptor and a data pointer of a corresponding spatiotemporal region;

[0049] The performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model includes:

[0050] Performing spatiotemporal division on the four-dimensional light field tensor data based on the spatiotemporal octree structure to generate multi-level spatiotemporal light field primitives;

[0051] Calculate the information entropy of space-time light field primitives at all levels and construct a space-time entropy pyramid;

[0052] Determine the redundancy distribution of light field data according to the spatiotemporal entropy pyramid and generate an adaptive entropy coding dictionary;

[0053] The light field entropy compression model is constructed based on the adaptive entropy coding dictionary.

[0054] Preferably, the server maintains a light field transmission quality prediction model, wherein the light field transmission quality prediction model is constructed based on the transmission characteristics of the quantum dot metasurface and channel state information;

[0055] The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes:

[0056] Predicting the light field transmission quality of each transmission channel using the light field transmission quality prediction model to generate a channel quality evaluation matrix;

[0057] Slicing the entropy-compressed multidimensional light field data according to the channel quality assessment matrix to generate a plurality of data slices;

[0058] Each data fragment is sent in parallel to the terminal device through the corresponding optimal transmission channel, and a time-space synchronization mark and quality control information are added.

[0059] Preferably, the method further comprises:

[0060] Build a dynamic light field caching mechanism to cache frequently accessed four-dimensional light field tensor data;

[0061] The performing entropy compression encoding on the four-dimensional light field tensor data to generate entropy-compressed multi-dimensional light field data comprises:

[0062] detecting whether a cached copy of the four-dimensional light field tensor data exists;

[0063] If a cached copy exists, generating incremental entropy coded data based on the difference between the cached data and the current data; if no cached copy exists, performing full entropy compression coding on the four-dimensional light field tensor data;

[0064] The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes:

[0065] If it is incremental entropy coded data, the incremental coding information and cached data pointer are sent; if it is complete entropy compression coded data, the complete coded data is sent directly.

[0066] Preferably, a multi-dimensional holographic image real-time rendering system based on dynamic light field modulation comprises: a terminal device and a server;

[0067] The terminal device is used to execute the above-mentioned method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation;

[0068] The server is used to execute the above-mentioned method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation;

[0069] The terminal device includes: a quantum dot metasurface light field modulator, a prediction compensation rendering engine, a spatiotemporal entropy decoder and a dynamic light field display;

[0070] The server includes: a NeRF-holographic joint encoder, a light field entropy compression module, a multi-channel light field transmission engine and a spatiotemporal octree database.

[0071] Preferably, a computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation is implemented.

[0072] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention discloses a real-time rendering method for multi-dimensional holographic images based on dynamic light field modulation, which is applied to terminal devices and servers, and solves the shortcomings of existing dynamic light field modulation holographic image rendering technology in local data utilization, transmission accuracy and detail restoration through innovative technical solutions; in terms of local data utilization, the method accurately determines the difference in spatial coordinate range between the target and the reference scene through the dynamic response characteristics of the quantum dot metasurface, and only generates an excitation sequence and requests data for the newly added part, avoiding repeated processing of the full scene data and improving data utilization efficiency. This is because the determination of the newly added spatial coordinate range is based on the comparison of the target and reference spatial coordinate ranges, thereby achieving local updates and reducing unnecessary data processing; in terms of transmission accuracy, the server organizes the target database using a spatiotemporal octree structure, which can accurately match the optical response function according to the target light field modulation instruction set, screen and retrieve the original holographic data that accurately matches the request parameters, and ensure the transmission of data The method is targeted and effective because the spatiotemporal octree structure can divide the four-dimensional light field tensor data into time and space, thereby realizing fine management and efficient retrieval of data and improving the accuracy of data transmission. In terms of detail restoration, the predictive compensation rendering engine of the terminal device combines the spatiotemporal attention mechanism and the light field fusion weight matrix to perform weighted fusion of the reconstructed light field data, and perform high-frequency detail enhancement and low-frequency noise suppression to generate the final rendered light field data. This is because the spatiotemporal attention mechanism can generate weights according to the target perspective parameter set and the target timestamp, thereby focusing on important details, and the light field fusion weight matrix ensures the consistency of new and old data, thereby improving the image detail restoration. In addition, this method caches frequently accessed four-dimensional light field tensor data by constructing a dynamic light field cache mechanism, reducing the transmission and processing of duplicate data, and further optimizing resource utilization. At the same time, the light field transmission quality prediction model can evaluate the quality of each transmission channel, realize the optimal transmission of data fragments, and improve the reliability and efficiency of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the implementation methods or the description of the prior art. Obviously, the drawings described below are merely exemplary. For ordinary technicians in this field, other implementation drawings can be derived based on the provided drawings without any creative work.

[0074] Figure 1 This is a flow chart of the terminal device method of the present invention;

[0075] Figure 2 This is a flow chart of the server method of the present invention;

[0076] Figure 3This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0077] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0078] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0079] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0080] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0081] Example 1

[0082] A multi-dimensional holographic image real-time rendering method based on dynamic light field modulation is applied to a terminal device, and the method comprises:

[0083] Obtaining the target space coordinate range, target viewing angle parameter set, and target timestamp of the target holographic scene to be rendered, and obtaining the reference space coordinate range and reference light field characteristics of the previous rendered scene of the target holographic scene;

[0084] Based on the dynamic response characteristics of the quantum dot metasurface, determining a newly added spatial coordinate range and an overlapping spatial coordinate range according to the target spatial coordinate range and the reference spatial coordinate range, and generating a quantum dot excitation sequence corresponding to the newly added spatial coordinate range;

[0085] The metasurface micro-nanostructure is regulated by the quantum dot excitation sequence to generate an initial light field modulation pattern, and a NeRF-holographic joint encoding model is constructed in combination with the target viewing angle parameter set;

[0086] Inputting the initial light field modulation pattern and the NeRF-holographic joint encoding model parameters into the prediction compensation rendering engine, performing spatiotemporal continuity optimization in combination with the reference light field characteristics, and generating a target light field modulation instruction set;

[0087] Sending a dynamic light field data request including the target light field modulation instruction set and the target timestamp to the server, wherein the dynamic light field data request is used to request multidimensional light field data corresponding to the newly added spatial coordinate range, the target viewing angle parameter set, and the target timestamp; the multidimensional light field data is four-dimensional light field tensor data encoded by entropy compression, and is generated by preprocessing the multi-source heterogeneous holographic data stored in the target database;

[0088] The entropy-compressed multidimensional light field data returned by the server is received, spatiotemporal entropy decoding and light field reconstruction are performed through a prediction compensation rendering engine, and fusion rendering is performed in combination with the historical light field data corresponding to the overlapping spatial coordinate range to generate the target holographic scene.

[0089] The reference light field features include multi-resolution spatiotemporal light field primitives and corresponding characteristic entropy values, and the target viewing angle parameter set includes spatial viewing angle coordinates, temporal phase parameters, and optical characteristic parameters of the observation device;

[0090] Before obtaining the target space coordinate range, target viewing angle parameter set and target timestamp of the target holographic scene to be rendered, the method further includes:

[0091] Performing spatiotemporal frequency analysis on the reference light field characteristics to construct a multi-scale light field prediction model;

[0092] generating pre-rendered light field data using the multi-scale light field prediction model according to the target timestamp;

[0093] The generating target light field modulation instruction set comprises:

[0094] Perform residual analysis on the pre-rendered light field data and the output of the NeRF-holographic joint encoding model to generate light field compensation parameters;

[0095] The target light field modulation instruction set is generated by combining the initial light field modulation mode and the light field compensation parameters.

[0096] After receiving the entropy-compressed multi-dimensional light field data returned by the server, the method further includes:

[0097] Based on the nonlinear optical response characteristics of quantum dot metasurfaces, a light field entropy decoding function is constructed;

[0098] Performing spatiotemporal entropy decoding on the multidimensional light field data using the light field entropy decoding function to restore four-dimensional light field tensor data;

[0099] Inputting the four-dimensional light field tensor data into a holographic reconstruction module, performing phase recovery and amplitude modulation in combination with the target light field modulation instruction set, and generating reconstructed light field data;

[0100] The performing fusion rendering in combination with the historical light field data corresponding to the overlapping spatial coordinate range includes:

[0101] Performing a spatiotemporal consistency check on the reconstructed light field data and the historical light field data to generate a light field fusion weight matrix;

[0102] Performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data;

[0103] The final rendered light field data is converted into a holographic image signal through a dynamic light field modulator, and the target holographic scene is generated by projection.

[0104] The step of performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data includes:

[0105] Constructing a spatiotemporal attention mechanism to generate spatiotemporal attention weights according to the target view parameter set and the target timestamp;

[0106] Combining the spatiotemporal attention weight and the light field fusion weight matrix, performing spatiotemporal attention fusion on the reconstructed light field data and the historical light field data to generate an initial fused light field;

[0107] The prediction compensation rendering engine is used to perform high-frequency detail enhancement and low-frequency noise suppression on the initial fused light field to generate the final rendered light field data.

[0108] See also Figure 1 After the terminal device is started, the target space coordinate range of the target holographic scene to be rendered is obtained through built-in sensors such as position sensors, posture sensors and user input devices such as mouse, keyboard, touch screen, etc. For example, the user can draw a cube area in the 3D modeling software or scan the actual physical space through the sensor to determine the target space coordinate range as X: [1.0m, 3.0m], Y: [2.0m, 4.0m], Z: [0.5m, 2.5m].

[0109] At the same time, the target perspective parameter set is obtained, where the spatial perspective coordinates are determined by the user's perspective selection operation in the virtual scene as (θ = 30°, ); the time phase parameter is set to t = 5s according to the system clock and the scene animation progress; the optical characteristic parameters of the observation device are obtained according to the display device parameters of the terminal device, such as camera parameters, focal length f = 50mm, aperture size F / 2.8, etc.

[0110] Get the target timestamp, which is used to identify the time information of the current rendering task. For example, get the current Network Time Protocol (NTP) timestamp as 2024-10-10T14:30:00Z.

[0111] The reference space coordinate range of the previous rendering scene of the target holographic scene is obtained from local storage or the previous rendering task cache, assuming it is X: [1.5m, 2.8m], Y: [2.3m, 3.8m], Z: [0.8m, 2.0m] and the reference light field features. The reference light field features include multi-resolution spatiotemporal light field primitives and corresponding feature entropy values.

[0112] The quantum dot metasurface light field modulator of the terminal device is compared and calculated based on the target space coordinate range (X: [1.0m, 3.0m], Y: [2.0m, 4.0m], Z: [0.5m, 2.5m]) and the reference space coordinate range (X: [1.5m, 2.8m], Y: [2.3m, 3.8m], Z: [0.8m, 2.0m]) through the built-in spatial coordinate analysis algorithm. The algorithm is based on the principle of geometric space division and comparison. The target space and reference space are gridded, and the coordinate range of the newly added space is determined to be X: [1.0m, 1.5m], X: [2.8m, 3.0m], Y: [2.0m, 2.3m], Y: [3.8m, 4.0m], Z: [0.5m, 0.8m], Z: [2.0m, 2.5m], and the coordinate range of the overlapping space is X: [1.5m, 2.8m], Y: [2.3m, 3.8m], Z: [0.8m, 2.0m].

[0113] Based on the dynamic response characteristics of the quantum dot metasurface, such as the excitation threshold of the quantum dot material being 2V and the response time being 10ps, a quantum dot excitation sequence corresponding to the newly added spatial coordinate range is generated. The excitation sequence is a pulse sequence containing a series of electric field strengths and timing information. For example, for each micro-nanostructure unit position within the newly added spatial coordinate range, an electric field pulse sequence with an amplitude of 2V-5V, a pulse width of 10ps-50ps, and a repetition frequency of 10MHz-50MHz is generated to drive the micro-nanostructure units of the quantum dot metasurface to change their optical properties.

[0114] After the quantum dot metasurface light field modulator receives the excitation sequence, its micro-nanostructure unit changes its own optical refractive index and phase delay characteristics according to the preset modulation rule. For example, when the micro-nanostructure unit receives the electric field pulse, the electronic energy level structure of the quantum dot material inside it changes, resulting in a refractive index change of Δn=0.01-0.05 and a phase delay change of In this way, the initial light field modulation pattern is generated, and the incident light field is spatially and phase modulated.

[0115] The prediction compensation rendering engine of the terminal device is based on the target viewing angle parameter set (spatial viewing angle coordinates (θ=30°, ), time phase parameters (t=5s), optical characteristic parameters of observation equipment (focal length f=50mm, aperture size F / 2.8, etc.)) to construct a NeRF-holographic joint coding model. The model first uses the multi-layer perceptron (MLP) network structure of NeRF to encode the spatial and perspective information of the light field, and maps the input initial light field modulation pattern data, such as light intensity, phase, etc. to a high-dimensional feature space. Then, combined with the physical optical model of the holographic image, such as the Fresnel diffraction formula, the data in the feature space is further processed to meet the generation requirements of the holographic image. For example, by adjusting the number of hidden layer nodes and activation function of the MLP network, and optimizing the parameters in the holographic physical model, such as wavelength, propagation distance, etc., a joint coding model that can accurately describe the light field characteristics of the target holographic scene is constructed.

[0116] The initial light field modulation pattern and the NeRF-holographic joint coding model parameters are input into the prediction compensation rendering engine, and the spatiotemporal continuity optimization is performed in combination with the reference light field characteristics (multi-resolution spatiotemporal light field primitives and corresponding characteristic entropy values). Specifically, the rendering engine first calculates the difference between the initial light field modulation pattern and the reference light field characteristics in the spatiotemporal domain. For example, by calculating indicators such as light intensity difference and phase difference, the difference area and degree of difference are obtained. Then, the light field interpolation algorithm, such as bicubic interpolation, is used to compensate for the difference area to generate a light field modulation pattern that is more continuous in time and space.

[0117] Through the light field compensation algorithm, the pre-rendered light field data (generated according to the target timestamp and the multi-scale light field prediction model) is subjected to residual analysis with the output of the NeRF-holographic joint coding model to generate light field compensation parameters. For example, the pre-rendered light field data is a preliminary light field estimation value generated based on the historical light field data and the time series prediction model, and there is a residual between it and the actual light field value output by the joint coding model. The rendering engine generates compensation parameters such as compensated light intensity value and compensated phase value by calculating the size and distribution of the residual, which are used to correct the initial light field modulation mode.

[0118] Combined with the initial light field modulation mode and light field compensation parameters, the target light field modulation instruction set is generated. This instruction set contains precise modulation instructions for the micro-nanostructure units of the quantum dot metasurface, such as the light intensity, phase and other modulation parameters of each unit at different time points, to ensure the continuity and consistency of the light field in time and space. For example, for each micro-nanostructure unit, the instruction set stipulates that the light intensity modulation value at t = 5s is I = 0.8-1.2 (normalized value), and the phase modulation value is And parameters such as the modulation change rate within the time interval Δt=0.01s.

[0119] The terminal device sends a dynamic light field data request including a target light field modulation instruction set and a target timestamp (2024-10-10T14:30:00Z) to the server through its communication module, such as a wireless network interface or a wired network interface. The request is used to apply to the server for obtaining a newly added spatial coordinate range (X: [1.0m, 1.5m], X: [2.8m, 3.0m], Y: [2.0m, 2.3m], Y: [3.8m, 4.0m], Z: [0.5m, 0.8m], Z: [2.0m, 2.5m]), a target viewing angle parameter set (spatial viewing angle coordinates (θ = 30°, ), time phase parameters (t = 5s), optical characteristic parameters of the observation equipment (focal length f = 50mm, aperture size F / 2.8, etc.) and multi-dimensional light field data corresponding to the target timestamp.

[0120] After receiving the request, the server processes the data according to the system's workflow and sends the entropy-compressed multidimensional light field data back to the terminal device through a multi-channel light field transmission protocol. The communication module of the terminal device receives the data returned by the server and passes it to the spatiotemporal entropy decoder.

[0121] The spatiotemporal entropy decoder performs spatiotemporal entropy decoding and light field reconstruction through a prediction compensation rendering engine. The decoder first determines the type of entropy coding algorithm, such as arithmetic coding, based on the data header information sent by the server, and then calls the corresponding decoding algorithm to decode the data and restore the four-dimensional light field tensor data. For example, for an entropy compressed data block that has been arithmetic encoded, the decoder initializes the decoding state machine and gradually restores the original four-dimensional light field data value, including light intensity, direction and other information, according to the coding probability model.

[0122] The decoded light field data is fused and rendered with the historical light field data corresponding to the overlapping spatial coordinate range (i.e., the light field data of the overlapping part of the previous rendered scene with the current target scene). The specific process is to first perform a spatiotemporal consistency check on the reconstructed light field data and the historical light field data, and generate a light field fusion weight matrix by calculating the correlation coefficient, overlap and other indicators of the data in the spatiotemporal domain. For example, for data points with high overlap and strong correlation in the spatiotemporal domain, a larger weight value, such as 0.8-1.0, is assigned, while for data points with low overlap and weak correlation, a smaller weight value, such as 0.2-0.5, is assigned.

[0123] The reconstructed light field data and the historical light field data are weightedly fused based on the light field fusion weight matrix to generate the final rendered light field data. For example, for each data point, the final rendered light field data = weight value × reconstructed light field data + (1-weight value) × historical light field data.

[0124] The final rendered light field data is converted into a holographic image signal through a dynamic light field modulator. The modulator controls the optical properties of its micro-nanostructure units according to the light intensity and phase information in the final rendered light field data, so that the incident light field is modulated according to a predetermined holographic image pattern. For example, for the micro-nanostructure unit corresponding to each pixel point, its phase delay is adjusted according to the phase value in the data, and its transmittance or reflectivity is adjusted according to the light intensity value, thereby converting the light field data into an observable holographic image signal and projecting it to generate the target holographic scene.

[0125] A multi-dimensional holographic image real-time rendering method based on dynamic light field modulation is applied to a server, and the method includes:

[0126] Receiving a dynamic light field data request sent by a terminal device, wherein the dynamic light field data request includes a target light field modulation instruction set, a newly added spatial coordinate range, a target viewing angle parameter set, and a target timestamp;

[0127] Retrieving original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from a target database, wherein the original holographic data includes a multi-view depth image sequence, light field intensity distribution data, and phase modulation information;

[0128] Decoding the target light field modulation instruction set through a quantum dot metasurface response model to generate light field modulation parameters;

[0129] Constructing a NeRF-holographic joint coding model based on the light field modulation parameters, performing feature extraction and encoding on the original holographic data, and generating a multi-dimensional light field feature tensor;

[0130] According to the target view parameter set and the target timestamp, the multi-dimensional light field feature tensor is optimized for spatiotemporal continuity by a prediction compensation rendering engine to generate four-dimensional light field tensor data;

[0131] performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model; performing entropy compression encoding on the four-dimensional light field tensor data based on the light field entropy compression model to generate entropy-compressed multidimensional light field data;

[0132] The entropy-compressed multi-dimensional light field data is sent to the terminal device through a multi-channel light field transmission protocol, so that the terminal device combines the historical light field data for fusion rendering to generate a target holographic scene.

[0133] Before receiving the dynamic light field data request sent by the terminal device, the method further includes:

[0134] Acquire multi-source heterogeneous holographic data sources, including image sequences acquired by multi-angle camera arrays, four-dimensional light field data acquired by light field cameras, and computer-generated virtual holographic data;

[0135] Performing spatiotemporal synchronization and registration on the multi-source heterogeneous holographic data sources to generate spatiotemporally aligned original holographic data;

[0136] Build a quantum dot metasurface database to store the optical response functions of metasurfaces under different excitation conditions;

[0137] The step of retrieving the original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from the target database includes:

[0138] According to the target light field modulation instruction set, matching a corresponding optical response function from the quantum dot metasurface database;

[0139] constructing a light field modulation filter based on the optical response function, and screening and retrieving matching original holographic data from the target database;

[0140] The target database is organized in a spatiotemporal octree structure, and each spatiotemporal node stores a light field feature descriptor and a data pointer of a corresponding spatiotemporal region;

[0141] The performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model includes:

[0142] Performing spatiotemporal division on the four-dimensional light field tensor data based on the spatiotemporal octree structure to generate multi-level spatiotemporal light field primitives;

[0143] Calculate the information entropy of space-time light field primitives at all levels and construct a space-time entropy pyramid;

[0144] Determine the redundancy distribution of light field data according to the spatiotemporal entropy pyramid and generate an adaptive entropy coding dictionary;

[0145] The light field entropy compression model is constructed based on the adaptive entropy coding dictionary.

[0146] The server maintains a light field transmission quality prediction model, wherein the light field transmission quality prediction model is constructed based on the transmission characteristics of the quantum dot metasurface and channel state information;

[0147] The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes:

[0148] Predicting the light field transmission quality of each transmission channel using the light field transmission quality prediction model to generate a channel quality evaluation matrix;

[0149] Slicing the entropy-compressed multidimensional light field data according to the channel quality assessment matrix to generate a plurality of data slices;

[0150] Each data fragment is sent in parallel to the terminal device through the corresponding optimal transmission channel, and a time-space synchronization mark and quality control information are added.

[0151] The method further comprises:

[0152] Build a dynamic light field caching mechanism to cache frequently accessed four-dimensional light field tensor data;

[0153] The performing entropy compression encoding on the four-dimensional light field tensor data to generate entropy-compressed multi-dimensional light field data comprises:

[0154] detecting whether a cached copy of the four-dimensional light field tensor data exists;

[0155] If a cached copy exists, generating incremental entropy coded data based on the difference between the cached data and the current data; if no cached copy exists, performing full entropy compression coding on the four-dimensional light field tensor data;

[0156] The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes:

[0157] If it is incremental entropy coded data, the incremental coding information and cached data pointer are sent; if it is complete entropy compression coded data, the complete coded data is sent directly.

[0158] See also Figure 2 The server receives a dynamic light field data request sent by the terminal device through its network interface, such as a high-speed Ethernet interface. The request includes a target light field modulation instruction set, a newly added spatial coordinate range, such as X: [1.0m, 1.5m], X: [2.8m, 3.0m], Y: [2.0m, 2.3m], Y: [3.8m, 4.0m], Z: [0.5m, 0.8m], Z: [2.0m, 2.5m]), a target viewing angle parameter set (spatial viewing angle coordinates (θ=30°, φ=45°), time phase parameters (t=5s), optical characteristic parameters of the observation device (focal length f=50mm, aperture size F / 2.8, etc.)), and a target timestamp (2024-10-10T14:30:00Z).

[0159] The server's request processing module parses the received request data and extracts key information. For example, it parses the target light field modulation instruction set into specific modulation parameters, including the light intensity and phase modulation requirements of each micro-nanostructure unit; stores the newly added spatial coordinate range and target viewing angle parameter set as a corresponding data structure for subsequent data query and processing; and records the target timestamp for data time synchronization and version management.

[0160] Based on the parsed request information, the server accesses its internal target database, which uses a spatiotemporal octree structure to organize and store the original holographic data. Each spatiotemporal octree node represents a specific spatiotemporal region and stores the light field feature descriptors of the region, such as the mean light intensity, directional vector distribution, and data pointer information.

[0161] The optical response function corresponding to the target light field modulation instruction set is matched from the quantum dot metasurface database. For example, according to the modulation requirements of light intensity and phase in the instruction set, an optical response function with similar response characteristics is found. Based on the matched optical response function, a light field modulation filter is constructed. This filter can filter out the original holographic data that meets the requirements.

[0162] Using the constructed light field modulation filter, the original holographic data that matches the newly added spatial coordinate range, target viewing angle parameter set and target timestamp is screened and retrieved from the target database. The original holographic data includes multi-perspective depth image sequences, such as image sequences with depth information taken from different angles. The resolution of each image is 4K, and the depth accuracy reaches the millimeter level; light field intensity distribution data, such as the light intensity distribution in different directions in three-dimensional space, with a data sampling density of 1000 sampling points per cubic meter of space; and phase modulation information (parameters used to describe the phase change of light waves, with an accuracy of 0.01 radians).

[0163] The server's NeRF-holographic joint encoder constructs a coding model based on the light field modulation parameters. The model uses the multi-layer perceptron (MLP) network structure of the neural radiation field (NeRF) to encode the spatial and perspective information of the light field, and maps the input original holographic data to a high-dimensional feature space. At the same time, it combines the physical optical model of the holographic image, such as the Fresnel diffraction formula, to further process the data in the feature space to meet the generation requirements of the holographic image. For example, by adjusting the number of hidden layer nodes and activation function of the MLP network, and optimizing the parameters in the holographic physical model, such as wavelength, propagation distance, etc., a joint coding model is constructed that can accurately describe the light field characteristics of the target holographic scene.

[0164] Based on the target perspective parameter set and target timestamp, the server's predictive compensation rendering engine optimizes the spatiotemporal continuity of the multi-dimensional light field feature tensor. The rendering engine analyzes the changing patterns of the light field in the spatiotemporal domain and uses light field interpolation and compensation algorithms to optimize the light field data to generate more continuous and natural four-dimensional light field tensor data. For example, in the time dimension, the light field data is interpolated and predicted according to the light field change trends of adjacent time points to reduce the abruptness of light field changes within the time interval; in the spatial dimension, the light field data is smoothed according to the light field correlation of adjacent spatial regions to improve spatial continuity.

[0165] The server performs spatiotemporal entropy analysis on the optimized four-dimensional light field tensor data, divides the data into spatiotemporal and spatial divisions based on the spatiotemporal octree structure, generates multi-level spatiotemporal light field primitives, and then calculates the information entropy of the spatiotemporal light field primitives at each level to construct a spatiotemporal entropy pyramid. Based on the spatiotemporal entropy pyramid, the redundancy distribution of the light field data is determined, an adaptive entropy coding dictionary is generated, and a light field entropy compression model is constructed.

[0166] The constructed light field entropy compression model is used to perform entropy compression encoding on the four-dimensional light field tensor data to generate entropy-compressed multi-dimensional light field data. For example, an entropy coding algorithm such as arithmetic coding or Huffman coding is used to encode the data according to the probability distribution model in the adaptive entropy coding dictionary, thereby reducing data redundancy and data volume.

[0167] The server's multi-channel light field transmission engine prepares to send entropy-compressed multi-dimensional light field data to the terminal device through the multi-channel light field transmission protocol. At the same time, the server uses the maintained light field transmission quality prediction model (based on the transmission characteristics of the quantum dot metasurface and the channel state information) to predict the light field transmission quality of each transmission channel and generate a channel quality evaluation matrix, which includes quality indicators such as bandwidth, delay, and packet loss rate of each channel.

[0168] According to the channel quality assessment matrix, the entropy-compressed multidimensional light field data is segmented to generate multiple data segments. Each data segment is assigned to the corresponding optimal transmission channel, and a spatiotemporal synchronization mark and quality control information are attached. Each data segment is sent in parallel to the terminal device through the corresponding optimal transmission channel to improve data transmission efficiency.

[0169] The server builds a dynamic light field caching mechanism to cache frequently accessed four-dimensional light field tensor data. When entropy compression encoding is performed on the four-dimensional light field tensor data, it first detects whether there is a cached copy of the data, for example, by searching and matching in the cache through the data's hash value or unique identifier.

[0170] If there is a cached copy, incremental entropy coded data is generated based on the difference between the cached data and the current data. The incremental data only contains the difference between the current data and the cached copy, which can effectively reduce the amount of data transmission. At the same time, the incremental coding information and the cached data pointer are sent to the terminal device so that the terminal device can restore the complete light field data based on the cached data and the incremental data. If there is no cached copy, the four-dimensional light field tensor data is fully entropy compressed and encoded, and the complete encoded data is sent directly to the terminal device.

[0171] Example 2

[0172] A multi-dimensional holographic image real-time rendering system based on dynamic light field modulation, the system comprising: a terminal device and a server;

[0173] The terminal device is used to execute the above-mentioned method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation;

[0174] The server is used to execute the above-mentioned method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation;

[0175] The terminal device includes: a quantum dot metasurface light field modulator, a prediction compensation rendering engine, a spatiotemporal entropy decoder and a dynamic light field display;

[0176] The server includes: a NeRF-holographic joint encoder, a light field entropy compression module, a multi-channel light field transmission engine and a spatiotemporal octree database.

[0177] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation.

[0178] Terminal equipment composition:

[0179] The quantum dot metasurface light field modulator includes a dynamic response plane made of high-precision quantum dot material, on the surface of which are distributed micro-nanostructure units with diameters of approximately 100-200 nanometers. These micro-nanostructure units can change their own optical refractive index, reflectivity, phase delay and other properties within a picosecond time scale under the excitation of an external electric field or a specific light signal, thereby realizing dynamic modulation of the incident light field. For example, when receiving a quantum dot excitation sequence, each micro-nanostructure unit can cause the different frequency components of the incident light field to change accordingly, thereby changing the spatial distribution and propagation direction of the light field.

[0180] The built-in high-resolution spatial light modulation driver chip can accurately control the excitation state of each micro-nanostructure unit. The chip has at least 12-bit grayscale control accuracy and can generate 4096 different excitation intensity levels to meet the needs of complex light field modulation.

[0181] Prediction compensation rendering engine: It adopts a high-performance graphics processing unit (GPU) cluster architecture, which includes multiple independent computing cores. Each core has a floating-point computing capability of 1012 times per second. These computing cores can process light field data and model parameters in parallel to achieve efficient image rendering calculations. For example, when building a NeRF-holographic joint coding model, multiple cores can simultaneously extract and encode features from data of different perspectives and spatial positions.

[0182] The spatiotemporal entropy decoder is equipped with a large-capacity cache, such as HBM-High Bandwidth Memory, with a bandwidth of more than 1TB per second. It can quickly read and write light field data and intermediate calculation results, reduce data transmission delays, and ensure the real-time performance of the rendering process.

[0183] Based on an advanced digital signal processor (DSP), it has a built-in specially designed entropy decoding algorithm library, which contains a variety of entropy decoding algorithms suitable for four-dimensional light field data, such as arithmetic decoding and Huffman decoding. For example, for the entropy coded data sent by the server after decorrelation processing, the decoder can select the optimal decoding algorithm based on the statistical characteristics of the data and quickly restore the original four-dimensional light field tensor data.

[0184] It has the function of adaptive decoding parameter adjustment, which can dynamically adjust the decoding parameters according to the bit rate, signal-to-noise ratio and other characteristics of the input entropy coded data to improve decoding efficiency and accuracy.

[0185] The dynamic light field display uses a high refresh rate holographic display with a pixel resolution of up to 8K. Each pixel can independently control light intensity and phase information. Through a special optical transmission system, such as a lens group with a high numerical aperture, the modulated light field is accurately projected into the observation space.

[0186] The display has a wide field of view, ensuring that users can view the complete holographic image at different positions, and can adjust the image display parameters in real time according to the user's head position and viewing angle changes to achieve dynamic viewing angle tracking display.

[0187] Server composition:

[0188] The NeRF-holographic joint encoder includes a deep learning training framework and a model inference engine. The training framework supports a variety of advanced neural network architectures, such as the Transformer-based architecture, which is used for feature extraction and encoding training of holographic data. The model inference engine can quickly load the trained model and perform real-time inference encoding on the input raw holographic data. For example, for multi-view depth image sequence data, the encoder can extract the depth correlation features between different viewpoints and encode them into a multi-dimensional light field feature tensor.

[0189] It is equipped with a large-scale storage array for storing training data sets and model parameters. The storage array adopts a distributed storage architecture with a total storage capacity of hundreds of TB, which can meet the storage needs of large-scale holographic data and has high-speed data reading and writing capabilities (data reading and writing bandwidth can reach several GB per second) to ensure data supply for the encoding process.

[0190] Light field entropy compression module: This module uses dedicated hardware compression chips, which are custom-designed based on advanced entropy coding algorithms such as context-adaptive binary arithmetic coding (CABAC). These chips can efficiently compress four-dimensional light field tensor data, achieving a compression ratio of 10:1-20:1 while maintaining high data fidelity. For example, for a four-dimensional light field tensor data of size 1000×1000×1000×1000 (assuming each dimension represents spatial coordinates and viewing angle coordinates), the data volume can be reduced to approximately 1 / 10-1 / 20 after compression.

[0191] It has the function of adaptive compression parameter adjustment, which can dynamically adjust the compression parameters according to the content characteristics of the data, such as the complexity of light field changes and data redundancy, to optimize the compression effect.

[0192] Multi-channel light field transmission engine: includes multiple high-speed network interface cards (NICs), each of which supports 10 Gigabit Ethernet (10GbE) or higher-bandwidth optical transmission network interfaces. It can simultaneously establish multiple transmission channels for data exchange with terminal devices. For example, when sending entropy-compressed multi-dimensional light field data to a terminal device, it can be transmitted in parallel through multiple channels to increase data transmission speed.

[0193] The built-in intelligent traffic scheduling algorithm can dynamically adjust the transmission order and routing path of data fragments based on network quality parameters such as real-time bandwidth, latency and packet loss rate of each transmission channel to ensure the reliability and real-time performance of data transmission.

[0194] Spatiotemporal octree database: A distributed database management system is used to organize and store the original holographic data according to the spatiotemporal octree structure. Each spatiotemporal octree node represents a specific spatiotemporal region and stores the light field feature descriptors of the region, such as the mean light intensity, directional vector distribution, and data pointer information. For example, an octree node may represent a spatiotemporal region with a spatial range of 1m×1m×1m and a time interval of 0.1s. Its light field feature descriptors can be used to quickly query and filter the original holographic data that matches the target light field modulation instruction set.

[0195] The database has an efficient indexing mechanism and query optimization algorithm, which can quickly respond to the server's query requests. For example, when the server queries the original holographic data of the corresponding space-time area according to the target light field modulation instruction set, the database can return the matching data results within milliseconds.

[0196] See also Figure 3When the terminal device starts the holographic scene rendering task, it first obtains the target space coordinate range of the target holographic scene through its internal sensors and user input devices, such as a rectangular space area specified by the user, with a coordinate range of X: [1.0m, 3.0m], Y: [2.0m, 4.0m], Z: [0.5m, 2.5m]), the target perspective parameter set, including the spatial perspective coordinates (θ = 30°, ), time phase parameters (t = 5s) and optical characteristic parameters of the observation equipment, such as the focal length of the camera f = 50mm, aperture size F / 2.8, etc. and the target timestamp, such as 2024-10-10T14:30:00Z.

[0197] The terminal device obtains the reference space coordinate range of the last rendered scene from local storage, assuming it is X: [1.5m, 2.8m], Y: [2.3m, 3.8m], Z: [0.8m, 2.0m]) and reference light field features, including multi-resolution spatiotemporal light field primitives, such as a low-resolution primitive size of 50mm×50mm×50mm spatial resolution and 0.01s temporal resolution, a high-resolution primitive size of 10mm×10mm×10mm spatial resolution and 0.001s temporal resolution, and the corresponding characteristic entropy values. The entropy value of the low-resolution primitive is 0.6bit / pixel, and the entropy value of the high-resolution primitive is 0.9bit / pixel.

[0198] The quantum dot metasurface light field modulator of the terminal device determines the newly added spatial coordinate range based on the target spatial coordinate range and the reference spatial coordinate range through the built-in spatial coordinate analysis algorithm. For example, after calculation, it is found that the newly added spatial regions are X: [1.0m, 1.5m], X: [2.8m, 3.0m], Y: [2.0m, 2.3m], Y: [3.8m, 4.0m], Z: [0.5m, 0.8m], Z: [2.0m, 2.5m]) and the overlapping spatial coordinate range, namely X: [1.5m, 2.8m], Y: [2.3m, 3.8m], Z: [0.8m, 2.0m]). Then, based on the dynamic response characteristics of the quantum dot metasurface, such as the excitation threshold and response time of the quantum dot material, a quantum dot excitation sequence corresponding to the newly added spatial coordinate range is generated. This excitation sequence is a pulse sequence containing a series of electric field strength and timing information, which is used to drive the micro-nanostructure units of the quantum dot metasurface to change the optical properties.

[0199] After the quantum dot metasurface light field modulator receives the excitation sequence, its micro-nanostructure unit changes its own optical refractive index, phase delay and other characteristics according to the preset modulation rules to generate an initial light field modulation pattern. At the same time, the predictive compensation rendering engine of the terminal device constructs a NeRF-holographic joint coding model based on the target perspective parameter set. This model encodes and optimizes the initial light field modulation pattern by combining the perspective modeling capability of the neural radiation field and the physical optical properties of the holographic image to adapt to the target perspective observation requirements.

[0200] The predictive compensation rendering engine inputs the initial light field modulation pattern and the NeRF-holographic joint coding model parameters, and combines the reference light field characteristics to optimize the spatiotemporal continuity. Specifically, by comparing the differences between the initial light field modulation pattern and the reference light field characteristics in the spatiotemporal domain, the light field interpolation and compensation algorithm is used to generate the target light field modulation instruction set. This instruction set contains precise modulation instructions for the micro-nanostructure units of the quantum dot metasurface, such as the modulation parameters of each unit such as light intensity and phase at different time points, to ensure the continuity and consistency of the light field in time and space.

[0201] The terminal device sends a dynamic light field data request containing a target light field modulation instruction set and a target timestamp to the server through its communication module. The request is used to apply to the server for multi-dimensional light field data corresponding to the newly added spatial coordinate range, target viewing angle parameter set and target timestamp. After receiving the request, the server first parses the target light field modulation instruction set and retrieves the corresponding original holographic data from the spatiotemporal octree database. These original holographic data include multi-perspective depth image sequences, such as image sequences with depth information taken from different angles, each image with a resolution of 4K and a depth accuracy of millimeter level, light field intensity distribution data, such as the light intensity distribution in different directions in three-dimensional space, with a data sampling density of 1000 sampling points per cubic meter of space, and phase modulation information, which is used to describe the parameters of the phase change of the light wave with an accuracy of 0.01 radians.

[0202] The server's NeRF-holographic joint encoder decodes the target light field modulation instruction set through the quantum dot metasurface response model to generate light field modulation parameters. These parameters are used to guide the encoder to extract and encode the features of the original holographic data to generate a multi-dimensional light field feature tensor. Then, based on the target viewing angle parameter set and target timestamp, the prediction compensation rendering engine optimizes the spatiotemporal continuity of the multi-dimensional light field feature tensor to generate four-dimensional light field tensor data. The light field entropy compression module performs spatiotemporal entropy analysis on the four-dimensional light field tensor data, constructs a light field entropy compression model, and performs entropy compression encoding based on the model to generate entropy-compressed multi-dimensional light field data. Finally, the multi-channel light field transmission engine sends the entropy-compressed multi-dimensional light field data back to the terminal device through the multi-channel light field transmission protocol.

[0203] After the terminal device receives the entropy-compressed multi-dimensional light field data returned by the server, the spatiotemporal entropy decoder performs spatiotemporal entropy decoding and light field reconstruction through the prediction compensation rendering engine. The decoded light field data is fused with the historical light field data corresponding to the overlapping spatial coordinate range (that is, the light field data of the overlapping part of the previous rendered scene with the current target scene) for rendering. The specific process is to first perform spatiotemporal consistency verification on the reconstructed light field data and the historical light field data, and generate a light field fusion weight matrix. The matrix assigns a weight value to each data point according to the correlation and reliability of the data in the spatiotemporal domain. Then, based on the light field fusion weight matrix, the reconstructed light field data and the historical light field data are weightedly fused to generate the final rendered light field data. Finally, the final rendered light field data is converted into a holographic image signal through a dynamic light field modulator, and the target holographic scene is projected.

[0204] The same or similar reference numerals correspond to the same or similar components;

[0205] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0206] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation, characterized in that: Applied to a terminal device, the method includes: Obtaining the target space coordinate range, target viewing angle parameter set, and target timestamp of the target holographic scene to be rendered, and obtaining the reference space coordinate range and reference light field characteristics of the previous rendered scene of the target holographic scene; Based on the dynamic response characteristics of the quantum dot metasurface, determining a newly added spatial coordinate range and an overlapping spatial coordinate range according to the target spatial coordinate range and the reference spatial coordinate range, and generating a quantum dot excitation sequence corresponding to the newly added spatial coordinate range; The metasurface micro-nanostructure is regulated by the quantum dot excitation sequence to generate an initial light field modulation pattern, and a NeRF-holographic joint encoding model is constructed in combination with the target viewing angle parameter set; Inputting the initial light field modulation pattern and the NeRF-holographic joint encoding model parameters into the prediction compensation rendering engine, performing spatiotemporal continuity optimization in combination with the reference light field characteristics, and generating a target light field modulation instruction set; Sending a dynamic light field data request including the target light field modulation instruction set and the target timestamp to the server, wherein the dynamic light field data request is used to request multidimensional light field data corresponding to the newly added spatial coordinate range, the target viewing angle parameter set, and the target timestamp; the multidimensional light field data is four-dimensional light field tensor data encoded by entropy compression, and is generated by preprocessing the multi-source heterogeneous holographic data stored in the target database; The entropy-compressed multidimensional light field data returned by the server is received, spatiotemporal entropy decoding and light field reconstruction are performed through a prediction compensation rendering engine, and fusion rendering is performed in combination with the historical light field data corresponding to the overlapping spatial coordinate range to generate the target holographic scene.

2. The method according to claim 1, characterized in that The reference light field features include multi-resolution spatiotemporal light field primitives and corresponding characteristic entropy values, and the target viewing angle parameter set includes spatial viewing angle coordinates, temporal phase parameters, and optical characteristic parameters of the observation device; Before obtaining the target space coordinate range, target viewing angle parameter set and target timestamp of the target holographic scene to be rendered, the method further includes: Performing spatiotemporal frequency analysis on the reference light field characteristics to construct a multi-scale light field prediction model; generating pre-rendered light field data using the multi-scale light field prediction model according to the target timestamp; The generating target light field modulation instruction set comprises: Perform residual analysis on the pre-rendered light field data and the output of the NeRF-holographic joint encoding model to generate light field compensation parameters; The target light field modulation instruction set is generated by combining the initial light field modulation mode and the light field compensation parameters.

3. The method according to claim 1, characterized in that After receiving the entropy-compressed multi-dimensional light field data returned by the server, the method further includes: Based on the nonlinear optical response characteristics of quantum dot metasurfaces, a light field entropy decoding function is constructed; Performing spatiotemporal entropy decoding on the multidimensional light field data using the light field entropy decoding function to restore four-dimensional light field tensor data; Inputting the four-dimensional light field tensor data into a holographic reconstruction module, performing phase recovery and amplitude modulation in combination with the target light field modulation instruction set, and generating reconstructed light field data; The performing fusion rendering in combination with the historical light field data corresponding to the overlapping spatial coordinate range includes: Performing a spatiotemporal consistency check on the reconstructed light field data and the historical light field data to generate a light field fusion weight matrix; Performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data; The final rendered light field data is converted into a holographic image signal through a dynamic light field modulator, and the target holographic scene is generated by projection.

4. The method according to claim 3, characterized in that The step of performing weighted fusion on the reconstructed light field data and the historical light field data based on the light field fusion weight matrix to generate final rendered light field data includes: Constructing a spatiotemporal attention mechanism to generate spatiotemporal attention weights according to the target view parameter set and the target timestamp; Combining the spatiotemporal attention weight and the light field fusion weight matrix, performing spatiotemporal attention fusion on the reconstructed light field data and the historical light field data to generate an initial fused light field; The prediction compensation rendering engine is used to perform high-frequency detail enhancement and low-frequency noise suppression on the initial fused light field to generate the final rendered light field data.

5. A method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation, characterized in that: Applied to a server, the method includes: Receiving a dynamic light field data request sent by a terminal device, wherein the dynamic light field data request includes a target light field modulation instruction set, a newly added spatial coordinate range, a target viewing angle parameter set, and a target timestamp; Retrieving original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from a target database, wherein the original holographic data includes a multi-view depth image sequence, light field intensity distribution data, and phase modulation information; Decoding the target light field modulation instruction set through a quantum dot metasurface response model to generate light field modulation parameters; Constructing a NeRF-holographic joint coding model based on the light field modulation parameters, performing feature extraction and encoding on the original holographic data, and generating a multi-dimensional light field feature tensor; According to the target view parameter set and the target timestamp, the multi-dimensional light field feature tensor is optimized for spatiotemporal continuity by a prediction compensation rendering engine to generate four-dimensional light field tensor data; performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model; performing entropy compression encoding on the four-dimensional light field tensor data based on the light field entropy compression model to generate entropy-compressed multidimensional light field data; The entropy-compressed multi-dimensional light field data is sent to the terminal device through a multi-channel light field transmission protocol, so that the terminal device combines the historical light field data for fusion rendering to generate a target holographic scene.

6. The method according to claim 5, characterized in that Before receiving the dynamic light field data request sent by the terminal device, the method further includes: Acquire multi-source heterogeneous holographic data sources, including image sequences acquired by multi-angle camera arrays, four-dimensional light field data acquired by light field cameras, and computer-generated virtual holographic data; Performing spatiotemporal synchronization and registration on the multi-source heterogeneous holographic data sources to generate spatiotemporally aligned original holographic data; Build a quantum dot metasurface database to store the optical response functions of metasurfaces under different excitation conditions; The step of retrieving the original holographic data corresponding to the newly added spatial coordinate range, target viewing angle parameter set, and target timestamp from the target database includes: According to the target light field modulation instruction set, matching a corresponding optical response function from the quantum dot metasurface database; constructing a light field modulation filter based on the optical response function, and screening and retrieving matching original holographic data from the target database; The target database is organized in a spatiotemporal octree structure, and each spatiotemporal node stores a light field feature descriptor and a data pointer of a corresponding spatiotemporal region; The performing spatiotemporal entropy analysis on the four-dimensional light field tensor data to construct a light field entropy compression model includes: Performing spatiotemporal division on the four-dimensional light field tensor data based on the spatiotemporal octree structure to generate multi-level spatiotemporal light field primitives; Calculate the information entropy of space-time light field primitives at all levels and construct a space-time entropy pyramid; Determine the redundancy distribution of light field data according to the spatiotemporal entropy pyramid and generate an adaptive entropy coding dictionary; The light field entropy compression model is constructed based on the adaptive entropy coding dictionary.

7. The method according to claim 6, characterized in that The server maintains a light field transmission quality prediction model, wherein the light field transmission quality prediction model is constructed based on the transmission characteristics of the quantum dot metasurface and channel state information; The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes: Predicting the light field transmission quality of each transmission channel using the light field transmission quality prediction model to generate a channel quality evaluation matrix; Slicing the entropy-compressed multidimensional light field data according to the channel quality assessment matrix to generate a plurality of data slices; Each data fragment is sent in parallel to the terminal device through the corresponding optimal transmission channel, and a time-space synchronization mark and quality control information are added.

8. The method according to claim 7, characterized in that The method further comprises: Build a dynamic light field caching mechanism to cache frequently accessed four-dimensional light field tensor data; The performing entropy compression encoding on the four-dimensional light field tensor data to generate entropy-compressed multi-dimensional light field data comprises: detecting whether a cached copy of the four-dimensional light field tensor data exists; If a cached copy exists, generating incremental entropy coded data based on the difference between the cached data and the current data; if no cached copy exists, performing full entropy compression coding on the four-dimensional light field tensor data; The sending of the entropy-compressed multi-dimensional light field data to the terminal device through a multi-channel light field transmission protocol includes: If it is incremental entropy coded data, the incremental coding information and cached data pointer are sent; if it is complete entropy compression coded data, the complete coded data is sent directly.

9. A multi-dimensional holographic image real-time rendering system based on dynamic light field modulation, characterized in that: The system includes: a terminal device and a server; The terminal device is used to execute the real-time rendering method of multi-dimensional holographic images based on dynamic light field modulation according to any one of claims 1 to 4; The server is used to execute the method for real-time rendering of multi-dimensional holographic images based on dynamic light field modulation according to any one of claims 5 to 8; The terminal device includes: a quantum dot metasurface light field modulator, a prediction compensation rendering engine, a spatiotemporal entropy decoder and a dynamic light field display; The server includes: a NeRF-holographic joint encoder, a light field entropy compression module, a multi-channel light field transmission engine and a spatiotemporal octree database.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method for real-time rendering of a multi-dimensional holographic image based on dynamic light field modulation according to any one of claims 1 to 8 is implemented.

Images