VR display PPD breakthrough method and system based on multi-level perception optimization, and medium

Abstract

The invention relates to a VR display PPD breakthrough method and system based on multi-level perception optimization and a medium, and the method comprises the steps: carrying out the adaptive non-uniform super-resolution rendering of an input image based on retina perception distribution, and carrying out the dynamic adjustment of a rendering zoom factor through the eccentricity of a visual angle deviating from a fixation point; performing dynamic optical compensation on the image after super-resolution rendering according to eccentricity of different areas relative to the fixation point; performing dispersion prediction and pixel correction on the image after dynamic optical compensation by using a pre-trained light offset model, and outputting the image; the system comprises a retina characteristic perception and adaptive rendering module, a dynamic optical compensation module and a dispersion prediction and pixel correction module. A medium is also implemented based on the method. According to the method, the subjective definition and the sense of reality of a picture are remarkably improved, visual fatigue caused by rendering delay or global high-load processing can be reduced, the overall color fidelity and visual coherence of display are improved, and the display potential of hardware is mined to the maximum extent from the system level.

Description

Technical Field

[0001] This invention relates to the technical field of general image data processing or generation, and in particular to a breakthrough method, system and medium for VR display PPD based on multi-level perception optimization. Background Technology

[0002] In VR / AR display systems, physical PPD (Pixels Per Degree) is the fundamental physical parameter determining visual sharpness, and its calculation formula is as follows:

[0003]

[0004] in, This refers to the number of physical pixels horizontally per eye. This is the horizontal field of view for a single eye.

[0005] To overcome the limitations of physical pixel density (PPD) and improve perceived clarity, existing technologies mainly attempt to address these limitations through super-resolution rendering, optical transfer function compensation, and dispersion correction techniques, but all of these have significant limitations.

[0006] Super-resolution rendering technology aims to generate high-detail images that exceed physical resolution through algorithms. Traditional methods, such as super-sampling anti-aliasing (SSAA, MSAA), render scenes at a higher resolution than the display resolution and then downsample to smooth jagged edges. In recent years, deep learning-based super-resolution technologies, such as NVIDIA DLSS and AMD FSR, have been widely used in traditional gaming. However, these technologies face severe challenges in VR applications: First, the computational overhead is enormous. VR experiences require extremely high frame rates (typically ≥90Hz) and extremely low latency, and the additional computational load from super-resolution processing severely restricts the system's real-time performance. Second, there is insufficient temporal stability. Existing algorithms perform well in static scenes, but in dynamic scenes with continuous changes in perspective caused by head movements, they are prone to causing temporal artifacts such as screen flickering and ghosting. Third, there is a lack of perceptual adaptation: the non-uniform characteristics of human retinal perception are not fully considered, and a uniform processing strategy is used for the visually acute foveal region and the visually coarser peripheral regions, resulting in inefficient allocation of computational resources.

[0007] The modulation transfer function (MTF) of VR optical systems (especially Fresnel and Pancake lenses) attenuates significantly at the edges of the field of view, resulting in image blurring. Existing optical transfer function (MTF) compensation techniques employ global inverse filtering, image edge enhancement algorithms, and static correction models based on fixed parameters. Global inverse filtering applies a uniform inverse filter across the entire field of view, which can improve overall sharpness, but it is prone to over-amplifying noise at the edges of the field of view due to excessively low MTF values, producing a ringing effect. Image edge enhancement algorithms, such as traditional sharpening techniques like unsharpened masking, are computationally simple but tend to amplify image noise and produce unnatural halo artifacts. Static correction models based on fixed parameters cannot adapt to changes in system optical characteristics caused by differences in user interpupillary distance, refractive power, and accommodation, resulting in inconsistent compensation effects and poor universality.

[0008] Chromatic aberration (dispersion) is an inherent problem in VR optical systems, particularly when using high-refractive-index lenses or compact optical designs. Existing dispersion correction solutions include independent RGB channel distortion correction, post-processing dispersion reduction, and optical material and design optimization. Among these, independent RGB channel distortion correction applies pre-distortion to the red, green, and blue channels separately, but it is mostly based on static optical models and is difficult to adapt to dynamic dispersion under different viewing angles and focal lengths. Post-processing dispersion reduction applies filtering at the end of the rendering pipeline to reduce color fringing, but often leads to a decrease in overall image saturation and loss of detail. Optical material and design optimization uses low-dispersion glass or apochromatic lens designs, which can reduce dispersion at the physical level, but significantly increase the manufacturing cost, weight, and size of the device.

[0009] In summary, existing technical solutions share a common systemic flaw: they are mostly isolated, static "single-point" optimizations, lacking overall coordination. Summary of the Invention

[0010] This invention solves the problems existing in the prior art and provides a breakthrough method, system and medium for VR display PPD based on multi-level perception optimization.

[0011] The technical solution adopted in this invention is a VR display PPD breakthrough method based on multi-level perception optimization. The method is based on retinal perception distribution and adaptively performs non-uniform super-resolution rendering on the input image. The scaling factor of the rendering is dynamically adjusted according to the eccentricity of the viewpoint deviating from the gaze point.

[0012] For the super-resolution rendered image, dynamic optical compensation is performed based on the eccentricity of different regions relative to the gaze point.

[0013] The image after dynamic optical compensation is output after performing dispersion prediction and pixel correction using a pre-trained ray shift model.

[0014] Preferably, the scaling factor is

[0015]

[0016] in, The maximum scaling factor for the central region of the retina. Minimum scaling factor for the retinal limb region. To perceive importance weights, satisfying,

[0017]

[0018] in, The attenuation coefficient is... denoted as eccentricity.

[0019] Preferably, super-resolution rendering also includes non-uniform resampling, using a polar coordinate sampling grid with the foveated point as the origin, and a sampling density of The angular radius of the pixel relative to the point of gaze The viewing radius increases and decreases. With eccentricity Related.

[0020] Preferably, in the dynamic optical compensation, the field of view is determined according to the eccentricity. The system is divided into a central region, a transition region, and an edge region. A full-intensity MTF inverse filter is used for the central region, while a filter based on eccentricity is used for the transition region. An inverse filter with adaptive attenuation intensity is used, and basic compensation or skipping the inverse filter processing is applied to the edge region.

[0021] Preferably, Wiener filtering is used in the transition region, and the filtering parameters are based on the eccentricity. Dynamic adjustment to meet requirements.

[0022]

[0023] in, β is the basic noise suppression factor, and β is the growth coefficient.

[0024] Preferably, with eccentricity The central region is less than 5°, with eccentricity as the criterion. An eccentricity greater than 15° is considered the edge region, while the remaining eccentricities correspond to the transition region.

[0025] Preferably, the light shift model is related to the wavelength. Relevant, satisfying

[0026]

[0027] Where (x, y) are screen coordinates and (θ, φ) are the viewing direction;

[0028] Based on the predicted offset The image is inversely shifted to achieve dispersion prediction and pixel correction.

[0029] Preferably, in pixel correction, an inter-frame consistency constraint is introduced, and an objective function based on this constraint is constructed. The objective function is related to inter-frame dispersion consistency and dispersion field gradient consistency.

[0030] A VR display PPD breakthrough system based on multi-level perception optimization, executing the VR display PPD breakthrough method based on multi-level perception optimization, the system comprising:

[0031] The retinal feature perception and adaptive rendering module is used to adaptively perform non-uniform super-resolution rendering on the input image, and the rendering scaling factor is dynamically adjusted according to the eccentricity of the viewpoint from the gaze point.

[0032] The dynamic optical compensation module is used to perform dynamic optical compensation based on the eccentricity of different regions relative to the gaze point.

[0033] The dispersion prediction and pixel correction module is used to perform dispersion prediction and pixel correction on the dynamically optically compensated image before outputting it.

[0034] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned VR display PPD breakthrough method based on multi-level perception optimization.

[0035] This invention relates to a VR display PPD breakthrough method, system, and medium based on multi-level perception optimization. Based on retinal perception distribution, the input image is adaptively rendered using non-uniform super-resolution rendering, with the rendering scaling factor dynamically adjusted according to the eccentricity of the viewing angle relative to the gaze point. For the super-resolution rendered image, dynamic optical compensation is performed based on the eccentricity of different regions relative to the gaze point. A pre-trained ray shift model is used to perform dispersion prediction and pixel correction on the dynamically optically compensated image before output. The system includes a retinal characteristic perception and adaptive rendering module, a dynamic optical compensation module, and a dispersion prediction and pixel correction module. A medium based on the method is also implemented.

[0036] The beneficial effects of this invention are as follows:

[0037] (1) By deeply integrating the non-uniform perception characteristics of the human eye retina into image generation, the limited rendering and pixel resources are non-uniformly allocated according to visual importance. Combined with gaze tracking, it provides a detail performance far exceeding the physical PPD in the area where the user's subjective perception is most sensitive, significantly improving the subjective clarity and realism of the picture.

[0038] (2) Through retinal perception adaptive super-resolution rendering, high-intensity calculations are performed only in the core area of ​​visual attention, while more efficient processing methods are used in the peripheral field of vision. While ensuring the quality of the central vision, the overall rendering computational complexity and power consumption are greatly reduced, providing a feasible path for mobile VR / AR devices to achieve high-quality display and helping to reduce visual fatigue caused by rendering delays or global high-load processing.

[0039] (3) Based on gaze-guided regionalized MTF inverse filtering, the central visual imaging quality is improved efficiently while avoiding unnecessary processing overhead and potential side effects;

[0040] (4) Use AI dispersion prediction model to accurately predict the spatial offset trajectory of light of different wavelengths and pre-correct it. Introduce inter-frame dispersion consistency constraints to effectively eliminate inter-frame flicker or jitter that may occur during the correction process, ensure the stability and clarity of color edges in dynamic visual scenes, and improve the overall color fidelity and visual coherence of the display.

[0041] (5) Closed-loop optimization enables super-resolution, aberration compensation and dispersion correction strategies to be deeply adapted to the current visual attention state of the user, maximizing the display potential of existing hardware from the system level. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0045] This invention relates to a breakthrough method for VR display PPD based on multi-level perception optimization, the method comprising the following steps:

[0046] (1) Based on the retinal perception distribution, the input image is adaptively rendered in a non-uniform super-resolution manner, and the rendering scaling factor is dynamically adjusted according to the eccentricity of the viewpoint away from the gaze point.

[0047] (2) For the super-resolution rendered image, dynamic optical compensation is performed based on the eccentricity of different regions relative to the gaze point.

[0048] (3) The image after dynamic optical compensation is output after performing dispersion prediction and pixel correction using a pre-trained ray shift model.

[0049] The method is explained below with specific steps.

[0050] (1) Based on the retinal perception distribution, the input image is adaptively rendered in a non-uniform super-resolution manner, and the rendering scaling factor is dynamically adjusted according to the eccentricity of the viewpoint away from the gaze point.

[0051] Unlike existing technologies that use uniform processing for super-resolution, this invention introduces a non-uniform processing model based on retinal cell distribution; and defines retinal eccentricity. The angle at which the viewpoint deviates from the point of fixation;

[0052] The (super-resolution) scaling factor is:

[0053]

[0054] in, The maximum scaling factor for the central region of the retina. Minimum scaling factor for the retinal limb region. To perceive importance weights, satisfying,

[0055]

[0056] in, The attenuation coefficient is... Eccentricity;

[0057] In practice, , , .

[0058] Super-resolution rendering also includes non-uniform resampling, using a polar coordinate sampling grid with the viewpoint as the origin, and a sampling density of... The angular radius of the pixel relative to the point of gaze The viewing radius increases and decreases. With eccentricity Related.

[0059] In this invention, non-uniform resampling is based on retinal characteristics.

[0060]

[0061] in, Sampling density in the central region This is the attenuation factor.

[0062] (2) For the super-resolution rendered image, dynamic optical compensation is performed based on the eccentricity of different regions relative to the gaze point.

[0063] In the aforementioned dynamic optical compensation, the field of view is determined based on eccentricity. The system is divided into a central region, a transition region, and an edge region. A full-intensity MTF inverse filter is used for the central region, while a filter based on eccentricity is used for the transition region. An inverse filter with adaptive attenuation intensity is used, and basic compensation or skipping the inverse filter processing is applied to the edge region.

[0064] Wiener filtering is used in the transition region, and its filtering parameters are based on the eccentricity. Dynamic adjustment to meet requirements.

[0065]

[0066] in, β is the basic noise suppression factor, and β is the growth coefficient.

[0067] With eccentricity The central region is less than 5°, with eccentricity as the criterion. An eccentricity greater than 15° is considered the edge region, while the remaining eccentricities correspond to the transition region.

[0068] (3) The image after dynamic optical compensation is output after performing dispersion prediction and pixel correction using a pre-trained ray shift model.

[0069] The light deflection model and wavelength Relevant, satisfying

[0070]

[0071] Where (x, y) are screen coordinates and (θ, φ) are the viewing direction;

[0072] Based on the predicted offset The image is inversely shifted to achieve dispersion prediction and pixel correction.

[0073] In pixel correction, an inter-frame consistency constraint is introduced, and an objective function based on this constraint is constructed. This objective function is related to inter-frame dispersion consistency and dispersion field gradient consistency.

[0074]

[0075] in, This is the dispersion correction result for the current frame. This is the result of the previous frame. For a dispersive field, This is the adjustment coefficient;

[0076] Minimize the objective function E.

[0077] A VR display PPD breakthrough system based on multi-level perception optimization, executing the VR display PPD breakthrough method based on multi-level perception optimization, the system comprising:

[0078] The retinal feature perception and adaptive rendering module is used to adaptively perform non-uniform super-resolution rendering on the input image, and the rendering scaling factor is dynamically adjusted according to the eccentricity of the viewpoint from the gaze point.

[0079] The dynamic optical compensation module is used to perform dynamic optical compensation based on the eccentricity of different regions relative to the gaze point.

[0080] The dispersion prediction and pixel correction module is used to perform dispersion prediction and pixel correction on the dynamically optically compensated image before outputting it.

[0081] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned VR display PPD breakthrough method based on multi-level perception optimization.

[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0087] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A breakthrough method for VR display PPD based on multi-level perception optimization, characterized in that: The method is based on retinal perception distribution and adaptively performs non-uniform super-resolution rendering on the input image. The scaling factor of the rendering is dynamically adjusted according to the eccentricity of the viewpoint from the fixation point. For the super-resolution rendered image, dynamic optical compensation is performed based on the eccentricity of different regions relative to the gaze point. The image after dynamic optical compensation is output after performing dispersion prediction and pixel correction using a pre-trained ray shift model.

2. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 1, characterized in that: The scaling factor is ， in, The maximum scaling factor for the central region of the retina. Minimum scaling factor for the retinal limb region. To perceive importance weights, satisfying, ， in, The attenuation coefficient is... denoted as eccentricity.

3. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 2, characterized in that: Super-resolution rendering also includes non-uniform resampling, using a polar coordinate sampling grid with the viewpoint as the origin, and a sampling density of... The angular radius of the pixel relative to the point of gaze The viewing radius increases and decreases. With eccentricity Related.

4. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 1, characterized in that: In the aforementioned dynamic optical compensation, the field of view is determined based on eccentricity. The system is divided into a central region, a transition region, and an edge region. A full-intensity MTF inverse filter is used for the central region, while a filter based on eccentricity is used for the transition region. An inverse filter with adaptive attenuation intensity is used, and basic compensation or skipping the inverse filter processing is applied to the edge region.

5. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 4, characterized in that: Wiener filtering is used in the transition region, and its filtering parameters are based on the eccentricity. Dynamic adjustment to meet requirements. ， in, β is the basic noise suppression factor, and β is the growth coefficient.

6. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 4, characterized in that: With eccentricity The central region is less than 5°, with eccentricity as the criterion. An eccentricity greater than 15° is considered the edge region, while the remaining eccentricities correspond to the transition region.

7. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 1, characterized in that: The light deflection model and wavelength Relevant, satisfying ， Where (x, y) are screen coordinates and (θ, φ) are the viewing direction; Based on the predicted offset The image is inversely shifted to achieve dispersion prediction and pixel correction.

8. The VR display PPD breakthrough method based on multi-level perception optimization according to claim 1, characterized in that: In pixel correction, an inter-frame consistency constraint is introduced, and an objective function based on this constraint is constructed. The objective function is related to inter-frame dispersion consistency and dispersion field gradient consistency.

9. A VR display PPD breakthrough system based on multi-level perception optimization, characterized in that: The VR display PPD breakthrough method based on multi-level perception optimization as described in any one of claims 1 to 8, wherein the system comprises: The retinal feature perception and adaptive rendering module is used to adaptively perform non-uniform super-resolution rendering on the input image, and the rendering scaling factor is dynamically adjusted according to the eccentricity of the viewpoint from the gaze point. The dynamic optical compensation module is used to perform dynamic optical compensation based on the eccentricity of different regions relative to the gaze point. The dispersion prediction and pixel correction module is used to perform dispersion prediction and pixel correction on the dynamically optically compensated image before outputting it.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the VR display PPD breakthrough method based on multi-level perception optimization as described in any one of claims 1 to 8.

Images