A multi-dimensional perception foveated rendering method, system, and medium

Abstract

This invention discloses a multi-dimensional perceptual gaze-point rendering method, system, and medium. The multi-dimensional perceptual gaze-point rendering method is characterized by the following steps: S1, acquiring the user's eye-tracking data in real-time within a virtual reality environment; S2, dividing the rendering sequence into saccade windows, recovery windows, and stable gaze windows; S3, constructing a multi-dimensional visual perception model that integrates saccade sequence features; S4, acquiring the perception critical frequency and mapping it to a resolution scaling factor to guide rasterization operations in the graphics pipeline; S5, invoking the resolution scaling factor and performing dynamic resolution rendering in the graphics rendering pipeline based on the resolution scaling factor. The system includes a data acquisition module, an event detection module, a model calculation module, a rendering scheduling module, and a graphics drawing module. This invention is rationally designed, effectively utilizing computing resources, improving the rendering system's frame rate, and enhancing the user experience.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and virtual reality technology, specifically to a multidimensional perceptual gaze-point rendering method, system, and medium. Background Technology

[0002] As is well known, foveated rendering is a key technology for solving the high-load rendering problem in VR. Its core idea is to leverage the imaging characteristics of the human retina, using eye-tracking technology to acquire the user's gaze point in real time, performing high-resolution rendering only in the foveal region where the gaze point is located, while significantly reducing the rendering resolution in the less visually sensitive peripheral regions. Existing mainstream methods typically rely on static contrast sensitivity function models, establishing a mapping relationship between spatial frequency and retinal eccentricity to guide the resolution descent gradient, which mainly suffers from the following problems: Current foveated rendering techniques suffer from insufficient utilization of temporal awareness during saccades, leading to redundant computational resources. The core assumption of existing technologies is that the foveal region consistently requires the highest quality rendering, an assumption that ignores the inherent time-varying characteristics of the visual system. Physiological evidence shows that the human eye experiences significant inhibition during saccades, with visual sensitivity dropping sharply. Furthermore, the visual sensitivity of the foveal region does not instantly recover to its peak after the saccade ends, but rather undergoes a gradual recovery process lasting hundreds of milliseconds due to microsaccades and eye drift. However, mainstream rendering pipelines often ignore this dynamic physiological period, maintaining a constant high-specification rendering for the foveal region. This ineffective high-quality rendering during the low point of visual sensitivity is essentially a waste of computational resources, limiting further improvements in system frame rate and energy efficiency. Summary of the Invention

[0003] This invention discloses a multi-dimensional foveated rendering method, system, and medium. It solves the technical problems of existing foveated rendering techniques, such as high computational resource consumption, reduced system frame rate, and negatively impacted user experience. The invention features a reasonable design, efficient use of computational resources, and improved rendering system frame rate and user experience. The adopted technical solution is as follows: A multidimensional perceptual foveated rendering method, including the following steps S1. Acquire the user's eye-tracking data in real time in a virtual reality environment, wherein the eye-tracking data includes the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point; S2. Call and perform real-time temporal analysis on the eye-tracking data, use a dual threshold detection algorithm based on speed and position to identify eye-tracking events, and synchronously divide the rendering time sequence into saccade windows, recovery windows and stable gaze windows based on the occurrence sequence, duration and state transition relationship of the eye-tracking events; the rendering time sequence is synchronized with the frame clock of the graphics rendering pipeline, and its time nodes are determined in real time by the eye-tracking events. S3. Obtain scene feature parameters of the current rendering frame. The eye movement events include saccade events, saccade recovery events, and stable fixation events. The eye movement state is the instantaneous state corresponding to the eye movement event at any time, including saccade state, saccade recovery state, and stable fixation state. Construct a multi-dimensional visual perception model that integrates saccade temporal features. The model takes the current eye movement state and the elapsed time after saccade as dynamic inputs, and combines the scene's brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters in the scene feature parameters to output the contrast sensitivity threshold of the fovea region of the human eye at the current time. The current rendering frame is a single frame image in the graphics rendering pipeline that is currently undergoing rasterization. It is frame-synchronized with the rendering timing in step S2, and its frame period corresponds one-to-one with the time nodes of the rendering timing. S4: Call the contrast sensitivity threshold to obtain the highest spatial frequency that meets the visual lossless condition, i.e. the perception critical frequency, and map the perception critical frequency as a resolution scaling factor to guide the rasterization operation of the graphics pipeline. S5: Invoke the resolution scaling factor and perform dynamic resolution rendering in the graphics rendering pipeline based on the resolution scaling factor; in the dynamic resolution rendering, full-screen downsampling is performed in the panning window, progressive resolution upsampling is performed over time in the recovery window, and standard foveated rendering is performed in the stable foveated window.

[0004] Based on the above technical solution, step S2, which involves dividing the rendering sequence into the scanning window, the recovery window, and the stable gaze window, includes: S21. Preset angular velocity threshold and fixation point deviation range threshold, used to distinguish between saccade events and non-saccade events. S22. Sagging start determination and saccade window activation: The instantaneous angular velocity of the eyeball and the screen coordinates of the fixation point obtained in step S1 are called in real time. When the instantaneous angular velocity of the eyeball is higher than the angular velocity threshold for multiple consecutive frames, and the offset of the screen coordinates of the fixation point relative to the initial screen coordinates of the fixation point at the start of the saccade exceeds the fixation point deviation range threshold, the saccade is determined to have started, and the saccade window described in step S2 is called in sync. S23. Sagling Landing Judgment and Recovery Window Activation: Continuously monitor the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point. When the instantaneous angular velocity of the eyeball falls below the angular velocity threshold and the screen coordinates of the gaze point remain within the preset range of the target gaze area for multiple consecutive frames, it is determined to be a saccade landing. The saccade window is closed and the recovery window described in step S2 is activated. At the same time, the saccade landing time is recorded as the starting point for the elapsed time after the saccade described in step S3. S24. Restore Window Termination and Stabilized Gaze Window Startup: During the restore window period, the elapsed time after saccade as described in step S3 and the resolution scaling factor as described in step S4 are monitored in real time. When the resolution scaling factor is restored to the preset reference value, the restore window is closed and the stabilized gaze window as described in step S2 is started.

[0005] Based on the above technical solution, the multidimensional visual perception model that integrates saccade timing features also incorporates a saccade timing modulation module, which outputs a saccade timing modulation factor; the value of the saccade timing modulation factor is designed as follows: If the current state is within the saccade window defined in step S2, the saccade timing modulation factor is set to a preset saccade suppression constant, which is used to characterize the extreme attenuation of human visual sensitivity under saccade conditions. If the current state is within the recovery window defined in step S2, the saccade timing modulation factor is a function of the elapsed time after saccade as described in step S3. This function exhibits a non-linear growth trend and is used to characterize the process by which visual sensitivity gradually recovers to a normal level over time after saccade. If the current state is within the stable gaze window defined in step S2, the saccade timing modulation factor is set to 1.0, indicating that the visual sensitivity is at a stable peak level. The contrast sensitivity threshold output by the multidimensional visual perception model that integrates saccade temporal features is obtained by multiplying the baseline sensitivity calculated from the static multidimensional visual parameters with the saccade temporal modulation factor. The static multidimensional visual parameters correspond to the scene brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters in step S3.

[0006] Based on the above technical solution, within the recovery window, the saccade timing modulation factor and the elapsed time after saccade follow a power function recovery law, and the recovery speed gradually slows down over time. The power function expression is:

[0007] t represents the time after scanning the landing, in seconds; SF(t) represents the highest resolvable spatial frequency at time t, in cpd; k1, k2, and k3 are dynamic parameters, k1 = 1.9471~1.9483, k2 = 0.3470~0.3477, and k3 = 9.5065~9.5088.

[0008] Based on the above technical solution, the sensing critical frequency is expressed as the product of the static cutoff frequency reference and the saccade timing modulation factor. The static cutoff frequency reference It is designed to be the highest spatial frequency that the human eye can reach during the stable gaze period formed by the existence of the stable gaze window; The saccade timing modulation factor The output factor is the saccade timing modulation module built into the multidimensional visual perception model that integrates saccade timing features; the perception critical frequency. : When the current eye movement state s=SACCADING, the eye movement event is a saccade event, and the saccade window persists for a period of time. (t,s)= min; When the current eye movement state s = POST - SACCADIC, the eye movement event is a saccade recovery event. During the duration of the recovery window, 0 ≤ t < hour, (t,s)= ·( ) / max, This is the duration of the restore window; When the current eye movement state s=FIXATING, the eye movement event is a stable fixation event, and the stable fixation window lasts for a period of time. (t,s)= ; Based on the above technical solution, in step S4, the resolution scaling factor k(t) is calculated using the following mapping function:

[0009] η∈(0,1] is a safety factor used to compensate for model fitting residuals and inter-individual physiological differences. The upper limit of the frequency corresponding to the spatial sampling rate during full-resolution rendering.

[0010] Based on the above technical solution, the dynamic resolution rendering step in step S5 includes: During the saccade window, the resolution scaling factor of the entire viewport is set to the lowest level, and the Kalman filter algorithm is used to predict the saccade landing point based on the current eye movement trajectory, and the area around the predicted saccade landing point is pre-rendered. During the recovery window, a time-dimensional low-pass filter is applied to the calculated resolution scaling factor to smooth the resolution improvement trajectory over time and avoid visual flicker caused by sudden changes in resolution between frames. At the same time, the coverage radius of the central high-resolution region dynamically expands as the perception critical frequency increases. During the stable gaze window, the resolution scaling factor is kept constant at 1.0, and the surrounding area is downsampled only based on spatial eccentricity.

[0011] Based on the above technical solution, the overall contrast sensitivity of the multidimensional visual perception model that integrates saccade temporal features is improved. The output expression is:

[0012] in, This represents the overall contrast sensitivity scalar value. These represent the response increments of L, M, and S type cone cells at the sensitivity threshold, respectively. These are the baseline response values ​​for L, M, and S type cone cells under the corresponding background light intensities.

[0013] A multidimensional foveated rendering system, the system comprising: The data acquisition module is used to acquire the user's eye movement signals through an eye-tracking device. The eye movement signals include at least the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point. The event detection module, connected to the data acquisition module, is used to identify three types of eye movement states in real time: saccade events, saccade recovery events, and stable gaze events, based on the dual threshold judgment logic of eye movement signals. It synchronously outputs the current eye movement state and the elapsed time after saccade, and divides the saccade window, recovery window, and stable gaze window in the rendering sequence based on the timing and duration of the eye movement events. The model calculation module has a built-in multi-dimensional visual perception model that integrates saccade timing features. It is connected to the event detection module and is used to calculate the contrast sensitivity threshold of the human eye in real time based on scene environment parameters, current eye movement state and elapsed time after saccade, and to calculate the perception critical frequency. The rendering scheduling module, connected to the model calculation module, is used to convert the perception critical frequency into a resolution scaling instruction and generate a dynamic multi-resolution rendering configuration that is synchronized with the rendering timing and eye-tracking state. The graphics rendering module, connected to the rendering scheduling module, is used to receive rendering configurations and perform image rendering with differentiated sampling rates for different rendering windows in the graphics rendering pipeline.

[0014] A storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it can implement the multidimensional perceptual gaze point rendering method as described above.

[0015] Beneficial effects This invention is rationally designed, employing a real-time detection algorithm based on dual thresholds of eye angular velocity and gaze point screen coordinate offset. It performs high-frequency sampling and analysis of the user's eye movement behavior. Based on the sampling and analysis process, the user's visual experience is divided into a saccade window, a recovery window, and a stable gaze window. Within the saccade window, the visual system is in a state of depth suppression, providing maximum theoretical redundancy for rendering optimization. Within the recovery window, the sensitivity of the visual system dynamically increases over time, requiring the rendering quality to recover synchronously and progressively. In other words, this application overcomes the limitations of traditional methods that only downsample spatially, fully utilizing the physiological characteristics of the human eye during saccades and recovery. By significantly reducing the shading resolution during periods of low visual sensitivity, it effectively reduces the computational load on invalid pixels, while significantly reducing frame rendering time in dynamic browsing tasks, greatly increasing the frame rate, and reducing device power consumption.

[0016] Thus, by constructing a spatiotemporal multidimensional perception model that conforms to the physiological characteristics of the human eye and an adaptive dynamic resolution scheduling mechanism, the bottleneck of traditional static foveated rendering being unable to fully utilize temporal perception redundancy is broken. This provides a high-efficiency solution for high-load virtual reality applications that significantly reduces computing resource consumption and improves rendering frame rate without sacrificing the user's visual experience.

[0017] In this invention, a multi-dimensional visual perception model integrating saccade temporal features is constructed. This model not only integrates multiple parameters such as the average brightness of the current scene, color channels, visual stimulus area, retinal eccentricity, and spatial frequency, but also introduces a saccade temporal modulation factor that changes with the elapsed time after saccade. This simulates the extreme inhibition state of the human eye during the saccade window and the gradual recovery process following a power function during the recovery window. Based on this, the model can output in real time the visual contrast sensitivity threshold of the human eye to different spatial frequency content at any time point, eye movement state, and lighting conditions, thereby accurately determining the highest cutoff frequency that the human eye can perceive, i.e., the perception critical frequency. This ensures visual perception quality by ensuring that all downsampling operations are below the human eye's perception threshold, achieving lossless rendering at the perception level.

[0018] In this invention, based on the perception critical frequency calculated by the model, the system dynamically calculates the resolution scaling factor of the current frame through a mapping function, driving the rendering pipeline to perform differentiated scheduling. Specifically, during the saccade window, the full viewport resolution is reduced to the lowest threshold by utilizing the characteristics of visual blind spots, and a Kalman filter algorithm is introduced to predict the eye's landing point to perform region pre-rendering, thereby eliminating the visual lag caused by system latency. During the recovery window, a progressive resolution recovery strategy is executed, making the rendering resolution strictly follow the recovery curve of visual sensitivity for non-linear improvement. At the same time, low-pass filtering in the time dimension is used to smooth the image and prevent screen flicker, and the coverage radius of the central foveal high-resolution region dynamically expands as sensitivity increases until a stable state is reached. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For those skilled in the art, other embodiments can be derived from the provided drawings without creative effort.

[0020] Figure 1 : Overall architecture diagram of the multi-dimensional perceptual gaze point rendering system in this invention; Figure 2 : Framework diagram of the multidimensional visual perception model integrating saccade temporal features in this invention Figure 3 : Flowchart of the single-trial time sequence of the eye-tracking psychophysical experiment used to calibrate model parameters in this invention; Figure 4 : Schematic diagram of the visual sensitivity recovery curve fitting results after saccade in this invention. Detailed Implementation

[0021] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims, as well as all available equivalents of the claims.

[0022] like Figure 1 and 2 The multidimensional perceptual gaze point rendering method shown includes the following steps: S1. In a virtual reality environment, eye-tracking data of the user is collected in real time through an eye-tracking device. The eye-tracking device is existing technology and will not be described in detail. The eye-tracking data includes at least the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point. The eye-tracking data is used to provide basic data support for the recognition of eye movement events and the determination of eye movement state in step S2. The acquisition frequency is kept in sync with the frame synchronization clock of the graphics rendering pipeline to ensure that the data acquisition timing is synchronized with the rendering timing described in step S2, so as to meet the requirements of real-time detection and rendering scheduling.

[0023] S2. The eye-tracking data is invoked and analyzed in real-time. An eye-tracking event detection algorithm based on both velocity and position is used to identify eye-tracking events. Based on the occurrence sequence, duration, and state transition relationships of these events, the corresponding saccade window, recovery window, and stable gaze window are synchronously divided within the rendering timeline. The rendering timeline is synchronized with the frame clock of the graphics rendering pipeline, and its time nodes are determined in real-time by the eye-tracking events. Specifically: S21. Preset angular velocity threshold and fixation point deviation range threshold, used to distinguish between saccade events and non-saccade events. S22. Sagling Start Determination and Sagling Window Activation: The real-time monitoring data of instantaneous eye angular velocity and fixation point screen coordinates obtained in step S1 are invoked. When the instantaneous eye angular velocity is higher than the angular velocity threshold for multiple consecutive frames, and the offset of the fixation point screen coordinates relative to the initial fixation point screen coordinates at the start of the saccade exceeds the fixation point deviation range threshold, a saccade is determined to have started, and a precise start timestamp is recorded. The scanning window described in step S2 is invoked synchronously; S23. The saccade landing determination and recovery window is activated, continuously monitoring the instantaneous angular velocity of the eye and the screen coordinates of the gaze point. When the instantaneous angular velocity of the eye falls below the angular velocity threshold, and the screen coordinates of the gaze point remain within a preset range of the target gaze area for multiple consecutive frames, it is determined to be a saccade landing, and the landing timestamp is recorded. Close the scanning window and start the recovery window described in step S2, while using the landing timestamp. , which serves as the starting point for timing the elapsed time t after the scan described in step S3; S24. The recovery window terminates and the stable gaze window begins. During the recovery window, high-precision timing is continuously maintained, and the landing timestamp is continuously output. The elapsed time t is calculated from the start, and the elapsed time t after the saccade described in step S3 and the resolution scaling factor described in step S4 are monitored in real time. When the resolution scaling factor recovers to the preset reference value, the recovery window is closed and the stable gaze window described in step S2 is started; or when the elapsed time t exceeds the preset maximum physiological recovery time, the recovery window is closed and the stable gaze window described in step S2 is started. S3. Obtain the scene feature parameters of the current rendering frame. In this embodiment, the scene feature parameters of the current rendering frame include the scene's brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters. That is, by introducing multi-dimensional scene feature parameters, the limitations of relying solely on spatial eccentricity for static prediction in traditional methods are overcome. This allows for accurate adaptation to dynamically changing visual scenes in virtual reality environments, providing a precise theoretical basis for perception optimization in dynamic visual environments.

[0024] The eye movement events include saccade events, saccade recovery events, and stable fixation events. The eye movement state is the instantaneous state corresponding to the eye movement event at any given time, including saccade state, saccade recovery state, and stable fixation state, which correspond one-to-one with the aforementioned three types of eye movement events. That is, when the eye movement event is a saccade event, the corresponding eye movement state is a saccade state; when the eye movement event is a saccade recovery event, the corresponding eye movement state is a saccade recovery state; and when the eye movement event is a stable fixation event, the corresponding eye movement state is a stable fixation state. The eye movement state is synchronized in real time with the update of the rendering sequence to ensure consistency with the current eye movement behavior.

[0025] A multidimensional visual perception model integrating saccade temporal features is constructed (hereinafter referred to as the model). The model also incorporates a saccade timing modulation module to achieve collaborative perception of eye movement timing features and multi-dimensional scene features; this saccade timing modulation module is used to output saccade timing modulation factors. ;

[0026] The eye-tracking state variable describes the current eye-tracking state; t represents the time after saccade landing, in seconds; It is the inhibition intensity during saccades, and its value is usually between 0.2 and 0.4; This refers to the recovery window duration. That is, the value of this saccade timing modulation factor is designed as follows: If the current window is the scanning window defined in step S2, The saccadic timing modulation factor is a preset saccadic suppression constant, which is used to characterize the extreme attenuation of human visual sensitivity under saccadic conditions. If we are currently in the recovery window defined in step S2, The saccade timing modulation factor is a function of the elapsed time after saccade in step S3. This function has a non-linear growth trend and is used to characterize the process by which visual sensitivity gradually recovers to a normal level over time after saccade. If we are currently in the stable gaze window defined in step S2, The saccade timing modulation factor is set to 1.0, which indicates that the visual sensitivity is at a stable peak level. The contrast sensitivity threshold output by the multidimensional visual perception model that integrates saccade temporal features is obtained by multiplying the baseline sensitivity calculated from the static multidimensional visual parameters with the saccade temporal modulation factor. The static multidimensional visual parameters correspond to the scene brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters in step S3.

[0027] Furthermore, within the recovery window, the saccade timing modulation factor and the elapsed time after saccade follow a power function recovery law, with the recovery speed gradually decreasing over time. The power function expression is as follows:

[0028] t represents the time after scanning the landing, in seconds; SF(t) represents the highest resolvable spatial frequency at time t, in cpd; k1, k2, and k3 are dynamic parameters, k1 = 1.9471~1.9483, k2 = 0.3470~0.3477, and k3 = 9.5065~9.5088.

[0029] In practice, the multidimensional visual perception model that integrates saccade timing features takes the current eye movement state and the elapsed time t after saccade as dynamic inputs. The elapsed time t after saccade is the duration calculated from the saccade landing time recorded in step S2, and is updated synchronously with the rendering timing. Simultaneously, by combining the scene feature parameters of the current rendering frame, such as the brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters of the scene mentioned above, the built-in perception calculation logic calculates and outputs the contrast sensitivity threshold of the fovea region of the human eye at the current moment in real time. This contrast sensitivity threshold is directly used as the core basic parameter for calculating the perception critical frequency in step S4, ensuring the calculation accuracy and dynamic adaptability of the perception critical frequency.

[0030] The current rendering frame is a single frame image in the graphics rendering pipeline that is performing rasterization operations. It is frame synchronized with the rendering timing in step S2, and its frame period corresponds one-to-one with the time node of the rendering timing. S4. Call the contrast sensitivity threshold, combine it with the multi-dimensional visual perception model that integrates saccade timing features, and obtain the highest spatial frequency that meets the visual lossless condition, i.e., the perception critical frequency. Through a preset mapping function, map the perception critical frequency into a resolution scaling factor that guides the rasterization operation of the graphics pipeline. The resolution scaling factor keeps frame synchronization with the aforementioned rendering timing and rendering frames, and corresponds one-to-one with the divided saccade window, recovery window and stable gaze window. Furthermore, the value of the resolution scaling factor is updated in real time according to the eye movement state and the elapsed time t after saccade, providing accurate and real-time parameter basis for the dynamic resolution rendering in step 5, and ensuring that the rendering strategy, eye movement perception characteristics, and rendering rhythm are coordinated and consistent.

[0031] In this embodiment, the sensing critical frequency is expressed as the product of the static cutoff frequency reference and the scanning timing modulation factor. Static cutoff frequency reference It is designed to be the highest spatial frequency that the human eye can reach during the stable fixation period formed by the existence of a stable fixation window; Sagling time modulation factor , is the output factor of the saccade temporal modulation module built into the multidimensional visual perception model that integrates saccade temporal features; As mentioned above, the saccade timing modulation factor and the elapsed time after saccade follow a power function recovery law, and the sensing critical frequency... : When the current eye movement state s=SACCADING, the eye movement event is a saccade event, and the saccade window lasts for a period of time. (t,s)= min; When the current eye movement state s = POST - SACCADIC, the eye movement event is a saccade recovery event, which occurs during the recovery window and 0 ≤ t < t. hour, (t,s)= ·( ) / max; When the current eye movement state s=FIXATING, the eye movement event is a stable fixation event. During the duration of the stable fixation window... (t,s)= ; For example, min This represents the lower limit of visual sensitivity during the saccade inhibition period. The stable value of visual sensitivity after the visual system has fully recovered; To restore window duration; These are the fitting parameters.

[0032] Furthermore, in obtaining the critical frequency of perception Then, using a preset mapping function, this perceptual critical frequency is mapped to a resolution scaling factor that guides the rasterization operation of the graphics pipeline, specifically: Determining the rendering resolution requires considering both the limits of human visual perception and the physical limitations of the display device. Let the spatial sampling rate of the target rendered image in the concave region be... According to the sampling theorem, to avoid wasting computational resources on frequencies that are imperceptible to the visual sense due to undersampling and introduce visible aliasing, the optimal rendering frequency is... It should meet the following requirements:

[0033] in, This is a conservative threshold set to avoid visible aliasing, typically slightly lower than [previous threshold]. Therefore, the theoretical resolution scaling factor for the central concave region... Defined as:

[0034] in, The upper limit of the frequency corresponding to the spatial sampling rate during full-resolution rendering. To simplify calculations and enhance robustness, in this invention, the resolution scaling factor k(t) guiding the rasterization operation of the graphics pipeline is calculated using the following mapping function:

[0035] η∈(0,1] is a safety factor used to compensate for model fitting residuals and physiological differences between individuals; The upper limit of the frequency corresponding to the spatial sampling rate during full-resolution rendering.

[0036] This mapping function ensures that within the scan window, due to At a lower resolution, the system can significantly reduce the rendering resolution. The value is relatively small; in the recovery window, the rendering quality recovers synchronously with the restoration of visual sensitivity in the time domain. The value increases monotonically; when the sensing capability reaches or exceeds the device's limit, the system uses full-resolution rendering. .

[0037] S5. Invoke the resolution scaling factor and perform dynamic resolution rendering in the graphics rendering pipeline based on the resolution scaling factor; wherein the rendering strategy is synchronized with the aforementioned eye-tracking state and rendering timing to achieve synergistic optimization of visual losslessness and rendering efficiency.

[0038] In dynamic resolution rendering, full-screen downsampling is performed within the saccade window, which means setting the full-screen resolution scaling factor to the lowest level. The Kalman filter algorithm is then used to predict the saccade landing point based on the current eye movement trajectory, and the area surrounding the predicted saccade landing point is pre-rendered to minimize computational resource consumption. Specifically: When an eye-movement event is determined to enter the saccade window, the saccade suppression effect almost completely blocks the foveal visual input. Therefore, the system adopts a proactive and aggressive downsampling strategy: directly setting the resolution scaling factor for the entire viewport to its minimum value. To minimize pixel color consumption, and considering that the end-to-end system latency from eye movement acquisition and signal processing to image display is typically between 20ms and 30ms, relying solely on real-time signal triggering could result in users seeing a blurry image with resolution not yet restored at the moment of saccade landing. Therefore, a Kalman filter algorithm is introduced to predict the saccade landing point based on the current eye movement trajectory. The landing time, and a preset radius centered on that point. A circular region is used as a potential central concave region for pre-rendering. To address the spatial uncertainty caused by prediction errors and system latency, the radius of this pre-rendered region is... Slightly larger than the standard central concave radius.

[0039] Within the recovery window, a gradual resolution upscaling is performed over time to adapt to the recovery process of visual brightness. Specifically, a time-dimensional low-pass filter is applied to the calculated resolution scaling factor to smooth the resolution upscaling trajectory over time and avoid visual flickering caused by sudden resolution changes between frames. Simultaneously, the coverage radius of the central foveal high-resolution region dynamically expands as the perception critical frequency increases. When gaze point is detected, with landing timestamp To enter the recovery window as an anchor point, each frame is adjusted based on the current elapsed time. Dynamically calculating the minimum resolution required at the moment allows the system to maintain a low rendering load during the recovery window. Furthermore, to prevent perceptible inter-frame flicker or detail pop-up artifacts caused by inter-frame resolution jumps, a first-order inertial smoothing filter is introduced.

[0040] in The actual scaling factor applied to the current frame, where i represents the frame number; Smoothing coefficient Based on frame time Adaptive adjustment based on the time integral characteristics of the visual system.

[0041] Based on the persistence of vision (approximately 100ms) and the display refresh cycle, this paper adopts the following empirical formula to achieve adaptive smoothing:

[0042] Furthermore, the concave region during the recovery window no longer uses a fixed radius, but is dynamically adjusted based on the sensing critical frequency. During the recovery window, the radius of the concave region... Compared to the highest resolvable spatial frequency Inversely proportional. Let the viewing angle corresponding to the central concave region at full resolution be... Then the dynamic radius can be approximated as:

[0043] According to the current The minimum acute angle of view that the human eye can distinguish at a given contrast can be calculated, and the radius of the fovea region can then be dynamically adjusted. This ensures that it always covers the range where visual sensitivity is above the set threshold.

[0044] Standard foveated rendering is performed within a stable foveated window, with a constant resolution scaling factor of 1.0 to ensure visual clarity in the foveal region while also considering rendering efficiency in the surrounding areas. Specifically: The current eye movement event is identified as entering a stable fixation window. To extend temporal optimization to the full-screen viewport, the system constructs a dynamic multi-resolution field. In each frame, the entire viewport is divided into multiple concentric rings, and the resolution scaling factor for each region is determined by the eccentricity of its center point. This is determined in conjunction with the current eye movement state s. Specifically, a resolution mapping function is defined. :

[0045] in, The minimum background resolution allowed by the system; The normalized static eccentricity sensitivity decay function is the standard foveation rendering curve.

[0046] This mapping function indicates the resolution scaling factor. Based on modulation, the sensitivity attenuation effect caused by spatial eccentricity is further superimposed. Therefore, in the early stage of restoration, not only is the resolution of the central concave region reduced, but its surrounding areas are also reduced proportionally, forming a rendering field with a smooth resolution gradient centered on the foveation point.

[0047] Furthermore, the overall contrast sensitivity of this multidimensional visual perception model that integrates saccade temporal features is... The output expression is:

[0048] in, This represents the overall contrast sensitivity scalar value. These represent the response increments of L, M, and S type cone cells at the sensitivity threshold, respectively. These represent the baseline response values ​​of L, M, and S-type cone cells under corresponding background light intensities. The overall contrast sensitivity is also included. The scalar value is used to verify the physiological rationality of the contrast sensitivity threshold output in step S3, ensuring that the output contrast sensitivity threshold result conforms to the actual perceptual characteristics of the human visual system.

[0049] A multidimensional foveated rendering system, the system comprising: The data acquisition module is used to acquire the user's eye movement signals through an eye-tracking device. The eye movement signals include at least the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point. The event detection module, connected to the data acquisition module, is used to identify three types of eye movement states in real time: saccade events, saccade recovery events, and stable gaze events, based on the dual threshold judgment logic of eye movement signals. It synchronously outputs the current eye movement state and the elapsed time after saccade, and divides the saccade window, recovery window, and stable gaze window in the rendering sequence based on the timing and duration of the eye movement events. The model calculation module has a built-in multi-dimensional visual perception model that integrates saccade timing features. It is connected to the event detection module and is used to calculate the contrast sensitivity threshold of the human eye in real time based on scene environment parameters, current eye movement state and elapsed time after saccade, and to calculate the perception critical frequency. The rendering scheduling module, connected to the model calculation module, is used to convert the perception critical frequency into a resolution scaling instruction and generate a dynamic multi-resolution rendering configuration that is synchronized with the rendering timing and eye-tracking state. The graphics rendering module, connected to the rendering scheduling module, is used to receive rendering configurations and perform image rendering with differentiated sampling rates for different rendering windows in the graphics rendering pipeline.

[0050] A storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it can implement the multidimensional perceptual gaze point rendering method as described above.

[0051] To verify the feasibility and effectiveness of the multi-dimensional perceptual gaze point rendering method and system that integrates saccade temporal features provided by this invention, this embodiment comprehensively verifies the model rationality, perceptual accuracy, and rendering performance of the method through systematic psychophysical experiments and rendering performance tests. The specific experimental process, test conditions, and verification results are as follows: I. Experimental Testing Environment Description The hardware and software environment used in this experiment is described in detail below: 1. Hardware environment: VR immersive roaming client (resolution 3840×2160, refresh rate 90Hz), built-in high-frequency eye-tracking device (sampling frequency 120Hz, gaze tracking error ≤0.5°); test host configuration GPU (NVIDIA RTX 4090), CPU (Intel Core i9-13900K), memory (32GB DDR5) to ensure no hardware bottleneck in rendering performance testing; 2. Software environment: The graphics rendering pipeline is built on the Unity engine, the eye-tracking data acquisition and analysis program is developed in C++, the model calculation module is integrated into the GPU core computing unit, and the experimental data statistics and fitting are performed using MATLAB R2023a software.

[0052] II. Model Parameter Calibration Experiment The core objective of this experiment is to calibrate the key parameters (k1, k2, k3) of the power function, verify the rationality of the power function recovery model, and provide parameter support for the accurate calculation of the multidimensional visual perception model that integrates saccade temporal features.

[0053] 1. Experimental Design: Design a single-trial eye-tracking psychophysical experiment. The single-trial procedure is as follows: Figure 3As shown; 12 subjects with normal vision (uncorrected visual acuity ≥1.0, no eye diseases) were recruited. All subjects signed informed consent forms and underwent pre-experiment training to familiarize themselves with the experimental procedures. 2. Experimental procedure: The scene parameters such as brightness (500 lux) and color contrast in the experimental environment were kept constant. Subjects were guided to complete the saccade action. The highest spatial frequency (SF, unit: cpd) that the human eye could distinguish was measured at different delay times after the saccade (0~0.5s, interval 0.05s). The test was repeated 10 times at each delay time, and the stable measurement value of the subject was taken as the single trial data. 3. Data Processing and Results: Nonlinear fitting was performed on the population median data of the 12 subjects. The fitting results are as follows: Figure 4 As shown, the expression for the power function is: Key parameters in the model: k1=1.9483, k2=0.3470, k3=9.5088, goodness of fit R²>0.87; 4. Experimental conclusions: The goodness of fit R²>0.87 indicates that the power function used can accurately fit the recovery law of visual sensitivity after saccades, verifying the rationality of the power function recovery model. The calibrated parameters can be directly applied to the calculation of the saccade temporal modulation factor of the multidimensional visual perception model that integrates saccade temporal features, ensuring the accuracy of the model input parameters.

[0054] III. Perception Model Validation Experiment This experiment is used to verify the perceptual accuracy and generalization ability of the multidimensional visual perception model that integrates saccade temporal features. By comparing it with the existing mainstream contrast sensitivity function (CSF) model, the advantages of the model of this invention are highlighted.

[0055] 1. Experimental Design: A publicly available multidimensional contrast sensitivity dataset (covering test samples of brightness, color, retinal eccentricity and spatial frequency in different scenes) was used to conduct five-fold cross-validation on the multidimensional visual perception model that integrates saccade temporal features. At the same time, existing models and other mainstream CSF models were used as controls, and tests were conducted under the same conditions on the same dataset to compare the test errors of each model. 2. Testing metrics: The contrast sensitivity test error (unit: dB) is used as the core evaluation metric. The smaller the error, the higher the perceptual accuracy of the model. At the same time, the generalization ability of the model is tested in an 8-dimensional full parameter space covering saccade temporal features (scene brightness, color, stimulus area, retinal eccentricity, spatial frequency, eye movement state, elapsed time after saccade, fixation point offset). 3. Test Results: The five-fold cross-validation results show that the average test error of the multidimensional visual perception model that integrates saccade temporal features in this embodiment is 3.37 dB, which is better than the existing model (average test error 3.64 dB) and other mainstream CSF models; in the 8-dimensional full-parameter space test, the average test error of the model is further reduced to 3.34 dB. 4. Experimental Conclusion: The test results demonstrate that the multidimensional visual perception model constructed in this invention, which integrates saccade temporal features, effectively improves the calculation accuracy of the contrast sensitivity threshold by fusing saccade temporal features with multidimensional scene features. It has good generalization ability, can adapt to dynamically changing visual scenes in virtual reality environments, and provides reliable support for the accurate calculation of perception critical frequencies.

[0056] IV. Rendering Performance and Perceptual Non-destructive Testing This experiment is used to verify the performance advantages of the aforementioned dynamic resolution rendering strategy, and at the same time to verify the perceptual non-destructive nature of the rendering method, ensuring that the present invention improves rendering efficiency without affecting the user's visual experience.

[0057] 1. Test scenario: A complex virtual reality scene (Sponza Atrium scene, which includes high-density polygon mesh and complex lighting effects, fits the actual application scenario of digital factory visualization, and is an existing technology) was selected, and the method of this invention and the traditional fixed foveated rendering method were compared and tested respectively. 2. Test metrics: (1) Rendering performance metrics: average frame rate (FPS) and computation savings ratio (ECSR). The computation savings ratio is used to quantify the degree of reduction in rendering load. The higher the value, the more significant the improvement in rendering efficiency. (2) Perceptual non-destructive index: The objective index adopts structural similarity (SSIM) to measure the similarity between the rendered image and the full-resolution reference image. The closer the SSIM value is to 1.0, the smaller the visual difference. The subjective index adopts user subjective rating (1~5 points, 5 points is no visual difference, 1 point is significant visual difference). 3. Test Results: (1) Performance test results: Compared with the traditional fixed foveated rendering method, the average frame rate (FPS) of the method of this invention increased from 75 to 92, an increase of about 22%; when the user performs frequent scanning search tasks (simulating the foveated switching scene in the digital factory roaming), the computational savings ratio (ECSR) can reach more than 50%, which effectively reduces the GPU rendering load; (2) Results of perceptual non-destructive testing: The SSIM values ​​of the rendered image and the full-resolution reference image remained above 0.92, indicating that there was no significant difference in image structure and details; In the user subjective rating experiment, the vast majority of subjects (10 out of 12) gave 4 to 5 points, and could not perceive the dynamic changes in resolution; 4. Experimental Conclusion: The dynamic resolution rendering strategy adopted in this invention can significantly improve rendering performance and reduce rendering load while maintaining good visual perception effect, achieving synergistic optimization of visual losslessness and rendering efficiency, and verifying the feasibility and superiority of the method in practical applications.

[0058] In summary, the three sets of experiments comprehensively verified the technical solution of this invention from three dimensions: the rationality of model parameters, the accuracy of the perceptual model, and the rendering performance and perceptual non-destructiveness. The experimental results show that the multi-dimensional perceptual foveated rendering method and system providing this invention, which integrates saccade temporal features, has reasonable parameter calibration, high perceptual accuracy, and superior rendering performance, while ensuring visual non-destructiveness. It fully meets the practical application needs of virtual reality environments (such as digital factory visualization) and solves the technical problems of low perceptual accuracy and insufficient rendering efficiency in traditional foveated rendering methods.

[0059] The present invention has been described above by way of example, but the present invention is not limited to the specific embodiments described above. Any modifications or variations made based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A multi-dimensional perceptual gaze-based rendering method, characterized in that, Including steps S1. Acquire the user's eye-tracking data in real time in a virtual reality environment, wherein the eye-tracking data includes the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point; S2. Call and perform real-time temporal analysis on the eye tracking data, use a dual threshold detection algorithm based on speed and position to identify eye movement events, and synchronously divide the corresponding saccade window, recovery window and stable gaze window in the rendering time sequence based on the occurrence sequence, duration and state transition relationship of the eye movement events. The rendering timing is synchronized with the frame clock of the graphics rendering pipeline, and its timing nodes are determined in real time by eye-tracking events. S3. Obtain scene feature parameters of the current rendering frame. The eye movement events include saccade events, saccade recovery events, and stable gaze events. The eye movement state is the instantaneous state corresponding to the eye movement event at any time, including saccade state, saccade recovery state, and stable gaze state. A multidimensional visual perception model integrating saccade temporal features is constructed. The model takes the current eye movement state and the elapsed time after saccade as dynamic inputs, and combines scene feature parameters such as scene brightness, color, stimulus area, retinal eccentricity and spatial frequency parameters to output the contrast sensitivity threshold of the foveal region of the human eye at the current moment. The current rendering frame is a single frame image in the graphics rendering pipeline that is currently undergoing rasterization. It is frame-synchronized with the rendering timing in step S2, and its frame period corresponds one-to-one with the time nodes of the rendering timing. S4. Call the contrast sensitivity threshold to obtain the highest spatial frequency that meets the visual lossless condition, i.e. the perception critical frequency, and map the perception critical frequency as a resolution scaling factor to guide the rasterization operation of the graphics pipeline. S5. Invoke the resolution scaling factor and perform dynamic resolution rendering in the graphics rendering pipeline based on the resolution scaling factor; in the dynamic resolution rendering, full-screen downsampling is performed in the panning window, progressive resolution upsampling is performed over time in the recovery window, and standard foveated rendering is performed in the stable foveated window.

2. The multi-dimensional perceptual foveated rendering method according to claim 1, characterized in that, In step S2, the step of dividing the rendering sequence into the scanning window, the recovery window, and the stable gaze window includes: S21. Preset angular velocity threshold and fixation point deviation range threshold, used to distinguish between saccade events and non-saccade events; S22. Sagging start determination and saccade window activation: The instantaneous angular velocity of the eyeball and the screen coordinates of the fixation point obtained in step S1 are called in real time. When the instantaneous angular velocity of the eyeball is higher than the angular velocity threshold for multiple consecutive frames, and the offset of the screen coordinates of the fixation point relative to the initial screen coordinates of the fixation point at the start of the saccade exceeds the fixation point deviation range threshold, the saccade is determined to have started, and the saccade window described in step S2 is called in sync. S23. The saccade landing determination and recovery window is started. The instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point are continuously monitored. When the instantaneous angular velocity of the eyeball falls below the angular velocity threshold and the screen coordinates of the gaze point remain within the preset range of the target gaze area for multiple consecutive frames, it is determined to be a saccade landing. The saccade window is closed and the recovery window described in step S2 is started. At the same time, the saccade landing time is recorded as the starting point for the elapsed time after the saccade described in step S3. S24. Restore window terminates and stabilize gaze window starts; During the restore window, monitor the elapsed time after saccade as described in step S3 and the resolution scaling factor as described in step S4 in real time. When the resolution scaling factor returns to the preset reference value, close the restore window and start the stabilize gaze window as described in step S2.

3. The multi-dimensional-aware foveated rendering method of claim 1, wherein, The multidimensional visual perception model that integrates saccade temporal features also incorporates a saccade temporal modulation module, which outputs a saccade temporal modulation factor. The value of the saccade temporal modulation factor is designed as follows: If the current state is within the saccade window defined in step S2, the saccade timing modulation factor is set to a preset saccade suppression constant, which is used to characterize the extreme attenuation of human visual sensitivity under saccade conditions. If the current state is within the recovery window defined in step S2, the saccade timing modulation factor is a function of the elapsed time after saccade as described in step S3. This function exhibits a non-linear growth trend and is used to characterize the process by which visual sensitivity gradually recovers to a normal level over time after saccade. If the current state is within the stable gaze window defined in step S2, the saccade timing modulation factor is set to 1.0, indicating that the visual sensitivity is at a stable peak level. The contrast sensitivity threshold output by the multidimensional visual perception model that integrates saccade temporal features is obtained by multiplying the baseline sensitivity calculated from the static multidimensional visual parameters with the saccade temporal modulation factor. The static multidimensional visual parameters correspond to the scene brightness, color, stimulus area, retinal eccentricity, and spatial frequency parameters in step S3.

4. The multi-dimensional-aware foveated rendering method of claim 3, wherein, Within the recovery window, the saccade timing modulation factor and the elapsed time after saccade follow a power function recovery law, with the recovery speed gradually decreasing over time. The power function expression is: t represents the time after scanning the landing, in seconds; SF(t) represents the highest resolvable spatial frequency at time t, in cpd; k1, k2, and k3 are dynamic parameters, k1 = 1.9471~1.9483, k2 = 0.3470~0.3477, and k3 = 9.5065~9.5088.

5. The multi-dimensional-aware foveated rendering method of claim 1, wherein, The sensing critical frequency is expressed as the product of a static cutoff frequency reference and a scanning timing modulation factor. The static cut-off frequency reference is designed to be the highest spatial frequency that the human eye can achieve within a steady fixation period formed while the steady fixation window persists. The saccade timing modulation factor , is the output factor of the saccade temporal modulation module built into the multidimensional visual perception model that integrates saccade temporal features; Sensing critical frequency : When the current eye movement state s=SACCADING, the eye movement event is a saccade event, and the saccade window persists for a period of time. (t,s)= min; When the current eye movement state s = POST - SACCADIC, the eye movement event is a saccade recovery event. During the duration of the recovery window, 0 ≤ t < hour, (t,s)= ·( ) / max; This is the duration of the restore window; When the current eye movement state s=FIXATING, the eye movement event is a stable fixation event, and the stable fixation window lasts for a period of time. (t,s)= .

6. The multi-dimensional perceptual gaze point rendering method according to claim 5, characterized in that, In step S4, the resolution scaling factor k(t) is calculated using the following mapping function: η∈(0,1] is a safety factor used to compensate for model fitting residuals and inter-individual physiological differences. The upper limit of the frequency corresponding to the spatial sampling rate during full-resolution rendering.

7. The multi-dimensional perceptual foveated rendering method according to claim 1, characterized in that, The dynamic resolution rendering step in step S5 includes: During the saccade window, the resolution scaling factor of the entire viewport is set to the lowest level, and the Kalman filter algorithm is used to predict the saccade landing point based on the current eye movement trajectory, and the area around the predicted saccade landing point is pre-rendered. During the recovery window, a time-dimensional low-pass filter is applied to the calculated resolution scaling factor to smooth the resolution improvement trajectory over time and avoid visual flicker caused by sudden changes in resolution between frames. At the same time, the coverage radius of the central high-resolution region dynamically expands as the perception critical frequency increases. During the stable gaze window, the resolution scaling factor is kept constant at 1.0, and the surrounding area is downsampled only based on spatial eccentricity.

8. The multi-dimensional perceptual foveated rendering method according to claim 1, characterized in that, The overall contrast sensitivity of the multidimensional visual perception model that integrates saccadic temporal features The output expression is: in, This represents the overall contrast sensitivity scalar value. These represent the response increments of L, M, and S type cone cells at the sensitivity threshold, respectively. These are the baseline response values ​​of L, M, and S type cone cells under the corresponding background light intensity.

9. A multi-dimensional foveated rendering system, characterized in that, The system includes: The data acquisition module is used to acquire the user's eye movement signals through an eye-tracking device. The eye movement signals include at least the instantaneous angular velocity of the eyeball and the screen coordinates of the gaze point. The event detection module, connected to the data acquisition module, is used to identify three types of eye movement states in real time: saccade events, saccade recovery events, and stable gaze events, based on the dual threshold judgment logic of eye movement signals. It synchronously outputs the current eye movement state and the elapsed time after saccade, and divides the saccade window, recovery window, and stable gaze window in the rendering sequence based on the timing and duration of the eye movement events. The model calculation module has a built-in multi-dimensional visual perception model that integrates saccade timing features. It is connected to the event detection module and is used to calculate the contrast sensitivity threshold of the human eye in real time based on scene environment parameters, current eye movement state and elapsed time after saccade, and to calculate the perception critical frequency. The rendering scheduling module, connected to the model calculation module, is used to convert the perception critical frequency into a resolution scaling instruction and generate a dynamic multi-resolution rendering configuration that is synchronized with the rendering timing and eye-tracking state. The graphics rendering module, connected to the rendering scheduling module, is used to receive rendering configurations and perform image rendering with differentiated sampling rates for different rendering windows in the graphics rendering pipeline.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the multidimensional perceptual foveated rendering method as described in any one of claims 1 to 8.

Images