A real-time ray tracing method and system for VR large-space immersive tours

Abstract

This invention discloses a real-time ray tracing method and system for VR large-space immersive tours. The method includes: acquiring ray data information and a near-eye photograph of the user to determine the rendering range; rendering the image using ray tracing based on the rendering range; and projecting the rendered image onto a display device for output. This invention significantly improves the rendering quality and efficiency of VR large-space immersive tours by employing a real-time ray tracing method and system based on foveal rendering. Through gaze point region segmentation and sampling strategies, combined with pre-trained neural networks and ultra-high-definition frame generation technology, it achieves accurate determination and optimized sampling of the rendering range, greatly reducing computational load while ensuring the realism and immersion of the visual effects.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, specifically to a real-time ray tracing method and system for VR large-space immersive tours. Background Technology

[0002] Currently, there are a large number of large-space immersive VR offline experience centers based on LBE. Users expect to be immersed in a realistic and interactive virtual environment. This environment not only requires a high degree of visual realism, but also needs to respond quickly to the user's actions to ensure a smooth user experience.

[0003] Ray tracing is the most widely used rendering technique for high-end, realistic graphics. It approximates the behavior of light interacting with objects in a virtual environment through complex Monte Carlo estimation, resulting in physically accurate and realistic effects such as global illumination, soft shadows, caustics, reflection, refraction, and diffuse reflection. These lighting effects are difficult to simulate using rasterization rendering techniques, making ray tracing dominant in offline rendering fields such as film, architectural visualization, and product design. However, ray tracing has a slow convergence time and is accompanied by dense noise. Combined with rendering environments involving pixel density, geometric complexity, advanced rendering materials, and multiple light sources, it remains difficult to apply to real-time interactive applications.

[0004] Slow ray tracing convergence is directly related to the number of samples, the number of ray bounces, and the complexity of the virtual scene. Using a small number of samples results in a noisy final rendered image. To obtain a better final rendered image, denoising operations with more samples are generally used. In fact, to reduce noise by half, four times the number of samples is needed. This may only take some time for offline rendering, but it is very challenging for real-time rendering. In addition to the number of samples and denoising, the number of ray bounces also contributes to frame rate and computational overhead. In complex scenes with many object and surface intersections, rays need to bounce multiple times, which greatly increases the computational load and convergence time.

[0005] Meanwhile, the demand for displays with higher pixel density and refresh rates is increasing, further increasing the computational load on rendering. For example, the average angular resolution of commercial VR devices is currently 35 PPD, with a refresh rate of 90Hz. Taking Pimax Crystal, released by Pimax Technology in 2024, as an example, it offers approximately 35 PPD and a maximum field of view of 120°, with 35 PPD * 120° = 4200 pixels in the horizontal direction. Because the human field of vision is not perfectly square, the vertical field of view is usually smaller than the horizontal field of view. If we assume the vertical field of view is approximately 100°, the number of pixels in the vertical direction is 35 PPD * 100° = 3500 pixels. Therefore, the theoretical resolution of Pimax Crystal is close to 4200x3500 pixels. However, Pimax Crystal is actually a binocular 8K device, meaning that each eye has a resolution of 4K (i.e., 3840x2160), so the total resolution of the entire display system is 7680x2160 (binocular total). In reality, humans can detect up to 60 PPD in the concave region. To achieve a fully immersive experience for human vision, we need higher pixel density and refresh rate. Summary of the Invention

[0006] To address the challenges of immersive rendering in large-space VR environments, this method employs ray tracing technology. To overcome issues such as slow ray tracing convergence, the high pixel density and refresh rate of VR devices, and the inherent limitations of VR device performance (compared to the NVIDIA RTX 5090), this method proposes a real-time ray tracing approach and system based on concave rendering. This aims to improve VR rendering quality and adaptively adjust rendering strategies according to hardware specifications and current rendering conditions, resulting in an overall rendering performance improvement of over 10 times.

[0007] To achieve the above objectives, the present invention provides a real-time ray tracing method for VR large-space immersive tours, the method comprising:

[0008] Acquire lighting data and close-up photos of the user to confirm the rendering range of the image;

[0009] Based on the aforementioned rendering area, ray tracing is used for rendering the image;

[0010] The rendered image is then projected onto a display device for output.

[0011] Preferably, the near-eye photo is provided by an eye photo taken by the near-eye camera of the VR device; the near-eye photo is input into a first pre-trained neural network for feature extraction, and after feature extraction, a gaze point region segmentation and sampling strategy is used to determine the final rendering range of the image.

[0012] Preferably, the fixation point region segmentation and sampling strategy includes:

[0013] The main camera screen is divided into three regions: the first gaze region, the second gaze region, and the third gaze region, and the radii of the three gaze regions are calculated based on the eccentricity angle.

[0014] The ratio of the total number of samples in the first fixation region, the second fixation region, and the third fixation region was set to [value missing].

[0015] Rays are emitted for each pixel in the first gaze region; rays are emitted for the second gaze region according to N. near ×N near Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N near This represents the number of pixels in the length or width of the superpixel block in the region adjacent to the fixation point; for the third fixation region, it is calculated according to N. far ×N far Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N far This indicates the number of pixels in length or width of the superpixel block located far from the point of view.

[0016] Preferably, the method for calculating the radii of the three fixation regions includes:

[0017]

[0018] Where θ represents the eccentricity angle, in degrees; the eccentricity angle of the "first fixation area" is θ1 = 5.2°; the eccentricity angle of the "second fixation area" is θ2 = 9°; and the eccentricity angle of the "third fixation area" is θ3 = 17°. d represents the distance from the eye, in centimeters, and in this embodiment, d = 60cm; A represents the display size, in centimeters, and in this embodiment, the display represents 4K resolution, i.e., 70.848 × 39.852 square centimeters.

[0019] Preferably, before projecting the ray, bounding box detection is performed to check whether the initial position of the ray in the screen space falls within the boundary of its area. Bounding box detection is a binary operation. If the ray does not intersect with the bounding box of the virtual scene, it is discarded; otherwise, the ray is retained to propagate further.

[0020] Preferably, before projecting to the display device, an ultra-high-definition frame generation step is added to improve rendering quality; the step includes: using a dual-path parallel input method, inputting single-path images and multiple-path images into the second pre-trained neural network in parallel, and obtaining their respective results.

[0021] Preferably, the output of the second pre-trained neural network is transmitted to the second discriminator. The second discriminator, in conjunction with the first discriminator, adjusts the rendering output based on the hardware performance, the complexity of the current scene, and user-controllable parameters to achieve a balance between rendering performance and immersive effects.

[0022] The present invention also provides a real-time ray tracing system for VR large-space immersive tours. The system is used to implement the above method and includes: an input module, a rendering module, and an output module.

[0023] The input module is used to acquire light data information and a close-up photo of the user's eye to confirm the rendering range of the image.

[0024] The rendering module is used to render the image using ray tracing based on the image rendering range;

[0025] The output module is used to project the rendered image onto a display device to complete the output.

[0026] Preferably, the near-eye photo is provided by an eye photo taken by the near-eye camera of the VR device; the input module inputs the near-eye photo into a first pre-trained neural network for feature extraction, and after feature extraction, a gaze point region segmentation and sampling strategy is used to determine the final image rendering range.

[0027] Preferably, the fixation point region segmentation and sampling strategy includes:

[0028] The main camera screen is divided into three regions: the first gaze region, the second gaze region, and the third gaze region, and the radii of the three gaze regions are calculated based on the eccentricity angle.

[0029] The ratio of the total number of samples in the first fixation region, the second fixation region, and the third fixation region was set to [value missing].

[0030] Rays are emitted for each pixel in the first gaze region; rays are emitted for the second gaze region according to N. near ×N near Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N near This represents the number of pixels in the length or width of the superpixel block in the region adjacent to the fixation point; for the third fixation region, it is calculated according to N. far ×N far Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N far This indicates the number of pixels in length or width of the superpixel block located far from the point of view.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention significantly improves the rendering quality and efficiency of large-space immersive VR tours by employing a real-time ray tracing method and system based on concave rendering. Through foveated region segmentation and sampling strategies, combined with pre-trained neural networks and ultra-high-definition frame generation technology, it achieves precise determination and optimized sampling of the rendering range, greatly reducing computational load while ensuring the realism and immersion of the visual effects. Furthermore, the system can adaptively adjust the rendering strategy based on hardware performance, scene complexity, and user parameters, effectively balancing rendering performance and immersive experience. Compared to traditional methods, this invention improves overall rendering performance by more than 10 times while enhancing rendering quality, providing strong support for the widespread application of VR immersive experiences. Attached Figure Description

[0033] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the gaze region division according to an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0038] First, some technical terms used in this invention will be explained:

[0039] The fovea: The photoreceptor cells in the human retina are unevenly distributed, making it impossible to maintain the same level of sharpness across the entire visual field. The retina contains approximately 6 million cone cells, densely packed in the central region with an off-center angle of about 5.2 degrees. This region is called the fovea, and it covers an area approximately 1.5 mm in diameter within the retina. The off-center angles in the two adjacent regions are 5.2–9 degrees and 9–17 degrees, respectively.

[0040] Ray recursion depth: that is, the number of times the ray bounces. In this embodiment, it is fixed at 3 times, instead of an infinite number of times, otherwise the convergence speed will be too slow.

[0041] Multiple importance sampling: a sampling strategy that involves sampling more in important areas and less in unimportant areas.

[0042] Multipath fusion: Combines the colors sampled from multiple rays to form the final pixel color.

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] This embodiment provides a real-time ray tracing method for VR large-space immersive tours, the steps of which include:

[0046] S1. Obtain lighting data and a close-up photo of the user to confirm the rendering range of the image.

[0047] The current ray tracing workflow is: main camera → ray emission → rendering pipeline → multipath blending → explicit device; in the conventional method, during the input phase, each pixel on the main camera screen emits a certain number of rays into the virtual scene. However, as... Figure 1 As shown, this embodiment adds "foveation prediction" in parallel during the input stage. The purpose is to reduce the computational load and improve rendering efficiency through foveal rendering technology. That is, by acquiring the user's eye data, the user's visual field is predicted; in subsequent steps, only the image within the user's visual field is rendered to reduce the computational load and improve rendering quality.

[0048] Specifically, a near-eye photo is first acquired (provided by a photo of the eye taken by the near-eye camera of the VR device), and then this photo is input into a first pre-trained neural network. The first pre-trained neural network is a pre-trained ViT (VisionTransformer) or Swin Transformer model, used to extract features from the "near-eye photo". A fully connected layer is then added after the Transformer to output the two-dimensional gaze direction.

[0049] After feature extraction, a foveated region segmentation and sampling strategy is used to determine the final rendering range. The purpose of the foveated region segmentation and sampling strategy is to divide the screen into different regions, and use a different sampling strategy for each region, increasing the sampling frequency in high-frequency regions and reducing the sampling frequency in low-frequency regions, thereby reducing the overall number of samples, reducing the computational load, and improving rendering efficiency.

[0050] The specific details of the fixation point region division and sampling strategy are as follows:

[0051] like Figure 2 As shown, the "camera screen" (which is also the main camera screen) is divided into three areas: the "first gaze area", the "second gaze area", and the "third gaze area".

[0052] The radii of the three fixation regions are calculated based on the eccentricity angle using the following formula:

[0053]

[0054] Where θ represents the eccentricity angle, in degrees; the eccentricity angle of the "first fixation area" is θ1 = 5.2°; the eccentricity angle of the "second fixation area" is θ2 = 9°; and the eccentricity angle of the "third fixation area" is θ3 = 17°. d represents the distance from the eye, in centimeters, and in this embodiment, d = 60cm; A represents the display size, in centimeters, and in this embodiment, the display represents 4K resolution, i.e., 70.848 × 39.852 square centimeters.

[0055] To achieve good ray tracing rendering effects and prevent visual artifacts, each pixel typically requires at least 32 samples. However, this high sampling count severely impacts the frame rate of real-time rendering. To address this issue, this embodiment employs a segmented, block-based sampling strategy:

[0056] 1) The total sampling ratio of the "first fixation region", "second fixation region" and "third fixation region" is:

[0057] 2) To further optimize the sampling quantity, rays are emitted for each pixel of the "first gaze region"; rays are emitted for the "second gaze region" according to N... near ×N near The superpixel blocks are used to emit rays (in this embodiment, N is selected). near =2), instead of emitting rays from every pixel; for the "third gaze region" according to N far ×N far The superpixel blocks are used to emit rays (in this embodiment, N is selected). far =4), instead of emitting a ray from every pixel. , where N near N represents the number of pixels in the length or width of the superpixel block in the region adjacent to the fixation point, i.e., the number of pixels in the length or width of the superpixel block in the "second fixation region". near ×N near N represents the size of the superpixel block in the "second gaze region". far N represents the number of pixels in length or width of the superpixel block furthest from the foveation point, i.e., the number of pixels in length or width of the superpixel block in the "third foveation region". far ×N far The size of the superpixel block representing the "third gaze region".

[0058] Specifically, each pixel in the "first gaze region" emits a ray, the "second gaze region" emits a ray every 4 pixels, and the "third gaze region" emits a ray every 16 pixels.

[0059] 3) The number of samples per pixel is highest only in the "first gaze region," which is τ (τ1 = 32 in this embodiment); in the "second gaze region," each N near ×N near The number of samples for each superpixel block is τ2 (τ2 = 16 in this embodiment); in the "third gaze region", each N far ×N far The number of samples for the superpixel block is τ3 (τ3 = 8 is selected in this embodiment).

[0060] 4) By employing the aforementioned segmented sampling strategy, the number of samples in the central concave region can be increased, ensuring rendering quality. Simultaneously, the number of samples in other regions is gradually reduced, resulting in a more reasonable sampling distribution, reducing the overall number of samples, and improving rendering efficiency. This embodiment also sets the ray recursion depth to 3 times to avoid further computational load. Furthermore, multiple importance sampling can be used to dynamically adjust the number of samples, thereby further optimizing the balance between visual effects and real-time performance.

[0061] Before projecting a ray, bounding box detection is performed, which checks whether the ray's initial position in screen space falls within the boundary of its assigned area. This boundary check effectively reduces unnecessary computation. The check is a binary operation: if the ray does not intersect the bounding box of the virtual scene, it is discarded; otherwise, the ray is preserved for further propagation.

[0062] S2. Based on the aforementioned rendering range, ray tracing is used for rendering the image.

[0063] After the annotation area (i.e. the rendering range) is determined, the main camera emits rays that are multi-channeled and input into the rendering pipeline through the first discriminator.

[0064] "Multi-path parallel input" means that each ray must pass through the "rendering pipeline," which involves first performing an "intersection test," then "ray tracing," then "pixel filling," and finally "post-processing." The rendering process for each ray is independent and does not affect others, allowing for parallel processing. It's important to note that during parallel processing, the final color of a pixel can only be determined after all pixel colors for each ray have been calculated. However, since the processing speed of each ray is different, the color of a pixel can only be determined after the slowest ray has finished processing.

[0065] Pixel filling: In this embodiment, the total number of projected rays is less than the number of pixels (i.e., less than the number of rays emitted by each pixel). In order to give each pixel a final color, these pixels are filled with color using linear interpolation.

[0066] Post-processing: In order to obtain a photorealistic appearance rendering effect, this embodiment uses the bidirectional reflectance distribution function (BRDF) as the lighting model.

[0067] (2) This method uses subpixel dithering, exposure compensation and tone mapping in post-processing to obtain the fused output result.

[0068] Subpixel dithering is an anti-aliasing technique designed to reduce stepped edges or jaggedness caused by pixelation. It simulates a finer resolution by slightly shifting the sampling location, resulting in smoother, more natural object edges. During rendering, the actual sampling location of each pixel is slightly shifted randomly or pseudo-randomly around its theoretical center. This tiny movement helps capture more detail and, when averaged across multiple samples, effectively reduces aliasing artifacts in geometry, such as hard edges or flickering.

[0069] Exposure compensation refers to the process of adjusting image brightness to ensure that brightness information in a scene is correctly displayed on the screen. Since computer-generated images typically use linear color spaces, while monitors employ non-linear gamma-corrected color spaces, appropriate exposure adjustments are needed to match real-world lighting conditions as perceived by the human eye. Renderers may adjust brightness globally or locally based on the overall brightness of the scene or the brightness of specific areas. For example, in a very bright environment, it may be necessary to reduce the exposure value to avoid overexposure; while in a dark environment, it may be necessary to increase the exposure value to make shadow details visible. This process can be achieved through a simple multiplication operation: multiplying by a number greater than 1 to increase brightness, or multiplying by a number less than 1 to decrease brightness.

[0070] Tone mapping is a process of converting a high dynamic range (HDR) image to a low dynamic range (LVR) image so that it can be displayed correctly on standard display devices. This algorithm is particularly suitable for maintaining visual fidelity of colors while compressing the brightness range, making the image appear neither too bright nor too dark. Reinhard tone mapping is a physically based algorithm that mimics how the human eye adapts to different lighting conditions. The core idea of ​​the algorithm is to map the brightness values ​​in an HDR image to a reasonable range, typically between 0 and 1, while preserving as much of the original image's color relationships and contrast as possible. Specifically, it is calculated according to the following formula: Where L represents the original luminance value; W represents the white point, which represents the maximum luminance. This formula ensures a smooth transition even under extreme luminance conditions.

[0071] Finally, three images are rendered for the "first gaze region", "second gaze region" and "third gaze region" respectively, and then merged into a single output image.

[0072] S3. Project the rendered image onto a display device to complete the output.

[0073] Traditional ray tracing methods render the image in the rendering pipeline and then project it onto the display device after multipath fusion. However, this embodiment adds "ultra-high-definition frame generation" in parallel during the "output stage" to improve rendering quality. Specifically, it adopts a dual-path parallel input method, inputting the single-path image (i.e., the image of pixel colors output by a single ray) and the multipath image (i.e., the image after "multipath fusion") into the second pre-trained neural network in parallel, obtaining their respective results without interference.

[0074] The second pre-trained neural network is a pre-trained model, commonly including FSRCNN (Fast Super-Resolution Convolutional Neural Network), FALSR (Fast, Accurate and Lightweight Super-Resolution models), Bicubic++, ESPCN (Efficient Sub-Pixel Convolutional Neural Network), LESRCNN (Lightweight Efficient Single Image Super-Resolution CNN), etc. These lightweight models are suitable for real-time application scenarios and can generate high-quality high-definition frame images.

[0075] The output of the second pre-trained neural network is then transmitted to the second discriminator and finally projected onto the display device.

[0076] The first and second discriminators can adaptively adjust the rendering output based on factors such as hardware performance, current scene complexity, and user-controllable parameters to achieve a balance between rendering performance and immersive effects. The second discriminator receives three inputs: one is the image I directly input after multi-path fusion. origin One is a single-channel image that has undergone "ultra-high-definition frame generation" to generate an image I. high One is an image I generated from multiple images through ultra-high-definition frame mapping. Higher The three images need to be judged:

[0077] The generation time of the three images: I high <I origin <I Higher The generation quality of the three images: I high ≈I origin <<I Higher Therefore, in the first discriminator, only I is used. high and I Higher Because I origin The generation time is longer and the quality is higher than I.Higher Poor, therefore there is I high The existence of I does not require origin I need to origin The generation path is disabled. Even so, on some low-end devices without GPUs, foveation prediction and ultra-high-resolution frame generation are unusable (because they have neural networks and Transformers, which require a GPU to run). In this case, it is necessary to use I... origin This is the main reason it is retained, as it is used to output rendering frames normally.

[0078] (1) Judgment Criterion 1: Hardware Performance. The system will check hardware performance; if there is no GPU, it will use I... origin As output, the overall hardware performance score is calculated by checking system parameters (such as GPU and CPU models, resolution, refresh rate, etc.).

[0079] P hardware =ω CPU ×S CPU +ω GPU ×S GPU +ω resolution ×S resolution +ω Refresh ×S Refresh +ω Others ×S Others

[0080] Where, ω CPU and S CPU The weighting coefficients and standardized performance scores for the CPU; ω GPU and S GPU ω represents the weighting coefficients and standardized performance score of the GPU. resoliution and S resolution The weighting coefficients for resolution and the standardized score for performance; ω Refresh and S Refresh The weighting coefficients for refresh rate and the standardized score for performance; ω Others and S Others The weighting coefficients for other parameters and the standardized score for performance.

[0081] For CPUs and GPUs, standardized scores can be obtained using benchmark tools such as Geekbench, PassMark, or VR-specific benchmarks. For resolution, pixel density (PPD) or the total number of pixels can be considered and normalized to a reasonable maximum. For refresh rate, it can be compared to the expected minimum refresh rate (e.g., 60Hz or 90Hz) and scored based on the highest supported refresh rate. Other factors, such as tracking accuracy and latency, can be scored and standardized through professional evaluation or user feedback. For example:

[0082] ω CPU =0.1, ω GPU =0.4, ω resolution =0.2, ω Refresh =0.2, ω Others =0.1

[0083] The corresponding standardized score is:

[0084] S CPU =0.85, S GPU =0.9, S resolution =0.75, S Refresh =0.95, S Others =0.8

[0085] Then P hardware =86.5%.

[0086] In this embodiment, the overall hardware performance score of the benchmark is:

[0087] P baseline (Who determines P in this embodiment) baseline =80%), for P hardware <P baseline The hardware device does not require judgment; it can be used directly with I. high As the final output, to improve output performance.

[0088] (2) Judgment Criterion Two: System Delay Time. The second discriminator records the time T from the end of the previous frame rendering by the main camera to the present. cur and according to a certain frame rate standard FPS τ Determine the maximum system delay time T τ For example, VR typically renders at a frame rate of FPS. τ =60FPS, then the total latency per image (maximum system latency) is T. τ = 1 / 60 ≈ 16.67ms, while VR high frame rate rendering is FPS τ =120FPS, then the total latency per image (maximum system latency) is T.τ = 1 / 120 ≈ 8.33 ms. When T cur ≥T τ When, if I Higher When rendered, use I. Higher Otherwise use U high .

[0089] (3) Judgment Criterion Three: User-Controllable Parameters. When the user sets specific parameters in the system, such as setting the frame rate (FPS). user Then the maximum delay time T of the automatic time system is calculated. τ =1 / FPS user When T cur ≥T τ When, if I Higher When rendered, use I. Higher Otherwise use I high .

[0090] It is important to note that for complex scenes, whether the scene has a large number of objects, a large number of object faces, complex lighting, or complex materials, the "criterion two" can be used for judgment.

[0091] The first and second discriminators record the previous configuration result. If factors such as hardware performance, current scenario complexity, and user-controllable parameters remain unchanged, the previous configuration will still be used; otherwise, the second discriminator will be reactivated for further discrimination. For example, the second discriminator records the I used this time. high In the next rendering iteration, if the environmental factors remain unchanged, the first discriminator will not perform multi-path parallel input, but will directly determine the result based on I. high It only uses one input path (i.e., one ray is emitted per pixel, passing through only one rendering pipeline), and does not require multi-path fusion or dual-path parallel input. Instead, it directly generates the final result I through ultra-high-definition frame generation. high (Assuming environmental factors remain unchanged). For example, the second discriminator records the I used this time. Higher In the next rendering iteration, if environmental factors remain unchanged, the first discriminator will generate the final result I through multi-path parallel input, multi-path fusion, and then ultra-high-definition frame generation. Higher .

[0092] After being processed by the second discriminator, the final image is output to the display device.

[0093] Example 2

[0094] This embodiment also provides a real-time ray tracing system for VR large-space immersive tours, including: an input module, a rendering module, and an output module; the input module is used to acquire ray data information and a near-eye photo of the user to confirm the rendering range of the screen; the rendering module is used to render the screen using ray tracing based on the rendering range of the screen; the output module is used to project the rendered screen onto a display device to complete the output.

[0095] The following will, in conjunction with this embodiment, explain in detail how the present invention solves technical problems in real life.

[0096] First, the input module is used to obtain light data and a close-up photo of the user to confirm the rendering range of the image.

[0097] The current ray tracing workflow is: main camera → ray emission → rendering pipeline → multipath blending → explicit device; in the conventional method, during the input phase, each pixel on the main camera screen emits a certain number of rays into the virtual scene. However, as... Figure 1 As shown, this embodiment adds "foveation prediction" in parallel during the input stage. The purpose is to reduce the computational load and improve rendering efficiency through foveal rendering technology. That is, by acquiring the user's eye data, the user's visual field is predicted; in subsequent steps, only the image within the user's visual field is rendered to reduce the computational load and improve rendering quality.

[0098] Specifically, a near-eye photo is first acquired (provided by a photo of the eye taken by the near-eye camera of the VR device), and then this photo is input into a first pre-trained neural network. The first pre-trained neural network is a pre-trained ViT (VisionTransformer) or Swin Transformer model, used to extract features from the "near-eye photo". A fully connected layer is then added after the Transformer to output the two-dimensional gaze direction.

[0099] After feature extraction, a foveated region segmentation and sampling strategy is used to determine the final rendering range. The purpose of the foveated region segmentation and sampling strategy is to divide the screen into different regions, and use a different sampling strategy for each region, increasing the sampling frequency in high-frequency regions and reducing the sampling frequency in low-frequency regions, thereby reducing the overall number of samples, reducing the computational load, and improving rendering efficiency.

[0100] The specific details of the fixation point region division and sampling strategy are as follows:

[0101] like Figure 2 As shown, the "camera screen" (which is also the main camera screen) is divided into three areas: the "first gaze area", the "second gaze area", and the "third gaze area".

[0102] The radii of the three fixation regions are calculated based on the eccentricity angle using the following formula:

[0103]

[0104] Where θ represents the eccentricity angle, in degrees; the eccentricity angle of the "first fixation area" is θ1 = 5.2°; the eccentricity angle of the "second fixation area" is θ2 = 9°; and the eccentricity angle of the "third fixation area" is θ3 = 17°. d represents the distance from the eye, in centimeters, and in this embodiment, d = 60cm; A represents the display size, in centimeters, and in this embodiment, the display represents 4K resolution, i.e., 70.848 × 39.852 square centimeters.

[0105] To achieve good ray tracing rendering effects and prevent visual artifacts, each pixel typically requires at least 32 samples. However, this high sampling count severely impacts the frame rate of real-time rendering. To address this issue, this embodiment employs a segmented, block-based sampling strategy:

[0106] 1) The total sampling ratio of the "first fixation region", "second fixation region" and "third fixation region" is:

[0107] To further optimize the sampling number, rays are emitted for each pixel in the "first gaze region"; and for the "second gaze region" according to N... near ×N near The superpixel blocks are used to emit rays (in this embodiment, N is selected). near =2), instead of emitting rays from every pixel; for the "third gaze region" according to N far ×N far The superpixel blocks are used to emit rays (in this embodiment, N is selected). far =4), instead of emitting a ray from every pixel. , where N near N represents the number of pixels in the length or width of the superpixel block in the region adjacent to the fixation point, i.e., the number of pixels in the length or width of the superpixel block in the "second fixation region". near ×N near N represents the size of the superpixel block in the "second gaze region". far N represents the number of pixels in length or width of the superpixel block furthest from the foveation point, i.e., the number of pixels in length or width of the superpixel block in the "third foveation region". far ×N far The size of the superpixel block representing the "third gaze region".

[0108] Specifically, each pixel in the "first gaze region" emits a ray, the "second gaze region" emits a ray every 4 pixels, and the "third gaze region" emits a ray every 16 pixels.

[0109] 3) The number of samples per pixel is highest only in the "first gaze region," which is τ1 (τ1 = 32 in this embodiment); in the "second gaze region," the number of samples per N is higher. near ×N near The number of samples for each superpixel block is τ2 (τ2 = 16 in this embodiment); in the "third gaze region", each N far ×N far The number of samples for the superpixel block is τ3 (τ3 = 8 is selected in this embodiment).

[0110] 4) By employing the aforementioned segmented sampling strategy, the number of samples in the central concave region can be increased, ensuring rendering quality. Simultaneously, the number of samples in other regions is gradually reduced, resulting in a more reasonable sampling distribution, reducing the overall number of samples, and improving rendering efficiency. This embodiment also sets the ray recursion depth to 3 times to avoid further computational load. Furthermore, multiple importance sampling can be used to dynamically adjust the number of samples, thereby further optimizing the balance between visual effects and real-time performance.

[0111] Before projecting a ray, bounding box detection is performed, which checks whether the ray's initial position in screen space falls within the boundary of its assigned area. This boundary check effectively reduces unnecessary computation. The check is a binary operation: if the ray does not intersect the bounding box of the virtual scene, it is discarded; otherwise, the ray is preserved for further propagation.

[0112] Then, the rendering module renders the image using ray tracing based on the image rendering range.

[0113] After the annotation area (i.e. the rendering range) is determined, the main camera emits rays that are multi-channeled and input into the rendering pipeline through the first discriminator.

[0114] "Multi-path parallel input" means that each ray must pass through the "rendering pipeline," which involves first performing an "intersection test," then "ray tracing," then "pixel filling," and finally "post-processing." The rendering process for each ray is independent and does not affect others, allowing for parallel processing. It's important to note that during parallel processing, the final color of a pixel can only be determined after all pixel colors for each ray have been calculated. However, since the processing speed of each ray is different, the color of a pixel can only be determined after the slowest ray has finished processing.

[0115] Pixel filling: In this embodiment, the total number of projected rays is less than the number of pixels (i.e., less than the number of rays emitted by each pixel). In order to give each pixel a final color, these pixels are filled with color using linear interpolation.

[0116] Post-processing: In order to obtain a photorealistic appearance rendering effect, this embodiment uses the bidirectional reflectance distribution function (BRDF) as the lighting model.

[0117] (2) This method uses subpixel dithering, exposure compensation and tone mapping in post-processing to obtain the fused output result.

[0118] Subpixel dithering is an anti-aliasing technique designed to reduce stepped edges or jaggedness caused by pixelation. It simulates a finer resolution by slightly shifting the sampling location, resulting in smoother, more natural object edges. During rendering, the actual sampling location of each pixel is slightly shifted randomly or pseudo-randomly around its theoretical center. This tiny movement helps capture more detail and, when averaged across multiple samples, effectively reduces aliasing artifacts in geometry, such as hard edges or flickering.

[0119] Exposure compensation refers to the process of adjusting image brightness to ensure that brightness information in a scene is correctly displayed on the screen. Since computer-generated images typically use linear color spaces, while monitors employ non-linear gamma-corrected color spaces, appropriate exposure adjustments are needed to match real-world lighting conditions as perceived by the human eye. Renderers may adjust brightness globally or locally based on the overall brightness of the scene or the brightness of specific areas. For example, in a very bright environment, it may be necessary to reduce the exposure value to avoid overexposure; while in a dark environment, it may be necessary to increase the exposure value to make shadow details visible. This process can be achieved through a simple multiplication operation: multiplying by a number greater than 1 to increase brightness, or multiplying by a number less than 1 to decrease brightness.

[0120] Tone mapping is a process of converting a high dynamic range (HDR) image to a low dynamic range (LVR) image so that it can be displayed correctly on standard display devices. This algorithm is particularly suitable for maintaining visual fidelity of colors while compressing the brightness range, making the image appear neither too bright nor too dark. Reinhard tone mapping is a physically based algorithm that mimics how the human eye adapts to different lighting conditions. The core idea of ​​the algorithm is to map the brightness values ​​in an HDR image to a reasonable range, typically between 0 and 1, while preserving as much of the original image's color relationships and contrast as possible. Specifically, it is calculated according to the following formula: Where L represents the original luminance value; W represents the white point, which represents the maximum luminance. This formula ensures a smooth transition even under extreme luminance conditions.

[0121] Finally, three images are rendered for the "first gaze region", "second gaze region" and "third gaze region" respectively, and then merged into a single output image.

[0122] Finally, the output module projects the rendered image onto the display device to complete the output.

[0123] Traditional ray tracing methods render the image in the rendering pipeline and then project it onto the display device after multipath fusion. However, this embodiment adds "ultra-high-definition frame generation" in parallel during the "output stage" to improve rendering quality. Specifically, it adopts a dual-path parallel input method, inputting the single-path image (i.e., the image of pixel colors output by a single ray) and the multipath image (i.e., the image after "multipath fusion") into the second pre-trained neural network in parallel, obtaining their respective results without interference.

[0124] The second pre-trained neural network is a pre-trained model, commonly including FSRCNN (Fast Super-Resolution Convolutional Neural Network), FALSR (Fast, Accurate and Lightweight Super-Resolution models), Bicubic++, ESPCN (Efficient Sub-Pixel Convolutional Neural Network), LESRCNN (Lightweight Efficient Single Image Super-Resolution CNN), etc. These lightweight models are suitable for real-time application scenarios and can generate high-quality high-definition frame images.

[0125] The output of the second pre-trained neural network is then transmitted to the second discriminator and finally projected onto the display device.

[0126] The first and second discriminators can adaptively adjust the rendering output based on factors such as hardware performance, current scene complexity, and user-controllable parameters to achieve a balance between rendering performance and immersive effects. The second discriminator receives three inputs: one is the image I directly input after multi-path fusion. origin One is a single-channel image that has undergone "ultra-high-definition frame generation" to generate an image I. high One is an image I generated from multiple images through ultra-high-definition frame mapping. Higher The three images need to be judged:

[0127] The generation time of the three images: I high <I origin <I HigherThe generation quality of the three images: I high ≈I origin <<I Higher Therefore, in the first discriminator, only I is used. high and I Higher Because I origin The generation time is longer and the quality is higher than I. high er is different, so I have high The existence of I does not require origin I need to origin The generation path is disabled. Even so, on some low-end devices without GPUs, foveation prediction and ultra-high-resolution frame generation are unusable (because they have neural networks and Transformers, which require a GPU to run). In this case, it is necessary to use I... origin This is the main reason it is retained, as it is used to output rendering frames normally.

[0128] (1) Judgment Criterion 1: Hardware Performance. The system will check hardware performance; if there is no GPU, it will use I... origin As output, the overall hardware performance score is calculated by checking system parameters (such as GPU and CPU models, resolution, refresh rate, etc.).

[0129] P hardware =ω CPU ×S CPU +ω GPU ×S GPU +ω resolution ×S resolution +ω Refresh ×S Refresh +ω Others ×S Others

[0130] Where, ω CPU and S CPU The weighting coefficients and standardized performance scores for the CPU; ω GPU and S GPU ω represents the weighting coefficients and standardized performance score of the GPU. resolution and S resolution The weighting coefficients for resolution and the standardized score for performance; ω Refresh and S Refresh The weighting coefficients for refresh rate and the standardized score for performance; ω Ohters and S Others The weighting coefficients for other parameters and the standardized score for performance.

[0131] For CPUs and GPUs, standardized scores can be obtained using benchmark tools such as Geekbench, PassMark, or VR-specific benchmarks. For resolution, pixel density (PPD) or the total number of pixels can be considered and normalized to a reasonable maximum. For refresh rate, it can be compared to the expected minimum refresh rate (e.g., 60Hz or 90Hz) and scored based on the highest supported refresh rate. Other factors, such as tracking accuracy and latency, can be scored and standardized through professional evaluation or user feedback. For example:

[0132] ω CPU =0.1, ω GPU =0.4, ω resolution =0.2, ω Refresh =0.2, ω Others =0.1

[0133] The corresponding standardized score is:

[0134] S CPU =0.85, S GP U = 0.9, S resolution =0.75, S Refresh =0.95, S Others =0.8

[0135] Then P hardware =86.5%.

[0136] In this embodiment, the overall hardware performance score of the benchmark is:

[0137] P baseline (Who determines P in this embodiment) baseline =80%), for P hardware <P baseline The hardware device does not require judgment; it can be used directly with I. high As the final output, to improve output performance.

[0138] (2) Judgment Criterion Two: System Delay Time. The second discriminator records the time T from the end of the previous frame rendering by the main camera to the present. cur and according to a certain frame rate standard FPS τ Determine the maximum system delay time T τ For example, VR typically renders at a frame rate of FPS. τ =60FPS, then the total latency per image (maximum system latency) is T. τ = 1 / 60 ≈ 16.67ms, while VR high frame rate rendering is FPS τ =120FPS, then the total latency per image (maximum system latency) is T.τ = 1 / 120 ≈ 8.33 ms. When T cur ≥T τ When, if I Higher When rendered, use I. Higher Otherwise use I high .

[0139] (3) Judgment Criterion Three: User-Controllable Parameters. When the user sets specific parameters in the system, such as setting the frame rate (FPS). user Then the maximum delay time T of the automatic time system is calculated. τ =1 / FPS user When T cur ≥T τ When, if I Higher When rendered, use I. Higher Otherwise use I high .

[0140] It is important to note that for complex scenes, whether the scene has a large number of objects, a large number of object faces, complex lighting, or complex materials, the "criterion two" can be used for judgment.

[0141] The first and second discriminators record the previous configuration result. If factors such as hardware performance, current scenario complexity, and user-controllable parameters remain unchanged, the previous configuration will still be used; otherwise, the second discriminator will be reactivated for further discrimination. For example, the second discriminator records the I used this time. high In the next rendering iteration, if the environmental factors remain unchanged, the first discriminator will not perform multi-path parallel input, but will directly determine the result based on I. high It only uses one input path (i.e., one ray is emitted per pixel, passing through only one rendering pipeline), and does not require multi-path fusion or dual-path parallel input. Instead, it directly generates the final result I through ultra-high-definition frame generation. high (Assuming environmental factors remain unchanged). For example, the second discriminator records the I used this time. Higher In the next rendering iteration, if environmental factors remain unchanged, the first discriminator will generate the final result I through multi-path parallel input, multi-path fusion, and then ultra-high-definition frame generation. Higher .

[0142] After being processed by the second discriminator, the final image is output to the display device.

[0143] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-time ray tracing method for VR large-space immersive tours, characterized in that, The method includes: The system acquires near-eye photos of the user and extracts their features. After feature extraction, a gaze point region segmentation and sampling strategy is used to determine the final image rendering range. The gaze point region segmentation and sampling strategy includes: The main camera screen is divided into three regions: the first gaze region, the second gaze region, and the third gaze region, and the radii of the three gaze regions are calculated based on the eccentricity angle. The ratio of the total number of samples in the first fixation region, the second fixation region, and the third fixation region was set to [value missing]. ; Rays are emitted for each pixel of the first gaze region; rays are emitted for the second gaze region according to... Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N near This represents the number of pixels in the length or width of the superpixel block in the region adjacent to the fixation point; for the third fixation region, according to... Instead of each pixel emitting a ray, a superpixel block is used to emit the ray, where N far The number of pixels in length or width of a superpixel block located far from the point of view; Methods for calculating the radii of the three fixation regions include: in, This refers to the eccentricity angle, measured in degrees; the eccentricity angle of the "first fixation area". ; the eccentricity angle of the "second fixation area" The eccentricity of the "third gaze area" ;d represents the distance from the eye, in centimeters;A represents the display size, in centimeters; Based on the aforementioned rendering area, ray tracing is used for rendering the image; The rendered image is then projected onto a display device for output. Before projecting onto the display device, an ultra-high-definition frame generation step is added to improve the rendering quality. The steps include: using a dual-path parallel input method, the single-path image and the multi-path image are input into the second pre-trained neural network in parallel to obtain their respective results. The single-path image is the pixel color image output by a single ray, and the multi-path image is the image after "multi-path fusion". The output of the second pre-trained neural network is transmitted to the second discriminator. The second discriminator, in conjunction with the first discriminator, adjusts the rendering output based on the hardware performance, the complexity of the current scene, and user-controllable parameters to achieve a balance between rendering performance and immersive effects. The input of the second discriminator also includes the image directly input after multi-path fusion.

2. The real-time ray tracing method for VR large-space immersive tours according to claim 1, characterized in that, The near-eye photo is provided by the near-eye camera of the VR device; the near-eye photo is input into the first pre-trained neural network for feature extraction.

3. The real-time ray tracing method for VR large-space immersive tours according to claim 1, characterized in that, Before projecting a ray, bounding box detection is performed to check whether the ray's initial position in screen space falls within the boundary of its region. Bounding box detection is a binary operation; if the ray does not intersect with the bounding box of the virtual scene, it is discarded; otherwise, the ray is preserved to propagate further.

4. A real-time ray tracing system for VR large-space immersive tours, the system being used to implement the method described in any one of claims 1-3, characterized in that, include: Input module, rendering module, and output module.

Images