Simulator real-time rendering scene generation method and system

Abstract

The invention provides a simulation simulator real-time rendering scene generation method and system. The method relates to the technical field of computer graphics and virtual reality, and comprises the following steps: S1, receiving geometric model data and texture data of a simulation scene, and segmenting the scene into a plurality of logic sectors based on a target display resolution and a field of view; and S2, acquiring a rotation attitude parameter output by a simulation platform attitude sensor, and calculating and updating a perspective projection matrix corresponding to the logic sector in real time by combining the center azimuth angle of each sector. According to the simulation simulator real-time rendering scene generation method and system, based on end-to-end hardware clock synchronization of a precision time protocol, tearing-free output of multiple projection surfaces can be achieved without additional frame buffer transmission, synchronization delay and hardware investment are remarkably reduced, the total cost of ownership is greatly reduced, and the mobile deployment capacity is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics and virtual reality, in particular to a simulation simulator real-time rendering scene generation method and system. BACKGROUND

[0002] The existing simulation simulator usually adopts two types of real-time rendering schemes: one is multi-viewport rendering of single GPU, which calculates multiple view cones in the same frame and splices the results to output to a ring screen or multiple projection surfaces; the other is parallel rendering of multi-GPU or rendering cluster, which synchronizes the frame buffers of each node through network or high-speed bus and then synthesizes and displays. The former has a simple pipeline and is easy to deploy, but is limited by the computing power and memory capacity of the graphics card, making it difficult to ensure both image quality and latency at high resolution and high frame rate; the latter can expand performance, but requires precise clock synchronization and high-speed bandwidth support, significantly increasing hardware investment, power consumption and operation complexity. In addition, the ring screen / cylindrical screen display end mostly relies on uniform brightness or static Gamma correction, and the mainstream engine generally uses a fixed perspective projection matrix, lacking quick adaptation to dynamic postures such as rotating tables and six-degree-of-freedom seats.

[0003] However, the existing technology still has many deficiencies: it is difficult to achieve high-resolution, low-latency 360° multi-sector real-time rendering in a single machine or lightweight hardware environment, and the computing power utilization rate is low. In view of these defects, the industry urgently needs a simulation simulator real-time rendering scene generation method and system with low latency, high consistency and controllable hardware cost. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a simulation simulator real-time rendering scene generation method and system, which solves the problem of how to achieve high-resolution, low-latency real-time panoramic rendering of a simulation simulator through scene sectorization, gaze-driven variable resolution parallel rendering, projection matrix preloading and end-to-end hardware clock synchronization.

[0005] To achieve the above purpose, the present application is implemented by the following technical scheme: a simulation simulator real-time rendering scene generation method, comprising: S1. receiving geometric model data and texture data of a simulation scene, and dividing the scene into a plurality of logical sectors based on a target display resolution and a field of view range; S2. obtaining a rotating posture parameter output by a simulation platform posture sensor, and combining a central azimuth angle of each sector to calculate and update a perspective projection matrix of the corresponding logical sector in real time; S3. setting an infrared eye tracking module, obtaining a user gaze point coordinate in real time through the infrared eye tracking module, and further dividing the logical sector into a gaze area and a peripheral area, the gaze area being rendered by full-resolution rasterization, and the peripheral area being rendered by variable-resolution ray tracing based on depth information; S4. Obtain the color buffer and depth buffer of the logical sector rendering output, and generate a panoramic video frame by seamlessly splicing the sector boundaries through a seamless splicing module; S5. Output the panoramic video frame to a ring-shaped or multi-projection surface display unit without tearing through an end-to-end hardware clock synchronization mechanism.

[0006] Preferably, the real-time calculation process in S2 establishes an independent perspective projection matrix cache queue for each logical sector, and preloads the perspective projection matrix of the next frame in the GPU asynchronous command stream to reduce the blocking of the rendering pipeline caused by the change in pose.

[0007] Preferably, the rendering process dynamically schedules rendering sub-tasks for parallel execution through a GPU task graph to reduce the overall pixel shading workload while maintaining the clarity of the core view area, thereby achieving high resolution and low latency.

[0008] Preferably, the division of the logical sector is performed by calculating the gaze weight of the i-th pixel with screen coordinates According to the following model formula, the gaze weight is determined: wherein, is the Euclidean distance between pixel i and the real-time gaze point is a scale coefficient for adjusting the radius of the gaze area, is an exponential factor for controlling the decay rate and the sharpness of the core-edge transition, and when is a preset threshold, pixel i is classified into the gaze area, otherwise it is classified into the peripheral area. Preferably, when performing variable resolution ray tracing on the peripheral area pixel i, the number of emitted rays

[0009] is adaptively adjusted according to the following formula wherein, is the maximum number of rays used in the gaze area, is the minimum number of rays allowed in the peripheral area, is an exponential coefficient for controlling the smoothness of the ray decay curve, is the gaze weight of the i-th pixel.

[0010] Preferably, the rendering is scheduled through a graphics processor task graph, and the gaze area task is given a higher real-time priority than the peripheral area task to ensure the frame time stability of the core view.

[0011] ​​Preferably, the end-to-end hardware clock synchronization mechanism adopts an end-to-end frame-level synchronization mode based on the precision time protocol, so that the time deviation of the source clock and the target clock is not more than 100 µs.

[0012] A simulation simulator real-time rendering scene generation system comprises: A scene sector division module is configured to receive geometric model data and texture data of a simulation scene, and divide the scene into a plurality of logical sectors based on a target display resolution and a field of view range; A projection matrix generation module is configured to obtain rotation attitude parameters output by an attitude sensor of a simulation platform, and calculate and update a perspective projection matrix of each logical sector in real time in combination with a central azimuth angle of the logical sector; A rendering module is integrated with an infrared eye movement tracker and a graphics processing unit, the infrared eye movement tracker is configured to obtain a user gaze point coordinate in real time, and the graphics processing unit adopts GPU task graph dynamic scheduling, divides each logical sector into a gaze area and a peripheral area, performs full-resolution rasterization rendering on the gaze area, and performs variable-resolution ray tracing rendering on the peripheral area based on depth information, so as to reduce the overall pixel shading workload while ensuring the clarity of the core view area; A splicing module is configured to receive color buffer and depth buffer output by rendering of each logical sector, and seamlessly splices the color buffer and the depth buffer to generate a panoramic video frame; A clock synchronization output module is configured to output the panoramic video frame to a ring-shaped or multi-projection surface display unit without tearing through an end-to-end hardware clock synchronization mechanism based on the precision time protocol.

[0013] The application provides a simulation simulator real-time rendering scene generation method and system. The simulation simulator real-time rendering scene generation method and system combine full-resolution rasterization of a user visual focus area and variable-resolution ray tracing of a peripheral area through gaze-driven variable-resolution parallel rendering, reduce the pixel shading load outside the core view area, and achieve a balance between high resolution and low latency; at the same time, the blocking of a simulation platform attitude change on a rendering pipeline is reduced to a minimum through an independent perspective projection matrix cache queue and a GPU asynchronous command stream preloading mechanism, so that the response to view angle switching is rapid and smooth; in addition, end-to-end hardware clock synchronization based on the precision time protocol can realize multi-projection surface non-tearing output without additional frame buffer transmission, and significantly reduces synchronization delay and hardware investment.

[0014] The system of the present application realizes high decoupling of each function in a modular architecture, and scene sector division, projection matrix calculation, gaze point capture and rendering scheduling, splicing and synthesis and clock synchronization output each perform its own function in parallel cooperation, facilitating subsequent expansion and maintenance; the built-in GPU task graph dynamic scheduling mechanism can intelligently adjust the rendering priority according to the runtime gaze point and system load, ensuring stable frame time of the core view area; at the same time, the splicing module integrates multi-band wavelet luminance fusion and adaptive color correction, which can maintain picture consistency under the conditions of projector aging or environmental light changes; more importantly, the system does not need to rely on large-scale rendering clusters, and a single machine or small-scale hardware platform can already meet the immersive simulation requirements, greatly reducing the total cost of ownership and improving the mobile deployment capability. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of the implementation of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0017] Embodiment one

[0018] As shown in Figure 1 , the present application provides a simulation simulator real-time rendering scene generation method, which comprises: S1. receiving geometric model data and texture data of a simulation scene, and dividing the scene into a plurality of logical sectors based on a target display resolution and a field of view range.

[0019] S2. obtaining rotation attitude parameters output by an attitude sensor of a simulation platform, combining the center azimuth angle of each sector, and real-time calculating and updating the perspective projection matrix of the corresponding logical sector. The real-time calculation process establishes an independent perspective projection matrix cache queue for each logical sector, and preloads the perspective projection matrix of the next frame in the GPU asynchronous command stream, so as to reduce the blocking of the rendering pipeline caused by attitude changes.

[0020] The specific implementation is as follows: Hardware and preset parameters: The simulation platform is a six-degree-of-freedom dynamic cockpit, and the attitude sensor includes a three-axis gyroscope, an accelerometer and an optical encoder, with an output frequency of 1000 Hz and a measurement accuracy of 0.01 degrees.

[0021] The display resolution is 8192x2048, there are 8 horizontal logical sectors, and the single-sector horizontal field of view is 45 degrees.

[0022] The perspective matrix buffer queue has a depth of 3 frames.

[0023] The rendering platform uses an RTX 4090 and supports a two-level command buffer.

[0024] Data Acquisition and Fusion: IMU and encoder data are collected every millisecond.

[0025] Quaternion Kalman filtering is used to fuse the original sensor data, and the filtering time is 0.2 milliseconds.

[0026] Output the current quaternion after fusion.

[0027] Projection matrix generation and caching: The center azimuth angles of the eight sectors are preset to 22.5 degrees, 67.5 degrees, 112.5 degrees, 157.5 degrees, 202.5 degrees, 247.5 degrees, 292.5 degrees and 337.5 degrees.

[0028] The fused quaternion is transformed into a 3x3 rotation matrix and multiplied by the camera intrinsic matrix with a focal length of 4000 pixels and the principal point centered.

[0029] Calculate the four-row, four-column perspective projection matrix for each logical sector in each frame.

[0030] The latest three-frame matrix of each sector is stored sequentially into a buffer queue with a depth of three.

[0031] The matrix for the next two frames is asynchronously preloaded in the GPU's secondary command buffer, with each preload taking 0.3 milliseconds.

[0032] This solution reduces the average blocking time of the rendering pipeline due to matrix updates from 5 milliseconds to 1.2 milliseconds.

[0033] Real-time queue maintenance and matrix switching: When rendering starts, the GPU reads the current frame matrix directly from the cache queue.

[0034] After rendering is complete, the queue automatically moves forward one position and continues to preload the matrix of the new frame at the end.

[0035] The loop execution ensures that the queue is always full, keeping the response latency of the rendering pipeline to pose changes within 1.5 milliseconds.

[0036] S3. Set up an infrared eye tracking module, real-time acquire user gaze point coordinates through the infrared eye tracking module, and further divide the logical sector into a gaze area and a peripheral area, the gaze area uses full-resolution rasterization rendering, and the peripheral area uses variable-resolution ray tracing rendering based on depth information. The rendering process dynamically schedules rendering tasks in parallel execution through GPU task graph, to reduce the overall pixel shading workload while maintaining the clarity of the core view area, thereby achieving high resolution and low latency.

[0037] The division of the logical sector is obtained by calculating the gaze weight of the i-th pixel with screen coordinates The gaze weight is determined according to the following model formula: wherein, is the Euclidean distance between pixel i and real-time gaze point is a scale coefficient for adjusting the radius of the gaze area, is an exponential factor for controlling the decay rate and the sharpness of the core-edge transition, when the preset threshold, pixel i is divided into the gaze area, otherwise into the peripheral area.

[0038] When variable-resolution ray tracing is performed on the peripheral area pixel i, the number of emitted rays is adaptively adjusted according to the following formula.

[0039] wherein, is the maximum number of rays used in the gaze area, is the minimum number of rays allowed in the peripheral area, is an exponential coefficient for controlling the smoothness of the ray decay curve, is the gaze weight of the i-th pixel.

[0040] The rendering is scheduled through a graphics processor task graph, the gaze area task is given a higher real-time priority than the peripheral area task, to ensure the frame time stability of the core view.

[0041] The specific implementation is as follows: Application scenario: industrial cockpit training system.

[0042] Display and sector division: The resolution is selected as 4096x1024, the horizontal field of view is divided into 4 logical sectors, and each sector covers 90 degrees.

[0043] An overlap area of 10 pixels is reserved at the sector boundary to reduce the splicing gap.

[0044] Eye tracking and gaze area division: ​​The infrared eye tracking module adopts a 300Hz sampling rate and a measurement accuracy of 0.2 degrees.

[0045] The gaze weight is based on the number of pixels from the gaze point, with a gaze area radius σ = 80 pixels, a transition sharpness index v = 2, and a threshold δ = 0.5.

[0046] The actual measured gaze area diameter is about 160 pixels, covering key areas such as the driver's dashboard and steering wheel.

[0047] Rendering strategy and performance: The gaze area uses full-resolution rasterization rendering, with about 40 million pixels colored per frame.

[0048] The peripheral area performs adaptive ray tracing based on depth information, with the number of rays decaying to about 32-128.

[0049] On the RTX3060 platform, the GPU task graph dynamically schedules gaze area and peripheral area rendering sub-tasks, with the gaze area task priority set to real-time level and the peripheral area task to background level.

[0050] After optimization, the pixel coloring amount is reduced by about 38%, the frame rate is increased from the original 75fps to 90fps, and the gaze area frame rendering time is stable within 7ms.

[0051] Visual effects and user experience: There is no significant difference in the clarity of perception when the driver operates the instrument and observes the environment.

[0052] Although the resolution of the peripheral scene is slightly reduced, it has no effect on the driving perception of peripheral objects, and the student feedback is smooth and not easy to fatigue.

[0053] S4. Obtain the color buffer and depth buffer of the logical sector rendering output, and combine them according to the sector boundary through the seamless splicing module to generate a panoramic video frame.

[0054] S5. The panoramic video frame is output to the ring or multi-projection surface display unit without tearing through the end-to-end hardware clock synchronization mechanism. The end-to-end hardware clock synchronization mechanism uses an end-to-end frame-level synchronization method based on the Precision Time Protocol, so that the time deviation between the source clock and the target clock is not more than 100µs.

[0055] A simulation simulator real-time rendering scene generation system, comprising: A scene sector division module is used to receive geometric model data and texture data of a simulation scene, and divide the scene into a plurality of logical sectors based on a target display resolution and a field of view range.

[0056] A projection matrix generation module is configured to acquire a rotation attitude parameter output by an attitude sensor of the simulation platform, and in combination with a center azimuth angle of each logical sector, to calculate and update a perspective projection matrix of the corresponding logical sector in real time.

[0057] A rendering module is integrated with an infrared eye movement tracker and a graphics processing unit. The infrared eye movement tracker is configured to acquire a user gaze point coordinate in real time, and the graphics processing unit is configured to divide each logical sector into a gaze area and a peripheral area by using GPU task graph dynamic scheduling. The graphics processing unit is configured to perform full-resolution rasterization rendering on the gaze area and perform variable-resolution ray tracing rendering on the peripheral area based on depth information, so as to reduce the overall pixel shading workload while ensuring the clarity of the core view area.

[0058] A splicing module is configured to receive a color buffer and a depth buffer output by rendering of each logical sector, and to seamlessly splice the color buffer and the depth buffer to generate a panoramic video frame.

[0059] A clock synchronization output module is configured to output the panoramic video frame to a ring-shaped or multi-projection surface display unit without tearing by using an end-to-end hardware clock synchronization mechanism based on a precision time protocol.

[0060] Embodiment Two

[0061] Different from the first embodiment, the application scenario of the embodiment is a large-scale ring-shaped immersive display platform.

[0062] System scale and display parameters: The resolution of the ring-shaped screen is set to 8192x2048, and the screen is horizontally divided into 8 sectors, each sector having a field of view of about 45 degrees.

[0063] The splicing areas are staggered and overlapped by 20 pixels, and a multi-band wavelet fusion is applied to smooth the brightness.

[0064] High-precision eye movement and gaze area setting: The sampling rate of the infrared eye movement tracking module is 500 Hz, and the measurement accuracy is 0.05 degrees.

[0065] The gaze area has a radius σ=200 pixels, a transition index v=3, and a threshold δ=0.7. The diameter of the gaze area is about 400 pixels, covering the main line of sight of the user.

[0066] Rendering performance scheduling: The gaze area is rendered by full-resolution rasterization, and the number of pixels per frame exceeds 160 million.

[0067] The peripheral area is rendered by variable-resolution ray tracing, and the number of rays is dynamically adjusted between 16 and 256 according to the weight.

[0068] In the RTX4090 cluster environment, the GPU task graph scheduling framework distributes the fovea tasks to the master GPU and the peripheral tasks to the slave GPUs for parallel execution.

[0069] The actual frame rate is maintained at 120fps, and the core fovea frame rendering delay is controlled within 6ms, and the average rendering time of the peripheral area is about 4ms.

[0070] Synchronization and stability: End-to-end frame-level clock synchronization based on PTP is adopted, and the signal delay does not exceed 50 microseconds.

[0071] The spliced output is free of tearing and jitter, and the long-time running brightness and color consistency is better than ΔE00=1.8, which is suitable for high requirement display and training.

[0072] Example three

[0073] Unlike example one, the application scenario of this embodiment is a tactical air combat simulation training platform.

[0074] System architecture and hardware selection: This embodiment builds a 12-sided ring projection domain driven simulation platform for tactical air combat training. The projection resolution is 1920x1200 per side, covering 360° horizontally. The attitude sensor is integrated into a full-motion 6-degree-of-freedom platform, supporting high-speed rolling, pitching, and heading changes. The rendering cluster uses 4 RTX4090s connected by interconnection to handle different sector rendering tasks in parallel. Each GPU uses a two-level command buffer to asynchronously preload and distribute projection matrices and rendering instructions.

[0075] Eye movement capture and sector subdivision: The infrared eye tracking module has a sampling rate of 1000Hz and a measurement accuracy of 0.01°, and can obtain the pilot's gaze point coordinates in real time. The logical sector is further divided into a fovea and a peripheral area based on the original 12 sides. The fovea radius is set to 240 pixels, the transition sharpness factor is 2.5, and the weight threshold is 0.7. The test shows that the fovea covers an area of about 30° in the center of the field of view, effectively surrounding the instrument panel and target area; the peripheral area uses variable resolution ray tracing based on depth, and the number of peripheral area rays is adaptively controlled between 20 and 320 during rendering.

[0076] Rendering task scheduling and performance improvement: GPU task graph dynamic scheduling is adopted, the fovea subtask in the framework is given real-time highest priority, and the peripheral task runs in the secondary queue to ensure millisecond-level response in the key area. Based on this scheme, under the conditions of complex weather, million-level triangle terrain, and dense target aircraft model rendering, the overall pixel shading amount is reduced by about 40% compared with traditional full rasterization, the inter-frame delay is stably controlled within 8ms, while ensuring that the texture details and lighting and shadow in the fovea are lossless.

[0077] Clock synchronization and visual consistency: End-to-end frame-level synchronization is achieved between projectors through precision time protocol, with the source clock and target clock deviation controlled within 80 μs. The seamless splicing module uses multi-band wavelet luminance fusion and real-time correction according to ambient light, so that the splicing seam ΔE00 color difference is always less than 2.5. Under the conditions of all-weather light changes and equipment aging, the picture brightness and color remain highly consistent, significantly improving the pilot's immersion and training credibility.

[0078] Training effect and expansion: In real tactical air combat training, the pilot's reaction time for locking and aiming at the key target is shortened by about 20%, and the identification rate of medium and long-range targets is increased by about 15 percentage points. The system runs stably in the long term and can be extended to various air-sea-land composite simulation scenes, with good universality and upgrading potential.

[0079] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for generating real-time rendering scenes in a simulation simulator, characterized in that, include: S1. Receive the geometric model data and texture data of the simulation scene, and divide the scene into several logical sectors based on the target display resolution and field of view; S2. Obtain the rotational attitude parameters output by the attitude sensor of the simulation platform, and calculate and update the perspective projection matrix of the corresponding logical sector in real time by combining the center azimuth angle of each sector. S3. Set up an infrared eye-tracking module to obtain the coordinates of the user's gaze point in real time through the infrared eye-tracking module, and further divide the logical sector into a gaze area and a peripheral area. The gaze area adopts full-resolution rasterization rendering, and the peripheral area adopts variable-resolution ray tracing rendering based on depth information. S4. Obtain the color buffer and depth buffer of the logical sector rendering output, and stitch them together according to the sector boundary through the seamless stitching module to generate panoramic video frames; S5. Output the panoramic video frame to the ring or multi-projection surface display unit without tearing through an end-to-end hardware clock synchronization mechanism.

2. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: The real-time computing process described in S2 establishes an independent perspective projection matrix cache queue for each logical sector and preloads the perspective projection matrix of the next frame in the GPU asynchronous command stream.

3. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: The rendering process utilizes the GPU task graph to dynamically schedule rendering tasks for parallel execution.

4. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: The logical sector division is achieved by dividing the screen coordinates as follows: Calculate gaze weight for the i-th pixel Therefore, the gaze weight is determined according to the following model formula: in, For pixel i and real-time gaze point Euclidean distance, To adjust the scale factor of the fixation zone radius, To control the exponential factor of decay rate and core-edge transition sharpness, when When the preset threshold is met, pixel i is included in the gaze area; otherwise, it is included in the peripheral area.

5. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: When performing variable resolution ray tracing on pixel i in the surrounding area, the number of emitted rays Adaptive adjustment based on the following formula in, The maximum number of rays used in the fixation area. The minimum number of rays allowed in the surrounding area. The exponential coefficient used to control the smoothness of the ray attenuation curve, Let be the gaze weight for the i-th pixel.

6. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: The rendering is scheduled via a graphics processor task graph, and the gaze area task is given a higher real-time priority than the surrounding area task.

7. The method for generating a real-time rendering scene in a simulation simulator according to claim 1, characterized in that: The end-to-end hardware clock synchronization mechanism adopts an end-to-end frame-level synchronization method based on a precision time protocol.

8. A simulation simulator real-time rendering scene generation system, characterized in that, include: The scene sector division module is used to receive geometric model data and texture data of the simulation scene, and divide the scene into several logical sectors based on the target display resolution and field of view. The projection matrix generation module is used to obtain the rotational attitude parameters output by the attitude sensor of the simulation platform, and calculate and update the perspective projection matrix of the corresponding logical sector in real time by combining the center azimuth angle of each logical sector. The rendering module integrates an infrared eye tracker and a graphics processing unit. The infrared eye tracker is used to acquire the coordinates of the user's gaze point in real time. The graphics processing unit adopts dynamic scheduling of the GPU task graph to divide each logical sector into a gaze area and a peripheral area. The graphics processing unit performs full-resolution rasterization rendering on the gaze area and performs variable-resolution ray tracing rendering based on depth information on the peripheral area. The stitching module is used to receive the color buffer and depth buffer of each logical sector rendering output, and seamlessly stitch them together according to the sector boundaries to generate panoramic video frames. The clock synchronization output module is used to output the panoramic video frames seamlessly to the ring or multi-projection surface display unit through an end-to-end hardware clock synchronization mechanism based on a precision time protocol.

Images