Method and system for optimizing three-dimensional scene rendering based on spatiotemporal prediction and dynamic level of detail (LOD) in cooperation with view frustum culling

Abstract

The application relates to the field of scene rendering, and particularly discloses a three-dimensional scene optimization method and system for dynamic LOD and view frustum culling collaborative rendering based on space-time prediction, which comprises the following steps: dynamically calculating the visibility frequency of an object, dynamically adjusting the loading and rendering mode of different priority objects, predicting the view frustum range of multiple future frames by using an LSTM network architecture, dividing the scene into uniform grid blocks, then performing hierarchical culling, optimizing a heterogeneous computing pipeline, and performing a rendering execution process; the system comprises a dynamic LOD and view frustum culling collaborative optimization module, an LSTM view frustum prediction module, a block-level hybrid culling module, a heterogeneous computing pipeline module, an optimization CPU-GPU task allocation and data transmission module, and a rendering execution control module. The dynamic LOD and view frustum culling collaborative optimization, the block-level hybrid culling and the heterogeneous computing pipeline optimization technology can reduce unnecessary rendering calculation and resource loading, and improve the rendering efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene rendering, in particular to a three-dimensional scene optimization method and system for collaborative rendering of dynamic LOD and view frustum culling based on space-time prediction. BACKGROUND

[0002] With the rapid development of three-dimensional graphics technology, real-time rendering of large-scale three-dimensional scenes has become a core demand in the fields of digital content creation, virtual simulation, etc. The current mainstream three-dimensional scene rendering system mainly relies on multi-level detail hierarchy (LOD) and view frustum culling technology to optimize GPU load and improve frame rate, but there are still significant bottlenecks in dynamic scene processing, hardware resource scheduling, and predictive optimization.

[0003] 1. Hardware resource scheduling mismatch problem, the traditional rendering system has serious deficiencies in CPU and GPU task allocation strategy, which fails to fully consider the dynamic characteristics of the scene, resulting in frequent blocking of rendering threads by resource loading threads, causing frame rate fluctuations, PCIe bus congestion due to frequent transmission of LOD model data, unbalanced CPU and GPU computing load, and inability to fully utilize heterogeneous computing advantages. The existing management strategy is static and cannot adapt to dynamic scene requirements.

[0004] 2. Limitations of traditional view frustum culling technology, which uses frame-by-frame full calculation of occlusion relationships, fails to utilize the space-time continuity characteristics of camera motion, resulting in a large amount of unnecessary computational overhead. Experimental data shows that in a typical city scene, about 35% to 45% of the removal calculations of the traditional method are redundant operations. Potential visible objects cannot be preloaded, and when the camera moves quickly, objects often "pop out" suddenly, seriously affecting user experience. Especially in VR environments, this problem can exacerbate user dizziness.

[0005] 3. Deficiencies in LOD switching mechanism, improper handling of high-frequency visible objects: high-frequency visible objects such as building groups on both sides of city main roads are prone to cause frame rate to drop sharply (typically causing a frame time peak of 20-30ms) when high-precision models are suddenly loaded. Low-frequency visible object resource waste: long-range mountains and other low-frequency visible objects occupy high-precision video memory resources for a long time, resulting in a decrease in video memory utilization. Static threshold problem: using a fixed distance threshold for LOD switching cannot adapt to different scene characteristics and hardware configurations.

[0006] Therefore, there is an urgent need for a three-dimensional scene optimization method and system for collaborative rendering of dynamic LOD and view frustum culling based on space-time prediction to solve the above problems. SUMMARY

[0007] The purpose of the present application is to solve the shortcomings of sudden frame rate drop, redundant view frustum culling calculation and insufficient hardware resource utilization caused by LOD switching in the prior art three-dimensional scene rendering. The three-dimensional scene optimization method and system for dynamic LOD and view frustum culling collaborative rendering based on space-time prediction provided by the present application realizes the above-mentioned rendering frame rate improvement and memory occupation reduction, and eliminates sudden stuttering through adaptive loading strategy driven by space-time prediction, block-level hybrid culling architecture and heterogeneous computing pipeline optimization.

[0008] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] The three-dimensional scene optimization method for dynamic LOD and view frustum culling collaborative rendering based on space-time prediction comprises:

[0010] S1: Dynamic LOD and view frustum culling collaborative optimization: the visibility frequency of the object is dynamically calculated through historical data and current frame information, and then the loading and rendering mode of the object of different priority is dynamically adjusted according to the visibility frequency of the object and the system memory usage;

[0011] S2: View frustum prediction based on LSTM: first, data preprocessing is performed, then the LSTM network architecture is designed according to the specific application scene and prediction requirements, the LSTM network architecture is trained and optimized, and finally the view frustum range of future multiple frames is predicted by using the LSTM network architecture;

[0012] S3: Block-level hybrid culling: first, the scene is divided into uniform grid blocks (such as 10m×10m×10m), and then hierarchical culling is performed;

[0013] S4: Heterogeneous computing pipeline optimization:

[0014] S41: In terms of GPU computing optimization, Compute Shader is used to perform view frustum culling and LOD switching operation. Compute Shader has high flexibility and parallel computing capability, and can efficiently process large-scale culling and LOD switching tasks. At the same time, Wave Intrinsics technology is used to more finely control the execution flow of threads, further optimize parallel efficiency, reduce thread synchronization overhead, and improve the utilization rate of GPU resources;

[0015] S42: In terms of CPU multi-threaded resource loading management, shared memory pool is used to reduce data transmission. The shared memory pool uniformly manages model resources, adopts LRU (Least Recently Used) cache replacement strategy to ensure that frequently used model resources can be quickly accessed. At the same time, atomic operation is used to ensure thread safety, avoiding data inconsistency when multiple threads access the shared memory pool at the same time;

[0016] S5: Rendering Execution Process: The rendering process is executed sequentially according to steps S2-S4.

[0017] As a further technical solution of the present invention, S1 specifically includes:

[0018] S11: Visibility Probability Statistics: The visibility probability is calculated using the following formula:

[0019] ,

[0020] in: For visibility probability, This represents the total number of visible frames of an object in the history of frames. Indicates the total number of frames. It is the visibility frequency calculated from the previous frame. It is the attenuation coefficient, the attenuation coefficient Its function is to control the degree of influence of historical data, when When the data is large, historical data has a significant impact on the current visibility frequency, making changes in the visibility frequency smoother and avoiding frequency abrupt changes due to anomalies in a single frame; when When the current frame is smaller, the information in the current frame has a greater impact on the visibility frequency, enabling a faster response to dynamic changes in the scene. In practical implementation, it is necessary to determine the appropriate frequency through experimentation and adjustment based on different scene characteristics and requirements. value;

[0021] S12: Hierarchical LOD loading strategy: For high-frequency visible objects ( >0.7), implement a preloading strategy, loading high-precision LOD models 1-2 frames in advance and ensuring that the models are not replaced in video memory, while giving them the highest rendering priority. This ensures that high-frequency visible objects can be rendered quickly, reducing loading latency and improving rendering smoothness; for mid-frequency objects (0.3≤... ≤0.7), a background thread is used for on-demand loading, automatically adjusting the cache size based on memory pressure. When memory is tight, the number of cached models is reduced appropriately, and when memory is sufficient, the cache size is increased to improve model loading speed. Medium rendering priority ensures that mid-frequency objects are handled appropriately during rendering, preventing stuttering due to loading delays and avoiding excessive VRAM consumption. For low-frequency objects (…),… < 0.3), using the lowest detail level, and loading last when memory is tight. It can be preempted by high-priority tasks. When system memory usage > 85%, adaptive adjustment is triggered, automatically downgrading the memory level. ,in: For low-frequency objects, the loading coefficient is... Memory usage rate; in this way, it can be ensured that high-priority objects have priority to obtain video memory resources, and the system automatically degrades under video memory pressure to avoid performance collapse caused by video memory overflow; in terms of memory management, the memory usage of the system needs to be monitored in real time, and video memory resources are dynamically allocated and released according to the loading strategy of different priority objects; at the same time, an effective model resource index and cache mechanism needs to be established to quickly find and load the required model.

[0022] As a further technical solution of the application, S2 specifically comprises:

[0023] S21: data preprocessing: data preprocessing is a key link of the LSTM view prediction module, which directly affects the training effect and prediction accuracy of the model; inputting continuous 5-10 frames of camera 6DOF parameters (position x / y / z + rotation pitch / yaw / roll) can provide spatiotemporal continuity information of camera motion, so that the model can learn the law of camera motion; first, batch processing data is used to improve the training efficiency, and the batch size is 64 samples; at the same time, batch normalization operation is used to map data of different dimensions and ranges to a unified interval, reduce the sensitivity of the model to initial parameters, and improve the generalization ability of the model; the formula of the normalization operation is:

[0024] ,

[0025] Among them: is the mean of the past 30 frames, is the standard deviation of the past 30 frames, is the original camera 6DOF parameter (position x / y / z + rotation pitch / yaw / roll), with a dimension of 6, After the normalization operation, the data, the normalization operation makes it easier for the model to learn the features in the data, and the normalization can not only improve the training speed of the model, but also prevent the problem of gradient vanishing or explosion caused by too large or too small data range;

[0026] In practical application, the data preprocessing also needs to consider the noise and outliers of the data, and finally uses a filtering algorithm to smooth the camera parameters and remove noise interference; for outliers, interpolation or replacement methods are used for processing to ensure the quality of the input data;

[0027] S22: LSTM network architecture: the input layer is 6-dimensional camera parameters, with a batch size of 64, which can fully receive the information of camera motion; the hidden layer uses 2 layers of LSTM, with 64 neurons in each layer, which can capture the complex spatio-temporal features of camera motion; the setting of Dropout=0.2 can effectively prevent overfitting, by randomly discarding a part of neurons, making the model pay more attention to important features during the learning process, and improving the generalization ability of the model; the output layer is the 8-vertex coordinates of the future 3 frames of the viewing cone (24-dimensional vector), which can accurately predict the future viewing cone range of the camera;

[0028] S23: Training optimization: in terms of training optimization, the loss function is:

[0029] ,

[0030] where: is the predicted segmentation region of the model; is the true segmentation label, this part of the loss function measures the difference between the predicted result and the true result, and improves the prediction accuracy by maximizing the Intersection over Union (IoU); is the regularization term, which controls the strength of regularization, prevents model overfitting, and makes the model maintain good performance on both training data and test data;

[0031] S24: Prediction process algorithm steps:

[0032] S241: Input and output:

[0033] Input: 5-10 consecutive frames of camera 6DOF parameter sequence (sliding window)

[0034] (length=10 frames)

[0035] Output: the 8-vertex coordinates of the viewing cone of the future 3 frames

[0036] ,

[0037] where each contains the coordinates of 8 vertices;

[0038] S242: Execution process:

[0039] Data preprocessing: normalize the input sequence frame by frame (use the past 30 frames of ); outlier processing: replace the sudden data with linear interpolation; smoothing filter: apply Kalman filter to the camera trajectory;

[0040] LSTM inference: ​

[0041] Input dimensions: (time step feature)

[0042] LSTM layer computation (simplified formula):

[0043] (forget gate)

[0044] (input gate)

[0045] (output gate)

[0046] (candidate memory)

[0047] (new memory)

[0048] (hidden state)

[0049] where: is the hidden state vector of the previous time step; is the memory state vector of the previous time step; is the concatenated vector (64+6); , , , are the forget gate weights, input gate weights, output gate weights, and candidate memory weights, respectively; , , , are the bias vectors; is the Sigmoid activation function: ; is the hyperbolic tangent activation function: ; is the element-wise multiplication; is the forget gate output, which determines how much old memory to keep ; is the input gate output, which determines how much new candidate memory to receive ; is the output gate output, which determines how much memory to expose to the hidden state; is the candidate memory state; is the updated memory state: new memory = forget old memory + remember new information; is the current hidden state output;

[0050] Output interpretation: Split the 24-dimensional output vector into 3 groups of 8 vertex coordinates and map them to the world coordinate system (need to combine with the camera projection matrix).

[0051] As a further technical solution of the present application, in S3, each grid block contains a static object list, a dynamic object list and occlusion metadata; the static object list records the relevant information of all static objects in the grid block, such as model ID, position, rotation, etc.; the dynamic object list updates the state of dynamic objects in the grid block in real time; and the occlusion metadata is the key to improve the occlusion efficiency.

[0052] As a further technical solution of the present application, the scene is divided into uniform grid blocks, which specifically includes:

[0053] S311: For a static scene, a pre-baked Occlusion Map is loaded, and a PVS (Potential Visible Set) is used to optimize the potential visible set; the Occlusion Map records the occlusion relationship of different regions in the scene, and by querying the Occlusion Map, it can be quickly judged whether some objects are occluded or not, thereby avoiding unnecessary rendering calculation; the PVS further optimizes the visibility judgment, only retaining objects that may be visible, and reducing the size of the rendering queue;

[0054] S312: In terms of dynamic object management, after the scene is divided into grid blocks, only the blocks in the camera motion direction are subjected to accurate occlusion; for the blocks in the non-motion direction, a simpler occlusion strategy or reduced occlusion accuracy is adopted to reduce the calculation overhead; at the same time, the positions and states of dynamic objects are updated in real time to ensure the accuracy of the occlusion metadata.

[0055] As a further technical solution of the present application, the hierarchical occlusion specifically includes:

[0056] S321: Coarse occlusion: based on the fast intersection test of block AABB, in the occlusion process, a coarse occlusion stage is first performed, a fast intersection test is performed based on the block's AABB (Axis Aligned Bounding Box), a parallel processing mode is adopted, each thread processes a certain number of blocks to improve processing efficiency; through coarse occlusion, the blocks that may be visible are quickly screened out to generate a visible block list;

[0057] S322: Fine occlusion: entering the fine occlusion stage, object-by-object occlusion query is performed on the objects in the visible block, HiZ occlusion removal technology is adopted, and GPU Compute Shader is used for accelerated processing; the Compute Shader fully utilizes the parallel computing capability of the GPU to perform efficient occlusion judgment on each object, and finally generates a final visible object list.

[0058] As a further technical scheme of the present application, in the hierarchical culling process, if the camera motion speed is greater than the set threshold, the block culling calculation in the non-motion direction is skipped, the LOD level of these blocks is reduced, and they are marked as possibly visible to reduce unnecessary calculation overhead.

[0059] As a further technical scheme of the present application, in the S4, the CPU and the GPU share a memory region through a PCIe bus, avoiding frequent data copying operations, further reducing communication overhead, and improving the overall performance of the system.

[0060] As a further technical scheme of the present application, the S5 specifically includes:

[0061] S51: In the prediction stage, the LSTM network is called every frame, and the continuous multi-frame camera 6DOF parameters are input. After network inference, the future 3 frames of the view cone range are output. This prediction result provides an important basis for subsequent resource preloading;

[0062] S52: In the resource preloading stage, according to the prediction result and the visibility frequency of the object (Vf) ), asynchronously load the LOD model with high visibility probability. By loading high-precision models in advance, the loading time during rendering can be reduced, and sudden stuttering can be avoided;

[0063] S53: In the culling and rendering stage, the CPU first performs coarse culling operation to quickly filter the blocks in the scene and output the visible block list. Then, the GPU performs HiZ fine culling to perform object-by-object occlusion query on the objects in the visible block and generates the final rendering queue. For low-frequency objects (Vf <0.3), the GPU Instancing batch rendering technology is enabled to combine multiple identical object instances into one rendering batch, reduce the number of rendering calls, and improve rendering efficiency.

[0064] The three-dimensional scene optimization system based on dynamic LOD and view cone culling collaborative rendering of space-time prediction is used to realize a three-dimensional scene optimization method based on dynamic LOD and view cone culling collaborative rendering of space-time prediction, which includes:

[0065] Dynamic LOD and view cone culling collaborative optimization module: realizes visibility probability statistics and hierarchical LOD loading;

[0066] LSTM view cone prediction module: predicts the view cone range of multiple future frames;

[0067] Block-level hybrid culling module: realizes coarse and fine two-level culling optimization;

[0068] Heterogeneous computing pipeline module: optimizes CPU-GPU task allocation and data transmission;

[0069] Render execution control module: coordinate the timing and data flow of each module.

[0070] The beneficial effects of the present application are:

[0071] 1. By dynamically adjusting the LOD loading priority (high-frequency object preloading) and blocking-level culling, the sudden stuttering problem is solved.

[0072] 2. At the same time, by predicting the future 3 frames of the viewing frustum range through a lightweight LSTM, and introducing a volume constraint loss function, the prediction accuracy is significantly improved.

[0073] 3. Simply combining LOD and culling techniques can cause memory overflow, and the shared memory pool design of the present application can effectively reduce the CPU-GPU communication overhead. BRIEF DESCRIPTION OF DRAWINGS

[0074] Fig. 1 is the overall system architecture diagram of the present application;

[0075] Fig. 2 is the dynamic LOD and frustum culling collaborative flowchart of the present application;

[0076] Fig. 3 is the LSTM network architecture diagram of the present application. DETAILED DESCRIPTION

[0077] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the present application will be further described below in conjunction with specific embodiments.

[0078] Please refer to Figs. 1-3 , the three-dimensional scene optimization method based on spatiotemporal prediction and dynamic LOD and frustum culling collaborative rendering, comprising:

[0079] S1: Dynamic LOD and frustum culling collaborative optimization: through historical data and current frame information, the visibility frequency of the object is dynamically calculated, and then according to the visibility frequency of the object and the system memory usage, the loading and rendering mode of different priority objects is dynamically adjusted;

[0080] S11: Visibility probability statistics: the visibility probability is calculated by the following formula:

[0081] ,

[0082] Wherein: is the visibility probability, represents the total number of visible frames of the object in the historical frames, represents the total number of frames, is the visibility frequency calculated in the last frame, is the decay coefficient, and the decay coefficient Its function is to control the degree of influence of historical data, when When the data is large, historical data has a significant impact on the current visibility frequency, making changes in the visibility frequency smoother and avoiding frequency abrupt changes due to anomalies in a single frame; when When the current frame is smaller, the information in the current frame has a greater impact on the visibility frequency, enabling a faster response to dynamic changes in the scene. In practical implementation, it is necessary to determine the appropriate frequency through experimentation and adjustment based on different scene characteristics and requirements. value;

[0083] S12: Hierarchical LOD loading strategy: For high-frequency visible objects ( >0.7), implement a preloading strategy, loading high-precision LOD models 1-2 frames in advance and ensuring that the models are not replaced in video memory, while giving them the highest rendering priority. This ensures that high-frequency visible objects can be rendered quickly, reducing loading latency and improving rendering smoothness; for mid-frequency objects (0.3≤... ≤0.7), a background thread is used for on-demand loading, automatically adjusting the cache size based on memory pressure. When memory is tight, the number of cached models is reduced appropriately, and when memory is sufficient, the cache size is increased to improve model loading speed. Medium rendering priority ensures that mid-frequency objects are handled appropriately during rendering, preventing stuttering due to loading delays and avoiding excessive VRAM consumption. For low-frequency objects (…),… < 0.3), using the lowest detail level, and loading last when memory is tight. It can be preempted by high-priority tasks. When system memory usage > 85%, adaptive adjustment is triggered, automatically downgrading the memory level. ,in: For low-frequency objects, the loading coefficient is... This is to monitor memory usage. In this way, high-priority objects can be guaranteed to receive video memory resources first, and the system will automatically degrade under video memory pressure to avoid performance crashes caused by video memory overflow. In terms of memory management, it is necessary to monitor the system's memory usage in real time and dynamically allocate and release video memory resources according to the loading strategy of objects with different priorities. At the same time, it is necessary to establish an effective model resource index and caching mechanism to quickly find and load the required models.

[0084] S2: LSTM-based frustum prediction: First, data preprocessing is performed, then the LSTM network architecture is designed according to the specific application scenario and prediction requirements, the LSTM network architecture is trained and optimized, and finally the LSTM network architecture is used to predict the frustum range of future multiple frames.

[0085] S21: Data preprocessing: Data preprocessing is a key step of the LSTM frustum prediction module, which directly affects the training effect and prediction accuracy of the model; inputting 5-10 continuous frames of camera 6DOF parameters (position x / y / z + rotation pitch / yaw / roll) can provide spatiotemporal continuity information of camera motion, so that the model can learn the law of camera motion; first, batch processing data can improve training efficiency, and the batch size is 64 samples; at the same time, batch normalization operation is used to map data of different dimensions and ranges to a unified interval, reduce the sensitivity of the model to initial parameters, and improve the generalization ability of the model; the formula of the normalization operation is:

[0086] ,

[0087] wherein: is the mean of the past 30 frames, is the standard deviation of the past 30 frames, is the original camera 6DOF parameter (position x / y / z + rotation pitch / yaw / roll), with a dimension of 6, After normalization, the data, the normalization operation makes it easier for the model to learn the features in the data, and normalization can not only improve the training speed of the model, but also prevent the problem of gradient vanishing or explosion caused by too large or too small data range;

[0088] In practical applications, data preprocessing also needs to consider the noise and outliers of the data, and finally uses a filtering algorithm to smooth the camera parameters and remove noise interference; for outliers, interpolation or replacement methods are used for processing to ensure the quality of the input data;

[0089] S22: LSTM network architecture: the input layer is 6-dimensional camera parameters, and the batch size is 64, which can fully receive the information of camera motion; the hidden layer uses 2-layer LSTM, with 64 neurons in each layer, which can capture the complex spatiotemporal features of camera motion, and the setting of Dropout=0.2 can effectively prevent overfitting, which can make the model pay more attention to important features during the learning process, and improve the generalization ability of the model; the output layer is the 8-vertex coordinates (24-dimensional vector) of the future 3 frames of frustum, which can accurately predict the future frustum range of the camera;

[0090] S23: Training optimization: in terms of training optimization, the loss function is:

[0091] ,

[0092] wherein: is the segmented region predicted by the model, For the real segmentation label, this part of the loss function measures the difference between the prediction result and the true result, and improves the accuracy of the prediction by maximizing the intersection over union (IoU); For the regularization term, the strength of regularization is controlled to prevent model overfitting, so that the model can maintain good performance on both training data and test data;

[0093] S24: Prediction process algorithm steps:

[0094] S241: Input and output:

[0095] Input: 5-10 continuous frame camera 6DOF parameter sequence (sliding window)

[0096] (length=10 frames)

[0097] Output: future 3 frames Voxels 8 vertex coordinates

[0098] ,

[0099] Each of them contains 8 vertex coordinates;

[0100] S242: Execution process:

[0101] Data preprocessing: normalize the input sequence frame by frame (use the past 30 frames ); Outlier processing: replace the sudden data with linear interpolation; Smoothing filter: apply Kalman filter to the camera trajectory;

[0102] LSTM inference:

[0103] Input dimension: (time step feature)

[0104] LSTM layer calculation (simplified formula):

[0105] (forget gate)

[0106] (input gate)

[0107] (output gate)

[0108] (candidate memory)

[0109] (new memory)

[0110] (hidden state)

[0111] where: is the hidden state vector of the previous time step; is the memory state vector of the previous time step; is the concatenation vector (64+6); , , , are the forget gate weight, input gate weight, output gate weight, and candidate memory weight, respectively; , , , is the bias vector; is the Sigmoid activation function: ; is the hyperbolic tangent activation function: ; is the element-wise multiplication; is the forget gate output, which determines how much old memory to retain ; is the input gate output, which determines how much new candidate memory to receive ; is the output gate output, which determines how much memory to expose to the hidden state; is the candidate memory state; is the updated memory state: new memory = forget old memory + remember new information; is the current hidden state output;

[0112] Output analysis: split the 24-dimensional output vector into 3 groups of 8 vertex coordinates, and map them to the world coordinate system (need to combine the camera projection matrix).

[0113] S3: Block-level hybrid culling: first divide the scene into uniform grid blocks (such as 10m x 10m x 10m), and then perform hierarchical culling;

[0114] Each grid block contains a static object list, a dynamic object list, and occlusion metadata; the static object list records the relevant information of all static objects within the grid block, such as model ID, position, rotation, etc.; the dynamic object list updates the status of dynamic objects within the grid block in real time; occlusion metadata is the key to improving culling efficiency;

[0115] Dividing the scene into uniform grid blocks specifically includes:

[0116] S311: For static scenes, load pre-baked Occlusion Map and use PVS (Potentially Visible Set) to optimize the potentially visible set; the Occlusion Map records the occlusion relationship between different regions in the scene, by querying the Occlusion Map, it can quickly determine whether some objects are occluded, thereby avoiding unnecessary rendering calculations; PVS further optimizes the visibility judgment, only retaining objects that may be visible, reducing the size of the rendering queue;

[0117] a. Load pre-baked Occlusion Map

[0118] Spatial grid division: divide the entire three-dimensional scene into uniform grid blocks, such as cubic grid blocks with a side length of 10m, the size of the grid block can be adjusted according to the complexity and requirements of the actual scene; assuming the size of the scene is LxWxH (lengthxwidthxheight), then the number of divided grid blocks is (L / 10)x(W / 10)x(H / 10), each grid block has a unique identifier for subsequent queries and management;

[0119] Occlusion Map generation: through pre-computation, obtain the occlusion relationship between each grid block and other grid blocks, and store these occlusion relationships in the Occlusion Map; the Occlusion Map can be represented as a three-dimensional array, where Occlusion_Map[i][j][k] represents whether grid block (i,j,k) is occluded by other grid blocks; the specific calculation method can use occlusion calculation algorithms such as ray casting or depth buffering, for example, from the center of grid block (i,j,k) to each direction, the light rays are emitted, the intersection of the light rays with the objects in the scene is detected, if the light rays intersect with the objects before reaching the camera or screen, it is considered that the grid block is occluded; the mathematical expression is: for grid block (i,j,k), the direction of the emitted light ray r is d, the origin is p0, the intersection point t of the light ray with the scene object is calculated, if there is an intersection point and t max (t max is the maximum propagation distance of the light ray), then Occlusion_Map[i][j][k]=1 (indicating occlusion), otherwise 0;

[0120] b. Use PVS to optimize the potentially visible set

[0121] PVS generation: Based on the Occlusion Map, the potential visible set (PVS) of each grid block is generated, which refers to the set of all grid blocks that can be seen from the current grid block. The process of calculating PVS can use recursive or iterative methods. Taking the recursive method as an example, starting from the current grid block, according to the occlusion relationship in the Occlusion Map, recursively find the visible grid blocks; the specific algorithm is as follows:

[0122] Input: current grid block (i, j, k);

[0123] Output: PVS[i][j][k] (the potential visible set of the current grid block, containing the set of visible grid blocks);

[0124] Initialize PVS[i][j][k] as an empty set;

[0125] Add the adjacent grid blocks of the current grid block to the candidate queue;

[0126] While the candidate queue is not empty: take out a grid block (x, y, z) from the queue;

[0127] If (x, y, z) is not in PVS[i][j][k] and Occlusion_Map[x][y][z]=0 (not occluded): add (x, y, z) to PVS[i][j][k]; add the adjacent grid blocks of (x, y, z) to the candidate queue;

[0128] Wherein, the definition of adjacent grid blocks can be determined according to the connection relationship of the actual scene, which usually includes the grid blocks in the six directions of up, down, left, right, front and back.

[0129] Visibility judgment: in the rendering process, according to the grid block where the camera is located, query its PVS, so as to quickly determine the visible grid blocks and reduce the rendering calculation; the specific calculation steps are as follows:

[0130] Assume that the camera is located in the grid block (cx, cy, cz), then the visible grid block set V=PVS[cx][cy][cz]; for each visible grid block (vx, vy, vz) ∈ V, further judge whether the object in the grid block is visible.

[0131] S312: In terms of dynamic object management, after dividing the scene into grid blocks, only the blocks in the direction of camera motion are implemented for accurate culling; for the blocks in the non-motion direction, a simpler culling strategy or lower culling precision is used to reduce the calculation overhead; at the same time, the position and state of dynamic objects are updated in real time to ensure the accuracy of the occlusion metadata;

[0132] Hierarchical culling specifically includes:

[0133] S321: coarse culling: based on the block AABB fast intersection test, in the culling process, first, the coarse culling stage is carried out, the block-based AABB (axis-aligned bounding box) is used for fast intersection test, the parallel processing mode is adopted, a certain number of blocks are processed by each thread to improve the processing efficiency; through coarse culling, the possible visible blocks can be quickly screened out to generate a visible block list;

[0134] Parameter definition:

[0135] AABB of block: the AABB of each block is defined by its minimum and maximum coordinates, which is represented as ;

[0136] Camera frustum: the camera frustum is defined by its six planes (left, right, up, down, near, far), each plane can be represented by a plane equation ; intersection test result: Boolean value, indicating whether the AABB intersects with the frustum.

[0137] Fast intersection test algorithm:

[0138] For the AABB of each block, the following steps are taken to determine whether it intersects with the camera frustum:

[0139] Get the six plane equations of the camera frustum: each plane equation is represented as , where is the normal vector of the plane, is the distance from the plane to the origin;

[0140] For each plane: calculate the distance of the eight vertices of the AABB on the plane, if all vertices are on the same side of the plane (all distances are positive or negative), the AABB is completely outside the plane, and does not intersect with the frustum, return non-intersection, specifically: let the eight vertices of the AABB be , the coordinates of each vertex are , for the plane , the distance of the vertex to the plane is calculated as: If for a certain plane , all vertex distances are positive or negative, the AABB does not intersect with the plane;

[0141] If the AABB intersects with all six planes, return intersection, indicating that the block may be visible.

[0142] Parallel processing:

[0143] In order to efficiently process multiple blocks, parallel processing is adopted:

[0144] Thread Assignment: Assign all blocks to multiple threads, each thread handles a certain number of blocks;

[0145] Thread Safety: Use thread-safe data structures (such as locks or atomic operations) to manage shared resources, such as the visible block list;

[0146] Task Scheduling: Dynamically adjust the number of blocks for each thread to balance the load.

[0147] S322: Precision Culling: Enter the precision culling phase, implement object-by-object occlusion query for objects within the visible block, use HiZ occlusion culling technology, and use GPU Compute Shader for accelerated processing. Compute Shader fully utilizes the parallel computing capability of GPU to efficiently judge occlusion for each object, and finally generates the final list of visible objects;

[0148] Parameter Definition:

[0149] HiZ Buffer: HiZ buffer is a hierarchical depth buffer, usually divided into multiple levels, each level has half the resolution of the previous level, the top level has the lowest resolution, and the lowest level has the highest resolution;

[0150] Depth Value: The depth value of each pixel represents the straight-line distance from the camera to that pixel;

[0151] Object's : Axis-aligned Bounding Box (AABB) of each object, represented as ;

[0152] HiZ Occlusion Culling Algorithm:

[0153] HiZ occlusion culling quickly determines whether an object is occluded by a hierarchical depth buffer:

[0154] Constructing HiZ Buffer: Construct HiZ buffer from depth buffer (Z-Buffer), each level generates the depth value of the previous level by taking the maximum depth value of the adjacent four pixels; set the resolution of the depth buffer as , the number of levels of the HiZ buffer is , where the resolution of each layer is , until the lowest level; Occlusion Query: For each object's , calculate its projection rectangle in screen space; query the maximum depth value in the rectangular region in the HiZ buffer; if the object's nearest depth value is greater than the maximum depth value queried, the object is occluded and can be culled; Specifically: set the projection rectangle of the object's AABB in screen space as Rect ​Query HiZ buffer level The deepest depth value in If the object's nearest depth value satisfy: If an object is obscured, it can be removed.

[0155] GPU Compute Shader acceleration:

[0156] Parallelized occlusion lookup using GPU Compute Shader:

[0157] Thread allocation: Assign each object to a thread or a group of threads; each thread is responsible for calculating the projection of an object's AABB onto the screen space and querying the HiZ buffer.

[0158] Parallel query: Utilizes the parallel computing capabilities of Compute Shader to process occlusion queries for multiple objects simultaneously; calculates the intersection of the AABB and HiZ buffer for each object to quickly determine its visibility.

[0159] During the hierarchical elimination process, if the camera's movement speed exceeds the set threshold, the elimination calculation for blocks in non-movement directions is skipped, the LOD level of these blocks is reduced, and they are marked as potentially visible to reduce unnecessary computational overhead.

[0160] S4: Heterogeneous Computing Pipeline Optimization

[0161] S41: In terms of GPU computing optimization, Compute Shader is used to perform frustum culling and LOD switching operations. Compute Shader has high flexibility and parallel computing capabilities, which can efficiently handle large-scale culling and LOD switching tasks. At the same time, Wave Intrinsics technology is used to control the execution flow of threads more finely, further optimize parallel efficiency, reduce synchronization overhead between threads, and improve the utilization of GPU resources.

[0162] S42: In terms of CPU multi-threaded resource loading management, a shared memory pool is used to reduce data transfer. The shared memory pool manages model resources uniformly and adopts the LRU (Least Recently Used) cache replacement strategy to ensure that frequently used model resources can be accessed quickly. At the same time, atomic operations are used to ensure thread safety and avoid data inconsistency when multiple threads access the shared memory pool at the same time.

[0163] The CPU and GPU share a memory area via the PCIe bus, avoiding frequent data copying operations, further reducing communication overhead, and improving the overall system performance.

[0164] S5: rendering execution flow: sequentially execute the rendering flow according to steps S2-S4;

[0165] S51: in the prediction stage, the LSTM network is called every frame, the camera 6DOF parameters of continuous multiple frames are input, and the frustum range of the future 3 frames is output after network inference, which provides an important basis for subsequent resource preloading;

[0166] S52: in the resource preloading stage, according to the prediction result and the visibility frequency of the object (Vf) , asynchronously load the LOD model with high visibility probability, by loading high-precision models in advance, the loading time during rendering can be reduced, and sudden stuttering can be avoided;

[0167] S53: in the culling and rendering stage, the CPU first performs coarse culling operation, quickly filters the blocks in the scene, and outputs the visible block list; then, the GPU performs HiZ fine culling, performs object-by-object occlusion query on the objects in the visible block, and generates the final rendering queue; for low-frequency objects (Vf <0.3), GPU Instancing batch rendering technology is enabled, multiple same object instances are merged into one rendering batch, the number of rendering calls is reduced, and the rendering efficiency is improved.

[0168] The three-dimensional scene optimization system based on dynamic LOD and frustum culling collaborative rendering of space-time prediction is used to realize the three-dimensional scene optimization method based on dynamic LOD and frustum culling collaborative rendering of space-time prediction, which comprises:

[0169] Dynamic LOD and frustum culling collaborative optimization module: realize visibility probability statistics and hierarchical LOD loading;

[0170] LSTM frustum prediction module: predict the frustum range of multiple future frames;

[0171] Block-level hybrid culling module: realize coarse and fine two-level culling optimization;

[0172] Heterogeneous computing pipeline module: optimize CPU-GPU task allocation and data transmission;

[0173] Rendering execution control module: coordinate the timing and data flow of each module.

[0174] Embodiment 1

[0175] I. initialization stage

[0176] a. scene preprocessing: divide the space grid (10m×10m×10m), generate the Occlusion Map of static objects, and precompute the LOD level of each object.

[0177] b. Model loading: Establish shared memory pool, initialize LRU cache, load initial visible region model.

[0178] c. Prediction model initialization: Load pre-trained LSTM model, initialize camera trajectory buffer, calibrate normalization parameters.

[0179] II. Running phase

[0180] a. Camera data acquisition: Obtain current frame camera parameters, update historical trajectory queue, perform data normalization.

[0181] b. Frustum prediction: Call LSTM inference, parse output vertex coordinates, build predicted frustum.

[0182] c. Visibility analysis: Calculate object visibility frequency, update LOD priority, adjust video memory allocation strategy.

[0183] d. Resource preloading: Schedule asynchronous loading tasks, manage memory pool allocation, handle loading completion events.

[0184] e. Culling calculation: CPU coarse culling (block level), GPU fine culling (object level), generate rendering queue.

[0185] f. Rendering execution.

[0186] Term explanation:

[0187] LOD (Level of Detail): A technique for dynamically adjusting model accuracy based on object distance from the camera.

[0188] Frustum Culling: A rendering process that removes objects outside the camera frustum that are not visible.

[0189] GPU Instancing: A GPU optimization technique for batch rendering of the same mesh.

[0190] Temporal-spatial prediction: A technique that combines historical view motion trajectory (time dimension) and scene spatial structure features to predict the future N-frame frustum range.

[0191] Heterogeneous computing pipeline: Through the clock cycle interleaved scheduling of CPU instruction set (SSE / AVX) and GPU computing unit (CU Core / StreamProcessor), the hardware resource utilization is maximized.

[0192] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects: by dynamically adjusting the LOD loading priority (high-frequency object preloading) and block-level culling, the sudden stuttering problem is solved.

[0193] At the same time, the future 3 frames of the view frustum range are predicted through the lightweight LSTM, and a volume constraint loss function is introduced, so that the prediction accuracy is improved significantly.

[0194] The simple combination of LOD and the rejection technology can cause memory overflow, and the shared memory pool design of the application can effectively reduce the communication overhead of CPU-GPU.

[0195] The overall effect is improved: the cooperative optimization of dynamic LOD and view frustum rejection, the block-level hybrid rejection, and the heterogeneous computing pipeline optimization, etc., reduce unnecessary rendering calculation and resource loading, and improve the efficiency of rendering.

[0196] In terms of memory occupation, the application reduces the memory occupation by more than 20% through block-level rejection and prediction loading, block-level rejection reduces the number of objects that need to be rendered, and prediction loading avoids unnecessary model loading, thereby reducing the use amount of video memory.

[0197] In terms of memory occupation, the design of the shared memory pool and the LRU cache replacement strategy effectively manage the model resources and reduce the waste of memory.

[0198] In terms of loading delay, the application reduces the burst stuttering through the dynamic LOD loading priority and resource preloading strategy, and the loading delay is obviously reduced.

[0199] Especially when the camera moves quickly, the application can quickly respond to ensure smooth rendering of the scene, and avoid the phenomenon of "sudden pop-up" of objects in the traditional method.

[0200] Those skilled in the art should understand that the above discussion of any embodiment is only exemplary and is not intended to suggest that the scope of the application is limited to these examples; under the idea of the application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the application as described above. In order to be brief, they are not provided in detail.

[0201] The present application is intended to cover all such alternatives, modifications and variations as fall within the broad scope of the specification. Therefore, any omission, modification, equivalent replacement, improvement, etc. made in the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for optimizing a three-dimensional scene based on spatio-temporal prediction, dynamic level of detail (LOD) and view frustum culling, characterized in that, Comprise: S1: Dynamic LOD and frustum culling optimization: dynamically calculate the visibility frequency of objects through historical data and current frame information, and then dynamically adjust the loading and rendering mode of objects of different priorities according to the visibility frequency of objects and system memory usage; S2: Frustum prediction based on LSTM: first data preprocessing, then according to the specific application scene and prediction requirements to design the LSTM network architecture, train and optimize the LSTM network architecture, and finally use the LSTM network architecture to predict the frustum range of multiple future frames; S3: Block-level hybrid culling: first divide the scene into uniform grid blocks, and then perform hierarchical culling; S4: Heterogeneous computing pipeline optimization: S41: In terms of GPU computing optimization, use Compute Shader to perform frustum culling and LOD switching operations; At the same time, use Wave Intrinsics technology to more finely control the execution flow of threads; S42: In terms of CPU multi-threaded resource loading management, use shared memory pool to uniformly manage model resources, use LRU cache replacement strategy; At the same time, use atomic operations to ensure thread safety; S5: Rendering execution process: execute the rendering process according to steps S2-S4 in turn; S1 specifically comprises: S11: Visibility probability statistics: The visibility probability is calculated by the following formula: where: is the visibility probability, denotes the total number of visible frames of the object in the history frames, denotes the total number of frames, is the visibility frequency calculated in the last frame, is the decay coefficient; S12: Hierarchical LOD loading strategy: for high-frequency visible objects, implement preloading strategy, load high-precision LOD model 1-2 frames in advance, and ensure that the model is not replaced in video memory, and at the same time give the highest rendering priority; For medium-frequency objects, use background threads to load on demand, automatically adjust the cache size according to memory pressure; For low-frequency objects, use the lowest level of detail, and load last when video memory is tight, which can be preempted by high-priority tasks.

2. The method of claim 1, wherein, S2 specifically comprises: S21: Data preprocessing: input 5-10 consecutive frames of camera 6DOF parameters, first process the data in batches, the batch size is 64 samples, and at the same time use batch normalization operation to map data of different dimensions and ranges to a unified interval; Finally, use a filtering algorithm to smooth the camera parameters; For outliers, use interpolation or replacement for processing; S22: LSTM network architecture: the input layer is 6-dimensional camera parameters, and the batch size is 64; The hidden layer uses 2 layers of LSTM, each layer has 64 neurons; The output layer is the 8 vertex coordinates of the future 3 frames of frustum; S23: Training optimization: In terms of training optimization, the loss function is wherein: is the predicted segmentation region by the model; is the real segmentation label; is the regularization term.

3. The method of claim 1, wherein, In S3, each grid block contains a static object list, a dynamic object list, and occlusion metadata; The static object list records the relevant information of all static objects in the grid block; The dynamic object list updates the state of dynamic objects in the grid block in real time; Occlusion metadata is the key to improving culling efficiency.

4. The method of claim 3, wherein, The scene is divided into uniform grid blocks, which specifically comprises: S311: For static scenes, load pre-baked Occlusion Map and use PVS to optimize the potential visible set; S312: In the dynamic object management aspect, after the scene is divided into grid blocks, only the blocks in the camera motion direction are implemented with accurate culling; for the blocks in the non-motion direction, a simpler culling strategy or reduced culling accuracy is adopted; meanwhile, the positions and states of the dynamic objects are updated in real time.

5. The method of claim 4, wherein, The hierarchical culling specifically includes: S321: coarse culling: based on the fast intersection test of the block AABB, in the culling process, a coarse culling stage is first performed, the fast intersection test is performed based on the block AABB, and a parallel processing mode is adopted, and each thread processes a certain number of blocks; S322: fine culling: entering the fine culling stage, object-by-object occlusion query is performed on the objects in the visible block, HiZ occlusion culling technology is adopted, and GPU Compute Shader is used for accelerated processing, the Compute Shader fully utilizes the parallel computing capability of the GPU, and efficient occlusion judgment is performed on each object, and finally a final visible object list is generated.

6. The method of claim 5, wherein, In the hierarchical culling process: if the camera motion speed is greater than the set threshold, the block culling calculation in the non-motion direction is skipped, the LOD level of these blocks is reduced, and they are marked as possibly visible.

7. The method of claim 1, wherein, In the S4, the CPU and the GPU share a memory region through a PCIe bus.

8. The method of claim 1, wherein, The S5 specifically includes: S51: in the prediction stage, an LSTM network is called every frame, and continuous multiple frames of camera 6DOF parameters are input, and the future 3 frames of view cone ranges are output after network inference; S52: in the resource preloading stage, according to the prediction result and the visibility frequency of the object, the LOD model with high visibility probability is asynchronously loaded; S53: in the culling and rendering stage, the CPU first performs a coarse culling operation to quickly screen the blocks in the scene and output a visible block list; then, the GPU performs HiZ fine culling to perform object-by-object occlusion query on the objects in the visible block and generate a final rendering queue; for low-frequency objects, the GPU Instancing batch rendering technology is enabled to combine multiple identical object instances into one rendering batch.

9. A system for optimizing a three-dimensional scene for collaborative rendering with dynamic LOD based on spatio-temporal prediction and view frustum culling, characterized in that, The three-dimensional scene optimization method for realizing the dynamic LOD and view cone culling cooperative rendering based on spatiotemporal prediction according to any one of claims 1-8, comprising: a dynamic LOD and view cone culling cooperative optimization module: realizing visibility probability statistics and hierarchical LOD loading; an LSTM view cone prediction module: predicting the view cone range of multiple future frames; a block-level hybrid culling module: realizing coarse and fine two-level culling optimization; a heterogeneous computing pipeline module: optimizing CPU-GPU task allocation and data transmission; a rendering execution control module: coordinating the timing and data flow of each module.

Images