A spatially-aware real-time rendering method for 3D scenes

Abstract

This invention discloses a real-time rendering method for 3D scenes based on spatial awareness. The method involves acquiring an initial sparse point cloud of the scene image, denoising the initial sparse point cloud, extracting the spatial boundaries of the scene, voxelizing the denoised point cloud, treating the center of each voxel as an anchor point, randomly selecting a camera pose, generating a rendered image using the current anchor point and its neural Gaussian properties, calculating the loss and performing backpropagation, performing a cropping operation on the original anchor points, and generating new anchor points based on spatial awareness. Finally, the iteration terminates when the termination condition is met, and the rendered image is generated using the current anchor point and its neural Gaussian properties. This invention ensures that anchor point growth only occurs within a reasonable spatial range, enabling anchor points to effectively cover under-reconstructed areas, optimizing the prediction of neural Gaussian related properties, and enhancing the ability to handle complex geometries and details while ensuring reconstruction quality.

Description

Technical Field

[0001] This application belongs to the field of 3D reconstruction, and in particular relates to a real-time rendering method for 3D scenes based on spatial perception. Background Technology

[0002] With the development of 3D scene rendering technology, 3D Gaussian Splatting (3DGS) has demonstrated superior rendering fidelity and efficiency. However, explicit 3DGS representation places very high demands on computational and memory resources, primarily because each Gaussian distribution needs to explicitly store its parameters (such as position, shape, color, and transparency), and these parameters need to be calculated and updated in real time during rendering. Furthermore, to accurately fit each viewpoint and render details, 3DGS often generates a large number of redundant Gaussian distributions. These redundant Gaussian repetitions characterize the same geometric features, significantly increasing memory usage and computational burden. This is particularly pronounced when processing complex scenes or rendering at high resolutions.

[0003] Scaffold-GS overcomes the aforementioned limitations by anchoring neural Gaussians. Scaffold-GS generates anchor points from a sparse point cloud, constructs a voxel mesh, and trains the neural Gaussians using a multilayer perceptron (MLP), significantly reducing memory consumption. Since the neural Gaussians at the anchor points only need to store a few parameters such as their mean and variance, and the computation mainly depends on the data distribution within the local voxel mesh, Scaffold-GS avoids global rendering computations for the entire scene. Therefore, even when dealing with scenes such as complex buildings or large open spaces, it can still operate with low and stable memory overhead.

[0004] However, the effectiveness of Scaffold-GS largely depends on obtaining high-quality sparse point clouds from Structure-from-Motion (SfM). If the SfM results are poor or the scene viewing perspective is limited, its anchor point growth strategy will face challenges, making detailed reconstruction of these regions still difficult. Summary of the Invention

[0005] The purpose of this application is to provide a spatially aware real-time rendering method for 3D scenes, so as to reduce Scaffold-GS's dependence on high-quality initial sparse point clouds, thereby achieving more efficient and accurate rendering in complex scenes.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] A spatially aware real-time rendering method for 3D scenes, comprising:

[0008] Step 1: Acquire multi-view images of the scene, estimate camera pose and generate initial sparse point cloud of the scene, and extract the corresponding depth map and edge map of the image;

[0009] Step 2: Denoise the initial sparse point cloud and then extract the spatial boundaries of the scene;

[0010] Step 3: Voxelize the denoised point cloud, with the center of each voxel as an anchor point. Derive a neural Gaussian for the anchor point based on the initialized anchor point offset and scaling factor, and use the initialized multilayer perceptrons to obtain the anchor point features and neural Gaussian properties.

[0011] Step 4: Randomly select a camera pose, generate a rendering image using the anchor features of the current anchor point and the Gaussian properties of the neural network, extract the depth map and edge map of the rendering image, calculate the loss and perform backpropagation, and train to obtain the anchor offset, scaling factor and weights of each multilayer perceptron.

[0012] Step 5: Perform a pruning operation on the original anchor points and generate new anchor points based on spatial perception. Merge the new anchor points with the pruned anchor points to obtain an anchor point set. Update the anchor point features and neural Gaussian properties of each anchor point in the anchor point set.

[0013] Step 6: Determine whether the iteration termination condition has been met. If not, return to step 4 to continue the iteration; otherwise, terminate the iteration and generate a rendered image using the anchor features and neural Gaussian properties of each anchor in the anchor set.

[0014] Furthermore, the denoising of the initial sparse point cloud includes:

[0015] For each point in the initial sparse point cloud, calculate its average distance to all points in its neighborhood;

[0016] If the average distance exceeds the standard deviation threshold, the corresponding point is considered a noise point and is removed.

[0017] Furthermore, the extraction of the spatial boundaries of the scene includes:

[0018] Calculate the convex hull of a point cloud;

[0019] According to the set scaling factor Expand each vertex of the convex hull outwards using the following formula:

[0020]

[0021] in, For convex hull vertices, is the expanded convex hull vertex, and c is the centroid of the convex hull vertex. The convex hull formed by the expanded vertices is used as the spatial boundary of the scene.

[0022] Furthermore, the joint loss function used to calculate the loss is as follows:

[0023]

[0024] in, For the joint loss function, For deep loss, For edge loss, The original loss function of Scaffold-GS , and These are the weighting coefficients.

[0025] Furthermore, the trimming operation on the original anchor points includes:

[0026] Record the opacity and access count of each anchor point used when rendering the image;

[0027] After a preset number of iterations, if the number of times an anchor point is accessed is greater than the minimum access threshold and its cumulative opacity is less than the minimum opacity threshold, then the anchor point is removed.

[0028] Furthermore, the generation of new anchor points based on spatial perception includes:

[0029] After a preset number of iterations, for each Gaussian neuron, the Euclidean distance to each anchor point is calculated. If the minimum Euclidean distance is greater than the distance threshold, the center position of the voxel containing the Gaussian neuron is marked as a candidate anchor point.

[0030] Based on the spatial boundaries of the scene, candidate anchor points that exceed the spatial boundaries are eliminated;

[0031] For the candidate anchor set, select the candidate anchor with the highest gradient value and add it to the new anchor set. Remove candidate anchors from the candidate anchor set whose distance from the selected candidate anchor with the highest gradient value is less than a distance threshold.

[0032] Repeat the previous step until the candidate anchor set is empty, and obtain the final set of new anchors;

[0033] For each new anchor point in the newly added anchor point set, initialize its local context features, scaling factor, and anchor point offset. The local context features are inherited from the candidate anchor points. Initialize a tensor with all zeros as the anchor point offset and a tensor with all 1s. Multiply the tensor by the voxel size of the current scene and take the logarithm to obtain the scaling factor.

[0034] Further, each of the multilayer perceptrons includes a first multilayer perceptron for anchor feature generation and a second to fourth multilayer perceptron for neural Gaussian attribute prediction. Updating the anchor features and neural Gaussian attributes of each anchor in the anchor set includes:

[0035] For newly added anchor points, a feature library is created. The viewpoint weights predicted by the first multilayer perceptron after training are used to mix the feature library to obtain anchor point features. Then, a scaling factor and offset are used to derive a neural Gaussian. The second to fourth multilayer perceptrons after training are used to predict the neural Gaussian attributes.

[0036] Further, each of the multilayer perceptrons includes a first multilayer perceptron for anchor feature generation and a second to fourth multilayer perceptron for neural Gaussian attribute prediction. Updating the anchor features and neural Gaussian attributes of each anchor in the anchor set includes:

[0037] For the clipped anchor points, the first multilayer perceptron trained after training is used to update the anchor point features; the scaling factor and offset trained after training are used to update the neural Gaussian position; and the second to fourth multilayer perceptrons trained after training are used to predict and update the neural Gaussian attributes.

[0038] The spatially-aware, real-time rendering method for 3D scenes proposed in this application has the following advantages compared with existing technologies:

[0039] Scene geometric constraints were extracted to ensure that anchor point growth only occurs within a reasonable spatial range, thus avoiding the generation of redundant anchor points. A spatially aware anchor point growth method was proposed, enabling anchor points to effectively cover under-reconstructed regions, thereby significantly reducing Scaffold-GS's dependence on high-quality initial point clouds. A joint loss function was designed, adding depth and edge losses to the original Scaffold-GS loss function, further optimizing the prediction of neural Gaussian correlation attributes. This enhances the ability to handle complex geometries and details while maintaining reconstruction quality. Attached Figure Description

[0040] Figure 1 This is a flowchart of the real-time rendering method for 3D scenes based on spatial awareness, as described in this application.

[0041] Figure 2 This is a flowchart of the anchor point growth process of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] One embodiment of this application, such as Figure 1 As shown, a real-time rendering method for 3D scenes based on spatial awareness is proposed, including:

[0044] Step S1: Acquire multi-view images of the scene, estimate the camera pose and generate the initial sparse point cloud of the scene, and extract the depth map and edge map corresponding to the image.

[0045] Structure from Motion (SfM) is a technique for reconstructing three-dimensional scenes from image sequences.

[0046] In this embodiment, the scene refers to the scene that needs to be rendered using this method. The camera captures images from various perspectives in the scene from different poses as real images. From the real images, the RGB pixel values, depth map, and edge map of the image can be obtained.

[0047] This embodiment uses the SfM method to process real images, estimate camera pose, and generate an initial sparse point cloud of the scene. This method generates an initial sparse point cloud by matching feature points in multi-view images to extract preliminary geometric information of the scene.

[0048] This embodiment uses a monocular depth estimator (Depth Anything V2) and the kornia.filters.Canny algorithm from the Kornia library to extract the depth map and edge map corresponding to each image.

[0049] Step S2: Denoise the initial sparse point cloud and then extract the spatial boundary of the scene.

[0050] This step involves extracting geometric constraints, performing geometric analysis on the generated initial sparse point cloud, and defining the spatial boundaries of the scene.

[0051] Specifically, it includes:

[0052] Step 2.1: Remove isolated noise points from the initial sparse point cloud.

[0053] Statistical Outlier Removal (SOR) is used to identify and remove isolated noise points in the initial sparse point cloud. The specific steps are as follows:

[0054] For the initial sparse point cloud For each point Calculate its relationship with its neighborhood Average distance of all points:

[0055]

[0056] If the average distance Exceeding the set standard deviation threshold Then the point is considered It is a noise point and it is removed.

[0057] Step 2.2: Extract the spatial boundaries of the scene.

[0058] The ConvexHull algorithm is used to compute the convex hull H of the point cloud. Specifically, the convex hull H is the minimal polyhedron containing all points in the point cloud, and its vertex set is... The following conditions must be met:

[0059]

[0060] Where ConvexHull is the algorithm identifier, and m represents the number of vertices. Represents the vertices in the vertex set U. Represents vertices Weights in the convex hull.

[0061] Based on this, to ensure that subsequent anchor points can cover the scene edges and their near-edge regions, the convex hull vertices need to be expanded. Let the centroid of the convex hull vertices be... Then each vertex is calculated using the following formula. Expand outward by a scaling factor :

[0062]

[0063] in, These are the vertices of the expanded convex hull. The convex hull formed by the expanded vertices serves as the spatial boundary of the scene, providing geometric constraints for the subsequent distribution of anchor points.

[0064] Step S3: Voxelize the denoised point cloud, with the center of each voxel as an anchor point. Derive a neural Gaussian for the anchor point based on the initialized anchor point offset and scaling factor, and use the initialized multilayer perceptrons to obtain the anchor point features and neural Gaussian properties.

[0065] This step uses the denoised point cloud as input, and processes the point cloud... The scene is voxelized as follows:

[0066]

[0067] in As a voxel center, Voxel size, Indicates rounding operation. This indicates the removal of duplicate entries to reduce redundancy and irregularities in P.

[0068] The center of each voxel It is considered as an anchor point, possessing a local contextual feature. A scaling factor k anchor point offsets (Obtained through initialization). Local context features are used to capture the geometric and appearance information of the environment surrounding the anchor point; scaling factors are used to adjust the size of the neural Gaussian associated with the anchor point; and anchor point offsets are used to describe the position of the neural Gaussian associated with the anchor point in space relative to the anchor point.

[0069] To further enhance contextual features Due to the dependency on multiple resolutions and views, a feature library is created for each anchor point v. ,in express In the channel dimension Factor downsampling.

[0070] Then the first multilayer perceptron ( The predicted viewpoint weights are used to mix the feature library to obtain anchor features. .

[0071] Specifically, given a location A camera and a location Calculate the relative distance and view orientation of the anchor points:

[0072]

[0073] Then use Predicted weights Anchor features are obtained by weighted summation with the feature library. :

[0074]

[0075] .

[0076] Next, a neural Gaussian is derived from the anchor point, and the neural Gaussian properties are obtained through the initialized second to fourth multilayer perceptrons.

[0077] The neural Gaussian is a highly efficient 3D representation unit that combines the mathematical properties of the Gaussian distribution with the learning capabilities of neural networks to accurately represent complex 3D scenes. (Using location...) Opacity Covariance-related quaternions and color To parameterize each Gaussian neuron.

[0078] For each anchor point in the scene, derive k neural Gaussians and obtain their properties.

[0079] Specifically, given the location The anchor point, and the position of its neural Gaussian are calculated as follows:

[0080]

[0081] in It is the anchor point offset. This is the scaling factor associated with the anchor point. These represent the positions of each Gaussian neural network. The anchor point offsets are initialized in this step. Specifically, for each anchor point, the position of its Gaussian neural network is calculated by adding the anchor point position to the offset multiplied by a scaling factor.

[0082] The opacity, color, and covariance-related quaternions of the neural Gaussian are respectively initialized through the second to fourth multilayer perceptrons (MLPs). , and To make predictions.

[0083] It's important to note that the attribute is decoded only once, and the opacity... ,color Covariance-related quaternions It is predicted by the following formula.

[0084]

[0085]

[0086]

[0087] It should be noted that this step uses a multilayer perceptron (MLP) to predict viewpoint weights and neural Gaussian properties. The MLP used in this step is the initialized MLP, which will be trained subsequently to obtain a more accurate predictive MLP. The initialization of the MLP, anchor offsets, and scaling factors are well-established techniques in this field and will not be elaborated upon here. These MLP weights are optimized during training to dynamically predict the properties of the neural Gaussian.

[0088] Step S4: Randomly select a camera pose, generate a rendering image using the anchor features of the current anchor point and the Gaussian properties of the neural network, extract the depth map and edge map of the rendering image, calculate the loss and perform backpropagation, and train to obtain the anchor offset, scaling factor and weights of each multilayer perceptron.

[0089] This step begins iterative training of each multilayer perceptron, starting with random viewpoint image rendering. Specifically, a camera pose obtained in step S1 is randomly selected, and a rendered image is generated using the Scaffold-GS method based on the current anchor point features and neural Gaussian properties. A monocular depth estimator is then used to obtain the depth map of the rendered image, and the edge map of the rendered image is extracted using the kornia.filters.Canny algorithm from the Kornia library.

[0090] Then, the loss and backpropagation are calculated to update the parameters of each multilayer perceptron.

[0091] This embodiment calculates the loss value using a joint loss function. It can be represented as:

[0092]

[0093] in, , and These are the weighting coefficients, used to balance the contributions of depth loss, edge loss, and other loss terms. Depth Loss It is calculated using the depth map of the real image obtained in step S1 and the depth map of the rendered image. Distance measures the consistency between two depth maps. Edge loss. Similar to the depth loss calculation method, it adopts Distance measures the consistency between two edge graphs. It is the original loss function of Scaffold-GS, which includes the rendering pixel color. Loss, structural similarity loss ) and volume regularization loss ( ):

[0094]

[0095] Next, the backpropagation phase begins. In this phase, based on the loss function value, the gradients of each MLP weight, anchor offset, and scaling factor in the model are calculated. Subsequently, the Adam optimizer for each parameter fine-tunes the parameters based on these gradient values ​​and pre-set hyperparameters such as the learning rate. Updating the MLP weights allows for more accurate prediction of the relevant properties of the neural Gaussian. Adjusting the anchor offset enables the neural Gaussian to more accurately locate key feature points in the scene. Optimizing the scaling factor allows for more precise control of the neural Gaussian's diffusion range, thus capturing scene details more meticulously. These updates work together to render more detailed scenes in the next iteration, reduce the loss value, and ultimately achieve a significant improvement in rendering quality.

[0096] The technique of backpropagation through loss, updating the weights of each MLP based on the loss value, and using the Adam optimizer to update the anchor offset and scaling factor is a relatively mature technique in this field, and will not be elaborated here.

[0097] Step S5: Perform a pruning operation on the original anchor points and generate new anchor points based on spatial perception. Merge the new anchor points with the pruned anchor points to obtain an anchor point set. Update the anchor point features and neural Gaussian attributes of each anchor point in the anchor point set.

[0098] Specifically, the first step is to perform a trimming operation on the existing anchor points, including:

[0099] Step 5.1.1: Record the opacity and access count of each anchor point used when rendering the image.

[0100] For each anchor point involved in rendering, sum the associated neural Gaussian opacities and add them to the cumulative opacity of these anchor points. Simultaneously, increment the visit count of all these anchor points by 1.

[0101] Step 5.1.2: After a preset number of iterations, if the number of times an anchor point is accessed is greater than the minimum number of accesses threshold and its cumulative opacity is less than the minimum opacity threshold, then the anchor point is removed.

[0102] After every N iterations, an anchor point pruning operation is performed. The cumulative opacity and access count of each anchor point during the previous N iterations are calculated. If the access count of an anchor point is greater than the minimum access count threshold, and its cumulative opacity is less than the minimum opacity threshold, then the anchor point is removed to eliminate active anchor points that contribute little to the scene. Afterwards, the calculated cumulative opacity and access count of the anchor points are reset to 0.

[0103] It should be noted that during the iteration process before reaching N iterations, no pruning operation is performed, and all current anchor points are retained for the next iteration.

[0104] Then, as Figure 2 As shown, new anchor points are generated based on spatial awareness, including:

[0105] Step 5.2.1: After a preset number of iterations, for each neuron Gaussian, calculate the Euclidean distance to each anchor point. If the minimum Euclidean distance is greater than the distance threshold, mark the center position of the voxel containing the neuron Gaussian as a candidate anchor point.

[0106] This step is used for candidate anchor point generation, defining a distance threshold D as the minimum spacing between candidate anchor points. During each anchor point growth process, the set of anchor points in the current scene is defined as... The neural Gaussian set is defined as Then iterate through the neural Gaussians in the scene. Calculate the nearest anchor point in the current anchor point set A. Euclidean distance If satisfied Then the neurogaussian The center position of the voxel is marked as a candidate anchor point to ensure that the candidate anchor points can cover areas where the anchor point distribution is insufficient.

[0107] Step 5.2.2: Based on the spatial boundaries of the scene, eliminate candidate anchor points that exceed the spatial boundaries.

[0108] Define the candidate anchor set as Then, based on the scene space boundary, the candidate anchor points are filtered out, and candidate anchor points that exceed the boundary range are removed.

[0109] Step 5.2.3: For the candidate anchor set, select the candidate anchor with the highest gradient value and add it to the new anchor set. Remove candidate anchors from the candidate anchor set whose distance from the selected candidate anchor with the highest gradient value is less than the distance threshold.

[0110] Then, a gradient-first greedy algorithm is used to control the density of candidate anchor points. Specifically, for each candidate anchor point in the candidate anchor point set C... All have gradient values In the subsequent screening process, the candidate anchor point with the highest gradient value is selected. And move it into a new set S, that is:

[0111]

[0112] Then, with the currently selected Centered on, all elements in set C that are related to... Candidate anchor points with a distance less than a threshold D are removed to control anchor point density, specifically expressed as follows:

[0113] .

[0114] Step 5.2.4: Repeat the previous step until the candidate anchor point set is empty, and obtain the final set of new anchor points.

[0115] Repeat the above steps until set C is empty. The final set S is the set of newly added anchor points after density control. For each new anchor point in the set S, initialize its local context features, scaling factor, and anchor point offset. Inherited from the candidate anchor point, initialize a tensor with all zeros as the anchor point offset, and initialize a tensor with all 1s. Multiply the tensor by the voxel size of the current scene, and then take the logarithm to obtain the scaling factor.

[0116] The clipped anchor points and the newly added anchor points are merged into an anchor point set. The anchor point features and neural Gaussian properties of each anchor point in the anchor point set are updated for use in the next iteration.

[0117] For newly added anchor points, a feature library is created to generate anchor point features, and a neural Gaussian is derived for the new anchor points. The trained multilayer perceptron is used to predict the attributes of the neural Gaussian, thus ensuring that the new anchor points can effectively express the geometric appearance features of the scene.

[0118] Specifically, creating a feature library Using the first multilayer perceptron after training ( The predicted viewpoint weights are used to mix the feature library to obtain anchor features. Then, scaling factors and offsets are used to derive neural Gaussians, and the trained second to fourth multilayer perceptrons (MLPs) are used. , and Prediction is performed to obtain the neural Gaussian properties.

[0119] For the existing anchor points after cropping, the trained first multilayer perceptron (MLP) is used. Update anchor features The neural Gaussian positions are updated using the trained scaling factor and offset, and the trained second to fourth multilayer perceptrons (MLPs) are used. , and To make predictions and update the neural Gaussian properties.

[0120] Step S6: Determine whether the iteration termination condition has been met. If not, return to step 4 to continue the iteration; otherwise, terminate the iteration and generate a rendering image using the anchor features and neural Gaussian properties of each anchor in the anchor set.

[0121] This step determines whether the iteration termination condition has been met. For example, the iteration termination condition is to reach the maximum number of iterations. If the maximum number of iterations is reached, the iteration ends. If not, the process returns to step S4 to continue iterating.

[0122] After iteration, the scene is rendered. A random camera pose is input, and anchor points within the camera's coverage area are determined. The spatial position of the neural Gaussian is determined based on its offset relative to the anchor points. The relative distance and view direction between the camera and anchor points are calculated, and these, along with the anchor point features, are input into a multilayer perceptron to infer the color, opacity, and covariance of the neural Gaussian. Subsequently, the contributions are weighted and accumulated according to the depth order of the neural Gaussian to calculate the color of each pixel, achieving real-time rendering of the 3D scene.

[0123] It should be noted that the specific method for generating rendered images using the Scaffold-GS method is a relatively mature technology in this field, and will not be elaborated here.

[0124] To verify the effectiveness of the technical solution, training and testing were conducted on three public datasets: Mip-NeRF360, Tanks & Temples, and Deep Blending. Similar to the 3DGS method, for each scene's input images, one-eighth of the total number of images was used as the test set, and the remaining images were used as the training set. To objectively evaluate visual fidelity, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Patch Similarity (LPIPS) were used as evaluation metrics. These metrics can compare the differences between images rendered by different methods and their corresponding real frames, thus providing a quantitative analysis of the rendering effect. Specific experimental results are shown in Table 1.

[0125] Table 1

[0126]

[0127] As shown in Table 1, the rendering quality of this application outperforms other methods in all three datasets mentioned above. As evaluation criteria, higher PSNR and SSIM values ​​indicate higher image similarity, while lower LPIPS values ​​indicate a smaller visual difference between the rendered image and the real image. Further analysis of the rendering results reveals that this application can preserve fine geometric details in complex scenes. For example, in the STUMP scene of the Mip-NeRF360 dataset, it accurately renders small tree branches; in the PLAYROOM scene of the Deep Blending dataset, it meticulously depicts the switches on the wall; and in the TRAIN scene of the Tanks & Temples dataset, it well reproduces the details of the truck license plate and windshield reflections.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A real-time rendering method for 3D scenes based on spatial awareness, characterized in that, The spatially-aware, real-time rendering method for 3D scenes includes: Step 1: Acquire multi-view images of the scene, estimate camera pose and generate initial sparse point cloud of the scene, and extract the corresponding depth map and edge map of the image; Step 2: Denoise the initial sparse point cloud and then extract the spatial boundaries of the scene; Step 3: Voxelize the denoised point cloud, with the center of each voxel as an anchor point. Derive a neural Gaussian for the anchor point based on the initialized anchor point offset and scaling factor, and use the initialized multilayer perceptrons to obtain the anchor point features and neural Gaussian properties. Step 4: Randomly select a camera pose, generate a rendering image using the anchor features and Gaussian neural properties of the current anchor point, extract the depth map and edge map of the rendering image, calculate the loss and perform backpropagation, and train to obtain the anchor offset, scaling factor and weights of each multilayer perceptron. Step 5: Perform a pruning operation on the existing anchor points and generate new anchor points based on spatial perception. Merge the new anchor points with the pruned anchor points to obtain an anchor point set. Update the anchor point features and neural Gaussian properties of each anchor point in the anchor point set. Step 6: Determine whether the iteration termination condition has been met. If not, return to step 4 to continue the iteration; otherwise, terminate the iteration and generate a rendering image using the anchor features and neural Gaussian properties of each anchor in the anchor set. The generation of new anchor points based on spatial perception includes: After a preset number of iterations, for each Gaussian neuron, the Euclidean distance to each anchor point is calculated. If the minimum Euclidean distance is greater than the distance threshold, the center position of the voxel containing the Gaussian neuron is marked as a candidate anchor point. Based on the spatial boundaries of the scene, candidate anchor points that exceed the spatial boundaries are eliminated; For the candidate anchor set, select the candidate anchor with the highest gradient value and add it to the new anchor set. Remove candidate anchors from the candidate anchor set whose distance from the candidate anchor with the highest gradient value is less than the distance threshold. Repeat the previous step until the candidate anchor set is empty, and obtain the final set of new anchors; For each new anchor point in the newly added anchor point set, initialize its local context features, scaling factor, and anchor point offset. The local context features are inherited from the candidate anchor points. Initialize a tensor with all zeros as the anchor point offset and a tensor with all 1s. Multiply the tensor by the voxel size of the current scene and take the logarithm to obtain the scaling factor.

2. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, The denoising of the initial sparse point cloud includes: For each point in the initial sparse point cloud, calculate its average distance to all points in its neighborhood; If the average distance exceeds the standard deviation threshold, the corresponding point is considered a noise point and is removed.

3. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, The extracted spatial boundaries of the scene include: Calculate the convex hull of a point cloud; According to the set scaling factor Expand each vertex of the convex hull outwards using the following formula: ； in, For convex hull vertices, is the expanded convex hull vertex, and c is the centroid of the convex hull vertex. The convex hull formed by the expanded vertices is used as the spatial boundary of the scene.

4. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, The joint loss function used to calculate the loss is as follows: ； in, For the joint loss function, For deep loss, For edge loss, The original loss function of Scaffold-GS , and These are the weighting coefficients.

5. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, The trimming operation on the original anchor points includes: Record the opacity and access count of each anchor point used when rendering the image; After a preset number of iterations, if the number of times an anchor point is accessed is greater than the minimum access threshold and its cumulative opacity is less than the minimum opacity threshold, then the anchor point is removed.

6. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, Each multilayer perceptron includes a first multilayer perceptron for anchor feature generation and a second to fourth multilayer perceptron for neural Gaussian attribute prediction. Updating the anchor features and neural Gaussian attributes of each anchor in the anchor set includes: For newly added anchor points, a feature library is created. The viewpoint weights predicted by the first multilayer perceptron after training are used to mix the feature library to obtain anchor point features. Then, a scaling factor and offset are used to derive a neural Gaussian. The second to fourth multilayer perceptrons after training are used to predict the neural Gaussian attributes.

7. The real-time rendering method for 3D scenes based on spatial awareness according to claim 1, characterized in that, Each multilayer perceptron includes a first multilayer perceptron for anchor feature generation and a second to fourth multilayer perceptron for neural Gaussian attribute prediction. Updating the anchor features and neural Gaussian attributes of each anchor in the anchor set includes: For the clipped anchor points, the first multilayer perceptron trained after training is used to update the anchor point features; the scaling factor and offset trained after training are used to update the neural Gaussian position; and the second to fourth multilayer perceptrons trained after training are used to predict and update the neural Gaussian attributes.

Images