A dynamic scene incremental reconstruction and rendering method based on 3DGS

Abstract

The application discloses a kind of dynamic scene incremental reconstruction and rendering method based on 3DGS, belong to specific computer model field, including steps: based on original image set, the 3DGS model of initial time is constructed;Any view image in dynamic scene time t is determined first update area by semantic segmentation and target identification after being obtained;Based on luminosity error, local similarity, global semantic feature generates incremental update area;Second update area is generated and modeling in it, and the optimized Gaussian point set is generated by minimizing area reconstruction error;With the real-time model of last time is fused and is optimized, obtains the real-time model G of time t t For real-time rendering.The application can effectively encode the geometry and texture information of the scene, support accurate detection and local incremental update of dynamic areas, optimize Gaussian point parameters to improve model continuity and visual quality, and realize low-latency real-time rendering, effectively improving the efficiency and quality of three-dimensional scene processing in dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional computer vision and graphics, and in particular to a dynamic scene incremental reconstruction and rendering method based on 3DGS. Background Technology

[0002] With the rapid development of applications such as augmented reality (AR), robot navigation, and intelligent monitoring, efficient reconstruction and real-time rendering technologies for dynamic 3D scenes have become increasingly popular research topics. Dynamic scenes typically contain various moving objects and environmental changes, making it difficult for traditional static 3D reconstruction methods to meet their demands for high update frequency and dynamic adaptability. Existing technologies mostly rely on dense 3D point clouds or mesh representations, resulting in significant data redundancy and high update costs, making it difficult to achieve real-time incremental updates. Furthermore, they suffer from insufficient reconstruction accuracy and poor texture continuity in complex dynamic environments.

[0003] To address the reconstruction of dynamic scenes, some studies employ incremental update methods based on voxels or meshes. However, limited by storage and computational resources, these methods struggle to support high resolution and large-scale dynamic changes. Furthermore, existing methods underutilize semantic information, making it difficult to accurately locate and distinguish dynamic objects from static backgrounds. This leads to excessively large update ranges or omissions of key change areas, reducing overall reconstruction efficiency and quality. In addition, real-time rendering often relies on traditional mesh-based algorithms, which suffer from high rendering latency in highly dynamic scenes, making it difficult to guarantee visual continuity and realism.

[0004] In recent years, scene representation based on 3D Gaussian point clouds (3DGS) has gradually attracted attention due to its continuity and flexibility. 3DGS can encode scene geometry and texture through the spatial distribution and attributes of Gaussian points, supporting efficient interpolation and sparse representation, facilitating local incremental updates and multi-level detail adjustments. However, how to combine change detection, semantic recognition, and high-quality fusion in dynamic scenes to construct a dynamic 3D reconstruction and rendering system that is both efficient and accurate remains a pressing technical challenge.

[0005] Therefore, developing a dynamic scene incremental reconstruction and rendering method based on 3DGS can effectively encode the geometric and texture information of the scene, support accurate detection and local incremental updates of dynamic regions, optimize Gaussian point parameters to improve model continuity and visual quality, and achieve low-latency real-time rendering. This is of great significance for improving the efficiency and quality of 3D scene processing in dynamic environments.

[0006] Definitions:

[0007] SIFT algorithm (Scale Invariant Feature Transformation, SIFT) is a classic algorithm for extracting "salient" local feature points (also called key points) from an image. It generates a series of SIFT descriptors, also known as local descriptors, through several steps including scale space extremum detection, key point precise localization, orientation assignment, and descriptor generation.

[0008] ORB algorithm (Oriented FAST and Rotated BRIEF) is a fast feature point extraction and description algorithm, consisting of two parts: Oriented FAST (oriented FAST key point detection) and Rotated BRIEF (rotated BRIEF descriptor).

[0009] The FAST algorithm (Features from Accelerated Segment Test) is a fast algorithm for detecting key points in images, originally used in computer vision and robotics. It is suitable for real-time processing, such as quickly identifying feature points in video streams or drone scenarios.

[0010] SuperPoint is a deep learning-based self-supervised feature point detection and descriptor extraction algorithm that can simultaneously perform keypoint localization and high-dimensional feature generation. It is commonly used in SLAM (Simultaneous Localization and Mapping).

[0011] SfM (Structure from Motion) is a 3D reconstruction method used to infer 3D information from a time-series of 2D images. The SfM method takes a series of images of the same scene as input and outputs the camera position and orientation for each image, as well as a 3D point cloud of the scene, typically a sparse point cloud. Similar 3D reconstruction methods include: MVS (Multi-View Stereo), SLAM (Simultaneous Localization and Mapping), and registration and mapping algorithms based on LiDAR point clouds.

[0012] 3DGS (3D Gaussian Scattering) is a method for representing and rendering 3D scenes based on Gaussian distributions. Its core idea is to represent a 3D scene as multiple learnable 3D Gaussian distributions (also known as 3D Gaussian spheres or Gaussian points), and to fit observation data (such as multi-view images) by optimizing these Gaussian distributions, ultimately achieving high-quality rendering effects. Each Gaussian contains the following parameters: (1) Position: the coordinates of the center of the 3D space; (2) Covariance matrix: controls the shape of the ellipsoid (scaling and rotation), and positive definiteness is ensured by decomposing the rotation matrix and scaling matrix; (3) Color: the view-dependent RGB values ​​are represented by spherical harmonics (SH), usually requiring 48 parameters; (4) Opacity: controls the contribution weight to the final image. Summary of the Invention

[0013] The purpose of this invention is to provide a 3DGS-based incremental reconstruction and rendering method for dynamic scenes that solves the problems of data redundancy, low efficiency of local updates, inconsistent textures, and difficulties in real-time rendering in current dynamic scene reconstruction, and can significantly improve reconstruction accuracy, rendering quality, and system response speed.

[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a dynamic scene incremental reconstruction and rendering method based on 3DGS, comprising the following steps;

[0015] S1, Construct an initial 3DGS model G0 based on the original image set, wherein the 3DGS model consists of multiple Gaussian points;

[0016] S2, acquire the image I from any viewpoint at time t in the dynamic scene. t And preprocessed into image frames After semantic segmentation and target recognition, the first update region is determined, including steps S21 to S24;

[0017] S21, preset the dynamic target category, specify the foreground category, and the background category, and acquire an image I from any viewpoint at the current time t from the dynamic scene. t Preprocessed into image frames ;

[0018] S22, based on Generate foreground region mask M seg and dynamic target mask M with semantic labels obj ;

[0019] S23, employing inter-frame optical flow calculation or image difference algorithms, in two consecutive frames I t-1 I tInter-pixel level motion vector estimation is performed, and a change confidence map C is constructed based on the motion vector. change ;

[0020] S24, C change M seg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is post-processed to update the region mask M. update and I t M update The region is designated as the first updated region;

[0021] S3 generates the incremental update region, including S31~S33;

[0022] S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ;

[0023] S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ;

[0024] S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region;

[0025] S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ;

[0026] S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ;

[0027] S6, using G t Real-time rendering generates renderings from any perspective.

[0028] As a preferred embodiment, S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model.

[0029] Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds;

[0030] Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

[0031] Preferably, S22 specifically includes:

[0032] right The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ;

[0033] right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

[0034] Preferably, in step S23, a change confidence map C is constructed according to the following formula. change ;

[0035] ,

[0036] Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at the x-coordinate. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

[0037] As a preferred option, in S24, C change M seg M obj Logical AND and fusion are performed in the pixel coordinate space, specifically by setting a preset change threshold θ. change , will C change C change (x)>θchange area, M seg The middle category is the foreground area, M obj Perform a logical AND operation on the regions of the semantically labeled dynamic target to obtain M. fusion ;

[0038] The post-processing involves smoothing M using connectivity analysis and morphological operations. fusion Edge, get M update .

[0039] Preferably, in S31, I is calculated pixel by pixel. t and I r Photometric error E at position x photo (x), and generate the difference confidence plot D according to the following formula. photo ;

[0040] ,

[0041] In the formula, D photo (x) is D photo At position x, the difference confidence level is given by exp(∙), where exp(∙) is the exp function and λ is the photometric error sensitivity adjustment factor.

[0042] Preferably, in S32, the local similarity matching is:

[0043] to I t and I r Keypoint detection and matching are performed to obtain sparse matching pairs and matching confidence scores. Then, a size and I are constructed. t For the same blank image, the matching confidence scores of sparse matching pairs are filled into the corresponding positions in the blank image, and the matching confidence scores are expanded to pixel-by-pixel matching confidence scores through guided interpolation to obtain the confidence map D. local And D local (x) is I t and I r Match confidence of the pixel at position x.

[0044] As a preferred option, S33 generates the probability change graph P according to the following formula. update And the threshold for judgment =0.5;

[0045] ,

[0046] In the formula, P update (x) is I t and I r The probability of change at position x, F multi (∙) is a multi-scale fusion function used to combine D local D semantic Dphoto Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), D photo (x) are respectively D local D semantic D photo The value at position x.

[0047] Preferably, S4 includes S41 to S43;

[0048] S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within;

[0049] S42, Preset optimization objective function L total and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ;

[0050] ,

[0051] In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point Color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

[0052] Preferably, step S5 includes steps S51 to S53;

[0053] S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ;

[0054] S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term Etex (G fusion ), calculate G according to the following formula fusion Geometric texture joint loss E fusion ;

[0055] ,

[0056] In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of );

[0057] S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

[0058] Regarding S2: This invention obtains the first updated region through S2, the purpose of which is to extract regions containing the possibility of dynamic targets or structural changes. This step generates a foreground region mask M through semantic segmentation and multi-object detection. seg and dynamic target mask M with semantic labels obj Change confidence map C is generated through motion estimation. change Then C change M seg M obj After fusion and post-processing, the updated region mask M is obtained. update .

[0059] Regarding S3: This invention obtains the incremental update region through S3, which is used to identify image regions that have significant deviations or added content from the original 3DGS model under the current image viewpoint, thereby limiting the subsequent modeling range and improving reconstruction efficiency. First, a difference confidence map D is generated based on the photometric error between the real image and the rendered image. photo D photo The value D at position x photo The larger (x) is, the more I t and I r The greater the difference, the more likely there are structural or texture changes; then, confidence map D is obtained through local similarity matching. local A semantic similarity feature map D is generated through global semantic feature extraction. semantic Finally, multi-scale fusion D photo D local D semantic Generation probability change graph P update , will P update Areas with values ​​greater than a preset threshold are identified as incremental update areas.

[0060] Regarding S4: This invention generates an optimized Gaussian point set using S4. .

[0061] In step S41, the first update region obtained from S2 and the incremental update region obtained from S3 are combined to generate the second update region, and a new set of points is generated within the second update region. And each The parameter definitions are consistent with the 3DGS model G0 at the initial moment in step S1. This process makes full use of the spatial continuity and structural redundancy of the existing model to quickly construct a Gaussian point initialization representation with geometric rationality and color approximation.

[0062] In S42, since the initialized Gaussian points are only approximately reasonable, an optimization objective function L is constructed to further improve the image reconstruction quality and geometric consistency. total ;

[0063] In S43, during the optimization process, a joint optimization algorithm based on image consistency is used to nonlinearly minimize the parameters of the candidate Gaussian points. This optimization process uses gradient descent to jointly adjust the parameters, so that the interpolation candidate points achieve a better state in terms of both visual effect and spatial arrangement, and finally outputs a high-quality optimized Gaussian point set. This provides data support for subsequent fusion and correction steps.

[0064] Regarding S5: The purpose is to... With G t-1 Integrate and optimize.

[0065] Among them, S51 will According to its spatial coordinates and G t-1 The positions of the Gaussian points are aligned. For point pairs with a distance less than a set threshold, spatial deduplication or weighted averaging strategies are used to retain only points with more stable features. New points in non-overlapping regions are directly merged into the global model. This fusion strategy ensures that information from newly added regions is seamlessly integrated into the original model while maintaining accuracy, and avoids redundant point clouds causing model bloat or rendering bottlenecks.

[0066] In S52, the boundary between the old and new regions may present issues such as abrupt color changes, inconsistent density, or geometric discontinuities. To address these issues, an energy function based on a graph structure is constructed, and the geometric smoothing term E is jointly optimized. geo (G fusion ) and texture consistency term E tex (G fusion This is to achieve a smooth transition in the boundary region. geo (G fusion Constraining the geometric smoothness between Gaussian points controls the structural continuity of the boundary region, E tex (G fusionThis measures the consistency of color distribution to avoid abrupt changes in texture stitching. By minimizing the energy function through graph optimization (such as the Gauss-Newton method or graph cut method), a smooth boundary transition and global texture fusion are ultimately achieved, resulting in an updated model with smooth boundaries and natural transitions.

[0067] Compared with the prior art, the advantages of the present invention are as follows:

[0068] (1) A sparse dynamic object representation structure is proposed. This structure uses sparsely distributed Gaussian points to uniformly model scene geometry and texture information, and supports region-level interpolation and density adjustment. In terms of point parameter settings, a joint encoding method that integrates spatial position, orientation, color and opacity is designed to make it have stronger representation capabilities while maintaining computational efficiency, and can better adapt to local modeling and reconstruction updates of dynamically changing regions.

[0069] (2) In step S2, a change detection method based on lightweight semantic prior and motion perception is designed. By introducing a lightweight semantic segmentation network and image difference-optical flow joint change estimation, a high-confidence update region mask M is constructed. update The first updated region was used to accurately locate the dynamic region. Combined with structure-preserving constraints, it effectively avoided spurious changes in the region from interfering with subsequent modeling.

[0070] (3) In step S3, a method is proposed to generate multiple generation probability change maps P by combining features such as photometric error, local similarity, and global semantic similarity. update The method of obtaining incremental update regions can effectively identify image regions where there are significant discrepancies or added content between the real image and the rendered image under the current image perspective, thereby limiting the scope of subsequent modeling and improving reconstruction efficiency.

[0071] (4) An incremental local reconstruction and optimization mechanism was constructed. A more accurate second update region was obtained by combining the first update region and the incremental update region. Candidate Gaussian points were generated in the second update region through neighborhood interpolation. Several loss constraint joint optimization strategies were proposed, including image reconstruction error, sparsity regularization and spatial density smoothing terms, which effectively improved the accuracy and structural consistency of local Gaussian point modeling.

[0072] (5) An efficient model fusion and consistency correction process was proposed. Redundant points were removed by spatial deduplication algorithm, and a graph optimization strategy was used to constrain the continuity of the fusion boundary, which solved the problems of model structure misalignment and texture breakage after multiple local updates, and enhanced the overall consistency and stability of the model.

[0073] In summary, this invention encodes the geometric and texture information of a scene through sparse dynamic object representation and structure, supports region-level interpolation and density adjustment, and achieves accurate positioning and high-quality local updates of dynamically changing regions. It takes into account both dynamic adaptability and real-time rendering requirements, and can significantly improve reconstruction accuracy, rendering quality and system response speed. Attached Figure Description

[0074] Figure 1 This is a flowchart of the present invention;

[0075] Figure 2 This is a flowchart of step S2 of the present invention;

[0076] Figure 3 This is a flowchart of step S3 of the present invention;

[0077] Figure 4 This is a flowchart of step S4 of the present invention;

[0078] Figure 5 This is a flowchart of step S5 of the present invention. Detailed Implementation

[0079] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0080] Example 1: See Figures 1 to 5 A dynamic scene incremental reconstruction and rendering method based on 3DGS includes the following steps;

[0081] S1, Construct an initial 3DGS model G0 based on the original image set, wherein the 3DGS model consists of multiple Gaussian points;

[0082] S2, acquire the image I from any viewpoint at time t in the dynamic scene. t And preprocessed into image frames After semantic segmentation and target recognition, the first update region is determined, including steps S21 to S24;

[0083] S21, preset the dynamic target category, specify the foreground category, and the background category, and acquire an image I from any viewpoint at the current time t from the dynamic scene. t Preprocessed into image frames ;

[0084] S22, based on Generate foreground region mask M seg and dynamic target mask M with semantic labels obj ;

[0085] S23, employing inter-frame optical flow calculation or image difference algorithms, in two consecutive frames I t-1 I tInter-pixel level motion vector estimation is performed, and a change confidence map C is constructed based on the motion vector. change ;

[0086] S24, C change M seg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is post-processed to update the region mask M. update and I t M update The region is designated as the first updated region;

[0087] S3 generates the incremental update region, including S31~S33;

[0088] S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ;

[0089] S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ;

[0090] S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region;

[0091] S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ;

[0092] S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ;

[0093] S6, using G t Real-time rendering generates renderings from any perspective.

[0094] Example 2: See Figures 1 to 5 S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model.

[0095] Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds;

[0096] Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

[0097] Specifically, S22 refers to: [the following text is incomplete and likely refers to a specific action or instruction]. The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ;right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

[0098] In step S23, a change confidence map C is constructed according to the following formula. change ;

[0099] ,

[0100] Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at coordinate x in the middle. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

[0101] In S24, C change M seg M obj Logical AND and fusion are performed in the pixel coordinate space, specifically by setting a preset change threshold θ. change , will C change C change (x)>θ change area, M seg The middle category is the foreground area, Mobj Perform a logical AND operation on the regions of the semantically labeled dynamic target to obtain M. fusion ;

[0102] The post-processing involves smoothing M using connectivity analysis and morphological operations. fusion Edge, get M update .

[0103] In S31, I is calculated pixel by pixel. t and I r Photometric error E at position x photo (x), and generate the difference confidence plot D according to the following formula. photo ;

[0104] ,

[0105] In the formula, D photo (x) is D photo At position x, the difference confidence level is given by exp(∙), where exp(∙) is the exp function and λ is the photometric error sensitivity adjustment factor.

[0106] In S32, the local similarity matching is:

[0107] to I t and I r Keypoint detection and matching are performed to obtain sparse matching pairs and matching confidence scores. Then, a size and I are constructed. t For the same blank image, the matching confidence scores of sparse matching pairs are filled into the corresponding positions in the blank image, and the matching confidence scores are expanded to pixel-by-pixel matching confidence scores through guided interpolation to obtain the confidence map D. local And D local (x) is I t and I r Match confidence of the pixel at position x.

[0108] S33 generates the probability change graph P according to the following formula. update And the threshold for judgment =0.5;

[0109] ,

[0110] In the formula, P update (x) is I t and I r The probability of change at position x, F multi (∙) is a multi-scale fusion function used to combine D local D semantic D photo Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), Dphoto (x) are respectively D local D semantic D photo The value at position x.

[0111] S4 includes S41~S43;

[0112] S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within;

[0113] S42, Preset optimization objective function L total and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ;

[0114] ,

[0115] In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point Color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

[0116] S5 as shown includes steps S51 to S53;

[0117] S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ;

[0118] S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term E tex (G fusion ), calculate G according to the following formula fusion Geometric texture joint loss E fusion ;

[0119] ,

[0120] In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of );

[0121] S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

[0122] Example 3: In this example, S6 specifically includes S61~S62, and the rest is the same as in Example 1 or Example 2.

[0123] S61, using G t When generating rendering images from any viewpoint in real time, a Gaussian Splatting rendering algorithm based on a GPU parallel architecture is used to render G... t Forward rendering is performed. Each frame of the image is generated by superimposing contributions from multiple Gaussian points. During the rendering process, the Gaussian contribution is rapidly accumulated for each pixel, achieving good real-time performance and anti-occlusion performance.

[0124] S62, to enhance realism, supports soft shadow simulation and ray reflection tracing (such as environment mapping and local reflections), controlling overall frame latency while ensuring visual quality: latency time Δt render ≤200ms ensures the system meets the requirements of highly responsive applications such as augmented reality and robot navigation.

[0125] Example 4: This example proposes a dynamic scene incremental reconstruction and rendering method based on 3DGS, designed as follows:

[0126] Dataset: Playground sequences from the industry-standard dynamic scene dataset Tanks and Temples and Campus sequences from the Dynamic Scene Dataset. The sequences contain moving pedestrians, vehicles, and camera motion, exhibiting high scene complexity, and consist of approximately 500 test images with a resolution of 1920×1080.

[0127] Experimental environment: The hardware environment consists of an NVIDIA RTX 4090 GPU and an Intel i9-13900K CPU, and the software environment consists of Ubuntu 20.04, PyTorch 2.0, and CUDA 11.8.

[0128] Experimental Groups:

[0129] Experimental group: The complete dynamic incremental reconstruction and rendering method proposed in this invention.

[0130] Control group: Model 1, using a simplified version of the 3DGS method that does not include incremental update region determination; Model 2, the classic voxel-Hashing incremental reconstruction method.

[0131] Experimental results: The common performance indicators of the two are compared and obtained in Table 1 below:

[0132] Table 1. Comparison of performance metrics for different models

[0133] Model Name Reconstruction Quality (PSNR) Rendering speed (FPS) Update region accuracy (IoU) Model update latency (ms) Model 1 30.1 55 0.72 52 Model 2 28.5 24 0.65 155 This invention model 32.8 59 0.85 48

[0134] Data Analysis: The model of this invention demonstrates significant advantages across all key performance indicators. In terms of reconstruction quality PSNR, this invention outperforms the two control groups by approximately 2.7 dB and 4.3 dB, respectively, proving that precise "second update region" determination and joint optimization effectively improve the geometric and texture fidelity of the model. In terms of update region accuracy IoU, this invention significantly outperforms the control group, indicating its ability to more accurately pinpoint areas of real change and avoid redundant calculations in unchanged areas. Regarding efficiency, the rendering speed (FPS) of this invention far exceeds that of the traditional voxel method, and the model update latency is only one-third of the latter, fully meeting the needs of real-time interactive applications. In summary, this invention, through its innovative dual-region determination and fusion mechanism, achieves highly efficient dynamic incremental updates while ensuring high-quality reconstruction.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic scene incremental reconstruction and rendering method based on 3DGS, characterized in that, Includes the following steps; S1, Construct an initial 3DGS model G0 based on the original image set, wherein the 3DGS model consists of multiple Gaussian points; S2, acquire the image I from any viewpoint at time t in the dynamic scene. t And preprocessed into image frames After semantic segmentation and target recognition, the first update region is determined, including steps S21 to S24; S21, preset the dynamic target category, specify the foreground category, and the background category, and acquire an image I from any viewpoint at the current time t from the dynamic scene. t Preprocessed into image frames ; S22, based on Generate foreground region mask M seg and dynamic target mask M with semantic labels obj ; S23, employing inter-frame optical flow calculation or image difference algorithms, in two consecutive frames I t-1 I t Inter-pixel level motion vector estimation is performed, and a change confidence map C is constructed based on the motion vector. change ; S24, C change M seg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is further processed to update the region mask M. update and I t M update The region is designated as the first updated region; S3 generates the incremental update region, including S31~S33; S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ; S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ; S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region; S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ; S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ; S6, using G t Real-time rendering generates rendered images from any perspective; S4 includes S41~S42; S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within; S42, Preset optimization objective function L total and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ; ， In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

2. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model. Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds; Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

3. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, Specifically, S22 is: right The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ; right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

4. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In step S23, a change confidence map C is constructed according to the following formula. change ; ， Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at the x-coordinate. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

5. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S24, C change M seg M obj Logical AND and fusion are performed in the pixel coordinate space, specifically by setting a preset change threshold θ. change , will C change C change (x)>θ change area, M seg The middle category is the foreground area, M obj Perform a logical AND operation on the regions of the semantically labeled dynamic target to obtain M. fusion ; The post-processing involves smoothing M using connectivity analysis and morphological operations. fusion Edge, get M update .

6. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S31, I is calculated pixel by pixel. t and I r Photometric error E at position x photo (x), and generate the difference confidence plot D according to the following formula. photo ; ， In the formula, D photo (x) is D photo At position x, the difference confidence level is given by exp(∙), where exp(∙) is the exp function and λ is the photometric error sensitivity adjustment factor.

7. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S32, the local similarity matching is: to I t and I r Keypoint detection and matching are performed to obtain sparse matching pairs and matching confidence scores. Then, a size and I are constructed. t For the same blank image, the matching confidence scores of sparse matching pairs are filled into the corresponding positions in the blank image, and the matching confidence scores are expanded to pixel-by-pixel matching confidence scores through guided interpolation to obtain the confidence map D. local And D local (x) is I t and I r Match confidence of the pixel at position x.

8. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S33 generates the probability change graph P according to the following formula. update And the threshold for judgment =0.5; ， In the formula, P update (x) is I t and I r The probability of change at position x, F multi (∙) is a multi-scale fusion function used to combine D local D semantic D photo Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), D photo (x) are respectively D local D semantic D photo The value at position x.

9. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S5 as shown includes steps S51 to S53; S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ; S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term E tex (G fusion ), calculate G according to the following formula fusion Geometric texture joint loss E fusion ; ， In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of ); S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

Images