Dynamic scene incremental reconstruction and rendering method based on 3DGS

Abstract

The invention discloses a dynamic scene incremental reconstruction and rendering method based on 3DGS, and belongs to the field of specific computer models, and the method comprises the steps: constructing a 3DGS model at an initial moment based on an original image set; obtaining any visual angle image at the moment t in the dynamic scene, and determining a first updating area through semantic segmentation and target recognition; generating an increment updating region based on the luminosity error, the local similarity and the global semantic feature; generating a second update region, modeling in the second update region, and minimizing region reconstruction errors to generate an optimized Gaussian point set; and fusing and optimizing the real-time model at the previous moment to obtain a real-time model Gt at the moment t for real-time rendering. According to the method, geometric and texture information of a scene can be effectively coded, accurate detection and local increment updating of a dynamic region are supported, Gaussian point parameters are optimized to improve the continuity and visual quality of a model, low-delay real-time rendering is realized, and the efficiency and quality of three-dimensional scene processing in a dynamic environment are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional computer vision and graphics, and in particular to a dynamic scene incremental reconstruction and rendering method based on 3DGS. BACKGROUND

[0002] With the rapid development of applications such as augmented reality (AR), robot navigation, and intelligent monitoring, efficient reconstruction and real-time rendering of dynamic three-dimensional scenes have become a research hotspot. Dynamic scenes usually contain multiple moving objects and environmental changes, and traditional static three-dimensional reconstruction methods cannot meet the requirements of high update frequency and dynamic adaptability. Existing technologies rely on dense three-dimensional point clouds or mesh representations, which have large data redundancy and high update costs, making it difficult to achieve real-time incremental updates. In addition, there are problems such as insufficient reconstruction accuracy and poor texture continuity in complex dynamic environments.

[0003] For the reconstruction problem of dynamic scenes, some studies use incremental update methods based on voxels or meshes, but are limited by storage and computing resources, making it difficult to support high resolution and large-scale dynamic changes. At the same time, existing methods do not make full use of semantic information, making it difficult to accurately locate and distinguish dynamic objects and static backgrounds, resulting in an excessively large update range or missing key change areas, reducing overall reconstruction efficiency and quality. In addition, real-time rendering methods mostly use traditional mesh-based rendering algorithms, which have high rendering delays when facing highly dynamic scenes, making it difficult to guarantee visual continuity and realism.

[0004] In recent years, scene representation based on three-dimensional Gaussian point cloud (3DGS) has gradually attracted attention due to its continuity and flexibility. 3DGS can encode scene geometry and texture through the spatial distribution and attributes of Gaussian points, support efficient interpolation and sparse representation, and facilitate local incremental update and multi-level detail adjustment. However, how to combine change detection, semantic recognition, and high-quality fusion in dynamic scenes to build an efficient and accurate dynamic three-dimensional reconstruction and rendering system remains a technical challenge.

[0005] Therefore, developing a dynamic scene incremental reconstruction and rendering method based on 3DGS can effectively encode the geometry and texture information of the scene, support accurate detection and local incremental update of dynamic regions, optimize Gaussian point parameters to improve model continuity and visual quality, and achieve low-delay real-time rendering, which is of great significance for improving the efficiency and quality of three-dimensional scene processing in dynamic environments.

[0006] Glossary: SIFT algorithm: (Scale Invariant Feature Transformation, SIFT) is a classic algorithm for extracting "significant" local feature points (also known as key points) from images. It generates a series of SIFT descriptors, also known as local descriptors, through several steps: scale space extreme detection, key point precise positioning, direction assignment, and descriptor generation.

[0007] ORB algorithm: (Oriented FAST and Rotated BRIEF) is an algorithm for fast feature point extraction and description, composed of two parts: Oriented FAST (Oriented FAST key point detection) and Rotated BRIEF (Rotated BRIEF descriptor).

[0008] FAST algorithm: (Features from Accelerated Segment Test) is a fast algorithm for detecting key points in images, originally used in computer vision and robotics. It is suitable for real-time processing, such as quickly identifying feature points in video streams or unmanned aerial vehicles.

[0009] SuperPoint is a deep learning-based self-supervised feature point detection and descriptor extraction algorithm that can simultaneously locate key points and generate high-dimensional features. It is commonly used in SLAM (Simultaneous Localization and Mapping).

[0010] SfM (Structure from motion) is a method of three-dimensional reconstruction, which is used to calculate 3D information from a series of 2D images. The input of the SfM method is a series of images of the same scene, and the output is the camera position and orientation corresponding to each image and the 3D point cloud in the scene, generally a sparse point cloud. Similar to SfM, three-dimensional reconstruction methods also include: MVS (Multi View Stereo) algorithm, SLAM (Simultaneous Localization and Mapping) algorithm, and laser radar point cloud registration and mapping algorithm.

[0011] 3DGS: (3D Gaussian Scattering, 3D high-spectral scattering) is a three-dimensional scene representation and rendering method based on Gaussian distribution. The core idea is to represent a three-dimensional scene as a plurality of learnable three-dimensional Gaussian distribution (also known as three-dimensional Gaussian ball, Gaussian point), and to optimize these Gaussian distributions to fit the observed data (such as multi-view images), and finally realize high-quality rendering effect. Each Gaussian contains the following parameters: (1) position: three-dimensional space center coordinates, (2) covariance matrix: control ellipsoid shape (scaling and rotation), ensure positive definiteness through rotation matrix and scaling matrix decomposition. (3) color: use spherical harmonics (Spherical Harmonics, SH) to represent the view-dependent RGB value, usually 48 parameters are needed. (4) opacity: control the contribution weight to the final image. SUMMARY

[0012] The purpose of the present application is to provide a kind of dynamic scene reconstruction based on 3DGS incremental reconstruction and rendering method for solving the problems of data redundancy, low local update efficiency, texture incoherence, real-time rendering difficulty and the like in current dynamic scene reconstruction, which can significantly improve reconstruction accuracy, rendering quality and system response speed.

[0013] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a kind of dynamic scene incremental reconstruction and rendering method based on 3DGS, comprising the following steps: S1, based on the initial time of the 3DGS model G0 of the original image set is constructed, the 3DGS model is composed of a plurality of Gaussian points; S2, any view image I t of time t in dynamic scene is acquired , preprocessed into image frame , after semantic segmentation and target identification, determine the first update area, including steps S21-S24; S21, preset dynamic target category, specify foreground category, background category, collect any view image I t of time t in dynamic scene , preprocessed into image frame ; S22, based on generate foreground region mask M seg And dynamic target mask M obj with semantic label; S23, adopt inter-frame optical flow calculation or image difference algorithm, estimate pixel-level motion vector between two continuous frames I t-1 , I t , construct change confidence map C change based on motion vector; S24, C change , Mseg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is further processed to update the region mask M. update and I t M update The region is designated as the first updated region; S3 generates the incremental update region, including S31~S33; S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ; S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ; S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region; S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ; S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ; S6, using G t Real-time rendering generates renderings from any perspective.

[0014] As a preferred embodiment, S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model. Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds; Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

[0015] Preferably, S22 specifically includes: right The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ; right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

[0016] Preferably, in step S23, a change confidence map C is constructed according to the following formula. change ; , Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at coordinate x in the middle. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

[0017] As a preferred option, in S24, C change M seg M obj Logical AND and fusion are performed in the pixel coordinate space, specifically by setting a preset change threshold θ. change , will C change C change (x)>θ change area, M seg The middle category is the foreground area, M obj Perform a logical AND operation on the regions of the semantically labeled dynamic target to obtain M. fusion ;

[0018] The post-processing involves smoothing M using connectivity analysis and morphological operations. fusion Edge, get M update .

[0019] Preferably, in S31, I is calculated pixel by pixel. t and I r Photometric error E at position xphoto (x), and generate the difference confidence plot D according to the following formula. photo ; , In the formula, D photo (x) is D photo At position x, the difference confidence level is given by exp(∙), where exp(∙) is the exp function and λ is the photometric error sensitivity adjustment factor.

[0020] Preferably, in S32, the local similarity matching is: to I t and I r Keypoint detection and matching are performed to obtain sparse matching pairs and matching confidence scores. Then, a size and I are constructed. t For the same blank image, the matching confidence scores of sparse matching pairs are filled into the corresponding positions in the blank image, and the matching confidence scores are expanded to pixel-by-pixel matching confidence scores through guided interpolation to obtain the confidence map D. local And D local (x) is I t and I r Match confidence of the pixel at position x.

[0021] As a preferred option, S33 generates the probability change graph P according to the following formula. update And the threshold for judgment =0.5; , In the formula, P update (x) is I t and I r The probability of change at position x, F multi (∙) is a multi-scale fusion function used to combine D local D semantic D photo Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), D photo (x) are respectively D local D semantic D photo The value at position x.

[0022] Preferably, S4 includes S41 to S43; S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within; S42, Preset optimization objective function Ltotal and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ; , In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

[0023] Preferably, step S5 includes steps S51 to S53; S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ; S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term E tex (G fusion ), calculate G according to the following formula fusion Geometric texture joint loss E fusion ; , In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of ); S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

[0024] Regarding S2: This invention obtains the first updated region through S2, the purpose of which is to extract regions containing the possibility of dynamic targets or structural changes. This step generates a foreground region mask M through semantic segmentation and multi-object detection. seg and dynamic target mask M with semantic labels obj, a change confidence map C is generated by motion estimation change , C is further processed change , M seg , M obj fusion and post-processing, to obtain an updated region mask M update .

[0025] Regarding S3: the present application obtains an incremental update region through S3, which is used to identify the image region with significant deviation from the original 3DGS model or new content under the current image view, so as to limit the subsequent modeling range and improve the reconstruction efficiency. First, a difference confidence map D is generated based on the photometric error of the real image and the rendered image photo , D photo The larger the value D photo (x) at position x is, the greater the difference between I t and I r is, and the more likely there is a structure or texture change; then a confidence map D local is obtained through local similarity matching, a semantic similarity feature map D semantic is generated by global semantic feature extraction, and finally a probability change map P photo , D local , D semantic is generated by multi-scale fusion update , and the region in P update greater than a predetermined decision threshold is determined as the incremental update region.

[0026] Regarding S4: the present application generates an optimized Gaussian point set through S4.

[0027] Among them, S41, the first update region obtained by combining S2 and the incremental update region obtained by S3 are used to generate a second update region, and a new point set is generated in the second update region, and each parameter definition is consistent with the initial 3DGS model G0 at the initial time in step S1, which fully utilizes the spatial continuity and structural redundancy of the existing model to quickly construct a Gaussian point initialization representation with geometric rationality and color approximation.

[0028] Among them, S42, since the initialized Gaussian point only has approximate rationality, in order to further improve its image reconstruction quality and geometric consistency, an optimization objective function L total is constructed. Among them, S43, when optimizing and solving, a joint optimization algorithm based on image consistency driving is used to nonlinearly minimize the candidate Gaussian point parameters. This optimization process jointly adjusts the parameters through the gradient descent method, so that the interpolation candidate points achieve an optimal state in terms of visual effect and spatial arrangement, and finally outputs a high-quality optimized Gaussian point set To provide data support for subsequent fusion and correction steps.

[0029] Regarding S5: the purpose is to align with G t-1 Fusion and optimization.

[0030] Wherein S51, will According to the spatial coordinates and the position relationship of the Gaussian points in G t-1 For the point pairs with a distance less than the set distance threshold, use spatial deduplication or weighted average strategy to retain only the points with more stable features; for the newly added points in the non-overlapping area, directly merge into the global model. This fusion strategy ensures that the new area information is seamlessly integrated into the original model while maintaining accuracy, and avoids model expansion or rendering bottlenecks caused by redundant point clouds.

[0031] Wherein S52, there may be color mutations, inconsistent densities, or geometric discontinuities at the junction of new and old areas. Therefore, an energy function based on graph structure is constructed to realize smooth transition of the boundary area by jointly optimizing the geometric smoothing term E geo (G fusion ) and the texture consistency term E tex (G fusion ). E geo (G fusion ) constrains the geometric smoothness between Gaussian points and controls the structural continuity of the boundary area, and E tex (G fusion ) measures the consistency of color distribution to avoid texture stitching mutations. Through graph optimization (such as Gauss-Newton method or graph cut method) to minimize the energy function, the final realization of boundary smooth transition and global texture fusion, get the boundary smooth, transition natural update model.

[0032] Compared with the prior art, the advantages of the present application are: (1) A sparse dynamic object representation structure is proposed. This structure uses sparse distributed Gaussian points to model scene geometry and texture information uniformly, and supports region-level interpolation and density adjustment. In the parameter setting of the points, a joint encoding method that fuses spatial position, direction, color and opacity is designed, which not only maintains computational efficiency but also has stronger representation ability, and can better adapt to local modeling and reconstruction update of dynamically changing areas.

[0033] (2) In step S2, a change detection method based on lightweight semantic prior and motion perception is designed. By introducing a lightweight semantic segmentation network and image difference-optical flow joint change estimation, a high-confidence update area mask M update and the first update area are constructed to realize accurate positioning of dynamic areas. Combined with the structure preservation constraint, the interference of false change areas on subsequent modeling is effectively avoided.

[0034] (3) In step S3, a multi-generation probability change map P is generated by combining photometric error, local similarity, global semantic similarity and other features update The method can effectively identify the image region with significant deviation or newly added content between the real image and the rendered image under the current image view angle, so as to limit the subsequent modeling range and improve the reconstruction efficiency.

[0035] (4) An incremental local reconstruction and optimization mechanism is constructed. A more accurate second update region is obtained by combining the first update region and the incremental update region, and a candidate Gaussian point is generated in the second update region by neighborhood interpolation. A multi-loss constraint joint optimization strategy is proposed, including image reconstruction error, sparsity regularization and spatial density smoothing term, which effectively improves the accuracy and structural consistency of local Gaussian point modeling.

[0036] (5) An efficient model fusion and consistency correction process is proposed. Redundant points are removed by a spatial deduplication algorithm, and a graph optimization strategy is used to constrain the continuity of the fusion boundary, solving the problem of model structure misplacement and texture breakage after multiple local updates, and enhancing the overall consistency and stability of the model.

[0037] In summary, the application encodes the geometric and texture information of the scene by sparse dynamic object representation and structure, supports regional interpolation and density adjustment, realizes accurate positioning and high-quality local update of dynamic changing regions, balances dynamic adaptability and real-time rendering requirements, and can significantly improve reconstruction accuracy, rendering quality and system response speed. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of the application; Figure 2 is a flowchart of step S2 of the application; Figure 3 is a flowchart of step S3 of the application; Figure 4 is a flowchart of step S4 of the application; Figure 5 is a flowchart of step S5 of the application. DETAILED DESCRIPTION

[0039] The application will be further described below with reference to the embodiments and the accompanying drawings.

[0040] Example 1: see Figures 1 to 5 A dynamic scene incremental reconstruction and rendering method based on 3DGS includes the following steps: S1, constructing an initial 3DGS model G0 based on an original image set, the 3DGS model being composed of a plurality of Gaussian points; S2, acquire the image I from any viewpoint at time t in the dynamic scene. t And preprocessed into image frames After semantic segmentation and target recognition, the first update region is determined, including steps S21 to S24; S21, preset the dynamic target category, specify the foreground category, and the background category, and acquire an image I from any viewpoint at the current time t from the dynamic scene. t Preprocessed into image frames ; S22, based on Generate foreground region mask M seg and dynamic target mask M with semantic labels obj ; S23, employing inter-frame optical flow calculation or image difference algorithms, in two consecutive frames I t-1 I t Inter-pixel level motion vector estimation is performed, and a change confidence map C is constructed based on the motion vector. change ; S24, C change M seg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is further processed to update the region mask M. update and I t M update The region is designated as the first updated region; S3 generates the incremental update region, including S31~S33; S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ; S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ; S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region; S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ; S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ; S6, using G t Real-time rendering generates renderings from any perspective.

[0041] Example 2: See Figures 1 to 5 S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model. Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds; Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

[0042] Specifically, S22 refers to: [the following text is incomplete and likely refers to a specific action or instruction]. The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ;right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

[0043] In step S23, a change confidence map C is constructed according to the following formula. change ; , Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at the x-coordinate. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

[0044] S24, C change , M seg , M obj Logical and fusion in pixel coordinate space, specifically, a preset change threshold θ change is set, C change , C change (x) > θ change , the region in M seg , the region in M obj , the region in M fusion ; The post-processing is to smooth the edge of M fusion using connectivity analysis and morphological operation, and obtain M update .

[0045] In S31, the light error E t (x) of I r and I photo at position x is calculated pixel by pixel, and the difference confidence map D photo is generated according to the following formula: , In the formula, D photo (x) is the difference confidence of D photo at position x, exp(·) is the exp function, and λ is the luminance error sensitivity adjustment factor.

[0046] In S32, the local similarity matching is: Key point detection and matching are performed on I t and I r to obtain sparse matching pairs and matching confidence, and a blank map with the same size as I t is constructed. The matching confidence of the sparse matching pairs is filled into the corresponding position of the blank map, and the matching confidence is expanded to pixel-by-pixel matching confidence through guided interpolation to obtain the confidence map D local , and D local (x) is the matching confidence of I t and I r at position x.

[0047] S33 generates the probability change map P update according to the following formula, and the determination threshold =0.5; , In the formula, P update (x) is the change probability of I t and I r at position x, and F multi is the Fisher information matrix.(∙) is a multi-scale fusion function used to combine D local D semantic D photo Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), D photo (x) are respectively D local D semantic D photo The value at position x.

[0048] S4 includes S41~S43; S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within; S42, Preset optimization objective function L total and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ; , In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

[0049] S5 as shown includes steps S51 to S53; S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ; S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term E tex (G fusion ), calculate G according to the following formula fusionGeometric texture joint loss E fusion ; , In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of ); S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

[0050] Example 3: In this example, S6 specifically includes S61~S62, and the rest is the same as in Example 1 or Example 2.

[0051] S61, using G t When generating rendering images from any viewpoint in real time, a Gaussian Splatting rendering algorithm based on a GPU parallel architecture is used to render G... t Forward rendering is performed. Each frame of the image is generated by superimposing contributions from multiple Gaussian points. During the rendering process, the Gaussian contribution is rapidly accumulated for each pixel, achieving good real-time performance and anti-occlusion performance.

[0052] S62, to enhance realism, supports soft shadow simulation and ray reflection tracing (such as environment mapping and local reflections), controlling overall frame latency while ensuring visual quality: latency time Δt render ≤200ms ensures the system meets the requirements of highly responsive applications such as augmented reality and robot navigation.

[0053] Example 4: This example proposes a dynamic scene incremental reconstruction and rendering method based on 3DGS, designed as follows: Dataset: Playground sequences from the industry-standard dynamic scene dataset Tanks and Temples and Campus sequences from the Dynamic Scene Dataset. The sequences contain moving pedestrians, vehicles, and camera motion, exhibiting high scene complexity, and consist of approximately 500 test images with a resolution of 1920×1080.

[0054] Experimental environment: The hardware environment consists of an NVIDIA RTX 4090 GPU and an Intel i9-13900K CPU, and the software environment consists of Ubuntu 20.04, PyTorch 2.0, and CUDA 11.8.

[0055] Experimental Groups: Experimental group: The complete dynamic incremental reconstruction and rendering method proposed in this invention.

[0056] Control group: Model 1, using a simplified version of 3DGS method without incremental update region determination, Model 2, the classic voxel-based incremental reconstruction method Voxel-Hashing.

[0057] Experimental results: The common performance indicators of the two are shown in the following Table 1: Table 1 Comparison of performance indicators of different models Model name Reconstruction quality (PSNR) Rendering speed (FPS) Update region accuracy (IoU) Model update delay (ms) Model one 30.1 55 0.72 52 Model two 28.5 24 0.65 155 Inventive model 32.8 59 0.85 48 Data analysis: The model of the present application shows significant advantages in all key performance indicators. In terms of reconstruction quality PSNR, the present application is about 2.7dB and 4.3dB higher than the two control groups respectively, which proves that through accurate "second update region" determination and joint optimization, the geometric and texture fidelity of the model can be effectively improved. In terms of update region accuracy IoU, the present application is far ahead, which shows that it can more accurately lock the real changed area and avoid redundant calculation in the unchanged area. In terms of efficiency, the rendering speed FPS of the present application is much higher than the traditional voxel method, and the model update delay is only one third of the latter, fully meeting the needs of real-time interactive applications. In summary, through the innovative dual region determination and fusion mechanism, the present application realizes high-efficiency dynamic incremental update while ensuring high-quality reconstruction.

[0058] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A dynamic scene incremental reconstruction and rendering method based on 3DGS, characterized in that, Includes the following steps; S1, Construct an initial 3DGS model G0 based on the original image set, wherein the 3DGS model consists of multiple Gaussian points; S2, acquire the image I from any viewpoint at time t in the dynamic scene. t And preprocessed into image frames After semantic segmentation and target recognition, the first update region is determined, including steps S21 to S24; S21, preset the dynamic target category, specify the foreground category, and the background category, and acquire an image I from any viewpoint at the current time t from the dynamic scene. t Preprocessed into image frames ; S22, based on Generate foreground region mask M seg and dynamic target mask M with semantic labels obj ; S23, employing inter-frame optical flow calculation or image difference algorithms, in two consecutive frames I t-1 I t Inter-pixel level motion vector estimation is performed, and a change confidence map C is constructed based on the motion vector. change ; S24, C change M seg M obj Perform a logical AND operation in the pixel coordinate space to generate a fusion mask M. fusion Then, it is further processed to update the region mask M. update and I t M update The region is designated as the first updated region; S3 generates the incremental update region, including S31~S33; S31, using the real-time model G at time t-1 t-1 Generate I t Renderings from the same perspective (I) r Based on the photometric errors of the two, a difference confidence map D is generated. photo ; S32, for I t and I r Perform local similarity matching to obtain the confidence graph D between the two. local , to I t and I r Global semantic features are extracted separately, and then semantic similarity is calculated pixel by pixel to generate a semantic similarity feature map D. semantic ; S33, for D photo D local D semantic Multi-scale fusion generates probability variation map P update , will P update Medium greater than the preset judgment threshold The region is identified as an incremental update region; S4. The overlapping area of ​​the first update region and the incremental update region is taken as the second update region and modeled within it. An optimization algorithm is used to minimize the region reconstruction error, generating an optimized Gaussian point set. ; S5, Real-time model G at time t-1 t-1 By fusing and optimizing the data, we obtain the real-time model G at time t. t ; S6, using G t Real-time rendering generates renderings from any perspective.

2. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S1 specifically involves: acquiring multi-view images of the initial moment of the dynamic scene to form an original image set, generating a sparse point cloud based on the original image set, selecting key spatial points from the sparse point cloud, and using the SfM algorithm to treat each key spatial point as a Gaussian point to generate an initial 3DGS model. Methods for generating sparse point clouds based on original image sets include SfM method, MVS method, SLAM method, and registration and mapping method based on lidar point clouds; Methods for extracting key spatial points include the SIFT algorithm, corner detection algorithm, ORB algorithm, FAST algorithm, and SuperPoint algorithm.

3. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, Specifically, S22 is: right The foreground is segmented using a semantic segmentation network, generating a foreground region mask M. seg ; right Perform multi-target detection, identify dynamic targets, and output a dynamic target mask M with semantic labels. obj .

4. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In step S23, a change confidence map C is constructed according to the following formula. change ; ， Among them, I t-1 For time t-1 and I t Images from the same or similar perspectives, C change (x) is C change The confidence level at position x in the middle. For I t-1 To I t The motion vector at position x, I t (x) is I t The color vector at coordinate x in the middle. For I t-1 In the middle, position The color vector at that location, Let be the magnitude of the vector.

5. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S24, C change M seg M obj Logical AND and fusion are performed in the pixel coordinate space, specifically by setting a preset change threshold θ. change , will C change C change (x)>θ change area, M seg The middle category is the foreground area, M obj Perform a logical AND operation on the regions of the semantically labeled dynamic target to obtain M. fusion ; The post-processing involves smoothing M using connectivity analysis and morphological operations. fusion Edge, get M update .

6. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S31, I is calculated pixel by pixel. t and I r Photometric error E at position x photo (x), and generate the difference confidence plot D according to the following formula. photo ; ， In the formula, D photo (x) is D photo At position x, the difference confidence level is given by exp(∙), where exp(∙) is the exp function and λ is the photometric error sensitivity adjustment factor.

7. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, In S32, the local similarity matching is: to I t and I r Keypoint detection and matching are performed to obtain sparse matching pairs and matching confidence scores. Then, a size and I are constructed. t For the same blank image, the matching confidence scores of sparse matching pairs are filled into the corresponding positions in the blank image, and the matching confidence scores are expanded to pixel-by-pixel matching confidence scores through guided interpolation to obtain the confidence map D. local And D local (x) is I t and I r Match confidence of the pixel at position x.

8. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S33 generates the probability change graph P according to the following formula. update And the threshold for judgment =0.5; ， In the formula, P update (x) is I t and I r The probability of change at position x, F multi (∙) is a multi-scale fusion function used to combine D local D semantic D photo Unify to the same scale, then perform pixel-by-pixel weighted fusion, D local (x), D semantic (x), D photo (x) are respectively D local D semantic D photo The value at position x.

9. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S4 includes S41~S43; S41, from G t-1 Extract the neighborhood of the second update region, and add Gaussian points within the neighborhood using a spatial interpolation algorithm to form a new point set. , for The i-th Gaussian point within; S42, Preset optimization objective function L total and minimize L total right By performing a nonlinear minimization solution, the optimized Gaussian point set is obtained. ; ， In the formula, L photo For G t-1 to I t and I r Reconstruction loss, α i α j They are respectively The i-th Gaussian point and the j-th Gaussian point Color value, x i x j They are respectively , The position, e is the natural constant, λ s λ is the weight of the sparsity regularization term. d This represents the weight of the spatial density continuity term.

10. The method for incremental reconstruction and rendering of dynamic scenes based on 3DGS according to claim 1, characterized in that, S5 as shown includes steps S51 to S53; S51, through a spatial deduplication strategy, With G t-1 Fusion into fusion model G fusion ; S52, Construct a graph-based geometric smoothing term E geo (G fusion ) and texture consistency term E tex (G fusion ), calculate G according to the following formula fusion Geometric texture joint loss E fusion ; ， In the formula, min(∙) is the minimum value, and λ tex For E tex (G fusion The weight of ); S53, to minimize E fusion Update G fusion The updated G fusion Marked as G t .

Images