A Method and System for Constructing Dynamic Environment 3D Gaussian Maps Based on Semantic Enhancement

Abstract

This invention relates to a method and system for constructing a dynamic 3D Gaussian map based on semantic enhancement, belonging to the field of computer vision and robot simultaneous localization and mapping technology. The method utilizes a keyframe scheduler to filter keyframes, performs semantic detection on the keyframes based on a semantic segmentation network to generate an initial semantic mask, and performs geometric verification and correction on the mask using depth reprojection residuals. The corrected mask is propagated to adjacent ordinary frames using camera pose-based reprojection, and temporal voting fusion is performed to obtain a temporally consistent semantic mask sequence. A 3D Gaussian sphere map is initialized and updated in static regions, and semantically guided adaptive density control is performed in dynamic regions to suppress dynamic artifacts. Lightweight map management is then performed based on information entropy scoring. Finally, under photometric error and geometric constraints, joint nonlinear optimization of camera pose and Gaussian sphere parameters is performed and iteratively updated in a closed loop.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and robot synchronous localization and mapping technology, and relates to a method and system for constructing dynamic environment 3D Gaussian maps based on semantic enhancement. Background Technology

[0002] With the rapid development of applications such as service robots, augmented reality / virtual reality (AR / VR), autonomous driving, and digital twins, systems need to achieve high-precision positioning and high-quality mapping in realistic 3D environments. In recent years, 3D representation methods driven by differentiable rendering have gradually emerged. Among them, scene representation based on 3D Gaussian spheres has gradually become an important technical route for new mapping and new perspective synthesis due to its high rendering efficiency and strong detail representation. In practical applications, the environment often contains dynamic targets such as pedestrians, vehicles, screens, and animals. Dynamic components can significantly interfere with pose estimation and map updates, resulting in map "ghosting," ghosting, local geometric collapse, and additional consumption of video memory and computing resources. Therefore, a 3D Gaussian map construction scheme that is oriented towards dynamic environments and balances robustness and lightweight design is needed.

[0003] However, existing technologies still have the following problems in 3D mapping of dynamic environments:

[0004] (i) A common approach in existing methods relies on semantic / instance detection of a single frame or a small number of key frames to remove dynamic regions. However, semantic detection is susceptible to occlusion, scale changes, motion blur and detection errors, which can lead to the omission of dynamic regions or the wrong deletion of static regions. Furthermore, if there is a lack of geometric consistency verification and cross-frame temporal constraints, mask jitter and error propagation can easily occur, causing dynamic targets to be incorrectly written into the map and forming obvious dynamic artifacts.

[0005] (ii) Another type of existing method focuses on geometric consistency (such as discrimination based on reprojection residuals, photometric consistency or depth consistency) to separate dynamics. However, this type of method is prone to misjudgment in scenes with weak texture, lighting changes or large depth noise. At the same time, for 3D Gaussian sphere maps, if adaptive density control and memory management are not performed for dynamic areas, the Gaussian sphere may continue to split / clone and accumulate in dynamic areas, resulting in rapid growth of memory usage, decreased real-time performance and difficulty in long-term stable operation. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for constructing a dynamic environment 3D Gaussian map based on semantic enhancement, so as to solve the problems mentioned in the background art, such as the easy introduction of artifacts by dynamic targets, missegmentation caused by single semantic / geometric discrimination, insufficient cross-frame consistency, and the easy expansion of Gaussian map in dynamic regions, which causes video memory pressure.

[0007] To achieve the above objectives, the present invention provides the following technical solution: On the one hand, a method for constructing dynamic environment 3D Gaussian maps based on semantic enhancement is provided, which includes the following steps: S1. Initialize camera intrinsic parameters and acquire image sequences in real time. Use a keyframe scheduler to filter keyframes based on displacement thresholds and the number of co-viewpoints to obtain a set of keyframes and a set of ordinary frames, and perform initial pose estimation. S2. Use the YOLOv11 instance segmentation and detection network to perform semantic detection on keyframes and generate an initial semantic mask. ; S3. Establish a geometric depth residual verification mechanism based on the depth reprojection residual of the initial semantic mask. Make corrections to the correction mask. Divide the region into dynamic and static areas; S4. Correct the keyframe mask by reprojecting it based on camera pose. Propagate to adjacent normal frames and perform temporal majority voting fusion within a sliding window to obtain a temporally consistent mask sequence; S5. Initialize or update the set of three-dimensional Gaussian spheres only within the static area. Each Gaussian sphere contains parameters such as position, covariance, opacity, and spherical harmonic function coefficients, forming a three-dimensional Gaussian map. S6. In the dynamic region, perform semantically guided adaptive density control on the 3D Gaussian sphere, including prohibiting splitting and cloning, and adjusting the opacity trimming threshold. S7. Calculate the information entropy score for all three-dimensional Gaussian spheres, and perform lightweight map management based on the information entropy score; S8. Using the photometric and depth errors obtained from dense differentiable rendering, a nonlinear optimization objective is constructed. Iteration is performed using a Riemannian manifold backtracking solver to achieve joint optimization of pose and 3D Gaussian sphere parameters, thus updating the current pose. Combined with map parameters, the optimization results are fed back to steps S4–S7 to form a closed loop, while the mask is also adjusted. Perform morphological operations and return to step S4 to continue propagation and optimization iterations.

[0008] Furthermore, in step S1, let the acquired RGB-D image stream be... ,in, It is an RGB image. For depth map, For frame index, Total number of frames; Keyframe scheduler filters keyframe set The criteria include motion amplitude and image quality. Motion amplitude includes translation distance and rotation angle indicators. Specifically, the translation distance between the current frame and the previous keyframe must be greater than a distance threshold, i.e.: The rotation angle between the current frame and the previous keyframe must be greater than the angle threshold, i.e.: Image quality refers to the gradient strength of a keyframe being greater than a gradient threshold, i.e.: gradient strength The keyframes selected based on the filtering criteria are then added to the keyframe set. Frames that do not meet the above filtering criteria are added to the set of adjacent normal frames. Adjacent ordinary frames refer to ordinary frames located between two keyframes in the time series. For each set of keyframes and adjacent normal frame set Initial pose estimation is performed on the image frames in the dataset. Specifically, an ensemble method is used to perform initial pose estimation on keyframes. The estimation result is obtained by minimizing the reprojection error for each key frame. The reprojection error is expressed as:

[0009] in, For perspective projection, From pixels With depth Inverse projection to a 3D point For the camera intrinsic parameter matrix, Weights for depth confidence; The set of adjacent normal frames is given by using a uniform motion model or motion tracking. A rough pose estimate.

[0010] Furthermore, in step S2, the process of performing semantic detection on keyframes using YOLOv11 is as follows: in each keyframe Perform YOLOv11 semantic detection on the RGB image. Adjust to YOLOv11 input size, perform forward propagation using the inference engine, and obtain each pixel. Belongs to the dynamic category initial probability And generate a binary mask. ,in, Initial threshold.

[0011] Furthermore, in step S3, the initial semantic mask is first checked using geometric depth residual verification. Perform geometric consistency verification and divide the region into dynamic and static areas, including: Depth residual calculation: for keyframes Internal markers are dynamic pixels Record its measured depth as In the previous keyframe Find the projection of the corresponding point of the pixel in the previous frame in 3D space. :

[0012] like Within the range of the previous frame, obtain the depth at that location. The reprojection depth is Calculate depth residual ;like If the image is not within the range of the previous frame, the corresponding depth cannot be obtained for geometric verification. In this case, the judgment result of the initial semantic mask is retained. ; Mask correction: If The pixel is determined to be truly dynamic and is retained. Otherwise, it is judged as a static background and corrected to... ; Dynamic and static area division: based on Divide the image region into static regions. With dynamic areas .

[0013] Furthermore, in step S4, the keyframe correction mask... The process of propagating to adjacent ordinary frames and performing temporal majority voting fusion to obtain a temporally consistent mask sequence is as follows: Set keyframes The adjacent normal frame sequences have , Set the propagation window size; obtain keyframes. Dynamic pixels on 3D coordinates The camera pose is transformed and then reprojected onto a normal frame. :

[0014] like Within the range of normal frames, the mask value of the keyframe is directly passed, i.e. ;like If a pixel is not within the range of a normal frame, its mask propagation is skipped; for cases where multiple source pixels project onto the same target pixel, the pixel with the highest probability is propagated. ; Then, in the sliding window Within, for ordinary frames Each pixel Collect the propagation mask from all visible keyframes. The final mask probability is formed using a time-series voting mechanism:

[0015] in, The number of keyframes in which this pixel is visible. By setting a binarization threshold, the dynamic mask of a normal frame can be obtained. .

[0016] Furthermore, in step S5, in the static region Within, pixels that are valid for depth values Perform 3D Gaussian sphere initialization or update: Initialization: If pixels Corresponding position in three-dimensional space If a Gaussian sphere does not exist, create a new Gaussian sphere. ,in The location of the three-dimensional center of the Gaussian sphere; For covariance, initialize to isotropic. ,in According to depth Adaptive setting of local gradient; Opacity; The coefficients are spherical harmonic function coefficients, which are determined by RGB values. The transformation yields the zeroth-order component: Higher-order coefficients are set to zero; Differentiable rendering: for the camera pose corresponding to the viewpoint , three-dimensional Gaussian sphere Projected onto screen space, set screen space pixels. The coordinates are The weights of the two-dimensional Gaussian distribution in screen space are defined as follows:

[0017] in, The projection of the three-dimensional center onto the screen. The projection of covariance onto screen space is expressed by the Jacobian. With rotation transformation calculate: ; For pixels Collect the set of all Gaussian spheres that affect this pixel. Colors are rendered by alpha blending and sorting by depth from front to back. With rendering depth :

[0018]

[0019] in For Gaussian sphere In the direction of observation The color of the spherical harmonic function below, The depth from the center of the Gaussian sphere to the camera; Parameter update: Under static masking, use the gradient pairs of photometric and geometric residuals. 、 、 、 Update; Furthermore, in step S6, in the dynamic region Internally, density control is performed on the 3D Gaussian sphere: Division and Cloning Control: For those located at Setting up a prohibition sign on a Gaussian ball Forcefully stop division and cloning operations; in the static region, the division trigger condition is... ; Opacity clipping threshold increase: Increases the opacity clipping threshold within the dynamic area from... Increase to , Output density-controlled 3D Gaussian maps.

[0020] Furthermore, in step S7, the lightweight map management based on information entropy includes: periodically calculating the information entropy score for all 3D Gaussian spheres.

[0021] in, For opacity, It can be defined as the volumetric scale of covariance. , 、 These are the weighting coefficients. The scale normalization constant is This is the scoring threshold; when If a value is deemed to have low contribution, it is removed; a simplified 3D Gaussian map is output.

[0022] Furthermore, in step S8, the camera pose and 3D Gaussian sphere parameters are jointly optimized nonlinearly, and the objective function is:

[0023] in, and In camera pose With 3D Gaussian sphere parameters Below, the rendering intensity and rendering depth are obtained through differentiable rendering. and These are the measured strength and the measured depth, respectively. For robust loss function, As a weighting factor, For the set of parameters of a 3D Gaussian sphere, The Lie algebra representation of the camera pose; Multi-scale bundle adjustment solution increment using Riemannian manifold backtracking solver and The Riemannian manifold backtracking solver iteratively optimizes across multiple resolution pyramid levels, establishing normal equations at each level:

[0024] in To unite Jacobi, For weighted matrices, For the residual vector, For LM damping factor; After obtaining the increment, the step size is determined using Armijo backtracking search. Each iteration verifies whether the Armijo condition is satisfied.

[0025] If the condition is not met, then contract. Until the Armijo condition is met; after determining the step size, update the camera pose using Lie algebras: in, for The hat operator, for parameters In 、 、 、 Synchronized Updates: ; For the mask Morphological operations include: morphological dilation and erosion of the semantic mask, and setting structuring elements. For a cross-shaped core, expansion and corrosion are defined as follows:

[0026] in , These represent expansion and corrosion operations, respectively.

[0027] On the other hand, a system for implementing the aforementioned semantically enhanced dynamic environment 3D Gaussian map construction method is also provided. This system includes an image acquisition module, a front-end perception module, a mask propagation module, a 3D Gaussian mapping module, a map management module, and a pose estimation module. The image acquisition module initializes the camera intrinsic parameters and acquires image sequences in real time. The keyframe scheduler filters keyframes based on displacement thresholds and the number of co-viewpoints to obtain a set of keyframes and a set of ordinary frames, and performs initial pose estimation. The front-end perception module uses the YOLOv11 instance segmentation and detection network to perform semantic detection on keyframes and generate an initial semantic mask. Furthermore, a geometric depth residual verification mechanism was established, based on the depth reprojection residual to verify the initial semantic mask. Make corrections to the correction mask. Divide the region into dynamic and static areas; The mask propagation module uses a camera pose-based reprojection method to correct the keyframe mask. Propagate to adjacent normal frames and perform temporal majority voting fusion within a sliding window to obtain a temporally consistent mask sequence; The 3D Gaussian mapping module initializes or updates a set of three-dimensional Gaussian spheres only in static regions. Each Gaussian sphere contains parameters such as position, covariance, opacity, and spherical harmonic function coefficients to form a three-dimensional Gaussian map. In dynamic regions, semantically guided adaptive density control is performed on the three-dimensional Gaussian spheres, including prohibiting splitting and cloning, and adjusting the opacity trimming threshold. The map management module calculates the information entropy score for all three-dimensional Gaussian spheres and performs lightweight map management based on the information entropy score. The pose estimation module performs joint nonlinear optimization of camera pose and 3D Gaussian sphere parameters based on photometric error and geometric constraints, and feeds the optimization results back to the mask propagation and mapping module to achieve closed loop.

[0028] The beneficial effects of this invention are as follows: (1) Based on semantic detection, this invention introduces geometric verification of depth reprojection residuals (GeoVerify) to correct the initial semantic mask, realize the complementarity of semantics and geometry, improve the accuracy of dynamic / static division, and suppress dynamic targets from being written into the map from the source, so as to solve the problem of "relying solely on semantic detection is prone to missed / false detection and leads to dynamic artifacts".

[0029] (2) This invention achieves mask propagation (MaskProp) based on camera pose reprojection and performs temporal majority voting fusion within a sliding window to obtain a temporally consistent semantic mask sequence, thereby improving the stability of dynamic region annotation, reducing false propagation and ghosting phenomena, and solving the problem of "mask jitter caused by lack of cross-frame constraints and accumulation of dynamic artifacts across frames".

[0030] (3) The present invention performs semantically guided adaptive density control in dynamic regions, prohibits Gaussian sphere splitting and cloning and increases the opacity trimming threshold, thereby avoiding disorderly growth of the number of Gaussian spheres in dynamic regions, reducing dynamic artifacts and redundancy, and solving the problem of "map expansion and real-time performance degradation caused by continuous splitting / cloning of Gaussian spheres in dynamic regions".

[0031] (4) In response to the problem of “high memory pressure during long-term operation and difficulty in stable online mapping”, this invention proposes a lightweight map management mechanism based on information entropy. By scoring the contribution of the Gaussian sphere and periodically and asynchronously removing low contribution elements, the memory usage can be controlled, thereby improving the stability and deployment feasibility of the system during long-term operation.

[0032] (5) This invention combines the Riemannian manifold backtracking solver (RMBS) framework with Armijo backtracking line search to perform joint nonlinear optimization of photometric error and geometric constraints. It can simultaneously optimize camera pose and Gaussian sphere parameters, forming a closed-loop iterative update, taking into account both positioning accuracy and map rendering quality. It is suitable for high-quality 3D Gaussian mapping and rendering output in dynamic environments.

[0033] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall process of the dynamic environment 3D Gaussian map construction method based on semantic enhancement according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the interactive correction process of semantic detection and geometric depth residual verification in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the timing consistency of mask propagation and cross-frame fusion in an embodiment of the present invention; Figure 4 This is a schematic diagram of adaptive Gaussian sphere density control and lightweight information entropy management according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the joint optimization closed loop based on photometric error and geometric constraints according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structural framework of the dynamic environment 3D Gaussian map construction system based on semantic enhancement according to an embodiment of the present invention. Detailed Implementation

[0035] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0037] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0038] Please see Figures 1-6 This is a method and system for constructing dynamic environment 3D Gaussian maps based on semantic enhancement.

[0039] Example 1 This embodiment first provides a method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement, such as... Figure 1 As shown, it includes at least the following steps: S1. System Initialization and Data Acquisition: Initialize the semantic detection model within the camera and acquire RGB data in real time. The D image sequence is processed by the keyframe scheduler (KFScheduler) to select keyframes based on the displacement threshold and the number of co-viewpoints, and outputs a set of keyframes, a set of adjacent ordinary frames, and an initial pose estimate for reprojection. S2. Semantic Detection: The YOLOv11 instance segmentation and detection network is used to generate initial semantic masks for keyframes. (Dynamic categories: people, vehicles, pets, cycling), input resolution 640×640; S3. GeoVerify: Based on the depth reprojection residual of the initial semantic mask. Make corrections to obtain the corrected semantic mask. And static / dynamic region division, where pixels satisfy Marked as dynamic; S4. Mask Propagation: This modifies the semantic mask. A camera pose-based reprojection method is used to propagate to adjacent normal frames, and temporal majority voting fusion is performed within a sliding window to obtain a temporally consistent mask sequence. S5. Initialization and Update of 3D Gaussian Spheres: Initializes or updates the set of 3D Gaussian spheres only within the static area. Each Gaussian sphere contains parameters such as position, covariance, opacity, and spherical harmonic function coefficients, forming a 3D Gaussian map. S6. Semantic-guided adaptive density control: Performs density control on a 3D Gaussian sphere within a dynamic region, including disabling splitting and cloning, and adjusting the opacity trimming threshold from a first threshold. Raise to the second threshold ,and ; S7. Lightweight Map Management Based on Information Entropy: Calculate information entropy scores for all 3D Gaussian spheres and periodically remove maps with scores below a threshold. A Gaussian sphere is used to control video memory usage; S8. Joint Optimization and Closed-Loop Feedback: A nonlinear optimization objective is constructed using the photometric and depth errors obtained from dense differentiable rendering. Iteration is performed using a Riemannian manifold backtracking solver (RMBS), with the step size determined by an Armijo backtracking line search. This completes the joint optimization of the pose and the 3D Gaussian sphere parameters, updating the current pose. The optimization results are fed back to steps S4–S7 to form a closed loop, along with map parameters. Morphological operations are performed on the mask to improve stability, allowing it to continue participating in propagation and optimization iterations.

[0040] In step S1 of this embodiment, image data input and initial pose estimation are performed.

[0041] Receive RGB-D image stream ,in It is an RGB image. For depth map, For frame index, To determine the total number of frames, a keyframe set is selected using an image quality-based keyframe scheduler (KF Scheduler). .

[0042] The criteria for keyframe selection are: Motion amplitude: Translation distance between the current frame and the previous keyframe or rotation angle In this embodiment m, rad.

[0043] Image quality: Calculating gradient strength ,Require In this embodiment, .

[0044] Initial pose estimation is performed on the keyframes using a geometric method: for the first keyframe... Each keyframe minimizes the reprojection error using RGB-D-ICP. ,in For perspective projection, For inverse projection (from pixel) With depth (to three-dimensional point) For the camera intrinsic parameter matrix, Weights for depth confidence. Estimation results. Used as initial values ​​for subsequent optimization.

[0045] Frames that do not meet the above filtering criteria are added to the set of adjacent normal frames. Adjacent ordinary frames refer to ordinary frames located between two keyframes in the time series; for non-keyframes (ordinary frames)... A rough pose is given by using a uniform motion model or motion tracking.

[0046] The interactive correction process of semantic detection and geometric depth residual verification in steps S2-S4 is as follows: Figure 2 As shown.

[0047] In step S2 of this embodiment, YOLOv11 semantic detection is performed. In each keyframe... Perform YOLOv11 semantic detection on the RGB image. Adjust the input size to 640×640 using YOLOv11, and perform forward propagation using an inference engine (such as TensorRT) to obtain the values ​​for each pixel. Belongs to the dynamic category Initial probability (such as pedestrians, pets, vehicles, etc.) And generate a binary mask. ,in The initial threshold is used. YOLOv11 output includes not only class probabilities but also bounding box coordinates and confidence scores; this embodiment directly uses the pixel-level mask output by the segmentation head. .

[0048] In step S3 of this embodiment, GeoVerify geometric depth residual verification is performed.

[0049] Using the GeoVerify module Perform a geometric consistency check. The specific steps are as follows: Depth residual calculation: for keyframes Internal markers are dynamic pixels Record its measured depth as In the previous keyframe Find the projection of the corresponding point in 3D space of the pixel in the previous frame:

[0050] like Within the range of the previous frame, obtain the depth at that location. The reprojection depth is Calculate depth residual .like If the image is not within the range of the previous frame, the corresponding depth cannot be obtained for geometric verification. In this case, the judgment result of the initial semantic mask is retained. .

[0051] Mask correction: If (This embodiment takes) m), determine that the pixel is truly dynamic, and retain it. Otherwise, it is judged as a static background and corrected to... This yields the corrected semantic mask. .

[0052] Static / dynamic zone division: based on Divide the image region into static regions. With dynamic areas This is for use by subsequent modules.

[0053] In step S4 of this embodiment, MaskProp is performed based on pose propagation and consistency timing voting fusion.

[0054] like Figure 3 As shown, the MaskProp module is used to apply a correction mask to the keyframe. Propagate to adjacent regular frames. Consider keyframes. With its subsequent ordinary frame sequence , For the propagation window size (in this embodiment) ).

[0055] Reprojection propagation based on camera pose: for keyframes Dynamic pixels on Obtain its three-dimensional coordinates The camera pose is transformed and then reprojected onto a normal frame. :

[0056] like Within the normal frame range, initialization is performed. (Passing the mask value of the keyframe directly), if If the pixel is not within the range of a normal frame, then the mask propagation for that pixel is skipped.

[0057] When multiple source pixels are projected onto the same target pixel, take the highest probability:

[0058] Timing-based voting fusion: in a sliding window Within, for ordinary frames Each pixel Collect the propagation mask from all visible keyframes. The final mask probability is formed using a time-series voting mechanism:

[0059] in This represents the number of keyframes in which this pixel is visible. The binarization threshold is set to... To obtain the dynamic mask of a normal frame. .

[0060] In step S5 of this embodiment, the 3D Gaussian sphere is initialized and updated.

[0061] In the static area Within, pixels that are valid for depth values Perform 3D Gaussian sphere initialization or update: 1) Initialization: If pixels Corresponding position in three-dimensional space If a Gaussian sphere does not already exist, create a new Gaussian sphere. Covariance is initialized to isotropic. ,in According to depth With local gradient adaptive setting, for example Opacity Initialized to 0.5; spherical harmonic coefficients From RGB values The transformation yields the zeroth-order component: Higher-order coefficients are set to zero.

[0062] 2) Differentiable rendering: for the camera pose corresponding to the viewpoint , three-dimensional Gaussian sphere Project onto screen space. Set screen space pixels. The coordinates are The weights of the two-dimensional Gaussian distribution in screen space are defined as follows:

[0063] in The projection of the three-dimensional center onto the screen. The projection of covariance onto screen space is expressed by the Jacobian. With rotation transformation calculate: For pixels Collect the set of all Gaussian spheres that affect this pixel. (Fast lookup via screen space bounding box index), sorted by depth from front to back, and rendered colors using alpha blending. With rendering depth :

[0064]

[0065] in For Gaussian sphere In the direction of observation The color of the spherical harmonic function below, The distance from the center of the Gaussian sphere to the camera.

[0066] 3) Parameter update: Under static masking, the gradients of the photometric residual and the geometric residual are used to update the parameters. 、 、 、 Update. Using photometric residuals. For example, the gradient at the center position is approximated as:

[0067] Update using Adam optimizer and (learning rate) , The pose and geometric parameters will be uniformly optimized in S8. Output the initial or updated 3D Gaussian map. .

[0068] Figure 4 The diagram illustrates the adaptive Gaussian sphere density control and lightweight information entropy management in steps S6-S7.

[0069] In step S6 of this embodiment, semantically guided adaptive Gaussian sphere density control is performed.

[0070] In the dynamic area Internally, density control is performed on the 3D Gaussian sphere: 1) Division and cloning control: for those located at Setting up a prohibition sign on a Gaussian ball Forcefully stop division and cloning operations; in the static region, the division trigger condition is... (This embodiment) ).

[0071] 2) Increased Opacity Clipping Threshold: The opacity clipping threshold within the dynamic area is increased from... Increase to ,For example , This more actively removes low-contribution dynamic Gaussian spheres, resulting in a density-controlled 3D Gaussian map.

[0072] In step S7 of this embodiment, lightweight map management based on information entropy is performed.

[0073] Periodic (every) For each frame, an information entropy score is calculated for all 3D Gaussian spheres, using the following evaluation function:

[0074] in, For opacity, It can be defined as the volumetric scale of covariance. , 、 Weighting coefficients (in this embodiment) , ), Let be the scale normalization constant (taken as 0.02m). The scoring threshold (take) ).when If an element is identified as having low contribution, it is removed from the rendering and optimization process. This removal is performed in a background thread to ensure the real-time performance of the main rendering and optimization thread. After removal, the screen space index is updated to avoid dangling references. A simplified 3D Gaussian map is then output.

[0075] In step S8 of this embodiment, RMBS+Armijo joint optimization is performed, and the specific process is as follows: Figure 5 As shown: Joint nonlinear optimization of camera pose and 3D Gaussian sphere parameters is performed. The optimization objective function is:

[0076] in, and In camera pose (correspond ) and 3D Gaussian sphere parameters Below, the rendering intensity and rendering depth are obtained through differentiable rendering. and These are the measured strength and the measured depth, respectively. For robust loss functions (Charbonnier is used in this embodiment): , ), The weighting factor (in this embodiment) ), For the set of parameters of a 3D Gaussian sphere, The Lie algebra representation of the camera pose.

[0077] Multi-scale bundle adjustment solution increments using the Riemannian manifold backtracking solver (RMBS) framework and RMBS iteratively optimizes across multiple resolution pyramid levels, establishing a normal equation for each level:

[0078] in To unite Jacobi, The weighted matrix (based on mask and depth confidence) For the residual vector, The LM damping factor is used (adaptive adjustment, initial value 1.0). After obtaining the increment, the step size is determined using an Armijo backtracking search. Let the initial step size be... Each iteration verifies whether the Armijo conditions are satisfied:

[0079] If the condition is not met, then contract. Until the Armijo condition is met, After determining the step size, update the camera pose using Lie algebra: in for The hat operator. For parameters In 、 、 、 Synchronized Updates: In this embodiment, , .

[0080] Optimization is performed within a sliding window (window size) The optimization results are fed back to the mask propagation module and the 3D Gaussian mapping module to form a closed loop, thereby achieving iterative refinement.

[0081] Perform morphological operations on the mask.

[0082] To further smooth the edges of the semantic mask and eliminate isolated noise, morphological dilation and erosion are performed on the semantic mask. Let the structuring element be... for The cross-shaped core, expansion, and corrosion are defined as follows:

[0083] in , These represent the dilation and erosion operations, respectively. This embodiment employs sequential operations: one round of dilation followed by two rounds of erosion to fill boundary holes and suppress false detections. The output mask from step S9 re-enters step S4 to participate in propagation and fusion.

[0084] Parameter settings and optimizer details (through S3 / S4 / S5 / S8) Learning rate and weights: The spherical harmonics of the rendered colors are optimized using the Adam optimizer. Opacity Using Adam, and apply Clamping; the pose adopts RMBS+Armijo, and the damping is automatically adjusted according to the error descent rate; , .

[0085] Sampling strategy: Uniform sampling per frame pixel subset (For example, 10k pixels per frame) are used for optimization to ensure real-time performance. Sampling is prioritized from high gradient regions and high-confidence depth regions.

[0086] Screen space index: Construct a two-dimensional grid hash index, registering the screen projection bounding box of each Gaussian sphere to a grid bucket, so as to... Approximate query .

[0087] Parallelization: Rendering, optimization, and culling run on independent CUDA streams, mask propagation is performed on the CPU, and pinned memory is used to reduce PCIe copy latency.

[0088] All steps rely on available open-source components and standard mathematical tools: the YOLO series of networks are used for semantic detection, ICP / PnP for pose initialization, and RMBS+Armijo for optimization; publicly available implementations of 3D Gaussian differentiable rendering and screen space indexing are available for reference. Parameter ranges and formulas are given clear physical meanings and units, and the interfaces between modules are... Figure 2 The invention is clearly defined, therefore it is clearly feasible.

[0089] Example 2 This embodiment provides a system for executing the aforementioned semantically enhanced dynamic environment 3D Gaussian map construction method. Specifically, as follows: Figure 6 As shown, the system includes an image acquisition module, a front-end perception module, a mask propagation module (MaskProp), a 3D Gaussian mapping module, a map management module, and a pose estimation module (RMBS+Armijo). The front-end perception module performs YOLOv11 semantic detection and GeoVerify geometric depth residual verification on keyframes, and publishes semantic masks and static / dynamic region divisions. The mask propagation module receives keyframe masks, propagates the masks to adjacent ordinary frames based on camera pose reprojection, performs temporal voting fusion, and publishes the mask sequence. The 3D Gaussian mapping module initializes or updates the 3D Gaussian sphere (position, covariance, opacity, spherical harmonic coefficients) in static regions and performs density control (split / cloning prohibition, opacity threshold increase) in dynamic regions. The map management module periodically calculates information entropy scores and removes low-contribution Gaussian spheres, controlling memory usage. The pose estimation module performs joint nonlinear optimization of camera pose and 3D Gaussian sphere parameters based on photometric error and geometric constraints, and feeds the optimization results back to the mask propagation and mapping modules to achieve a closed loop.

[0090] Image acquisition module: Subscribes to and publishes / rgb_raw and / depth_raw, provides initial pose estimation / pose_init; exchanges data with the GPU via PCIe x16, with a depth timestamp alignment error of <2ms.

[0091] Front-end perception module: Performs YOLOv11 semantic detection and GeoVerify geometric depth residual verification on keyframes, and publishes / semantic_mask and / static_dynamic_split.

[0092] Mask Propagation Module (MaskProp): Receives keyframe masks, propagates the masks to adjacent ordinary frames based on camera pose reprojection, performs temporal voting fusion, and publishes / mask_seq.

[0093] 3D Gaussian mapping module: Initializes or updates 3D Gaussian spheres (position, covariance, opacity, spherical harmonic coefficients) in static regions, and performs density control (split / cloning prohibition, opacity threshold increase) in dynamic regions.

[0094] Map management module: Periodically calculates information entropy scores and removes low-contribution Gaussian spheres, controlling video memory usage (target <18GB).

[0095] The pose estimation module (RMBS+Armijo) performs joint nonlinear optimization of camera pose and 3D Gaussian sphere parameters based on photometric error and geometric constraints, and feeds the optimization results back to the mask propagation and mapping module to achieve closed loop.

[0096] In indoor dynamic scene testing, the system ran at 20–30 FPS. Compared to the baseline method without semantic enhancement, the Gaussian map showed no obvious dynamic artifacts in the presence of moving pedestrians and pets. Joint photometric-geometric optimization reduced the pose error RMSE by approximately 28%, and the map's memory usage remained stable at 16–18 GB during long-term operation (>10 min). Information entropy-driven culling reduced redundant Gaussian spheres by approximately 6–9% per cycle. The resulting 3D Gaussian map can be used for AR rendering and robot navigation. The residual distribution between the rendered depth and the measured depth exhibits zero mean and finite variance, demonstrating geometric consistency.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement, characterized in that: The method includes the following steps: S1. Initialize camera intrinsic parameters and acquire image sequences in real time. Use a keyframe scheduler to filter keyframes based on displacement thresholds and the number of co-viewpoints to obtain a set of keyframes and a set of ordinary frames, and perform initial pose estimation. S2. Use the YOLOv11 instance segmentation and detection network to perform semantic detection on keyframes and generate an initial semantic mask. ; S3. Establish a geometric depth residual verification mechanism based on the depth reprojection residual of the initial semantic mask. Make corrections to the correction mask. Divide the region into dynamic and static areas; S4. Correct the keyframe mask by reprojecting it based on camera pose. Propagate to adjacent normal frames and perform temporal majority voting fusion within a sliding window to obtain a temporally consistent mask sequence; S5. Initialize or update the set of three-dimensional Gaussian spheres only within the static area. Each Gaussian sphere contains parameters such as position, covariance, opacity, and spherical harmonic function coefficients, forming a three-dimensional Gaussian map. S6. In the dynamic region, perform semantically guided adaptive density control on the 3D Gaussian sphere, including prohibiting splitting and cloning, and adjusting the opacity trimming threshold. S7. Calculate the information entropy score for all three-dimensional Gaussian spheres, and perform lightweight map management based on the information entropy score; S8. Using the photometric and depth errors obtained from dense differentiable rendering, a nonlinear optimization objective is constructed. Iteration is performed using a Riemannian manifold backtracking solver to achieve joint optimization of pose and 3D Gaussian sphere parameters, thus updating the current pose. Combined with map parameters, the optimization results are fed back to steps S4–S7 to form a closed loop, while the mask is also adjusted. Perform morphological operations and return to step S4 to continue propagation and optimization iterations.

2. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 1, characterized in that: In step S1, let the acquired RGB-D image stream be... ,in, It is an RGB image. For depth map, For frame index, Total number of frames; Keyframe scheduler filters keyframe set The criteria include motion amplitude and image quality. Motion amplitude includes translation distance and rotation angle indicators. Specifically, the translation distance between the current frame and the previous keyframe must be greater than a distance threshold, i.e.: The rotation angle between the current frame and the previous keyframe must be greater than the angle threshold, i.e.: Image quality refers to the gradient strength of a keyframe being greater than a gradient threshold, i.e.: gradient strength The keyframes selected based on the filtering criteria are then added to the keyframe set. Frames that do not meet the above filtering criteria are added to the set of adjacent normal frames. Adjacent ordinary frames refer to ordinary frames located between two keyframes in the time series. For each set of keyframes and adjacent normal frame set Initial pose estimation is performed on the image frames in the dataset. Specifically, an ensemble method is used to perform initial pose estimation on keyframes. The estimation result is obtained by minimizing the reprojection error for each key frame. The reprojection error is expressed as: in, For perspective projection, From pixels With depth Inverse projection to a 3D point For the camera intrinsic parameter matrix, Weights for depth confidence; The set of adjacent normal frames is given by using a uniform motion model or motion tracking. A rough pose estimate.

3. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 2, characterized in that: In step S2, the process of performing semantic detection on keyframes using YOLOv11 is as follows: In each keyframe Perform YOLOv11 semantic detection on the RGB image. Adjust to YOLOv11 input size, perform forward propagation using the inference engine, and obtain each pixel. Belongs to the dynamic category initial probability And generate a binary mask. ,in, Initial threshold.

4. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 3, characterized in that: In step S3, the initial semantic mask is first checked using geometric depth residual verification. Perform geometric consistency verification and divide the region into dynamic and static areas, including: Depth residual calculation: for keyframes Internal markers are dynamic pixels Record its measured depth as In the previous keyframe Find the projection of the corresponding point of the pixel in the previous frame in 3D space. : like Within the range of the previous frame, obtain the depth at that location. The reprojection depth is Calculate depth residual ;like If the image is not within the range of the previous frame, the corresponding depth cannot be obtained for geometric verification. In this case, the judgment result of the initial semantic mask is retained. ; Mask correction: If The pixel is determined to be truly dynamic and is retained. Otherwise, it is judged as a static background and corrected to... ; Dynamic and static area division: based on Divide the image region into static regions. With dynamic areas .

5. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 4, characterized in that: In step S4, the keyframe correction mask The process of propagating to adjacent ordinary frames and performing temporal majority voting fusion to obtain a temporally consistent mask sequence is as follows: Set keyframes The adjacent normal frame sequences have , Set the propagation window size; obtain keyframes. Dynamic pixels on 3D coordinates The camera pose is transformed and then reprojected onto a normal frame. : like Within the range of normal frames, the mask value of the keyframe is directly passed, i.e. ;like If a pixel is not within the range of a normal frame, its mask propagation is skipped; for cases where multiple source pixels project onto the same target pixel, the pixel with the highest probability is propagated. ; Then, in the sliding window Within, for ordinary frames Each pixel Collect the propagation mask from all visible keyframes. The final mask probability is formed using a time-series voting mechanism: in, The number of keyframes in which this pixel is visible. By setting a binarization threshold, the dynamic mask of a normal frame can be obtained. .

6. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 5, characterized in that: In step S5, in the static region Within, pixels that are valid for depth values Perform 3D Gaussian sphere initialization or update: Initialization: If pixels Corresponding position in three-dimensional space If a Gaussian sphere does not exist, create a new Gaussian sphere. ,in The location of the three-dimensional center of the Gaussian sphere; For covariance, initialize to isotropic. ,in According to depth Adaptive setting of local gradient; Opacity; The coefficients are spherical harmonic function coefficients, which are determined by RGB values. The transformation yields the zeroth-order component: Higher-order coefficients are set to zero; Differentiable rendering: for the camera pose corresponding to the viewpoint , three-dimensional Gaussian sphere Projected onto screen space, set screen space pixels. The coordinates are The weights of the two-dimensional Gaussian distribution in screen space are defined as follows: in, The projection of the three-dimensional center onto the screen. The projection of covariance onto screen space is expressed by the Jacobian. With rotation transformation calculate: ; For pixels Collect the set of all Gaussian spheres that affect this pixel. Colors are rendered by alpha blending and sorting by depth from front to back. With rendering depth : in For Gaussian sphere In the direction of observation The color of the spherical harmonic function below, The depth from the center of the Gaussian sphere to the camera; Parameter update: Under static masking, use the gradient pairs of photometric and geometric residuals. 、 、 、 Update.

7. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 6, characterized in that: In step S6, in the dynamic region Internally, density control is performed on the 3D Gaussian sphere: Division and Cloning Control: For those located at Setting up a prohibition sign on a Gaussian ball Forcefully stop division and cloning operations; in the static region, the division trigger condition is... ; Opacity clipping threshold increase: Increases the opacity clipping threshold within the dynamic area from... Increase to , Output density-controlled 3D Gaussian maps.

8. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 7, characterized in that: In step S7, lightweight map management based on information entropy includes: periodically calculating information entropy scores for all 3D Gaussian spheres. in, For opacity, It can be defined as the volumetric scale of covariance. , 、 These are the weighting coefficients. The scale normalization constant is This is the scoring threshold; when If a value is deemed to have low contribution, it is removed; a simplified 3D Gaussian map is output.

9. The method for constructing a dynamic environment 3D Gaussian map based on semantic enhancement according to claim 8, characterized in that: In step S8, the camera pose and 3D Gaussian sphere parameters are jointly optimized nonlinearly, and the objective function is: in, and In camera pose With 3D Gaussian sphere parameters Below, the rendering intensity and rendering depth are obtained through differentiable rendering. and These are the measured strength and the measured depth, respectively. For robust loss function, As a weighting factor, For the set of parameters of a 3D Gaussian sphere, The Lie algebra representation of the camera pose; Multi-scale bundle adjustment solution increment using Riemannian manifold backtracking solver and The Riemannian manifold backtracking solver iteratively optimizes across multiple resolution pyramid levels, establishing normal equations at each level: in To unite Jacobi, For weighted matrices, For the residual vector, For LM damping factor; After obtaining the increment, the step size is determined using Armijo backtracking search. Each iteration verifies whether the Armijo condition is satisfied. If the condition is not met, then contract. Until the Armijo condition is met; after determining the step size, update the camera pose using Lie algebras: in, for The hat operator, for parameters In 、 、 、 Synchronized Updates: ; For the mask Morphological operations include: morphological dilation and erosion of the semantic mask, and setting structuring elements. For a cross-shaped core, expansion and corrosion are defined as follows: in , These represent expansion and corrosion operations, respectively.

10. A system for performing the semantically enhanced dynamic environment 3D Gaussian map construction method according to any one of claims 1-9, characterized in that: The system includes an image acquisition module, a front-end perception module, a mask propagation module, a 3D Gaussian mapping module, a map management module, and a pose estimation module. The image acquisition module initializes the camera intrinsic parameters and acquires image sequences in real time. The keyframe scheduler filters keyframes based on displacement thresholds and the number of co-viewpoints to obtain a set of keyframes and a set of ordinary frames, and performs initial pose estimation. The front-end perception module uses the YOLOv11 instance segmentation and detection network to perform semantic detection on keyframes and generate an initial semantic mask. Furthermore, a geometric depth residual verification mechanism was established, based on the depth reprojection residual to verify the initial semantic mask. Make corrections to the correction mask. Divide the region into dynamic and static areas; The mask propagation module uses a camera pose-based reprojection method to correct the keyframe mask. Propagate to adjacent normal frames and perform temporal majority voting fusion within a sliding window to obtain a temporally consistent mask sequence; The 3D Gaussian mapping module initializes or updates a set of three-dimensional Gaussian spheres only in static regions. Each Gaussian sphere contains parameters such as position, covariance, opacity, and spherical harmonic function coefficients to form a three-dimensional Gaussian map. In dynamic regions, semantically guided adaptive density control is performed on the three-dimensional Gaussian spheres, including prohibiting splitting and cloning, and adjusting the opacity trimming threshold. The map management module calculates the information entropy score for all three-dimensional Gaussian spheres and performs lightweight map management based on the information entropy score. The pose estimation module performs joint nonlinear optimization of camera pose and 3D Gaussian sphere parameters based on photometric error and geometric constraints, and feeds the optimization results back to the mask propagation and mapping module to achieve closed loop.

Images