An anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance
By constructing an independent feature Gaussian field and generating a dynamic region mask, the problem of inaccurate dynamic region localization in existing technologies is solved, and higher quality static 3D reconstruction and new perspective synthesis are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2026-05-26
- Publication Date
- 2026-07-31
AI Technical Summary
Existing anti-interference static scene reconstruction methods are highly dependent on the geometric and appearance fitting quality of the RGB Gaussian field. When the RGB field is overfitted or underfitted, dynamic regions are easily missed. Furthermore, dynamic regions can disturb the surrounding static regions during the optimization process, resulting in incomplete mask boundaries or excessively large mask ranges.
An independent feature Gaussian field is constructed. High-dimensional semantic features of multi-view images are extracted by a pre-trained visual model and quantized and compressed into low-dimensional features. A dynamic region mask is generated and used to iteratively optimize the independent feature Gaussian field. Gradient backpropagation of the dynamic region is blocked during the optimization process of the static RGB 3D Gaussian field.
It effectively reduces false detections, missed detections, and background contamination caused by geometric errors, appearance disturbances, and changes in perspective, and significantly improves the quality of static 3D reconstruction and new perspective synthesis in dynamic scenes.
Smart Images
Figure CN122492936A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and 3D reconstruction technology, specifically to an anti-interference static 3D Gaussian reconstruction method, apparatus, electronic device, storage medium, and system based on independent feature Gaussian field guidance. Background Technology
[0002] 3D Gaussian Splatting (3DGS) has become an important technique for static scene reconstruction and new novel synthesis due to its explicit Gaussian primitive representation, high optimization efficiency, and fast rendering speed. However, existing 3DGS methods typically assume that the input image satisfies multi-view static consistency. But in real-world data acquisition, scenes often contain dynamic interference components such as pedestrians, vehicles, moving objects, and partial occlusions. These transient targets can disrupt cross-view consistency, causing their corresponding Gaussian primitives to be incorrectly absorbed into the static scene representation, leading to floating artifacts, blurred boundaries, background contamination, and new novel rendering distortion.
[0003] Existing anti-interference static scene reconstruction methods mostly locate dynamic regions within the RGB reconstruction space, for example, by generating dynamic masks based on the photometric residuals, feature differences, uncertainties, or heuristic priors between the rendered and observed images. The shortcomings of this approach are: firstly, the quality of dynamic localization is highly dependent on the geometric and appearance fitting quality of the RGB Gaussian field; when the RGB field is overfitted or underfitted, dynamic regions are easily missed; secondly, dynamic regions can disturb surrounding static regions during optimization, causing static backgrounds to be misjudged as dynamic, resulting in incomplete mask boundaries or excessively large mask ranges.
[0004] Therefore, there is an urgent need for a new scheme that can break free from the constraints of RGB reconstruction quality and directly perform dynamic target localization based on cross-view stable semantic information, so as to provide more reliable supervision for static 3DGS reconstruction under dynamic object interference. Summary of the Invention
[0005] This application discloses an anti-interference static 3D Gaussian reconstruction method based on independent feature Gaussian field guidance. It aims to solve the technical problems of existing anti-interference static scene reconstruction methods that highly depend on the geometric and appearance fitting quality of RGB Gaussian field. When the RGB field is overfitted or underfitted, dynamic regions are easily missed. Furthermore, the dynamic regions will disturb the surrounding static regions during the optimization process, resulting in incomplete mask boundaries or excessively large mask ranges.
[0006] To achieve the above objectives, the first aspect of this application provides an anti-interference static 3D Gaussian reconstruction method guided by an independent feature Gaussian field, comprising the following steps: acquiring multi-view images, corresponding point clouds, and camera parameters; extracting high-dimensional semantic features from each of the multi-view images using a pre-trained visual model, and quantizing and compressing the high-dimensional semantic features to obtain low-dimensional semantic features; constructing an independent feature Gaussian field based on the point clouds, the camera parameters, and the low-dimensional semantic features, and rendering the independent feature Gaussian field using a differentiable rendering engine to obtain a rendered low-dimensional feature map; and decoding and upscaling the rendered low-dimensional feature map to the original dimension to obtain a decoded high-dimensional feature map. The process involves calculating the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and generating a dynamic region mask based on the cosine similarity. The dynamic region mask is then used to iteratively optimize the independent feature Gaussian field, with gradient backpropagation of the dynamic region masked in each iteration to update the dynamic region mask, resulting in a refined dynamic region mask. A static RGB three-dimensional Gaussian field is constructed based on the multi-view images, the point cloud, and the camera parameters. The refined dynamic region mask is used to constrain the optimization process of the static RGB three-dimensional Gaussian field, blocking gradient backpropagation of the dynamic region and completing the anti-interference static three-dimensional scene reconstruction.
[0007] Optionally, the step of extracting high-dimensional semantic features from each of the multi-view images using a pre-trained visual model, and quantizing and compressing the high-dimensional semantic features to obtain low-dimensional semantic features, includes: inputting each of the multi-view images into a pre-trained two-dimensional basic visual encoder, and obtaining a high-dimensional semantic feature map of the corresponding view through an upsampling algorithm; initializing an adaptive feature codebook based on the high-dimensional semantic feature map, calculating the similarity between each spatial location feature vector in the high-dimensional semantic feature map and each codeword, assigning it to the codeword with the highest similarity, replacing the original high-dimensional semantic feature vector with the corresponding codeword, and obtaining a quantized feature representation; training a lightweight encoder and decoder combination using the quantized feature representation, and compressing the quantized feature representation using the encoder to obtain low-dimensional semantic features for each view.
[0008] Optionally, the step of constructing an independent feature Gaussian field based on the point cloud, the camera parameters, and the low-dimensional semantic features includes: initializing the spatial location, covariance, and opacity attributes of the independent feature Gaussian field using the point cloud of the multi-view input image and the camera parameters; removing the RGB color attributes from the original Gaussian primitives and adding the low-dimensional semantic features as feature attributes to obtain the initialized independent feature Gaussian field.
[0009] Optionally, calculating the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and generating a dynamic region mask based on the cosine similarity, includes: For the t-th viewpoint, obtain its high-dimensional semantic feature map. , and the decoded high-dimensional feature map obtained through differentiable rendering and decoding. ; calculate and Cosine similarity between And generate the dynamic region mask for the t-th view according to the following formula. :
[0010] in, This indicates an indicator function, which takes a value of 1 when the condition within the parentheses is true, and a value of 0 otherwise; τ is a preset cosine similarity threshold. .
[0011] Optionally, the step of iteratively optimizing the independent feature Gaussian field using the dynamic region mask, and masking the gradient backpropagation of the dynamic region in each iteration to update the dynamic region mask, includes: during the training process of the independent feature Gaussian field, every preset number of training iterations, regenerating an updated dynamic region mask based on the rendering result of the current iteration; introducing the updated dynamic region mask into the subsequent training loss calculation, applying masking weights to the pixels corresponding to the dynamic region, so that the independent feature Gaussian field mainly updates its parameters based on the observed features of the static region.
[0012] Optionally, the step of constraining the optimization process of the static RGB three-dimensional Gaussian field using the refined dynamic region mask to block gradient backpropagation of the dynamic region includes: inputting the multi-view image as a supervision signal into the static RGB three-dimensional Gaussian field; when calculating the rendering loss of the static RGB three-dimensional Gaussian field, performing a dot product operation between the refined dynamic region mask and the rendering error, so that the pixels corresponding to the dynamic region do not participate in gradient backpropagation, thereby suppressing the pollution of the static scene representation by dynamic interference.
[0013] To achieve the above objectives, a second aspect of this application also provides an anti-interference static 3D reconstruction device guided by an independent feature Gaussian field, comprising: a feature extraction and compression module for acquiring multi-view images, corresponding point clouds, and camera parameters; extracting high-dimensional semantic features of each of the multi-view images using a pre-trained visual model, and quantizing and compressing the high-dimensional semantic features to obtain low-dimensional semantic features; an independent feature field construction and rendering module for constructing an independent feature Gaussian field based on the point clouds, the camera parameters, and the low-dimensional semantic features, and rendering the independent feature Gaussian field using a differentiable rendering engine to obtain a rendered low-dimensional feature map; and a dynamic mask generation module for decoding and upscaling the rendered low-dimensional feature map to the original dimension. A high-dimensional feature map is decoded, and the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint is calculated. A dynamic region mask is generated based on the cosine similarity. An iterative optimization module is used to iteratively optimize the independent feature Gaussian field using the dynamic region mask. In each iteration, the gradient backpropagation of the dynamic region is blocked to update the dynamic region mask, resulting in a refined dynamic region mask. A static reconstruction guidance module is used to construct a static RGB three-dimensional Gaussian field based on the multi-view images, the point cloud, and the camera parameters. The optimization process of the static RGB three-dimensional Gaussian field is constrained using the refined dynamic region mask to block the gradient backpropagation of the dynamic region, thereby completing the anti-interference static three-dimensional scene reconstruction.
[0014] To achieve the above objectives, a third aspect of this application also provides an electronic device, comprising: a processor; and a memory for storing a computer program; wherein the processor is configured to perform the steps of the interference-resistant static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance as described above.
[0015] To achieve the above objectives, a fourth aspect of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance provided above.
[0016] To achieve the above objectives, the fifth aspect of this application also provides an anti-interference static 3D reconstruction system guided by an independent feature Gaussian field, comprising: a data acquisition terminal for acquiring multi-view images containing dynamic interference objects and corresponding camera parameters; an edge computing node or cloud server, internally integrating the aforementioned device for receiving the multi-view images and camera parameters, performing feature extraction, independent feature Gaussian field construction, dynamic mask generation and iterative optimization, and outputting a refined dynamic region mask; and a 3D reconstruction terminal for receiving the refined dynamic region mask and the original multi-view images and point cloud data, using the dynamic region mask to guide gradient blocking optimization of the static RGB 3D Gaussian field, and finally rendering and outputting a static 3D scene representation without dynamic interference.
[0017] This application discloses an anti-interference static 3D Gaussian reconstruction method, apparatus, electronic device, storage medium, and system based on independent feature Gaussian fields. The method includes acquiring multi-view images, corresponding point clouds, and camera parameters; extracting high-dimensional semantic features from each of the multi-view images using a pre-trained visual model, and quantizing and compressing the high-dimensional semantic features to obtain low-dimensional semantic features; constructing an independent feature Gaussian field based on the point clouds, camera parameters, and low-dimensional semantic features, and rendering the independent feature Gaussian field using a differentiable rendering engine to obtain a rendered low-dimensional feature map; and decoding and upscaling the rendered low-dimensional feature map to the original dimension to obtain a decoded high-dimensional feature map. The process involves calculating the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and generating a dynamic region mask based on the cosine similarity. The dynamic region mask is then used to iteratively optimize the independent feature Gaussian field, with gradient backpropagation of the dynamic region masked in each iteration to update the dynamic region mask, resulting in a refined dynamic region mask. A static RGB three-dimensional Gaussian field is constructed based on the multi-view images, the point cloud, and the camera parameters. The refined dynamic region mask is used to constrain the optimization process of the static RGB three-dimensional Gaussian field, blocking gradient backpropagation of the dynamic region and completing the anti-interference static three-dimensional scene reconstruction. This application constructs a feature Gaussian field independent of the RGB Gaussian field, carrying more stable high-level semantic features across viewpoints, effectively reducing false detections, missed detections, and background contamination caused by geometric errors, appearance perturbations, and viewpoint changes. Iterative optimization of the independent feature Gaussian field using the dynamic region mask and shielding gradient backpropagation in the dynamic region yields a higher-quality dynamic target mask. The refined dynamic region mask is then used in the optimization process of the static RGB 3D Gaussian field, applying supervision only to the static region, significantly improving the quality of static 3D reconstruction and new viewpoint synthesis in dynamic scenes. Attached Figure Description
[0018] Figure 1This is a flowchart illustrating an embodiment of the anti-interference static 3D Gaussian reconstruction method based on independent characteristic Gaussian fields provided in this application. Figure 2 This is a schematic diagram of an embodiment of the anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guided by this application; Figure 3 This image shows a comparison of the dynamic object localization results provided in an embodiment of the anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guided by this application.
[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0020] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0021] Explanation of key terms Target data: refers to multi-view images containing dynamic disturbances, corresponding point clouds, and camera parameters, which are used as inputs to pre-trained visual models and independent feature Gaussian fields for subsequent processing.
[0022] Training samples: refer to the low-dimensional semantic features and their corresponding spatial location information obtained after quantization and compression, which are used to construct and train independent feature Gaussian fields.
[0023] Feature vector: refers to the feature representation of each spatial location in the high-dimensional semantic feature map extracted by the pre-trained two-dimensional basic visual encoder, as well as the feature representation after adaptive codebook quantization.
[0024] Model inputs include low-dimensional semantic features, point clouds, and camera parameters from each viewpoint, which serve as the initialization and training inputs for the independent feature Gaussian field; and multi-view images and refined dynamic region masks, which serve as the supervision signal inputs for the static RGB 3D Gaussian field.
[0025] Inference results: refer to the low-dimensional feature map obtained by rendering the independent feature Gaussian field through a differentiable rendering engine, and the high-dimensional feature map obtained after decoding and dimensional upscaling.
[0026] Note: This application does not involve the construction of cue words for traditional large models. Here, it refers to constraint information used to guide model training and optimization, such as dynamic region masks and preset cosine similarity thresholds.
[0027] Constraint information: refers to the refined dynamic region mask, used to block gradient backpropagation in the dynamic region during the optimization process of independent feature Gaussian fields and static RGB 3D Gaussian fields.
[0028] Multimodal data: This application mainly involves visual image modalities (multi-view images) and geometric space modalities (point clouds, camera parameters). Semantic features are extracted through pre-trained models and combined with three-dimensional spatial attributes for modeling.
[0029] Edge devices / servers / terminals refer to the hardware carriers that perform data acquisition, feature extraction, independent feature Gaussian field construction, dynamic mask generation and iterative optimization, and finally static RGB three-dimensional Gaussian field reconstruction.
[0030] Business object / control object: The business object in this application is a multi-view image scene containing dynamic interference; the control object is the parameter update process of an independent feature Gaussian field and a static RGB three-dimensional Gaussian field, which suppresses dynamic regions and guides the reconstruction of static regions through gradient blocking.
[0031] Existing methods for reconstructing static scenes with robustness often locate dynamic regions within the RGB reconstruction space. For example, they generate dynamic masks based on the photometric residuals, feature differences, uncertainties, or heuristic priors between the rendered and observed images. However, these methods have several drawbacks: first, the quality of dynamic localization is highly dependent on the geometric and appearance fitting quality of the RGB Gaussian field. Overfitting or underfitting of the RGB field can easily lead to missed detections of dynamic regions; second, dynamic regions can disturb surrounding static regions during optimization, causing static backgrounds to be misjudged as dynamic, resulting in incomplete mask boundaries or excessively large mask areas. Therefore, a new approach is urgently needed that can break free from the constraints of RGB reconstruction quality and directly perform dynamic target localization based on stable cross-viewpoint semantic information.
[0032] This invention aims to provide a dynamic target localization method based on an independent feature Gaussian field and apply this method to interference-resistant static 3D Gaussian reconstruction. Unlike traditional methods that directly identify dynamic regions within the RGB space, this invention constructs a separate feature Gaussian field independent of the RGB Gaussian field to carry more stable high-level semantic features across different viewpoints. By utilizing the consistency differences between this feature field and the observation features from various viewpoints, dynamic targets and static backgrounds can be distinguished more accurately.
[0033] To achieve the above objectives, the present invention adopts the following technical solution: First, high-dimensional semantic features are extracted from the input multi-view images, and compressed into a low-dimensional latent space using scene-adaptive codebook quantization and a lightweight autoencoder; second, independent feature Gaussian fields are constructed using the compressed low-dimensional features, and low-dimensional feature maps corresponding to each viewpoint are generated through differentiable rendering; third, the low-dimensional feature maps are decoded back into the high-dimensional semantic space, compared pixel-by-pixel with the original observed features, and a dynamic mask is generated based on cosine inconsistency; then, the feature Gaussian field is further optimized using the dynamic mask with mask guidance to obtain a purer static feature field; finally, the refined dynamic mask is used in the optimization process of the static RGB three-dimensional Gaussian field, applying supervision only to static regions to obtain a static three-dimensional scene representation without dynamic interference.
[0034] The overall process of this invention is as described in the appendix to the specification. Figure 2 As shown, the goal of interference-resistant static 3D scene reconstruction is to eliminate the influence of dynamic interference on scene reconstruction and obtain a more complete, accurate, and consistent static scene representation. Given multi-view image input... T represents the number of viewpoints. The image represents a single viewpoint, with H and W representing the image's height and width, respectively. The goal of this invention is to achieve static scene reconstruction from multi-view images containing dynamic interference, thereby obtaining a more accurate, complete, and viewpoint-consistent static 3D scene representation to improve the accuracy of dynamic target localization and the quality of static scene reconstruction. To this end, an independent feature Gaussian field is first constructed for the input multi-view images. A pre-trained 2D basic vision model is introduced to extract high-dimensional features from each viewpoint image, and low-dimensional features are obtained through quantization and compression. These low-dimensional features are then used as supervision signals to train the feature Gaussian field. Compared to methods based on RGB reconstruction errors for dynamic localization, the feature Gaussian field can represent more stable and cross-viewpoint consistent semantic information, thereby reducing false detections, missed detections, and background contamination caused by geometric errors, appearance perturbations, and viewpoint changes. Furthermore, by comparing the consistency between the features rendered by the feature Gaussian field at the current viewpoint and the corresponding observed features of the image at that viewpoint, a dynamic region mask is generated. This dynamic region mask is then used to further optimize the feature Gaussian field to obtain a higher-quality dynamic target mask. Subsequently, an RGB Gaussian field is constructed based on the input multi-view images, and a dynamic region mask is used to suppress the dynamic interference regions under each viewpoint, so that the RGB Gaussian field is mainly optimized for static regions, thereby obtaining the initial static three-dimensional Gaussian representation of the scene.
[0035] This invention can be applied to multi-view static scene reconstruction tasks containing dynamic interference, and is applicable to fields such as intelligent driving, robot perception, 3D modeling, and virtual reality. This application introduces an independent characteristic Gaussian field to effectively identify dynamic targets, significantly improving the accuracy, stability, and generalization ability of static 3D reconstruction in complex scenes.
[0036] refer to Figure 1 and Figure 2 The first embodiment of this application provides an anti-interference static 3D Gaussian reconstruction method guided by independent feature Gaussian fields. This addresses the technical problems of existing anti-interference static scene reconstruction methods in the background art, which highly rely on the geometric and appearance fitting quality of the RGB Gaussian field. When the RGB field is overfitted or underfitted, dynamic regions are easily missed. Furthermore, during the optimization process, dynamic regions disturb the surrounding static regions, resulting in incomplete mask boundaries or excessively large mask ranges. This method can be executed by a processor, which can be located on a terminal or server. The execution process of the anti-interference static 3D Gaussian reconstruction method guided by independent feature Gaussian fields can be as follows: Step S101: Obtain multi-view images, corresponding point clouds, and camera parameters; use a pre-trained visual model to extract high-dimensional semantic features from each multi-view image, and quantize and compress the high-dimensional semantic features to obtain low-dimensional semantic features.
[0037] In one embodiment of this application, each of the multi-view images is input into a pre-trained two-dimensional basic visual encoder, and a high-dimensional semantic feature map of the corresponding view is obtained through an upsampling algorithm. An adaptive feature codebook is initialized based on the high-dimensional semantic feature map. For each spatial location feature vector in the high-dimensional semantic feature map, its similarity with each codeword is calculated, and it is assigned to the codeword with the highest similarity. The original high-dimensional semantic feature vector is replaced with the corresponding codeword to obtain a quantized feature representation. A lightweight encoder and decoder combination is trained using the quantized feature representation, and the encoder is used to compress the quantized feature representation to obtain low-dimensional semantic features of each view.
[0038] For example, the processor can process images from various perspectives. The input is fed into a pre-trained visual encoder (DINOv3), and the high-dimensional features corresponding to the viewpoint are obtained through an upsampling algorithm (AnyUp). ,in This represents the high-dimensional feature map from the t-th viewpoint. Secondly, an adaptive feature codebook is initialized based on the high-dimensional features. Where N refers to the codebook size, This refers to the k-th codeword. For each spatial location feature vector in the input high-dimensional feature map, its similarity to each codeword is calculated, and it is assigned to the codeword with the highest similarity. This replaces the original high-dimensional feature vector with the corresponding codeword, resulting in a quantized feature representation. Subsequently, a lightweight encoder-decoder pair is trained using the quantized features, and the encoder is used to compress the quantized features to obtain low-dimensional features from various perspectives. .
[0039] Step S102: Construct an independent feature Gaussian field based on the point cloud, the camera parameters, and the low-dimensional semantic features, and render the independent feature Gaussian field using a differentiable rendering engine to obtain a rendered low-dimensional feature map.
[0040] Furthermore, the spatial location, covariance, and opacity attributes of the independent feature Gaussian field are initialized using the point cloud of the multi-view input image and camera parameters; the RGB color attributes are removed from the original Gaussian primitives and the low-dimensional semantic features are added as feature attributes to obtain the initialized independent feature Gaussian field. During the training process, a selected view feature field is used to obtain the rendering features through differentiable rendering. Calculate the feature field reconstruction loss :
[0041] Step S103: Decode and upscale the rendered low-dimensional feature map to the original dimension to obtain a decoded high-dimensional feature map. Calculate the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint. Generate a dynamic region mask based on the cosine similarity.
[0042] In an optional embodiment of this application, the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint is calculated, and a dynamic region mask is generated based on the cosine similarity, including: For the t From multiple perspectives, obtain the high-dimensional semantic feature map of its observation. , and the decoded high-dimensional feature map obtained through differentiable rendering and decoding. ; calculate and Cosine similarity between And generate the dynamic region mask for the t-th view according to the following formula. :
[0043] in, This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. τ The preset cosine similarity threshold, .
[0044] Step S104: Iteratively optimize the independent feature Gaussian field using the dynamic region mask. In each iteration, the gradient backpropagation of the dynamic region is masked to update the dynamic region mask, resulting in a refined dynamic region mask.
[0045] Furthermore, during the training of this independent feature Gaussian field, every preset number of training iterations, an updated dynamic region mask is regenerated based on the rendering result of the current iteration. This updated dynamic region mask is then incorporated into subsequent training loss calculations, applying masking weights to the pixels corresponding to the dynamic regions. This ensures that the independent feature Gaussian field primarily updates its parameters based on the observed features of the static regions. The corresponding feature field loss update is as follows:
[0046] in, Pointer multiplication: Every certain number of training iterations, recalculate and execute step 3 based on the feature field of the current iteration, updating the dynamic mask. .
[0047] Step S105: Construct a static RGB three-dimensional Gaussian field based on the multi-view image, the point cloud, and the camera parameters. Use the refined dynamic region mask to constrain the optimization process of the static RGB three-dimensional Gaussian field, block the gradient backpropagation of the dynamic region, and complete the anti-interference static three-dimensional scene reconstruction.
[0048] For example, and multi-view input images The static RGB 3D Gaussian field is used as a monitoring signal input. When calculating the rendering loss of this static RGB 3D Gaussian field, the refined dynamic region mask is multiplied by the rendering error, ensuring that pixels corresponding to the dynamic region do not participate in gradient backpropagation, thereby suppressing the contamination of the static scene representation by dynamic interference. (Loss during reconstruction) The definition is as follows:
[0049] in, Point multiplication; To balance the weights, set them to 0.2; ; An image rendered using a static Gaussian field.
[0050] As per the instruction manual Figure 3As shown, the independent feature Gaussian field introduced in this invention successfully locates dynamic interference objects (balloons and crab dolls) in indoor scenes compared to the RGB Gaussian field. However, in outdoor scenes, the color field misidentifies static backgrounds as dynamic objects (sky and ridges), but the feature field proposed in this invention can still accurately locate dynamic objects.
[0051] To address the aforementioned technical problems, the second embodiment of this application provides an anti-interference static 3D reconstruction device guided by an independent feature Gaussian field, which solves the same technical problems as the method embodiment. The device 1000 may include the following modules: a feature extraction and compression module 1001, an independent feature field construction and rendering module 1002, a dynamic mask generation module 1003, an iterative optimization module 1004, and a static reconstruction guidance module 1005.
[0052] The feature extraction and compression module 1001 is used to acquire multi-view images, corresponding point clouds and camera parameters; extract high-dimensional semantic features of each multi-view image using a pre-trained visual model; and quantize and compress the high-dimensional semantic features to obtain low-dimensional semantic features.
[0053] The independent feature field construction and rendering module 1002 is used to construct an independent feature Gaussian field based on the point cloud, the camera parameters and the low-dimensional semantic features, and to render the independent feature Gaussian field through a differentiable rendering engine to obtain a rendered low-dimensional feature map.
[0054] The dynamic mask generation module 1003 is used to decode and upscale the rendered low-dimensional feature map to the original dimension to obtain a decoded high-dimensional feature map, calculate the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and generate a dynamic region mask based on the cosine similarity.
[0055] The iterative optimization module 1004 is used to iteratively optimize the independent feature Gaussian field using the dynamic region mask. In each iteration, the gradient backpropagation of the dynamic region is masked to update the dynamic region mask, resulting in a refined dynamic region mask.
[0056] The static reconstruction guidance module 1005 is used to construct a static RGB three-dimensional Gaussian field based on the multi-view image, the point cloud and the camera parameters. The refined dynamic region mask is used to constrain the optimization process of the static RGB three-dimensional Gaussian field, block the gradient backpropagation of the dynamic region, and complete the anti-interference static three-dimensional scene reconstruction.
[0057] It is understood that the above-described device embodiments correspond to the method embodiments. Therefore, the device embodiments have all the beneficial effects of the method embodiments, and will not be repeated here.
[0058] refer to Figure 1 and Figure 2The third embodiment of this application provides an anti-interference static three-dimensional reconstruction system based on independent feature Gaussian field guidance to solve the same technical problem as the method embodiment. The system may include the following parts: a data acquisition terminal, an edge computing node or cloud server, and a three-dimensional reconstruction terminal.
[0059] The data acquisition terminal is used to acquire multi-view images containing dynamic interference and the corresponding camera parameters. The edge computing node or cloud server integrates the aforementioned device to receive the multi-view image and camera parameters, perform feature extraction, independent feature Gaussian field construction, dynamic mask generation and iterative optimization, and output the refined dynamic region mask. The 3D reconstruction terminal receives the refined dynamic region mask, as well as the original multi-view images and point cloud data. It uses the dynamic region mask to guide the gradient blocking optimization of the static RGB 3D Gaussian field, and finally renders and outputs a static 3D scene representation without dynamic interference.
[0060] The fourth embodiment of this application provides an electronic device, including: a processor; and a memory for storing a computer program; wherein the processor is configured to perform the steps of the interference-resistant static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance as provided above.
[0061] In addition, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance provided above.
[0062] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance, characterized in that, Includes the following steps: Acquire multi-view images, corresponding point clouds, and camera parameters; extract high-dimensional semantic features of each multi-view image using a pre-trained visual model, and quantize and compress the high-dimensional semantic features to obtain low-dimensional semantic features. Based on the point cloud, the camera parameters, and the low-dimensional semantic features, an independent feature Gaussian field is constructed, and the independent feature Gaussian field is rendered using a differentiable rendering engine to obtain a rendered low-dimensional feature map. The rendered low-dimensional feature map is decoded and upscaled to the original dimension to obtain a decoded high-dimensional feature map. The cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint is calculated, and a dynamic region mask is generated based on the cosine similarity. The independent feature Gaussian field is iteratively optimized using the dynamic region mask. In each iteration, the gradient backpropagation of the dynamic region is masked to update the dynamic region mask, resulting in a refined dynamic region mask. A static RGB three-dimensional Gaussian field is constructed based on the multi-view images, the point cloud, and the camera parameters. The optimization process of the static RGB three-dimensional Gaussian field is constrained by the refined dynamic region mask to block gradient backpropagation in the dynamic region, thereby completing the anti-interference static three-dimensional scene reconstruction.
2. The anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guidance as described in claim 1, characterized in that, The step of extracting high-dimensional semantic features from each of the multi-view images using a pre-trained visual model, and then quantizing and compressing the high-dimensional semantic features to obtain low-dimensional semantic features, includes: Each of the multi-view images is input into a pre-trained two-dimensional basic visual encoder, and a high-dimensional semantic feature map of the corresponding view is obtained through an upsampling algorithm. An adaptive feature codebook is initialized based on the high-dimensional semantic feature map. For each spatial location feature vector in the high-dimensional semantic feature map, its similarity with each codeword is calculated and assigned to the codeword with the highest similarity. The original high-dimensional semantic feature vector is replaced with the corresponding codeword to obtain the quantized feature representation. A lightweight encoder and decoder combination is trained using the quantized feature representation, and the encoder is used to compress the quantized feature representation to obtain low-dimensional semantic features from each perspective.
3. The anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guidance as described in claim 1, characterized in that, The construction of an independent feature Gaussian field based on the point cloud, the camera parameters, and the low-dimensional semantic features includes: The spatial location, covariance, and opacity properties of the independent feature Gaussian field are initialized using the point cloud of the multi-view input image and camera parameters. The RGB color attributes are removed from the original Gaussian primitives and the low-dimensional semantic features are added as feature attributes to obtain the initialized independent feature Gaussian field.
4. The anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guidance as described in claim 1, characterized in that, The calculation of the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and the generation of a dynamic region mask based on the cosine similarity, includes: For the t-th viewpoint, obtain its high-dimensional semantic feature map. , and the decoded high-dimensional feature map obtained through differentiable rendering and decoding. ; calculate and Cosine similarity between And generate the dynamic region mask for the t-th view according to the following formula. : in, This indicates an indicator function, which takes a value of 1 when the condition within the parentheses is true, and a value of 0 otherwise; τ is a preset cosine similarity threshold. .
5. The anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guidance as described in claim 1, characterized in that, The iterative optimization of the independent feature Gaussian field using the dynamic region mask, wherein gradient backpropagation of the dynamic region is masked in each iteration to update the dynamic region mask, includes: During the training process of the independent feature Gaussian field, every preset number of training iterations, an updated dynamic region mask is regenerated based on the rendering result of the current iteration. The updated dynamic region mask is introduced into the subsequent training loss calculation, and a masking weight is applied to the pixels corresponding to the dynamic region, so that the independent feature Gaussian field is mainly updated based on the observed features of the static region.
6. The anti-interference static three-dimensional Gaussian reconstruction method based on independent characteristic Gaussian field guidance as described in claim 1, characterized in that, The process of optimizing the static RGB three-dimensional Gaussian field using the refined dynamic region mask to constrain gradient backpropagation in the dynamic region includes: The multi-view image is used as a supervision signal and input into the static RGB three-dimensional Gaussian field; When calculating the rendering loss of the static RGB 3D Gaussian field, the refined dynamic region mask is multiplied by the rendering error, so that the pixels corresponding to the dynamic region do not participate in gradient backpropagation, thereby suppressing the pollution of the static scene representation by dynamic interference.
7. An anti-interference static three-dimensional reconstruction device based on an independent characteristic Gaussian field, characterized in that, include: The feature extraction and compression module is used to acquire multi-view images, corresponding point clouds and camera parameters; extract high-dimensional semantic features of each multi-view image using a pre-trained visual model, and quantize and compress the high-dimensional semantic features to obtain low-dimensional semantic features. The independent feature field construction and rendering module is used to construct an independent feature Gaussian field based on the point cloud, the camera parameters and the low-dimensional semantic features, and to render the independent feature Gaussian field through a differentiable rendering engine to obtain a rendered low-dimensional feature map. The dynamic mask generation module is used to decode and upscale the rendered low-dimensional feature map to the original dimension to obtain a decoded high-dimensional feature map, calculate the cosine similarity between the decoded high-dimensional feature map and the observed high-dimensional semantic features corresponding to each viewpoint, and generate a dynamic region mask based on the cosine similarity. The iterative optimization module is used to iteratively optimize the independent feature Gaussian field using the dynamic region mask. In each iteration, the gradient backpropagation of the dynamic region is masked to update the dynamic region mask, so as to obtain a refined dynamic region mask. The static reconstruction guidance module is used to construct a static RGB three-dimensional Gaussian field based on the multi-view images, the point cloud, and the camera parameters. The refined dynamic region mask is used to constrain the optimization process of the static RGB three-dimensional Gaussian field, block the gradient backpropagation of the dynamic region, and complete the anti-interference static three-dimensional scene reconstruction.
8. An electronic device, characterized in that, include: processor; as well as Memory, used to store computer programs; The processor is configured to execute the anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the anti-interference static three-dimensional Gaussian reconstruction method based on independent feature Gaussian field guidance as described in any one of claims 1 to 6.
10. An anti-interference static three-dimensional reconstruction system guided by an independent characteristic Gaussian field, characterized in that, include: A data acquisition terminal is used to acquire multi-view images containing dynamic interference and corresponding camera parameters. An edge computing node or cloud server integrates the device described in claim 7, which is used to receive the multi-view images and camera parameters, perform feature extraction, independent feature Gaussian field construction, dynamic mask generation and iterative optimization, and output a refined dynamic region mask. The 3D reconstruction terminal is used to receive the refined dynamic region mask as well as the original multi-view images and point cloud data, and uses the dynamic region mask to guide the gradient blocking optimization of the static RGB 3D Gaussian field, and finally renders and outputs a static 3D scene representation without dynamic interference.