An online semantic three-dimensional Gaussian field delta reconstruction method

Abstract

This invention discloses an online semantic 3D Gaussian field incremental reconstruction method, characterized by a geometry-semantic decoupling dual-branch structure to achieve online scene understanding and reconstruction for long-sequence RGB image streams. Specifically, it includes steps such as geometric streaming modeling based on causal Transformer, geometric prior distillation constraints, construction of a geometry-semantic decoupling semantic branch and identity stabilization mechanism, semantic enhancement, and Gaussian-based rendering. Compared with existing technologies, this invention can continuously update the 3D representation in a scalable manner under strict causal conditions and stably output instance-level semantic understanding results, achieving scalable online scene understanding and reconstruction. It significantly reduces computation and memory usage growth under long-sequence input, possesses stable online reconstruction and instance-level semantic consistency, and effectively solves the problems of rapid growth in computation and memory usage and unstable instance identities under long-sequence input. It is suitable for scenarios such as robotics, AR / VR, and digital twins, and has good application prospects.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction and scene understanding technology, and in particular to an online semantic 3D Gaussian field incremental reconstruction method for streaming 3D reconstruction and instance-level semantic understanding of long-sequence RGB video streams under strict causal and non-repeating historical frame constraints. Background Technology

[0002] With the development of applications such as robot perception, augmented reality / virtual reality, and digital twins, systems need to continuously receive image streams from cameras in real-world environments and output the 3D structure and semantic understanding results of the environment in real time under online conditions, such as new perspective rendering, semantic segmentation, instance segmentation, and panoramic segmentation. Unlike traditional offline scene modeling, online systems have typical characteristics: inputs arrive continuously in chronological order, processing needs to meet low latency, and at any given moment can only rely on the current observation and historical accumulated states; they are also usually subject to strict limitations in terms of video memory and computing power. Therefore, how to achieve unified modeling of high-quality 3D reconstruction and semantic understanding in long-running image streams has always been an important research direction in the field of computer vision.

[0003] To achieve high-quality novel perspective synthesis and 3D representation, scene representation methods such as NeRF and 3D Gaussian scattering have received widespread attention in recent years. These methods can represent scene geometry and appearance well and achieve perspective synthesis through differentiable rendering. However, traditional schemes are mostly based on scene-by-scene optimization, which usually requires a long optimization time and is difficult to meet the needs of online real-time scenes. To improve efficiency, a large number of feedforward reconstruction methods have emerged, which use neural networks to encode the input image sequence and directly regress the 3D representation, significantly reducing the reconstruction time. Furthermore, some studies have begun to embed semantic features into the 3D representation or introduce query mechanisms to achieve joint modeling of reconstruction and understanding, enabling the system to output semantic segmentation or instance-level understanding results while reconstructing.

[0004] To address the scalability issues of long sequence processing, another type of research has begun exploring streaming 3D reconstruction. Its core idea is to process only the current frame at each time step and compress historical information into a reusable persistent state, achieving long sequence reasoning through incremental updates. Related research has demonstrated better latency and memory growth characteristics in geometric reconstruction or new perspective synthesis. However, existing streaming schemes mostly focus on geometric and appearance modeling, often lacking semantic representations that can be directly used for scene understanding, especially lacking instance-level semantic consistency mechanisms. In online scenarios, semantic observation naturally exhibits viewpoint dependence and occlusion variations. The visible area, appearance texture, and boundary shape of the same object may change significantly in different frames. Without effective cross-frame identity alignment and stabilization mechanisms, the system is prone to semantic jitter and frequent instance identity switching, leading to unstable output and affecting downstream applications such as robot interaction, navigation, and task planning. Furthermore, online systems often have strict causal constraints: at time t, the system can only access the current frame and historical states, and cannot use future frames for global review or correction; simultaneously, to meet real-time requirements, the system typically cannot perform repeated forward computation on historical frames at every time step. This strict causal constraint further exacerbates two core challenges: first, how to maintain the long-term stability of geometric reconstruction and avoid structural drift caused by error accumulation in the absence of future perspective correction; second, how to maintain cross-frame consistency of instance-level semantic identity and avoid identity drift and association errors when there is only semantic observation of the current frame and the observation changes with the perspective.

[0005] Existing technologies for 3D structure and semantic understanding have the following shortcomings and problems:

[0006] 1) The offline global paradigm requires repeated processing of historical frames, which leads to a rapid increase in computation and memory consumption under long sequences, making it difficult to meet the requirements of long-term online operation;

[0007] 2) Streaming reconstruction methods often lack instance-level semantic modeling capabilities, making it difficult to output stable semantic, instance, and panoramic understanding results;

[0008] 3) Under strict causal constraints, geometric errors and semantic identity drift tend to accumulate with sequence length, leading to a decrease in output quality;

[0009] 4) There is a lack of a unified framework that can simultaneously achieve incremental 3D reconstruction and instance-level semantic consistency maintenance without reprocessing historical frames.

[0010] In summary, existing technologies still fall short in their ability to balance 3D reconstruction quality and instance-level semantic consistency while ensuring strict causality and non-repeating historical frame processing. A novel online incremental modeling method is needed to continuously update the 3D representation in a scalable manner under strict causality and stably output instance-level semantic understanding results, thereby meeting the application requirements of long-sequence online scene understanding and reconstruction. Summary of the Invention

[0011] The purpose of this invention is to provide an online semantic 3D Gaussian field incremental reconstruction method to address the shortcomings of existing technologies. It adopts a geometry-semantic decoupled dual-branch structure to achieve online scene understanding and reconstruction for long-sequence RGB image streams. Under strict causal constraints without utilizing future frames or reprocessing historical frames, this method incrementally maintains a 3D Gaussian semantic field containing geometry, appearance, and instance-level semantics. The geometric branch performs temporal causal aggregation on the current frame based on a causal Transformer, predicting depth, camera parameters, and pixel-level Gaussian attributes, generating and incrementally updating a global 3D Gaussian field. The semantic branch extracts multi-scale semantic features based on a frozen 2D vision foundation model and outputs instance masks, category scores, and query embeddings through a query-driven mask classification decoder. During the training phase, query-level supervised contrastive learning is introduced to enhance cross-frame instance consistency. During the inference phase, an instance memory is introduced for lightweight online association, and instance prototypes are updated using momentum to reduce identity drift. The semantic confidence and instance information of each frame are elevated to pixel-aligned 3D Gaussian and differentiable Gaussian rendering is used to output new perspective synthesis, semantic segmentation, instance segmentation, and panoramic segmentation results. Furthermore, a query semantic projector is set up to align query embeddings to the visual-linguistic semantic space to support language-driven open-lexical segmentation. This invention effectively solves the problems of rapidly increasing computational load and GPU memory usage, as well as unstable instance identities, when dealing with long input sequences, thereby achieving scalable online scene understanding and reconstruction. It significantly reduces computation and memory usage growth under long sequence inputs, possesses stable online reconstruction and instance-level semantic consistency, and is suitable for scenarios such as robotics, AR / VR, and digital twins, showing promising application prospects.

[0012] The specific technical solution to achieve the purpose of this invention is: an online semantic three-dimensional Gaussian field incremental reconstruction method, characterized in that the method uses an uncalibrated RGB image stream. For input, where The total number of frames in the video sequence, at time [time value missing]. Utilize the current frame Incremental updates are performed on the persistent state accumulated from historical frames to achieve online scene understanding and 3D semantic Gaussian reconstruction for long sequences of RGB image streams. The specific steps include the following:

[0013] Step 1: Geometric Branch Causal Coding

[0014] Current frame Encode the visual tokens and input them into a causal Transformer encoder, using a temporal attention mask to make the time intervals... The token can only access historical prefixes Key-value information.

[0015] Step 2: Geometric Feature Prediction

[0016] Through deep prediction head Camera parameter prediction head Gaussian property prediction head It outputs the current frame depth map, camera pose parameters, and pixel-level Gaussian properties, respectively.

[0017] Step 3: Generate Gaussian by pixel back projection

[0018] in accordance with and The pixels are back-projected into three-dimensional space to obtain the pixel-aligned three-dimensional Gaussian center, and then compared with... By concatenating the sets, we obtain the Gaussian set of the current frame. .

[0019] Step 4: Incremental Global Gaussian Field Update

[0020] Will Incremental accumulation to the global three-dimensional Gaussian field In this process, the global three-dimensional Gaussian field is maintained as a persistent state.

[0021] Step 5: Semantic Branch Independent Feature Extraction

[0022] Using a frozen 2D vision base model Extract semantic features and input them into a lightweight adapter to form multi-scale features. .

[0023] Step 6: Query-driven instance segmentation and embedding generation

[0024] Multiple learnable queries Multiscale features Input the query decoder to obtain the instance mask, class score, and query embedding for each frame. , where query embedding For instance identity descriptors.

[0025] Step 7: Cross-frame identity stabilization

[0026] Cross-frame identity stabilization includes a training phase and an inference phase. The training phase aligns queries with real instances and performs query-level supervised contrastive learning, enabling the same instance to be clustered in query embeddings across different frames, thus separating different instances. The inference phase establishes an instance memory library, associates the current frame query with historical instance prototypes based on cosine similarity and binary matching, and updates the instance prototypes using momentum updates to obtain a real-time instance library.

[0027] Step 8: Semantic information is elevated to a three-dimensional Gaussian field

[0028] The instance prototype prediction obtained in step 7 is assigned to the pixel-aligned 3D Gaussian obtained in step 3, and multi-view evidence is fused in the global 3D Gaussian field to obtain the semantic information of the 3D Gaussian field.

[0029] Step 9: Differentiable Gaussian rendering output

[0030] Based on a global 3D Gaussian field, Gaussian rendering is used to output one or more of the following tasks: new perspective image, semantic segmentation, instance segmentation, and panoptic segmentation.

[0031] The causal Transformer uses a time mask matrix represented by the following formula. :

[0032] .

[0033] in, The time of the current frame It represents the time of the frame preceding the current frame.

[0034] In the reasoning stage of step 1), the key-value tensors of historical frames are cached and reused, so that when processing the current frame, there is no need to perform forward calculation on the historical frames again, thereby realizing non-reprocessing streaming reasoning.

[0035] Step 2) employs a strictly causal 3D basic model to output pseudo-depth. and pseudo camera parameters As a signal from the teacher, and Distillation constraints are applied to suppress the accumulation of errors in long sequences.

[0036] The query-level supervised contrastive learning loss in step 7) is as follows: the query embeddings for the same instance label are the positive sample set, and different instances are the negative sample set, using the temperature parameter... The objective function of comparison increases the similarity of positive samples and decreases the similarity of negative samples, thereby enhancing cross-frame instance consistency.

[0037] In step 7), the association matrix of the instance memory is calculated using cosine similarity, and the association results are obtained through binary matching.

[0038] The cosine similarity Calculated by the following formula:

[0039] .

[0040] in, For the current frame number Each query embeds a normalized vector; For the first time in history An instance prototype.

[0041] The instance memory is updated using an exponential moving average method to update the prototype embedding. :

[0042] .

[0043] in, This represents the update coefficient.

[0044] Step 8) maps the prediction mask, category score, or instance ID of each frame to Gaussian semantic attributes, and together with geometric visibility, participates in Gaussian rendering fusion, so that consistent semantic and instance results are obtained across viewpoints when rendering from any viewpoint.

[0045] Step 9) sets the query semantic projector The query embedding is mapped to the visual-language joint space to support language-driven open-lexical segmentation, and its similarity with the text embedding output by the text encoder is calculated. The mask corresponding to the query with the highest similarity is selected as the output.

[0046] The query semantic projector is trained using a prediction mask to crop or weight aggregate target regions from the input image, regional semantic distillation, and a frozen visual-language image encoder to generate teacher embeddings. Cosine regression is then performed on the projected query embeddings to achieve semantic space alignment.

[0047] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0048] 1) Strict causality and repetitive handling of historical frames: Inference only depends on the current frame and persistent state, avoiding latency and memory growth caused by repeated calculation of historical frames in the offline global paradigm;

[0049] 2) Strong scalability of long sequences: As the sequence length increases, the processing overhead per frame increases slowly, significantly reducing the risk of memory exhaustion during long sequence execution;

[0050] 3) Instance-level semantic stability: Reduce ID switching and improve cross-frame consistency through query-level comparative learning and instance memory association mechanisms;

[0051] 4) Unified multi-task output: The same 3D Gaussian semantic field can simultaneously support new perspective synthesis and multiple semantic understanding tasks;

[0052] 5) Scalable open vocabulary capability: Supports language-driven segmentation through query semantic projection and text retrieval mechanisms, making it more widely applicable. Attached Figure Description

[0053] Figure 1 This is a flowchart of Example 1. Detailed Implementation

[0054] This invention employs a geometrically and semantically decoupled dual-branch structure to achieve online scene understanding and reconstruction for long sequences of RGB image streams, specifically including the following steps:

[0055] (I) Geometric Flow Modeling Based on Causal Transformer

[0056] 1-1: Input is an uncalibrated RGB image stream The system is at all times Process only the current frame And persistent state accumulated from historical frames, without accessing future frames or reprocessing historical frames;

[0057] 1-2: Move the current frame Encode the visual tokens and input them into a causal Transformer encoder, constraining the time step by a temporal attention mask. Only historical prefixes of the token should be considered. Furthermore, the key-value tensors of historical frames are cached and reused during the inference phase, thereby avoiding repeated forward computation of historical frames.

[0058] 1-3: The depth prediction head, camera parameter prediction head, and Gaussian attribute prediction head are set up using the geometric branch, and the predicted depth maps are output respectively. Predict camera parameters And pixel-level Gaussian properties ;

[0059] 1-4: Based on and Backprojecting the pixels yields pixel-aligned 3D Gaussian centers, and then aligning these 3D centers with... The three-dimensional Gaussian set of the current frame is formed by splicing the two elements together. ;

[0060] 1-5: Will The increments are accumulated into a global 3D Gaussian field as a persistent state and used for subsequent rendering and multi-view evidence fusion, thereby achieving long-sequence streaming updates.

[0061] (ii) Geometric Prior Distillation Constraints

[0062] 2-1: In the training phase, a strictly causal 3D base model is introduced as the teacher model, and at each time step... Output pseudo-depth With pseudo camera parameters ;

[0063] 2-2: Output of the student model Apply a depth regression loss based on effective pixels to... Apply camera parameter regression loss to ensure that the online estimation results are consistent with the teacher's geometric prior;

[0064] 2-3: Improve the geometric stability and camera parameter consistency in long sequence scenarios by using the above distillation constraints.

[0065] (III) Constructing a semantic branch and identity stabilization mechanism with geometric-semantic decoupling

[0066] 3-1: The semantic branch extracts features independently of the geometric branch, using a frozen 2D visual base model for the current frame. Extract semantic features and convert them into multi-scale semantic features using a lightweight adapter. To avoid geometric update noise interfering with semantic representation;

[0067] 3-2: A fixed number of learnable queries A query-driven mask classification decoder with multi-scale semantic feature input outputs an instance mask, class score, and query embedding for each frame. The query embedding serves as the instance identity descriptor;

[0068] 3-3: During the training phase, the predicted query is matched with the real instance in each frame, and supervised contrastive learning is performed on the matched query embeddings to make the embeddings of the same instance in different frames close to each other, and the embeddings of different instances separate from each other, thereby improving cross-frame semantic consistency.

[0069] 3-4: During the inference phase, an instance memory is established to store the prototype embeddings of historical instances. The similarity between the current frame query embedding and the historical prototype embeddings is calculated, and binary matching is performed to achieve online instance association. For successfully matched instances, the prototype embedding is updated using the momentum update method to reduce instance identity drift and ID switching.

[0070] (iv) Semantic enhancement and Gaussian-based rendering

[0071] 4-1: Assign the semantic attributes such as category information, instance ID, and confidence score output from each frame's semantic branch to a pixel-aligned 3D Gaussian field, and fuse multi-view semantic evidence in the global 3D Gaussian field;

[0072] 4-2: Decode the global 3D Gaussian semantic field from any viewpoint using differentiable 3D Gaussian rendering, and output new viewpoint images as well as one or more results of semantic segmentation, instance segmentation, and panoptic segmentation;

[0073] 4-3: Further configure the query semantic projector to map the query embedding to the visual-language joint semantic space and calculate the similarity with the text embedding to achieve language-driven open vocabulary segmentation output.

[0074] The present invention will be further described in detail through the following specific implementation examples.

[0075] Example 1

[0076] See Figure 1 According to a preferred embodiment of the present invention, a strictly causal streaming three-dimensional semantic Gaussian reconstruction method specifically includes the following steps:

[0077] Step 1: Build a streaming input processing framework for unlabeled RGB image streams As input, the system initializes the global three-dimensional Gaussian field to an empty set. At any given time... The system only receives the current frame. It combines the persistent state accumulated from historical frames for processing. The processing strictly follows causal constraints, does not backtrack to modify historical frames, and does not read future frames.

[0078] Step 2: Establish a dual-stream architecture with parallel geometric and semantic branches. The geometric branch is responsible for recovering the scene's depth, camera pose, and 3D Gaussian properties from the image stream; the semantic branch is responsible for extracting semantic features from the image, generating instance masks, and maintaining the consistency of instance identities across frames.

[0079] Step 3: Perform causal encoding and prediction of the geometric branch, use the temporal masking mechanism to ensure strict causality of attention calculation, and output the geometric attributes of the current frame.

[0080] Step 4: Perform independent feature extraction and instance parsing for the semantic branch, and generate instance masks and query embeddings as identity identifiers through a query-driven approach.

[0081] Step 5: Perform cross-frame identity stabilization and association. Utilize instance memory and contrastive learning mechanisms to align the semantic prediction of the current frame with historical instances, thus resolving occlusion and viewpoint changes in the video stream.

[0082] Step 6: Fuse geometric and semantic information, generate a 3D Gaussian set with semantic attributes through back projection, and incrementally update it to the global 3D Gaussian field.

[0083] Step 7: Using differentiable Gaussian rendering technology, based on the updated global 3D Gaussian field, output new perspective images, semantic segmentation maps, instance segmentation maps, or panoramic segmentation maps in real time.

[0084] See Figure 1 The causal coding and prediction process of the geometric branch in step 3) specifically includes:

[0085] Step 3.1: Image Encoding and Tokenization

[0086] Current frame The input image encoder divides the image into multiple patches and maps them to a sequence of visual tokens.

[0087] Step 3.2: Causal Transformer Encoding

[0088] Construct a causal Transformer encoder and apply a temporal attention mask matrix defined by the following formula in the self-attention mechanism. :

[0089] .

[0090] This formula ensures that at time... When calculating attention, only the current moment and historical moments can be accessed. The key-value pair, while shielding future moments. Information.

[0091] Step 3.3: Key-Value Caching Mechanism

[0092] During the inference phase, to achieve streaming processing and avoid recomputation, the system maintains a key-value cache (KV Cache) to process the first... When a frame is being processed, only the Query, Key, and Value of the current frame are calculated, and the Key and Value are concatenated into the cache, without needing to recalculate the features of historical frames.

[0093] Step 3.4: Geometric Feature Prediction

[0094] The encoded features are input into a depth prediction head, a camera parameter prediction head, and a Gaussian attribute prediction head for geometric feature prediction. The depth prediction head outputs a pixel-level depth map of the current frame. The camera parameter prediction head regresses the camera pose parameters of the current frame relative to the global coordinate system. The Gaussian attribute prediction head predicts the Gaussian attribute corresponding to each pixel. (Including opacity, scaling factor, rotation quaternion, and spherical harmonic coefficients).

[0095] Step 3.5: Geometric Prior Distillation

[0096] To prevent long sequence drift, a pre-trained dense geometry model is introduced as the teacher model, which outputs pseudo-depth. and pseudo camera parameters The loss function is calculated during training, forcing the prediction head to output... and It approximates the teacher's signal, thereby suppressing cumulative error.

[0097] See Figure 1The instance parsing of the semantic branch in step 4) specifically includes:

[0098] Step 4.1: Independent semantic feature extraction

[0099] Using a two-dimensional visual basic model with frozen parameters as a feature extractor, the current frame... Extracting multi-scale semantic features Lightweight adapters are used to fine-tune features to adapt them to specific tasks, avoiding the gradient updates of geometric branches from interfering with the stability of semantic features.

[0100] Step 4.2: Query driver decoding

[0101] Initialize a fixed set of learnable queries. Combine it with image features The input is a query decoder, which outputs an instance mask, a class score, and a query embedding. The instance mask represents the image region corresponding to each query; the class score represents the semantic category to which each query belongs; and the query embedding... This serves as the identity feature vector of the instance in the current frame.

[0102] See Figure 1 Step 5) of cross-frame identity stabilization specifically includes:

[0103] Step 5.1: Supervised Comparative Learning during the Training Phase

[0104] To learn robust instance features, a contrastive loss function is constructed. For query embeddings of the same real instance generated in different frames, they are treated as positive sample pairs, bringing them closer together in the feature space; for query embeddings of different instances, they are treated as negative sample pairs, pushing them further apart.

[0105] Step 5.2: Instance memory construction during the inference phase

[0106] Establish an instance memory to store instance prototypes that have been confirmed at historical moments.

[0107] Step 5.3: Similarity-based instance association

[0108] At any moment The following formula is used to calculate all query embeddings in the current frame. With instance prototypes in the memory Cosine similarity matrix between : .

[0109] Based on this matrix, a binary matching algorithm (or greedy matching) is used to associate the detection result of the current frame with the historical ID.

[0110] Step 5.4: Momentum Update of the Prototype

[0111] For successfully matched instances The prototype in the memory is updated using the observation features of the current frame, employing the exponential moving average (EMA) method shown below: .

[0112] in, To update the coefficients, this step ensures that instance features evolve slowly over time to adapt to changes in appearance.

[0113] See Figure 1 Step 6) specifically includes:

[0114] Step 6.1: Pixel Back Projection

[0115] Using the predicted depth map Combined with camera intrinsics, the image pixel coordinates are back-projected into 3D space and combined with the predicted camera pose. We obtain the three-dimensional coordinates of the point in the world coordinate system, which serves as the center position of the newly generated Gaussian.

[0116] Step 6.2: Property Binding and Collection Generation

[0117] The Gaussian geometric properties predicted in step 3.4 The instance ID and semantic category label determined in step 5 are bound to the corresponding 3D center to form the Gaussian set of the current frame. .

[0118] Step 6.3: Incremental Global Update

[0119] Will Directly incorporated into the global Gaussian field middle.

[0120] See Figure 1 Step 7) specifically includes:

[0121] Step 7.1: Language-driven open lexical segmentation

[0122] Set up a query semantic projector Embed the query Mapped to the visual-language joint space.

[0123] Step 7.2: Text Retrieval

[0124] The user inputs a text prompt (such as "red cup"), which is then used to generate a text embedding via a text encoder. The similarity between the projected query embedding and the text embedding is calculated, and the mask with the highest similarity is selected as the target region, achieving open segmentation for unseen categories.

[0125] Step 7.3: Region Semantic Distillation

[0126] During projector training, features extracted from the image region to be predicted mask coverage by a pre-trained 2D base model are used as "teacher embeddings" to guide the projector. Through learning, it acquires the ability to generalize.

[0127] Step 7.4: Final Rendering

[0128] Based on user instructions, select the rendering mode. If the panorama segmentation mode is selected, the renderer will splash all Gaussian points within the view frustum, aggregating not only colors but also semantic IDs and instance IDs, and finally outputting a pixel-aligned panorama segmentation map.

[0129] Implementing all or part of the processes in the embodiments of this invention is accomplished by scheduling relevant hardware (such as a GPU equipped with a CUDA core) through computer software programming. The computer software program can be stored as an entity in a computer-readable storage medium, which can be any physical device capable of carrying computer software programs or code, including but not limited to: USB flash drives, external hard drives, disks, optical discs, server storage arrays, etc. This invention can effectively solve the challenges of online 3D reconstruction and semantic understanding in long-sequence video streams. Through strictly causal streaming processing and instance-level identity stabilization mechanisms, it provides real-time, consistent, and semantically rich 3D environment representations for applications such as robot navigation and AR / VR interaction.

[0130] The above is merely a further description of the present invention and is not intended to limit the scope of this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.

Claims

1. An online semantic 3D Gaussian field incremental reconstruction method, characterized in that, This method uses unlabeled RGB image streams For input, where For the total number of frames, at time... Utilize the current frame Incremental updates are performed on the persistent state accumulated from historical frames to achieve online scene understanding and 3D semantic Gaussian reconstruction for long sequences of RGB image streams. The specific steps include the following: Step 1: Geometric Branch Causal Coding Current frame Encode the visual tokens and input them into a causal Transformer encoder, using a temporal attention mask to make the time intervals... The token can only access historical prefixes Key-value information; Step 2: Geometric Feature Prediction Through deep prediction head Camera parameter prediction head Gaussian property prediction head Output the current frame depth map, camera pose parameters, and pixel-level Gaussian properties respectively; Step 3: Generate Gaussian by pixel back projection in accordance with and The pixels are back-projected into three-dimensional space to obtain the pixel-aligned three-dimensional Gaussian center, and then compared with... By concatenating the Gaussian sets of the current frame, we obtain the set of Gaussians. ; Step 4: Incremental Global Gaussian Field Update Will Incremental accumulation to the global three-dimensional Gaussian field In the meantime, this global three-dimensional Gaussian field is maintained as a persistent state; Step 5: Semantic Branch Independent Feature Extraction Using a frozen 2D vision base model Extract semantic features and input them into a lightweight adapter to form multi-scale features. ; Step 6: Query-driven instance segmentation and embedding generation Multiple learnable queries Multiscale features Input the query decoder to obtain the instance mask, class score, and query embedding for each frame. , where query embedding For instance identity descriptors; Step 7: Cross-frame identity stabilization Cross-frame identity stabilization includes a training phase and an inference phase. The training phase aligns queries with real instances and performs query-level supervised contrastive learning, enabling the same instance to be clustered in query embeddings across different frames, thus separating different instances. The inference phase establishes an instance memory bank, associates the current frame query with historical instance prototypes based on cosine similarity and binary matching, and updates the instance prototypes using momentum updates to obtain a real-time instance bank. Step 8: Semantic information is elevated to a three-dimensional Gaussian field The instance prototype prediction obtained in step 7 is assigned to the pixel-aligned 3D Gaussian obtained in step 3, and multi-view evidence is fused in the global 3D Gaussian field to obtain the semantic information of the 3D Gaussian field. Step 9: Differentiable Gaussian rendering output Based on a global 3D Gaussian field, Gaussian rendering is used to output one or more of the following tasks: new perspective image, semantic segmentation, instance segmentation, and panoptic segmentation.

2. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, The causal Transformer uses a time mask matrix represented by the following formula. : ； in, The moment of the current processing frame This refers to the moment before the current frame is being processed.

3. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, In the reasoning stage of step 1), the key-value tensors of historical frames are cached and reused, so that when processing the current frame, there is no need to perform forward calculation on the historical frames again, thereby realizing non-reprocessing streaming reasoning.

4. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, Step 2) employs a strictly causal 3D basic model to output pseudo-depth. and pseudo camera parameters As a signal from the teacher, and Distillation constraints are applied to suppress the accumulation of errors in long sequences.

5. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, The query-level supervised contrastive learning loss in step 7) is as follows: the query embeddings for the same instance label are the positive sample set, and different instances are the negative sample set, using the temperature parameter... The objective function of comparison increases the similarity of positive samples and decreases the similarity of negative samples, thereby enhancing cross-frame instance consistency.

6. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, The steps 7) The association matrix of the instance memory is calculated using cosine similarity, and the association results are obtained through binary matching. Similarity Calculated by the following formula: ； in, For the current frame number Each query embeds a normalized vector; For the first time in history An instance prototype.

7. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, The instance memory is updated using an exponential moving average method to update the prototype embedding. : in, This represents the update coefficient.

8. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, Step 8) maps the prediction mask, category score, or instance ID of each frame to Gaussian semantic attributes, and together with geometric visibility, participates in Gaussian rendering fusion, so that consistent semantic and instance results are obtained across viewpoints when rendering from any viewpoint.

9. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 1, characterized in that, Step 9) sets the query semantic projector The query embedding is mapped to the visual-language joint space to support language-driven open-lexical segmentation, and its similarity with the text embedding output by the text encoder is calculated. The mask corresponding to the query with the highest similarity is selected as the output.

10. The online semantic three-dimensional Gaussian field incremental reconstruction method according to claim 9, characterized in that, The query semantic projector is trained using a prediction mask to crop or weight aggregate target regions from the input image, regional semantic distillation, and a frozen visual-language image encoder to generate teacher embeddings. Cosine regression is then performed on the projected query embeddings to achieve semantic space alignment.

Images