Generalized real-time rendering 3D Gaussian splash semantic segmentation method and system

Abstract

The invention discloses a generalized real-time rendering 3D Gaussian splash semantic segmentation method and system, and relates to the field of computer vision. The method comprises the following steps: training a universal 3D feature encoder through self-supervised contrast learning by using multi-scene data; for a target scene to be segmented, using the encoder to extract view irrelevant features from the 3D Gaussian splash representation of the target scene and binding the view irrelevant features to obtain feature enhanced scene representation; performing synchronous rendering on the representation according to a specified camera pose to generate a pixel-level aligned RGB image and a view-independent feature image; and inputting the RGB image and the view-independent feature image pair into a pre-trained VDVI fusion semantic segmentation network, and outputting a semantic segmentation map through feature fusion and decoding. According to the method, real-time reasoning and strong scene generalization ability are simultaneously realized for the first time on 3D Gaussian splash representation, model fine adjustment of a new scene is not needed, and the contradiction that speed and generalization cannot be realized at the same time in the prior art is effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a generalizable real-time rendering 3D Gaussian splash semantic segmentation method and system. This invention also relates to electronic devices and storage media for implementing the above method. Background Technology

[0002] Scene understanding is a fundamental research area in computer vision, crucial for applications requiring seamless interaction with 3D environments, such as augmented reality (AR), autonomous robot navigation, and digital twins. To achieve semantic understanding of scenes (e.g., object category identification), an effective scene representation method is first needed.

[0003] 1. Evolution of scene representation methods:

[0004] Early work relied primarily on two-dimensional (2D) representations (such as RGB images). While 2D images are easy to acquire and process quickly, they are essentially projections of the 3D world onto a specific viewpoint, failing to fully capture the geometry and 3D properties of the scene. This results in 2D-based semantic segmentation results being heavily dependent on the shooting perspective and lacking consistency in 3D space.

[0005] To achieve consistent perceptual results, researchers have turned to three-dimensional (3D) representations, such as point clouds and polygon meshes. These representations can directly digitize the geometric information of 3D scenes. However, high-precision 3D data typically requires expensive equipment such as LiDAR for acquisition, and semantic segmentation models based on this data often require time-consuming and laborious 3D annotation, limiting their widespread application.

[0006] In recent years, Neural Radiation Field (NeRF) has emerged as a revolutionary implicit 3D scene representation technology. NeRF can train a multilayer perceptron (MLP) to model the geometry and appearance of a scene and synthesize high-quality new viewpoint images by inputting only a set of multi-view 2D images. This paradigm is significant: it not only avoids expensive 3D data acquisition but also allows the supervision signals for rendering losses and even downstream tasks (such as semantic segmentation) to be constructed in an easily accessible 2D domain, paving the way for viewpoint-consistent scene understanding without 3D annotation. However, NeRF relies on a time-consuming volume rendering process, resulting in slow inference speeds that are difficult to meet the needs of real-time interactive applications.

[0007] To address rendering speed bottlenecks, 3D Gaussian Splatting (3DGS) technology was proposed. 3D Gaussian Splatting explicitly represents a scene as a series of 3D Gaussian distributions with position, covariance, opacity, and spherical harmonic color coefficients. During rendering, these Gaussian distributions are efficiently "splashed" onto the image plane via rasterization, enabling real-time, high-quality synthesis of new perspectives. Essentially, 3DGS can be viewed as a structured, differentiable point cloud representation, making it naturally compatible with the processing paradigm of point cloud deep learning networks (such as PointNet, PointNet++, etc.).

[0008] 2. Development and existing limitations of semantic segmentation technology in 3D representation:

[0009] Imbuing the aforementioned 3D representation with semantic segmentation capabilities is a crucial step in achieving scene understanding. Currently, two main technical paths have emerged in this field, but both have significant drawbacks:

[0010] Path 1: Generalizable semantic segmentation based on NeRF.

[0011] To overcome the reliance of traditional methods on 3D annotation and to handle unseen scenes, researchers have proposed generalizable segmentation frameworks based on NeRF, such as GNeSF (Generalizable Neural Semantic Fields). These methods typically employ complex architectures (such as soft voting mechanisms and Transformer feature aggregation) to fuse multi-view 2D semantic information, enabling them to generalize to entirely new scenes beyond the training set. However, because they rely on the underlying NeRF rendering mechanism, they inherit and exacerbate the slow inference speed inherent in NeRF, making real-time segmentation impossible and severely limiting their application prospects in real-time systems.

[0012] Path 2: Efficient semantic segmentation based on 3DGS.

[0013] To pursue real-time performance, another type of work directly combines semantic features with 3DGS representations, such as CoSSegGaussians. This method associates semantic features with each 3D Gaussian and leverages the real-time rendering capabilities of 3DGS to achieve fast, viewpoint-consistent semantic segmentation in specific scenes. However, these methods typically employ a "per-scene optimization" training paradigm, where model parameters are highly coupled to the geometry and appearance of a single scene. This leads to severe overfitting of the model to the training scene and a lack of ability to generalize to new scenes. Once the scene changes, retraining or fine-tuning is required, failing to achieve "train once, apply everywhere".

[0014] 3. Summary of existing technologies and technical problems to be solved:

[0015] In summary, existing technologies present a contradictory situation where "speed and generalization ability cannot be simultaneously achieved":

[0016] NeRF-based methods, such as GNeSF, have good generalization capabilities, but they sacrifice real-time performance due to limitations in the rendering mechanism.

[0017] Methods based on 3DGS, such as CoSSegGaussians, achieve real-time segmentation through explicit representation, but lose generalization ability due to the scene-by-scene training mode.

[0018] Therefore, there is an urgent need in this field for an innovative technical solution that can simultaneously support real-time rendering capabilities and powerful scene generalization capabilities, that is, to achieve real-time, high-precision, and viewpoint-consistent semantic segmentation of unknown 3D scenes without the need for retraining or fine-tuning for new scenes. This is the core technical problem that this invention aims to solve. Summary of the Invention

[0019] To overcome the aforementioned problems, this invention proposes a generalizable real-time rendering 3D Gaussian splash semantic segmentation method and system. The proposed method combines the advantages of the previous two types of methods, possessing both real-time segmentation speed and excellent generalization ability.

[0020] Therefore, the first objective of this invention is to provide a generalizable real-time rendering method for 3D Gaussian splash semantic segmentation, which includes the following steps:

[0021] By using multi-scene data and self-supervised contrastive learning, a general 3D feature encoder capable of extracting view-independent features is trained, resulting in the trained general 3D feature encoder. For the target scene to be segmented, the trained general 3D feature encoder is used to extract view-independent feature vectors from the 3D Gaussian splatter representation of the target scene, and the feature vectors are bound to the corresponding 3D Gaussian to obtain a feature-enhanced 3D Gaussian scene representation. Based on the specified camera pose, the feature-enhanced 3D Gaussian scene representation is rendered synchronously to generate pixel-aligned RGB images and view-independent feature images. The RGB image and view-independent feature image are input into a pre-trained VDVI fusion semantic segmentation network. Through feature fusion and decoding, a semantic segmentation map of the current viewpoint is output.

[0022] Preferably, the step of training a general 3D feature encoder capable of extracting view-independent features using multi-scene data through self-supervised contrastive learning includes: The 3D Gaussian splash representation of each training scene is converted into point cloud data composed of the geometric and basic appearance attributes of each of its 3D Gaussians. Based on the point cloud data, the initial depth feature vector of each 3D Gaussian is extracted by the point cloud encoder; Based on the initial depth feature vector, and with the goal of ensuring consistent features across multiple viewpoints using the same 3D Gaussian vector, the general 3D feature encoder is obtained.

[0023] Preferably, for the target scene to be segmented, the trained general 3D feature encoder is used to extract view-independent feature vectors from the 3D Gaussian splatter representation of the target scene, and the feature vectors are bound to the corresponding 3D Gaussian splatter representations to obtain a feature-enhanced 3D Gaussian scene representation, including: Convert the 3D Gaussian splash representation of the target scene into point cloud data; The point cloud data is input into the trained general 3D feature encoder to generate a high-dimensional feature vector corresponding to each 3D Gaussian. The high-dimensional feature vector is bound to the corresponding 3D Gaussian vector to form the feature-enhanced 3D Gaussian scene representation.

[0024] Preferably, the step of synchronously rendering the feature-enhanced 3D Gaussian scene representation according to the specified camera pose to generate a pixel-aligned RGB image and a view-independent feature image includes: Based on the feature-enhanced 3D Gaussian scene representation and the camera pose, the 3D Gaussian is projected onto the image plane and sorted by depth. Based on the depth sorting, for each pixel location covered by the 3D Gaussian, color blending and feature vector blending based on its opacity are performed in parallel to generate color buffers and feature buffers respectively; The color buffer and feature buffer are respectively converted into the pixel-aligned RGB image and view-independent feature image.

[0025] Preferably, the RGB image and view-independent feature image are input into a pre-trained VDVI fusion semantic segmentation network. Through feature fusion and decoding, a semantic segmentation map of the current viewpoint is output, including: The RGB image is input into the view-related encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-related features; The view-independent feature image is input into the view-independent encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-independent features; At multiple feature scales with the same spatial resolution output by the view-dependent encoder and the view-independent encoder, the view-dependent features and view-independent features are fused to obtain multi-scale fused features; Decoding is performed based on the multi-scale fusion features to output a pixel-level semantic segmentation map from the current perspective.

[0026] The second objective of this invention is to provide a generalizable real-time rendering 3D Gaussian splash semantic segmentation system, implementing the steps of any of the above-mentioned generalizable real-time rendering 3D Gaussian splash semantic segmentation methods, including: An offline training subsystem is used to train the general 3D feature encoder; An online inference subsystem is used to perform semantic segmentation on the target scene.

[0027] Preferably, the offline training subsystem includes: The multi-scene 3D reconstruction and data management module is used to reconstruct multi-view images of various training scenes into 3D Gaussian splash representations and convert them into point cloud data. A self-supervised feature encoder training module is used to train the general 3D feature encoder through self-supervised contrastive learning.

[0028] Preferably, the online inference subsystem includes: The feature extraction and binding module is used to extract and bind view-independent feature vectors for the 3D Gaussian splash representation of the new scene, generating a feature-enhanced 3D Gaussian scene representation; The real-time multi-attribute rendering module is used to render the feature-enhanced 3D Gaussian scene representation based on the camera pose, generating aligned RGB images and view-independent feature images. The semantic decoding output module is used to call the pre-trained VDVI fusion network to process the image pairs and output the segmentation results.

[0029] A third objective of this invention is to provide an electronic device comprising: Memory, used to store computer programs; A processor is used to execute a program stored in memory to implement any of the steps of a generalizable real-time rendering 3D Gaussian splash semantic segmentation method described above.

[0030] The fourth objective of this invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned generalizable real-time rendering 3D Gaussian splash semantic segmentation methods.

[0031] The beneficial effects of this invention are as follows:

[0032] 1. It fundamentally solves the technical contradiction of the difficulty in achieving both "real-time performance" and "generalizability", and realizes high-performance real-time semantic segmentation for unseen scenarios.

[0033] As described in the background section, there is a clear divide between existing technologies: NeRF-based methods (such as GNeSF) can generalize but are slow; 3DGS-based methods (such as CoSSegGaussians) are fast but require scene-by-scene training and cannot generalize.

[0034] This invention creatively proposes a unified three-stage architecture, achieving for the first time generalizable semantic segmentation capabilities on a real-time rendering representation of 3D Gaussian splash. Through this method, the system can perform real-time, viewpoint-consistent semantic segmentation of entirely new scenes, relying solely on the 3DGS representation reconstructed from the input multi-view images, without requiring any fine-tuning or retraining for new scenes. This achieves, for the first time, both real-time inference capabilities for unseen scenes and powerful scene generalization capabilities on the 3D Gaussian splash representation, resolving the technical contradictions pointed out in the background section.

[0035] 2. A self-supervised 3D Gaussian view-independent feature extraction method was proposed and implemented, laying a key foundation for generalizable segmentation.

[0036] This invention innovatively introduces a self-supervised contrastive learning mechanism into the processing of 3D Gaussian representation. By designing a learning objective that "the features of the same 3D point should be consistent under different viewpoints," the point cloud encoding network is driven to learn robust, viewpoint-independent depth features. The core advantage of this method is:

[0037] 1) No 3D semantic annotation required: The feature extractor can be pre-trained using only multi-view images and their geometric relationships, which greatly reduces the cost of data annotation.

[0038] 2) Obtaining essential features: The learned features focus on the 3D geometry and inherent appearance of objects, rather than the imaging details from a specific viewpoint. This is the fundamental reason why the model can generalize to new scenes.

[0039] 3. A feature fusion mechanism for view-dependent and view-independent features was designed, which significantly improved segmentation accuracy and 3D consistency.

[0040] This invention does not use 3D features or 2D image features in isolation, but instead designs an asymmetric dual-encoder fusion module. This module captures rich texture, lighting, and other detailed information from RGB images through a view-dependent encoder; simultaneously, it encodes the rendered 3D feature map using a view-independent encoder, injecting strong geometric constraints. By fusing these two complementary features at multiple scales, the final segmentation result maintains high-precision boundary details while ensuring high 3D spatial consistency of the predicted semantics across different viewpoints, overcoming the viewpoint dependency problem of pure 2D segmentation methods.

[0041] 4. An efficient real-time inference pipeline was constructed, realizing the combination of theoretical innovation and engineering application.

[0042] The overall architecture of this invention has been carefully designed to meet real-time requirements:

[0043] 1) One-time 3D feature extraction: For any input scene, its 3D Gaussian view-independent features only need to be extracted and stored offline once.

[0044] 2) Lightweight 2D forward inference: In the online real-time segmentation stage, only efficient 3DGS rendering (to obtain RGB images and feature maps) and a lightweight 2D VDVI fusion semantic segmentation network need to be run. This design enables the overall inference speed of the system to be comparable to or even surpass traditional scene-by-scene 3DGS segmentation methods (such as CoSSegGaussians), while gaining generalization capabilities that the latter do not possess.

[0045] 5. It possesses strong practical application value and scalability.

[0046] Because the method of this invention simultaneously satisfies the two key requirements of "real-time" and "generalizability," it can be directly applied to fields with extremely high requirements for latency and universality, such as augmented reality, real-time environmental perception of robots, and rapid semantic mapping of large-scale scenes. Once the model is jointly trained on multi-scene data, it can be deployed as a general-purpose tool, greatly improving the practicality and cost-effectiveness of the technical solution. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] Figure 1 This is a flowchart illustrating a generalizable real-time rendering 3D Gaussian splash semantic segmentation method according to the present invention.

[0049] Figure 2 This is a flowchart illustrating the training steps of a general 3D feature encoder in an embodiment of the present invention.

[0050] Figure 3 This is a flowchart illustrating the feature extraction and binding steps in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the principle of the real-time multi-attribute synchronous rendering step in an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of the decoding and inference steps of the VDVI fusion semantic segmentation network in an embodiment of the present invention.

[0053] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0055] Terminology Explanation:

[0056] 3D Gaussian Splatting: An explicit, point-based 3D scene representation and real-time rendering technique. This technique models a 3D scene as a set of learnable 3D Gaussian distributions, each with spatial location, covariance (controlling anisotropic shape), opacity, and view-dependent color attributes encoded by spherical harmonics. During rendering, the 3D Gaussian distributions are projected onto a 2D image plane via differentiable rasterization ("splatting") and alpha blended, enabling high-quality, high-frame-rate real-time rendering of the scene from any viewpoint. Compared to implicit representations (such as NeRF), it significantly improves rendering speed through explicit geometric representation and an efficient rasterization process.

[0057] Neural Radiance Fields (NeRF) is an implicit, continuous 3D scene representation and modeling method. It uses a multilayer perceptron (MLP) neural network as a function approximator to map 3D spatial coordinates and 2D viewing angles to corresponding volume density and color values. By integrating this neural field along camera rays using volume rendering techniques, highly realistic images from any new viewpoint can be synthesized. NeRF's core advantage lies in its ability to reconstruct the geometry and appearance of complex scenes with high quality from sparse, multi-view 2D images. However, its rendering process, which requires dense sampling and neural network queries, results in slow inference (rendering) speeds, often making it difficult to meet the needs of real-time interactive applications.

[0058] Semantic segmentation: a computer vision task that aims to assign a semantic category label to each basic unit in the input data. Depending on the modality of the input data, it can be divided into:

[0059] 1.2D Image Semantic Segmentation: Assign a predefined semantic category (such as "person", "vehicle", "road") to each pixel in the image.

[0060] 2.3D Scene Semantic Segmentation: Assign a semantic category to each point, patch, or voxel in a 3D representation (such as a point cloud, mesh, voxel, or implicit field).

[0061] The semantic segmentation task of this invention specifically refers to understanding a 3D scene and ultimately outputting a two-dimensional semantic segmentation map corresponding to a specific viewing perspective, and the segmentation result should maintain consistency in 3D space under different perspectives.

[0062] Generalizability refers to the ability of a machine learning model or algorithm to adapt to and process new data or scenarios outside the training data distribution. In the context of this invention, it specifically refers to the trained semantic segmentation model.

[0063] 1. Scene generalization: After joint training using multi-scene data during the training phase, it can be directly applied to a brand new, unseen scene (B) without any additional model parameter optimization, fine-tuning, or retraining for scene B.

[0064] 2. Functional Consistency: In new scenarios, the model not only runs but also maintains segmentation accuracy, robustness, and inference speed similar to those in the training scenario. This is a key performance indicator for measuring whether a system can achieve "train once, apply everywhere".

[0065] Point cloud: A discrete, unstructured representation of 3D spatial data, consisting of a series of points distributed in a 3D coordinate system, typically possessing geometric or physical properties. Each point contains at least its coordinates (X, Y, Z) in 3D space and may have additional attributes such as color (R, G, B), normal, intensity, or reflectivity. Point cloud data is typically acquired by 3D scanning devices (such as LiDAR and structured light scanners) or reconstructed from multi-view 2D images using techniques such as Structure-from-Motion (SfM).

[0066] In the context of this invention:

[0067] 1. As a processing object: The core parameters (position, covariance, color, transparency) of each 3D Gaussian distribution in the "3D Gaussian splatter" representation can be extracted and reconstructed into a point with a multi-dimensional attribute vector, thus treating the entire 3D Gaussian set as a special point cloud with rich attributes. This allows deep learning networks designed for point clouds (such as PointNet, PointNeXt) to be used directly or adapted to process 3D Gaussian representations.

[0068] 2. Relationship with 3D Gaussian Splash: Point clouds are a more basic and general 3D data format, while 3D Gaussian splash is a structured, differentiable, and specific point cloud representation optimized for efficient rendering. Both share geometric origins, but 3D Gaussian splash, by introducing anisotropic covariance and optimizable opacity, possesses stronger scene fitting capabilities and rendering quality.

[0069] Structure-from-Motion (SfM) algorithm: A computer vision method that automatically reconstructs the sparse geometric structure (i.e., sparse point cloud) of a 3D scene from a series of unordered, visually overlapping 2D images and simultaneously estimates the camera pose (position and orientation) for each image. Its basic principle is to establish correspondences between images through feature matching, utilize multi-view geometric constraints, and simultaneously solve for 3D point positions and camera parameters using optimization algorithms (such as bundle adjustment). In this invention, the SfM algorithm or its output (the camera pose corresponding to the image) is one of the key prerequisite inputs required for the 3D Gaussian splash reconstruction algorithm, providing initial geometric constraints and viewpoint relationships for multi-view images.

[0070] 3D Gaussian Splatting Representation: A scene modeling and rendering paradigm based on a three-dimensional Gaussian distribution. In this invention, the term refers to the overall model formed by modeling a scene using 3D Gaussian splatting technology. 3D Gaussian splatting representation emphasizes the underlying representation technology upon which this method relies and is typically used for high-level technical roadmap descriptions and system overviews.

[0071] A 3D Gaussian Distribution Set is a concrete, computable data structure that constitutes the 3D Gaussian splash representation. Specifically, it refers to a mathematical set generated from multi-view images using the aforementioned "3D Gaussian splash reconstruction algorithm," consisting of a large number (typically tens of thousands to millions) of discrete, parameter-optimizable 3D Gaussian distributions. Each element (a 3D Gaussian distribution) in the 3D Gaussian Distribution Set is a basic unit carrying geometric, appearance, and feature information. In the specific implementation of this invention, the feature extraction, attribute binding, and real-time rendering operations performed on the "3D Gaussian splash representation" directly process this "3D Gaussian Distribution Set."

[0072] The 3D Gaussian Splatting Reconstruction Algorithm is an algorithm for efficiently reconstructing a high-quality, real-time-renderable explicit 3D scene representation from a set of multi-view color (RGB) images with known camera poses. The algorithm takes the images and their corresponding poses as input, and its core process and optimization objectives are as follows:

[0073] 1. Initialization: Typically, start with the sparse point cloud generated by the Structure for Motion Recovery (SfM) algorithm, and initialize each point as a 3D Gaussian distribution.

[0074] 2. Representation Parameters: The scene is represented as a large set of optimizable 3D Gaussian distributions. Each Gaussian distribution is defined by the following parameters:

[0075] 1) Geometric properties: 3D center position, a covariance matrix defined by scaling vectors and rotation quaternions (controlling its anisotropic shape and orientation in space).

[0076] 2) Appearance attributes: opacity, and a set of spherical harmonic coefficients (used to model view-dependent colors).

[0077] 3. Differentiable Rendering and Optimization: A differentiable, rasterization-based "splatter" rendering technique is employed to project a 3D Gaussian distribution onto a 2D image plane and generate a predicted 2D image through alpha blending. A loss function is constructed by comparing the differences (such as color loss) between the predicted image and the input ground image. Using the backpropagation algorithm, the aforementioned geometric and appearance parameters of all Gaussian distributions are continuously and iteratively optimized, including adaptively controlling the density of the Gaussian distribution (e.g., Gaussians with excessively large splits or excessively low cropping opacity), until the model can re-render the input image with high quality from any input viewpoint.

[0078] The algorithm ultimately outputs an optimized 3D Gaussian distribution set, which is a compact, explicit, and differentiable representation of the scene. This representation is the direct input for feature extraction in step S1 of the method described in this invention. Compared with implicit representations such as neural radiation fields, the results generated by this algorithm support extremely high rendering frame rates due to their explicitness and special rasterization method.

[0079] The general-purpose 3D feature encoder is the cornerstone of this patent's scene generalization capability. It is a feature extractor with a neural network at its core, specifically designed to extract robust, viewpoint-independent essential features from any 3D Gaussian splash representation.

[0080] VDVI Fusion Semantic Segmentation Network: This is the core two-dimensional neural network module responsible for achieving real-time, high-precision semantic segmentation in the system described in this patent.

[0081] VD (View-Dependent): Refers to features processed by the network that originate from RGB images, and these features vary with imaging conditions such as lighting, viewing angle, and texture.

[0082] VI (View-Independent): Refers to features processed by the network that originate from the geometry of the 3D scene and remain stable across different viewing perspectives.

[0083] Integration: This refers to the core mechanism of the network, which is to effectively integrate the two features that are different from each other and complementary in nature.

[0084] Semantic segmentation network: This refers to a network whose ultimate task is to output a pixel-level semantic category label map.

[0085] VDVI fusion semantic segmentation networks typically employ a dual encoder-single decoder architecture, including:

[0086] View-related encoder: A standard 2D image encoder (such as CNN or Transformer-based) is responsible for extracting multi-scale, detailed view-related features from the input new-view RGB image.

[0087] View-independent encoder: An encoder with a structure similar to the view-dependent encoder, whose input channels are adapted to the view-independent feature image (extracted and rendered by the general 3D feature encoder). This branch is responsible for extracting structural features with 3D spatial consistency.

[0088] Multi-scale fusion module: At corresponding levels of the encoder (e.g., 1 / 4, 1 / 8, 1 / 16 scales), feature maps from two branches are fused. The fusion method can be concatenation, addition, or through an attention mechanism, aiming to combine the detailed information of RGB with the structural information of 3D features.

[0089] Decoder: Based on the fused multi-scale features, the spatial resolution is gradually restored through upsampling and skip connections, and finally the semantic category of each pixel is output through the classification layer.

[0090] In this invention, the VDVI fusion semantic segmentation network directly adopts mature semantic segmentation models (such as AsymFormer) and their weights pre-trained on large-scale general-purpose 2D image datasets (such as Cityscapes, COCO). When the system is deployed to a new scene, the VDVI fusion semantic segmentation network is called as a fixed module without any fine-tuning or retraining for the new scene, which is an important prerequisite for achieving "generalization".

[0091] VDVI decouples the semantic segmentation network from another core module—a general-purpose 3D feature encoder that requires self-supervised training. The former is a powerful "2D semantic understanding and decision-making expert," while the latter is a "3D essential feature extraction expert." The two work together through a 3D Gaussian splashing rendering interface.

[0092] As a pure 2D forward inference network, VDVI fused semantic segmentation network has extremely high computational efficiency. When combined with real-time rendered 3DGS, the entire system can achieve end-to-end real-time semantic segmentation.

[0093] Core value: By deeply fusing view-related RGB appearance information with view-independent 3D geometric feature information in the 2D domain, the VDVI fusion semantic segmentation network can ensure high 3D consistency of segmentation results across different viewpoints while maintaining high segmentation accuracy and detail representation. This fundamentally overcomes the viewpoint dependency problem of pure 2D segmentation methods and avoids the lack of generalization ability caused by training 3D segmentation models scene by scene.

[0094] Example 1:

[0095] To better understand the idea behind the generalizable real-time rendering 3D Gaussian splash semantic segmentation method of this invention, a specific example will be used below. First, the model is trained on a "hotel lobby" scene dataset. Then, the trained model is applied to a brand-new, untrained "office corridor" scene for inference and segmentation. This will gradually illustrate the various steps of the method and its technical effects, and elaborate on the entire process of this invention.

[0096] This embodiment includes an offline training phase and an online inference phase.

[0097] Offline training phase: A general feature extraction encoder is trained using datasets containing multiple scenarios such as "hotel lobby," "shopping mall atrium," and "library." This example uses "hotel lobby" as a representative scenario for this phase.

[0098] Online inference stage: For the new target scene "office corridor", the offline-trained feature extraction encoder is used in conjunction with a VDVI fusion semantic segmentation network to perform real-time semantic segmentation without any fine-tuning.

[0099] This invention provides a generalizable real-time rendering method for 3D Gaussian splash semantic segmentation, such as... Figure 1 As shown, it includes the following steps:

[0100] S1. Using multi-scene data, a general 3D feature encoder capable of extracting view-independent features is trained through self-supervised contrastive learning, resulting in the trained general 3D feature encoder.

[0101] The core objective of this phase is to train a general 3D feature encoder using a dataset containing multiple scenes through self-supervised contrastive learning. Once trained, this encoder will possess a key capability: the ability to extract a set of stable, view-independent depth feature vectors from the 3D Gaussian splatter representation of any scene.

[0102] The technical principle and value of this technology lie in the fact that by enabling the encoder to learn the rule that "the same object should have the same features under different viewpoints," it is forced to ignore factors such as viewpoint and lighting changes, thereby extracting the feature representation of the object's essential geometric and appearance attributes. This view invariance is the fundamental basis for the model to generalize to entirely new scenes and support subsequent viewpoint-consistent semantic segmentation.

[0103] This training process is computationally intensive and is usually completed offline on a high-performance server. Only one training session is required, and the resulting encoder can be permanently deployed to serve any new scenario.

[0104] The entire training process is as follows Figure 2 As shown, it includes the following steps:

[0105] S11 converts the 3D Gaussian splash representation of each training scene into point cloud data composed of the geometric and basic appearance attributes of each of its 3D Gaussians.

[0106] The goal of this step is to prepare standardized, uniformly formatted input data for subsequent feature extraction and optimization. For each scenario in the multi-scenario data set, the specific implementation process is as follows:

[0107] 1) Data acquisition and 3D reconstruction.

[0108] In this embodiment, taking the "hotel lobby" scene as an example, a set of color (RGB) images observed from different perspectives are acquired through surround shooting with a camera. Simultaneously, the precise camera pose (including position and orientation) in the global coordinate system corresponding to each image is recorded or calculated using a motion reconstruction algorithm. Subsequently, a 3D Gaussian splatter reconstruction algorithm is used to process the multi-view images and their camera poses. This 3D Gaussian splatter reconstruction algorithm automatically learns and generates an explicit 3D model representing the scene through an optimization process. This model consists of a large number of discrete 3D Gaussian distributions, denoted as set G. Each element g_i in set G is an independent 3D Gaussian distribution, whose complete mathematical description includes the following optimizable parameters: 3D coordinates representing the center position, a covariance matrix controlling its shape and orientation (usually decomposed into a 3D scaling vector and quaternion rotation), an opacity value representing its transparency, and a set of spherical harmonic coefficients used to model the view-dependent appearance.

[0109] 2) Attribute extraction and vector construction.

[0110] To convert the 3D Gaussian representation suitable for rendering into a data format suitable for point cloud neural network processing, it is necessary to extract and recombine the key attributes of each Gaussian g_i. Specifically, the most essential geometric and appearance-based attributes, independent of the viewing perspective, are extracted from each Gaussian g_i, and these attributes are arranged in a fixed order to form a single attribute feature vector v_i. This vector v_i has a dimension of ten, and its attributes are as follows:

[0111] 3D position coordinates: that is, the X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the center point of the Gaussian distribution in the global coordinate system.

[0112] Basic color value: The value corresponding to the zeroth-order component is taken from the coefficients of the spherical harmonic function that describes the relationship between the Gaussian color and the viewing angle. This component represents the diffuse color, which is represented by the red channel intensity value, green channel intensity value, and blue channel intensity value in the RGB color space.

[0113] 3D scaling scale: This Gaussian scaling factor is distributed along the three principal axes of the X, Y and Z axes to characterize the extent of its extension in different directions.

[0114] Opacity value: This Gaussian distribution represents the ability of light to be blocked during rendering; it is a scalar value between completely transparent and completely opaque.

[0115] 3) Point cloud data assembly.

[0116] Iterate through all M Gaussians in the 3D Gaussian set G of the scene, and repeat the attribute extraction and vector construction operations described above for each Gaussian to obtain M ten-dimensional attribute vectors. Stack these M vectors as data rows in order to construct a two-dimensional data matrix P with M rows and 10 columns. This two-dimensional data matrix P is formally equivalent to a point cloud dataset with M points, each with 10 additional attributes. Through this series of operations, the transformation from a complex 3D Gaussian splash representation used for high-quality rendering to a regular point cloud data format suitable for general point cloud deep learning network processing is completed.

[0117] In this embodiment, the "hotel lobby" scenario during the offline training phase is taken as an example:

[0118] 1) Data Acquisition and 3D Reconstruction: A series of color (RGB) images of the "hotel lobby" scene were acquired from different perspectives through surround camera shooting, and the precise camera pose of each image was calculated. Subsequently, a 3D Gaussian splash reconstruction algorithm was used to process the images, generating an explicit 3D model set G_Hotel, which represents the lobby scene and consists of a large number of 3D Gaussian distributions. The number of Gaussians in the 3D model set G_Hotel is defined as M_Hotel.

[0119] 2) Attribute extraction and vector construction: Extract the core, view-independent basic attributes (3D position, basic diffuse color, 3D scaling scale, opacity) from each Gaussian of G_Hotel, and combine them in a fixed order to form a 10-dimensional attribute feature vector.

[0120] 3) Point cloud data assembly: Traverse all M_hotel Gaussian vectors in G_hotel to obtain M_hotel 10-dimensional vectors, and stack them to construct a two-dimensional data matrix P_hotel with M_hotel rows multiplied by 10 columns. This matrix is ​​the point cloud data after the "hotel lobby" scene transformation.

[0121] S12, Based on the point cloud data, the initial depth feature vector of each 3D Gaussian is extracted by the point cloud encoder.

[0122] The PointNeXt network is selected as the encoder part of the point cloud encoder, denoted as E_θ (parameter θ). Point cloud data is input into the encoder E_θ. The PointNeXt network performs feature learning and aggregation on the unstructured point cloud data, outputting an initial depth feature vector for each point in the input (corresponding to each 3D Gaussian). This step is the initial mapping for feature extraction.

[0123] In this embodiment, the point cloud data P_hotel of the "hotel lobby" is input into E_θ, and the network outputs an initial depth feature vector for each point in the input (corresponding to each 3D Gaussian).

[0124] S13, Based on the initial depth feature vector, and with the goal of ensuring that the features of the same 3D Gaussian are consistent across multiple viewpoints, optimization is performed to obtain the general 3D feature encoder.

[0125] The initial depth feature vector is used as the direct processing object. The encoder parameters are optimized using a self-supervised contrastive learning framework: for the same scene, two different camera views are randomly selected. Positive sample pairs are constructed using the feature vectors corresponding to the same set of 3D Gaussians visible in both views, encouraging similarity in their feature representations; negative samples are constructed using feature vectors corresponding to different 3D Gaussians, encouraging dissimilarity in their feature representations. By minimizing the contrastive loss function constructed in this way (such as InfoNCE loss), the encoder network is driven to ignore viewpoint changes, thereby extracting stable, view-independent features. After completing this optimization process on a cross-scene dataset, a trained, generalizable 3D feature encoder is obtained. The training data (multi-scene) of this general 3D feature encoder is completely different from the target inference scene (such as an office corridor).

[0126] The specific implementation is as follows:

[0127] 1) Constructing Comparative Samples: For the same scene, randomly sample two camera viewpoints, V_a and V_b. Select the same set of 3D Gaussian features visible from both viewpoints. Denote the initial depth feature vector set of the 3D Gaussian features under viewpoint V_a as {f_k_a}, and the corresponding feature vector set under viewpoint V_b as {f_k_b}. For each index k, (f_k_a, f_k_b) constitutes a positive sample pair, representing the features of the same 3D entity under different viewpoints. In the same training batch, all other mismatched 3D Gaussian feature vectors constitute negative samples.

[0128] 2) Calculate the contrastive loss: An InfoNCE-based contrastive loss function is used for optimization. The core objective of this loss function is to ensure that the similarity (e.g., cosine similarity) between two feature vectors in a positive sample pair (f_k_a, f_k_b) is significantly higher than the similarity between any feature in the positive sample and all feature vectors in the negative samples. By minimizing this contrastive loss, the parameter θ of the encoder E_θ is continuously updated, thereby gradually ensuring that the extracted features meet the requirement of "viewpoint invariance".

[0129] 3) Complete Training: Repeat the above process of constructing samples and optimizing the loss on a dataset containing multiple scenes. After training, the encoder parameters converge to θ', resulting in the optimized encoder E_θ'. This E_θ' is the general 3D feature encoder. The general 3D feature encoder has learned to ignore viewpoint changes and can output stable, view-independent depth feature representations for any input 3D Gaussian.

[0130] In this embodiment, taking the "hotel lobby" scenario as an example, the self-supervised training process is implemented as follows:

[0131] 1) Constructing contrast samples: For the "hotel lobby" scene, two camera views are randomly sampled. The same set of 3D Gaussian features visible from both views is selected. The initial depth feature vectors of these 3D Gaussians from the two views are used to construct sets, and these sets are then paired to form positive sample pairs (f_k_a, f_k_b). In the same training batch, all other unrelated 3D Gaussian feature vectors constitute negative samples.

[0132] 2) Calculate the contrastive loss: Input the positive and negative samples into the self-supervised contrastive learning framework. Optimize using the InfoNCE loss function, which aims to maximize the feature similarity between positive sample pairs (f_k_a, f_k_b) while minimizing the feature similarity between positive samples and all negative samples. Minimize this loss through backpropagation, driving the continuous update of the encoder E_θ's parameter θ.

[0133] 3) Complete Training: Repeat the above sample construction and optimization process on a dataset containing multiple scenes such as "hotel lobby," "shopping mall atrium," and "library." After training, the encoder E_θ' with optimized parameters is obtained. This E_θ' is the general 3D feature encoder. It has learned to capture view-invariant commonalities from multi-scene data and can extract general and stable view-independent feature representations for any new 3D Gaussian scene, including "hotel lobby."

[0134] S2, for the target scene to be segmented, the trained general 3D feature encoder is used to extract view-independent feature vectors from the 3D Gaussian splatter representation of the target scene, and the feature vectors are bound to the corresponding 3D Gaussian to obtain a feature-enhanced 3D Gaussian scene representation.

[0135] This phase is performed when a completely new target scene (e.g., an "office corridor" scene not used in training) needs to be segmented. In this embodiment, the target scene "office corridor" is used as an example. Figure 3 As shown, it includes the following steps:

[0136] S21. Convert the 3D Gaussian splash representation of the target scene into point cloud data.

[0137] The input to this sub-step is an explicit 3D representation of the target scene, namely, the 3D Gaussian distribution set G_target generated by the 3D Gaussian splatter reconstruction algorithm. The processing follows the method detailed in step S11: extracting the center coordinates, basic diffuse color values ​​(from the zeroth-order coefficients of the spherical harmonic function), scaling scales along each axis, and opacity values ​​of each 3D Gaussian in G_target, and combining them in the same order to form a ten-dimensional feature vector. Iterating through all Gaussians, stacking their ten-dimensional feature vectors, constructs a regular point cloud data, denoted as P_target. This process completes the conversion from rendering format to neural network processing format.

[0138] In this embodiment, the input object is the 3D Gaussian set G_corridor of the "office corridor" scene. Following the method in step S11, it is converted into point cloud data P_corridor.

[0139] S22. Input the point cloud data into the trained general 3D feature encoder to generate a high-dimensional feature vector corresponding to each 3D Gaussian.

[0140] The point cloud data P_target is input into the general 3D feature encoder E_θ', which has been optimized and fixed during the offline training phase. The general 3D feature encoder performs one forward propagation, generating a high-dimensional, dense feature vector for each data point in P_target (corresponding to a 3D Gaussian). The set of all feature vectors constitutes the view-independent feature vector set F_3d of the target scene.

[0141] In this embodiment, the point cloud data P_corridor is input into the post-trained encoder E_θ'. The post-trained encoder outputs a feature vector for each data point of P_corridor, which is then aggregated to form a view-independent high-dimensional feature vector set F3d_corridor for the "office corridor" scene. The tensor shape of F3d_corridor is (M_corridor, 256), where M_corridor is the total number of Gaussians.

[0142] S23. Bind the high-dimensional feature vector to the corresponding 3D Gaussian to form the feature-enhanced 3D Gaussian scene representation.

[0143] A precise and permanent one-to-one binding is performed between the high-dimensional feature vector set F_3d and the original 3D Gaussian set G_target. Each 3D Gaussian, in addition to its original geometric appearance parameters, is additionally associated with a feature vector representing its essential properties. The result of this binding is the feature-enhanced 3D Gaussian scene representation. This representation is fully stored in memory and serves as direct input to the subsequent real-time rendering module, providing a digital foundation rich in view-independent features for the semantic understanding of the entire scene.

[0144] In this embodiment, the feature vectors in F3d_corridor are bound one by one to the corresponding Gaussian vectors in G_corridor to form a feature-enhanced representation of the "office corridor" scene. The effect of this step is that, by utilizing general knowledge learned from scenes such as "hotel lobby", a "digital fingerprint" of each 3D point is generated for the new "office corridor".

[0145] S3, based on the specified camera pose, synchronously render the feature-enhanced 3D Gaussian scene representation to generate a pixel-aligned RGB image and a view-independent feature image.

[0146] For a given, arbitrarily specified target camera pose, the efficient differentiable rendering capability of 3D Gaussian splashing is utilized to project and fuse the geometry, appearance, and extracted depth features of 3D space onto a 2D image plane in a single operation, thereby generating paired input data for subsequent 2D semantic decoding network processing. For example... Figure 4 As shown, it includes the following steps:

[0147] S31, based on the feature-enhanced 3D Gaussian scene representation and the camera pose, project the 3D Gaussian onto the image plane and perform depth sorting.

[0148] The input is a feature-enhanced 3D Gaussian scene representation, in which each 3D Gaussian carries a high-dimensional view-independent feature vector. Based on the specified camera pose, the projection region of each 3D Gaussian on the image plane is calculated, and all visible Gaussians are sorted from front to back according to their depth information, establishing the correct geometric and occlusion relationships for subsequent blending rendering.

[0149] In this embodiment, an enhanced 3D Gaussian scene representation of an "office corridor" is input. Each 3D Gaussian in this representation carries a high-dimensional view-independent feature vector. Based on the specified camera pose, the projection region of each 3D Gaussian on the image plane is calculated, and all visible Gaussians are sorted from front to back according to their depth information to establish the correct geometric and occlusion relationships for subsequent blending rendering.

[0150] S32, according to the depth sorting, for each pixel location covered by the 3D Gaussian, perform color blending and feature vector blending based on its opacity in parallel to generate color buffer and feature buffer respectively.

[0151] The process involves a 3D Gaussian scene representation based on depth-ranked feature enhancement, and a user-specified virtual camera pose for rendering. The process comprises two logically parallel but procedurally unified stages:

[0152] 1) Projection and Depth Ranking: Based on the specified camera pose, the visibility of each 3D Gaussian distribution carrying feature attributes within the camera's view frustum is calculated, and its projection area on the 2D image plane is determined. Subsequently, based on the depth information of each Gaussian distribution's center point relative to the camera's optical center, all visible Gaussian distributions are strictly ranked from front to back, establishing the correct occlusion relationships for subsequent blending and overlay.

[0153] 2) Parallel Alpha Blending Sputtering Calculation: For each pixel in the image plane, a differentiable alpha blending calculation, similar to that used in classic 3D Gaussian splash rendering, is performed. This calculation process simultaneously applies to two types of attributes: a) Color Attribute Sputtering: Following depth sorting, all Gaussian distributions projected onto the pixel are used as input, with their 2D projection opacity and RGB color values ​​interpolated by spherical harmonics based on the viewing direction. The alpha blending formula is used to synthesize these values ​​layer by layer, calculating the final color value of the pixel and forming a color buffer that stores the final RGB color value of each pixel. b) Feature Attribute Sputtering: Under the same depth sorting and pixel coverage relationship, a parallel blending calculation is performed. Here, the "value" of each Gaussian distribution inputting the blending formula is its bound high-dimensional feature vector, not its RGB color value. The alpha blending formula uses the same weighted average and synthesis method to form a fused high-dimensional feature vector, i.e., a feature buffer, for each pixel.

[0154] The core of this sub-step lies in using the same set of geometric projection, sorting, and blending mechanisms to complete the mapping and fusion of two different attributes—color and features—to the image plane in one go. The computational result is the generation of two two-dimensional data fields in memory: one storing the final RGB color value of each pixel (the color buffer), and the other storing the fused high-dimensional feature vector corresponding to each pixel (the feature buffer).

[0155] In this embodiment, the input is the scene representation with enhanced feature attributes described above, and a user-specified virtual camera pose (e.g., located at the entrance of a corridor, facing forward).

[0156] 1) Projection and Depth Sort: Based on the specified pose, calculate the visibility of each Gaussian and its projection area on the image plane, and sort them from front to back according to depth.

[0157] 2) Parallel Alpha Blending Sputtering Calculation: For each pixel in the image plane, the following is performed in parallel:

[0158] Color attribute sputtering: Mixes all Gaussian colors covering the pixel, calculates the final RGB color value of the pixel, and forms a color buffer.

[0159] Feature sputtering: Under the same geometric relationship, all Gaussian-bound feature vectors covering the pixel are mixed to calculate the fused feature vector of the pixel, forming a feature buffer.

[0160] S33, the color buffer and feature buffer are converted into the pixel-aligned RGB image and view-independent feature image, respectively.

[0161] Based on the color buffer and feature buffer, this sub-step converts the above intermediate data into a standardized image format that can be directly processed by the subsequent semantic decoding module.

[0162] First, the pixel color values ​​stored in the color buffer are standardized: according to the spatial row and column order of the image plane, the floating-point value sequences representing the red, green, and blue channels are arranged and transformed to generate a scene RGB image that conforms to the general image standard.

[0163] Simultaneously, parallel processing is performed on the feature buffer: based on the exact same spatial correspondence as the generated RGB image, the high-dimensional feature vectors (with dimensions C, for example, C=256) corresponding to each pixel position in the buffer are arranged to construct a view-independent feature image. This feature image maintains a strictly consistent spatial alignment with the scene RGB image in the height and width dimensions, the difference being that each pixel position no longer stores three-channel color information, but instead stores a depth feature vector containing C components.

[0164] The final output of this sub-step is a set of RGB image-feature image pairs that are fully spatially aligned at the pixel level. This image pair serves as the unified input to the subsequent semantic decoding module, where the RGB images provide appearance detail information and the feature images provide viewpoint-independent semantic feature information derived from the 3D scene geometry.

[0165] In this embodiment, based on the color buffer and the feature buffer:

[0166] 1) Convert the values ​​in the color buffer to a standard image format to generate an RGB image of the scene as seen from the entrance of the corridor: I_rgb_corridor.

[0167] 2) Organize the feature buffer directly as a view-independent feature image I_feat_corridor, where each pixel stores a 256-dimensional feature vector.

[0168] The final output is a set of RGB-feature image pairs (I_rgb_corridor, I_feat_corridor) that are fully spatially aligned at the pixel level. The effect of this step is that it "packages" the 3D information and general features of the "office corridor" into a standard two-dimensional image pair, laying the foundation for efficient 2D processing later.

[0169] S4. Input the RGB image and view-independent feature image into a pre-trained VDVI fusion semantic segmentation network, and output the semantic segmentation map under the current view through feature fusion and decoding.

[0170] Using pixel-level strictly aligned RGB images and view-independent feature images as input, a VDVI fusion semantic segmentation network (e.g., using the publicly available AsymFormer model and its weights) pre-trained on a general 2D image dataset is invoked to achieve efficient and accurate semantic segmentation. The pre-trained VDVI fusion semantic segmentation network can be independently pre-trained on large public 2D segmentation datasets (such as Cityscapes and COCO), and its parameters remain fixed throughout the process of this invention, requiring no adjustment for new scenes.

[0171] The VDVI fusion semantic segmentation network adopts a dual-branch encoder architecture, which extracts and fuses two types of complementary features respectively: View-related features: derived from the RGB branch, encoding visual information related to viewpoint, such as texture and lighting; View-independent features: derived from the feature map branch, encoding stable structural semantic information derived from 3D geometry.

[0172] By deeply fusing and decoding these two types of features, the network can maintain detail accuracy while ensuring three-dimensional consistency of the segmentation results across different perspectives, thereby outputting a pixel-level semantic segmentation map from the current perspective.

[0173] This embodiment uses the AsymFormer as an example to illustrate the network architecture, which has powerful feature fusion and semantic understanding capabilities.

[0174] The whole process is as follows Figure 5 As shown, it includes the following steps:

[0175] S41, the RGB image is input into the view-related encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-related features.

[0176] The view-dependent encoder encodes the scene image (Image). The scene RGB image (denoted as Image) is input into the view-dependent encoder. The view-dependent encoder (denoted as Enc...) VD Typically built upon convolutional neural networks (CNNs) or visual Transformers, their function is to extract multi-scale hierarchical features from the input image. The encoding process can be represented as:

[0177] Among them, F VD This represents the view-related multi-scale feature pyramid representing the encoder output. Specifically, F VD Feature maps with different spatial resolutions are output from multiple network layers (at sampling magnifications of 2x, 4x, 8x, and 16x), forming a feature set: F VD ={f vd 1, f vd 12, ..., f vd 1L}, where L is the total number of feature scales. f vd L represents the view-related feature map at the Lth scale. Its spatial size decreases as L increases, but the number of feature channels increases.

[0178] S42, Input the view-independent feature image into the view-independent encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-independent features.

[0179] View-independent encoder F VI Encode the feature image. The view-independent feature image (denoted as Feature) is input into the view-independent encoder. This encoder (denoted as Enc...) VI ) has the same characteristics as Enc VDA similar downsampling structure, but with the number of input channels equal to the dimension C of the feature image (e.g., 256). The encoding process can be represented as:

[0180] Among them, F VI The view-independent multi-scale feature pyramid representing the encoder output, i.e., F VI = {f vi 1,f vi 2, ...,f vi L}, where L is the total number of feature scales. f vi L represents the view-independent feature map at the Lth scale.

[0181] In this embodiment, the encoder Enc related to the I_rgb_corridor input view is... VD The multi-scale feature pyramid F is obtained. VD FVD captures viewpoint-related visual information in an image, such as wall textures, door frame details, and light reflections.

[0182] Inputting I_feat_corridor into the view-independent encoder EncVI yields the multi-scale feature pyramid FVI. The FVI encodes stable structural information derived from 3D geometry, such as spatial layout and object surface orientation.

[0183] S43, on multiple feature scales with the same spatial resolution output by the view-dependent encoder and the view-independent encoder, the view-dependent features and the view-independent features are fused to obtain multi-scale fused features.

[0184] View-dependent features and view-independent features at multiple feature scales are effectively fused at the same spatial scale level to combine the complementary advantages of view-dependent and view-independent features across multiple feature scales. The fusion operation (denoted as Merge) is performed independently at each corresponding scale L. For the L-th scale, the fusion process can be formally represented as:

[0185] Among them, f fused L is the fused feature map at the Lth scale. The specific implementation of the Merge operation can be feature concatenation followed by a convolutional layer for dimensionality reduction, or it can be element-wise addition or weighted fusion through an attention mechanism.

[0186] In a specific example of the AsymFormer architecture, an asymmetric fusion attention module is used. For example, if a concatenation method is used, then f fused L=Conv_{1x1}(Concat(f vd L,fvi L)).

[0187] Among them, Concat(f vd L,f vi L) represents the operation of concatenation along the channel dimension: this operation merges two feature maps f that have identical spatial dimensions (height H and width W). vd L (number of channels is C_vd) and f vi L (with C_vi channels) is connected along its channel direction to generate a new fused feature map with (C_vd+C_vi) channels.

[0188] Conv_{1x1} represents a 1x1 convolutional layer used for cross-channel information fusion and dimensionality reduction of the concatenated high-dimensional features, and adjustment to the target number of channels. By performing this operation on all L scales, a multi-scale fused feature pyramid is obtained: F fused ={f fused 1, f fused 2 ...,f fused L}.

[0189] In this embodiment, under the influence of pre-trained network parameters, the fusion module automatically associates and fuses the RGB texture features describing the "door" with the 3D planar geometric features describing the "door," thereby obtaining a more robust and comprehensive representation. By performing this operation on all scales, a multi-scale fusion feature pyramid F is obtained. fused .

[0190] S44, Decode based on the multi-scale fusion features and output the pixel-level semantic segmentation map under the current view.

[0191] Multi-scale fusion feature pyramid F fused The input is fed into a decoder (Decoder). This decoder typically contains a series of upsampling (deconvolutional or interpolation) layers, skip connections at scales corresponding to the encoder, and a final classification convolutional layer. The processing can be represented as:

[0192] Here, Result is the output of the decoder, i.e., the semantic segmentation prediction map. Specifically, the workflow of the decoder Dec is: f is fused from the deepest layer (smallest spatial resolution). fused Starting with L, through upsampling and with data from F... fusedShallow features of the same scale are fused to gradually restore spatial resolution and refine segmentation details. Finally, after restoring the spatial resolution to the original input image I_rgb, a 1x1 convolutional layer (followed by a Softmax activation function) maps the feature vector of each pixel location to a predefined semantic category probability distribution. Therefore, the output Result is a tensor of dimension (H,W,K), where H and W are the height and width of the image, respectively, and K is the total number of semantic categories. The vector at position (i,j) in Result represents the probability that the pixel belongs to each of the K categories.

[0193] In this embodiment, F fused Input decoder Result = Dec(F fused The decoder progressively restores resolution through upsampling and skip connections, and finally calculates the probability of each pixel belonging to its respective semantic category (such as "wall", "floor", "door", "fire hydrant") through a classification layer. The output Result is a semantic segmentation map with the same resolution as I_rgb_corridor.

[0194] Example 2:

[0195] This embodiment provides a generalizable real-time rendering 3D Gaussian splash semantic segmentation system, whose architecture follows the design principle of decoupling "offline training" and "online inference".

[0196] A generalizable, real-time rendering 3D Gaussian splash semantic segmentation system achieves a complete process from learning general knowledge from multi-scene data to performing real-time, high-precision semantic segmentation on novel target scenes through the collaboration of two logically independent but functionally connected subsystems. The system is driven by a VDVI-fused semantic segmentation network based on general 3D features extracted through self-supervised learning, ensuring excellent generalization ability and real-time inference performance even in unseen scenes.

[0197] 1. An offline training subsystem for training the general 3D feature encoder.

[0198] The offline training subsystem is responsible for performing joint training on datasets containing multiple heterogeneous scenarios in a single run, producing a generalizable model. The offline training subsystem typically runs on servers with powerful computing capabilities.

[0199] 1) Multi-scene 3D reconstruction and data management module, used to reconstruct multi-view images of various training scenes into 3D Gaussian splash representation and convert them into point cloud data.

[0200] The multi-scene 3D reconstruction and data management module serves as the data preprocessing hub for offline training, responsible for converting raw multi-view images into a unified 3D representation format, including:

[0201] Scene data pool: Persistently stores the raw data of all scenes used for training. Each scene data package contains: a set of multi-view RGB images, and the precise camera pose corresponding to each image (obtained or synchronously recorded through the SfM algorithm).

[0202] 3D Gaussian Reconstruction Engine: For each scene in the scene data pool, this engine calls the 3D Gaussian splash reconstruction algorithm, taking the multi-view images and camera poses of the scene as input, and performs optimized reconstruction. The output is a high-quality, explicit 3D Gaussian splash representation of the scene, that is, a set G consisting of a large number of 3D Gaussian distributions with geometric and appearance parameters.

[0203] Point Cloud Transformation Unit: To adapt the 3D Gaussian representation to point cloud neural network processing, this unit performs a normalization transformation on the set G obtained from each scene reconstruction. It extracts the ten-dimensional basic attributes (position, basic color, scaling, and opacity) of each Gaussian and assembles them into a regular point cloud data matrix P. The transformed point cloud data from all scenes constitute the training dataset.

[0204] 2) A self-supervised general 3D feature encoder training module, used to train the general 3D feature encoder through self-supervised contrastive learning.

[0205] The self-supervised general 3D feature encoder training module is the cornerstone of this system's generalization capability. Through self-supervised contrastive learning, it enables the encoder to learn to capture essential 3D features, including:

[0206] Point cloud encoding network: PointNeXt is used as the backbone network architecture, and its parameter θ is the target to be optimized.

[0207] Contrastive learning trainer: drives the learning process of the encoding network. For point cloud data P of each training scene, the contrastive learning trainer randomly samples two virtual camera views; selects the same set of Gaussian points visible in both views to construct positive sample pairs, and other points in the same batch constitute negative samples; extracts features through the encoding network, calculates the InfoNCE contrastive loss, and optimizes the parameters θ through backpropagation with the goal of bringing the features of positive sample pairs closer together and pushing the features of negative samples apart.

[0208] After training on a cross-scene dataset, a fixed-parameter, general-purpose 3D feature encoder E_θ' is obtained. This general-purpose 3D feature encoder has the ability to output stable, view-independent depth feature vectors for any input 3D Gaussian point cloud.

[0209] 2. Online inference subsystem, used to perform semantic segmentation on the target scene.

[0210] The online inference subsystem is the lightweight deployment and real-time service unit of this invention. It loads the general-purpose 3D feature encoder produced by the offline training subsystem and calls a pre-trained VDVI fusion semantic segmentation network, which is directly applied to a novel target scene to achieve real-time semantic segmentation. The online inference subsystem can be deployed on edge computing devices or servers.

[0211] 1) Feature extraction and binding module, used to extract view-independent feature vectors for the 3D Gaussian splash representation of the new scene and bind them to generate a feature-enhanced 3D Gaussian scene representation.

[0212] The feature extraction and binding module is the core of the online inference process's initialization and preprocessing. It is responsible for calculating and solidifying a general view-independent feature representation for the new target scene. It processes a pre-generated 3D Gaussian splash representation G_target for the scene to be segmented (e.g., an "office corridor"). This 3D Gaussian splash representation fully characterizes the scene's geometry and appearance, serving as the foundation for all subsequent processing.

[0213] The core processing of the feature extraction and binding module consists of two stages:

[0214] Point cloud transformation and feature encoding pipeline: This pipeline first transforms G_target into point cloud data P_target. Then, it loads the offline-trained, fixed general-purpose 3D feature encoder E_θ' and performs a forward propagation on P_target.

[0215] Feature Binding and Storage Unit: This unit precisely and permanently binds the set of feature vectors output from the encoding pipeline to each distribution in the original 3D Gaussian set G_target. This generates a 3D scene representation with enhanced general feature attributes. This "feature-enhanced" scene representation is then fully stored in memory or a cache, forming a semantically rich scene feature database that can be efficiently accessed by the rendering module.

[0216] The output of the feature extraction and binding module is not the final segmentation result, but rather a shared feature foundation serving the entire online inference subsystem. Its core value lies in the fact that, for any new target scene, this feature extraction and binding operation only needs to be performed once during the initialization phase to provide consistent, high-quality 3D feature support for any number of subsequent rendering requests from any perspective, thereby ensuring the overall real-time inference efficiency of the system from a system design perspective.

[0217] 2) Real-time multi-attribute rendering module, used to render the feature-enhanced 3D Gaussian scene representation based on the camera pose, generating aligned RGB images and view-independent feature images.

[0218] The real-time multi-attribute rendering module extends the standard 3D Gaussian splash renderer to support parallel rasterization of feature attributes, including:

[0219] Rendering request receiving unit: Receives real-time camera pose requests specified by the user or upper-layer application.

[0220] Differentiable Feature Rendering Engine: This engine extracts an enhanced feature representation of the current target scene from the new scene 3D feature extraction module of the online inference subsystem. For the input camera pose, the engine performs 3D-to-2D projection and depth sorting, and then performs alpha blending sputtering in parallel for each pixel—one path generates a color buffer by mixing Gaussian RGB color values, and the other path generates a feature buffer by mixing Gaussian-bound feature vectors under the exact same pixel coverage and blending weights.

[0221] Image generation unit: Converts the color buffer to a standard RGB image I_rgb; simultaneously, it organizes the feature buffer into a view-independent feature image I_feat, where each pixel is a C-dimensional vector. Outputs pixel-level strictly aligned image pairs (I_rgb, I_feat).

[0222] 3) Semantic decoding output module, used to call the pre-trained VDVI fusion network to process the image pairs and output semantic segmentation maps.

[0223] The semantic decoding and output module performs efficient 2D network inference, including:

[0224] Model loading unit: Loads a pre-trained VDVI fusion semantic segmentation network.

[0225] Forward inference pipeline: Receives real-time image pairs (I_rgb, I_feat) from the real-time multi-attribute rendering module. Inputs them into the VDVI fusion semantic segmentation network: the Enc_VD branch processes I_rgb, the Enc_VI branch processes I_feat, the fusion module performs multi-scale feature fusion, and the decoder Dec upsamples and classifies the data.

[0226] Result Generation and Delivery Unit: The decoder outputs a pixel-level semantic segmentation probability map with the same resolution as I_rgb. After ArgMax operation, the final semantic segmentation map is generated. The semantic segmentation map can be displayed in real time through a graphical interface, or output as structured data (such as image masks, category lists) to upstream applications through an API interface to complete a real-time segmentation task.

[0227] Example 3:

[0228] This embodiment proposes an electronic device, including: Memory, used to store computer programs; The processor is used to execute the program stored in the memory to implement the steps of the above embodiment of a generalizable real-time rendering 3D Gaussian splash semantic segmentation method.

[0229] For details on the specific implementation of each step and related explanations, please refer to the aforementioned embodiment of a generalizable real-time rendering 3D Gaussian splash semantic segmentation method, which will not be repeated here.

[0230] The memory of the electronic device mentioned in this embodiment may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0231] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0232] Example 4:

[0233] This invention also proposes a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the steps of the above-described embodiment of a generalizable real-time rendering 3D Gaussian splash semantic segmentation method. For specific implementation details and explanations of each step, please refer to the aforementioned embodiment of a generalizable real-time rendering 3D Gaussian splash semantic segmentation method, which will not be repeated here.

[0234] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A generalizable real-time rendering method for 3D Gaussian splash semantic segmentation, characterized in that, include: By using multi-scene data and self-supervised contrastive learning, a general 3D feature encoder capable of extracting view-independent features is trained, resulting in the trained general 3D feature encoder. For the target scene to be segmented, the trained general 3D feature encoder is used to extract view-independent feature vectors from the 3D Gaussian splatter representation of the target scene, and the feature vectors are bound to the corresponding 3D Gaussian to obtain a feature-enhanced 3D Gaussian scene representation. Based on the specified camera pose, the feature-enhanced 3D Gaussian scene representation is rendered synchronously to generate pixel-aligned RGB images and view-independent feature images. The RGB image and view-independent feature image are input into a pre-trained VDVI fusion semantic segmentation network. Through feature fusion and decoding, a semantic segmentation map of the current viewpoint is output.

2. The method according to claim 1, characterized in that, The method of using multi-scene data to train a general 3D feature encoder capable of extracting view-independent features through self-supervised contrastive learning includes: The 3D Gaussian splash representation of each training scene is converted into point cloud data composed of the geometric and basic appearance attributes of each of its 3D Gaussians. Based on the point cloud data, the initial depth feature vector of each 3D Gaussian is extracted by the point cloud encoder; Based on the initial depth feature vector, and with the goal of ensuring consistent features across multiple viewpoints using the same 3D Gaussian vector, the general 3D feature encoder is obtained.

3. The method according to claim 1, characterized in that, For the target scene to be segmented, the trained general 3D feature encoder is used to extract view-independent feature vectors from the 3D Gaussian splatter representation of the target scene, and the feature vectors are bound to the corresponding 3D Gaussian symbols to obtain a feature-enhanced 3D Gaussian scene representation, including: Convert the 3D Gaussian splash representation of the target scene into point cloud data; The point cloud data is input into the trained general 3D feature encoder to generate a high-dimensional feature vector corresponding to each 3D Gaussian. The high-dimensional feature vector is bound to the corresponding 3D Gaussian vector to form the feature-enhanced 3D Gaussian scene representation.

4. The method according to claim 1, characterized in that, The step of synchronously rendering the feature-enhanced 3D Gaussian scene representation according to the specified camera pose to generate pixel-aligned RGB images and view-independent feature images includes: Based on the feature-enhanced 3D Gaussian scene representation and the camera pose, the 3D Gaussian is projected onto the image plane and sorted by depth. Based on the depth sorting, for each pixel location covered by the 3D Gaussian, color blending and feature vector blending based on its opacity are performed in parallel to generate color buffers and feature buffers respectively; The color buffer and feature buffer are respectively converted into the pixel-aligned RGB image and view-independent feature image.

5. The method according to claim 1, characterized in that, The RGB image and view-independent feature image are input into a pre-trained VDVI fusion semantic segmentation network. Through feature fusion and decoding, a semantic segmentation map of the current viewpoint is output, including: The RGB image is input into the view-related encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-related features; The view-independent feature image is input into the view-independent encoder of the VDVI fusion semantic segmentation network to extract multi-scale view-independent features; At multiple feature scales with the same spatial resolution output by the view-dependent encoder and the view-independent encoder, the view-dependent features and view-independent features are fused to obtain multi-scale fused features; Decoding is performed based on the multi-scale fusion features to output a pixel-level semantic segmentation map from the current perspective.

6. A generalizable real-time rendering 3D Gaussian splash semantic segmentation system, used to implement the method of any one of claims 1 to 5, characterized in that, include: An offline training subsystem is used to train the general 3D feature encoder; An online inference subsystem is used to perform semantic segmentation on the target scene.

7. The system according to claim 6, characterized in that, The offline training subsystem includes: The multi-scene 3D reconstruction and data management module is used to reconstruct multi-view images of various training scenes into 3D Gaussian splash representations and convert them into point cloud data. A self-supervised feature encoder training module is used to train the general 3D feature encoder through self-supervised contrastive learning.

8. The system according to claim 6, characterized in that, The online inference subsystem includes: The feature extraction and binding module is used to extract and bind view-independent feature vectors for the 3D Gaussian splash representation of the new scene, generating a feature-enhanced 3D Gaussian scene representation; The real-time multi-attribute rendering module is used to render the feature-enhanced 3D Gaussian scene representation based on the camera pose, generating aligned RGB images and view-independent feature images. The semantic decoding output module is used to call the pre-trained VDVI fusion network to process the aligned RGB image and view-independent feature image and output the segmentation result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to implement the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images