Three-dimensional Gaussian semantic segmentation and scene understanding method and system based on tree-guided contrast learning

Abstract

The invention provides a three-dimensional Gaussian semantic segmentation and scene understanding method based on tree-guided contrast learning, which comprises the following steps of: firstly, inputting a multi-view image and a camera pose, and constructing a multi-level object tree to model a hierarchical semantic relationship between an object and a part; performing three-dimensional Gaussian reconstruction, parameterizing Gaussian points, associating instance feature vectors, and generating a feature map through micro rendering; performing two-stage comparative learning: performing global feature optimization based on the object tree, and performing local fine-grained feature optimization for the component; a consistency segmentation detection mechanism is introduced, comparison loss is dynamically adjusted according to the multi-view segmentation number, and over-segmentation or under-segmentation is avoided; and finally, calculating Gaussian point connection strength based on the graph structure to remove noise points, and outputting a high-quality three-dimensional semantic segmentation result. The invention also provides a three-dimensional Gaussian semantic segmentation and scene understanding system based on tree-guided contrast learning. Therefore, the method effectively improves the segmentation precision and robustness in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and 3D scene understanding technology, and in particular to a method and system for 3D Gaussian semantic segmentation and scene understanding based on tree-guided contrastive learning. Background Technology

[0002] 3D scene understanding is widely used in virtual reality (VR), augmented reality (AR), autonomous driving, robot interaction, and digital twins. These applications require efficient and accurate 3D data representation to enable real-time understanding and interaction with scenes from different perspectives. However, traditional 3D representation methods, such as voxels, meshes, and point clouds, while achieving good results in certain scenarios, typically face challenges such as low rendering efficiency, high computational complexity, and a lack of end-to-end optimization. For example, voxel representation has very high computational requirements at high resolutions, limiting its real-time performance when handling large-scale complex scenes; mesh representation, while capable of representing scene structure well, suffers from low processing efficiency in dynamic scenes; and point cloud representation, although flexible and able to directly capture data points in 3D space, often lacks efficient rendering and computation methods when processing large-scale point cloud data.

[0003] Neural Radiance Field (NeRF), as an emerging method for representing 3D scenes, successfully overcomes some limitations of traditional methods through implicit rendering. NeRF uses neural networks to represent lighting and viewpoint changes in a scene, generating very high-quality images. However, NeRF also faces the problem of enormous computational overhead. Its rendering process requires significant computational resources and is relatively slow, making it difficult to meet the requirements of real-time applications, especially in scenarios requiring rapid response (such as autonomous driving or robot interaction).

[0004] In recent years, 3D Gaussian Splatting (3DGS), as a novel explicit 3D representation method, has gradually become an efficient solution suitable for real-time rendering. Unlike traditional methods, 3DGS represents the scene using explicit Gaussian points, where each point has explicit geometric parameters and color information in 3D space, providing high computational efficiency during rendering. 3DGS not only supports real-time rendering but can also be deeply integrated with downstream tasks (such as semantic segmentation and object detection) to form an end-to-end optimization framework. Compared to NeRF, 3DGS has lower computational overhead and faster rendering speed, thus possessing significant application potential in scenarios requiring high efficiency and real-time performance.

[0005] With the continuous development of 3DGS technology, more and more applications are adopting this method for 3D data processing, especially in fields such as autonomous driving, robot vision, and virtual reality. 3DGS provides strong technical support for real-time and efficient 3D scene understanding. In the future, with the improvement of hardware performance and the continuous improvement of optimization algorithms, 3DGS is expected to become an important tool in the field of 3D scene understanding. Existing semantic methods based on 3DGS attempt to introduce contrastive learning and linguistic features, but they have the following problems: flattened semantic assumption: ignoring the hierarchical relationship between objects and parts; contrast pair saturation: generating too many redundant contrast samples in complex scenes, leading to feature space saturation; label noise problem: supervision based on 2D segmentation prior introduces noise, especially in occluded or complex scenes.

[0006] Existing technologies generally suffer from several significant problems when handling 3D scene understanding and semantic segmentation tasks:

[0007] First, most current methods fail to effectively model the hierarchical structure between objects and parts, resulting in a lack of effective hierarchical reasoning capabilities in complex scenes. Traditional semantic segmentation methods typically assume that objects in a scene are independent, ignoring the fine-grained component structures within objects. This makes the models perform poorly when dealing with complex objects containing multiple parts; especially when the object's shape is complex and there is severe occlusion, it is impossible to obtain effective segmentation results through simple global feature extraction. Therefore, existing technologies lacking hierarchical reasoning capabilities cannot meet the segmentation needs of multiple objects and multiple parts in complex scenes.

[0008] Secondly, existing feature learning methods often exhibit instability when dealing with complex scenes, especially in multi-view or multi-scale scenarios, where the feature space is prone to saturation, leading to convergence issues or decreased accuracy during training. This instability stems from the over-reliance of traditional contrastive learning methods on redundant contrastive samples, resulting in poor generalization ability of the feature space and difficulty in maintaining robustness across diverse scenarios. For low-quality data, especially noisy images, existing methods demonstrate poor robustness, struggling to extract stable and highly discriminative features, thus impacting the overall performance of the model.

[0009] Furthermore, noise points are a significant challenge in existing technologies. In processing 3D point cloud data, the presence of noise points severely impacts segmentation quality. Traditional point cloud processing methods typically assume that every point in the data is valid. However, in real-world scenarios, point cloud data is often affected by various factors, such as sensor errors and occlusion effects. These noise points not only increase computational complexity but also lead to inaccurate and unstable segmentation results. Existing methods have failed to effectively suppress the influence of these noise points, resulting in insufficient accuracy in the final semantic segmentation results.

[0010] Therefore, existing technologies have significant shortcomings in hierarchical structure modeling, feature learning stability, and noise point processing, and cannot meet the requirements of high precision and high robustness in complex scenarios. Summary of the Invention

[0011] To address the issues of missing hierarchical relationships, redundant contrast samples, and noise interference in existing 3DGS semantic segmentation methods, this invention proposes a 3D Gaussian semantic segmentation and scene understanding method and system based on tree-guided contrastive learning, which can achieve more efficient and robust 3D scene semantic segmentation and understanding.

[0012] To achieve the aforementioned technical effects, this invention provides a method for 3D Gaussian semantic segmentation and scene understanding based on tree-guided contrastive learning, comprising the following steps:

[0013] Input multi-view image data, including image sets and corresponding camera pose information;

[0014] A tree-like semantic structure is constructed to generate multi-scale object masks from multi-view images, and a multi-level object tree is constructed using a depth-first search algorithm; wherein each tree node of the multi-level object tree represents a semantic mask of an object or part of an object.

[0015] Perform 3D Gaussian reconstruction, parameterize the position, scale, rotation, opacity, and color information of each Gaussian point, and configure a low-dimensional learnable instance feature vector for each Gaussian point to capture the structural information of its corresponding region; render the instance feature vectors of all Gaussian points onto a 2D planar image to generate a feature map.

[0016] Based on the feature map and the large-scale instance mask of the multi-level object tree, the feature prototype is calculated, and feature learning is performed through global contrastive loss to optimize global semantic features;

[0017] For each coarse-grained node in the multi-level object tree, its corresponding Gaussian points are rendered as independent sub-scenes, and local feature maps are calculated. Based on the local feature maps, feature learning is performed through local contrast loss to optimize fine-grained features.

[0018] A consistency segmentation detection mechanism is used to dynamically adjust the contrast loss based on the number of segments from multiple perspectives to avoid oversegmentation or undersegmentation.

[0019] An adjacency graph of the Gaussian points is established based on their spatial location and the instance feature vectors. The connection strength of each Gaussian point in the adjacency graph is calculated. Noise points are identified and removed based on the connection strength.

[0020] Output the final 3D semantic segmentation result.

[0021] Furthermore, in the step of constructing the tree-like semantic structure, the multi-scale object mask is generated from the multi-view image using the SAM (Segment Anything Model) method.

[0022] Furthermore, the mask corresponding to each child node of the multi-level object tree is contained in the mask corresponding to its parent node.

[0023] Furthermore, the feature map is represented as follows:

[0024] ;

[0025] in, This is represented as the instance feature vector. The visibility weight is calculated by projecting the Gaussian point. The projection rate is calculated recursively, and i represents the light transmission coefficient of each Gaussian point.

[0026] Furthermore, the feature prototype for each region is calculated based on the following formula:

[0027] ;

[0028] The local feature map is obtained based on binary region mapping and fine-grained masking, and its calculation formula is as follows:

[0029] ;

[0030] in, This is the mask for the large-scale instance. Mapping for the binarized region, For the fine-grained mask, This refers to the feature map.

[0031] Furthermore, both the global contrast loss and the local contrast loss include a zoom-in loss and a zoom-out loss; wherein, the zoom-in loss is used to minimize the feature distance within the same object, and the zoom-out loss is used to maximize the feature distance between different objects.

[0032] Furthermore, the dynamic adjustment of the contrast loss based on the number of segments from multiple perspectives through the consistency segmentation detection mechanism specifically includes:

[0033] Calculate the number of segments for each viewpoint;

[0034] When the number of segments is higher than the first threshold, feature learning is performed only by applying the close-in loss in the contrastive loss.

[0035] When the number of segments is lower than the second threshold, feature learning is performed only by applying the push-away loss in the contrastive loss.

[0036] When the number of segments is between the first threshold and the second threshold, feature learning is performed simultaneously using both the approach loss and the push-away loss.

[0037] Furthermore, the step of establishing the adjacency graph of the Gaussian points based on their spatial location and the instance feature vectors specifically involves:

[0038] Calculate the adjacency matrix between Gaussian point i and Gaussian point j using the following formula. :

[0039] ;

[0040] in, and Let i and j be the spatial locations corresponding to Gaussian points i and j, respectively. and These are the instance feature vectors corresponding to Gaussian points i and j, respectively. and These are the standard deviations calculated based on the pairwise distance distributions in the location space and feature space, respectively.

[0041] Furthermore, the connection strength of the Gaussian point Calculated based on the following formula:

[0042] ;

[0043] The step of identifying and removing noise points based on the connection strength specifically includes:

[0044] The connection strength Target Gaussian points below the preset connection threshold are identified as noise points.

[0045] On the other hand, the present invention also provides a 3D Gaussian semantic segmentation and scene understanding system based on tree-guided contrastive learning, for implementing the method described above, including:

[0046] The data preprocessing module is used for image acquisition and mask generation;

[0047] A tree-like semantic structure building module is used to construct multi-level object trees from multi-view images;

[0048] The feature rendering and contrast learning module is used for 3D Gaussian reconstruction, feature rendering, and global and local contrast learning.

[0049] The consistency detection module is used to implement the consistency segmentation detection mechanism and dynamically adjust the comparison loss.

[0050] The graph noise reduction module is used to remove noise points based on graph structures.

[0051] The 3D semantic segmentation and understanding output module is used to output the final 3D semantic segmentation result.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] Supports hierarchical semantic reasoning: By constructing a multi-level object tree, it effectively models the hierarchical relationship between objects and components, significantly improving the semantic segmentation quality in complex scenes.

[0054] Improved training stability: The two-stage comparative learning strategy significantly alleviates the feature space saturation problem and enhances the stability and convergence efficiency of model training.

[0055] Enhanced adaptability to open-vocabulary scenarios: Under open-vocabulary conditions, more accurate 3D object retrieval and segmentation can be achieved, expanding the practical scope of the method.

[0056] Superior overall performance: Experiments on multiple public datasets show that the present invention outperforms existing mainstream methods in key metrics such as mean intersection-union ratio (mIoU) and mean accuracy (mAcc). Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the steps of the tree-guided contrastive learning-based 3D Gaussian semantic segmentation and scene understanding method provided in an embodiment of the present invention.

[0058] Figure 2 This is an overall framework diagram of the 3D Gaussian semantic segmentation and scene understanding method based on tree-guided contrastive learning described in this invention;

[0059] Figure 3 This is a schematic diagram of object tree construction for the 3D Gaussian semantic segmentation and scene understanding method based on tree-guided contrastive learning described in this invention;

[0060] Figure 4 This is a schematic diagram of the consistency segmentation detection mechanism of the tree-guided contrastive learning-based 3D Gaussian semantic segmentation and scene understanding method described in this invention.

[0061] Figure 5 The structural block diagram of the three-dimensional Gaussian semantic segmentation and scene understanding system based on tree-guided contrastive learning provided in another embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0064] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0065] To address the common problems in existing 3D semantic segmentation methods, such as missing hierarchical structures, redundant contrastive samples, and noise interference, this invention proposes a comprehensive technical solution. First, to address the insufficient modeling of hierarchical relationships between objects and components, a tree-like multi-layered semantic structure is introduced to effectively organize the semantic relationships between objects and their components, thereby enhancing semantic expression capabilities in complex scenes and enabling efficient feature learning and differentiation at both the object and component levels. Second, to solve the overfitting and feature saturation problems caused by redundant samples in contrastive learning, a phased contrastive learning strategy of "global first, local later" is proposed, sequentially performing coarse-grained global feature learning and fine-grained local feature optimization to improve model stability and generalization ability. Finally, to suppress noise interference in Gaussian point clouds, a graph-based denoising mechanism is combined. An adjacency graph is constructed based on the spatial location and feature similarity between Gaussian points to identify and remove semantically unstable noise points, further improving the robustness and accuracy of semantic segmentation. Thus, this invention significantly improves the overall performance of 3D scene semantic segmentation and understanding through the organic combination of tree structures, phased contrastive learning, and graph denoising mechanisms.

[0066] To enable those skilled in the art to reproduce and implement the present invention, the various steps of the tree-guided contrastive learning-based 3D Gaussian semantic segmentation and scene understanding method will be described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are specific illustrations of the present invention, and their descriptions are helpful in understanding the present invention, but should not be construed as limiting the present invention.

[0067] Figures 1-2 This invention illustrates a method for 3D Gaussian semantic segmentation and scene understanding based on tree-guided contrastive learning, comprising the following steps:

[0068] S101: Input multi-view image data, including image set and corresponding camera pose information.

[0069] Specifically, the input data in this embodiment includes a set of images from multiple viewpoints. , where V represents the number of viewpoints; each viewpoint contains corresponding camera pose information, and the parameters containing the corresponding camera pose information include position, scale, rotation, opacity, and color information (represented by spherical harmonic coefficients). Each Gaussian point also has a low-dimensional learnable instance feature vector. This feature vector captures the structural information of the region corresponding to the point cloud.

[0070] S102: Construct a tree-like semantic structure to generate multi-scale object masks from multi-view images, and construct a multi-level object tree through a depth-first search algorithm; wherein, each tree node of the multi-level object tree represents the semantic mask of an object or part of an object, and the mask corresponding to each child node is contained in the mask corresponding to its parent node.

[0071] In practical implementation, to model the hierarchical semantic relationships between objects and parts, a preliminary multi-scale object mask is first generated from multi-view images using the SAM method; then, a depth-first search (DFS) algorithm is used to construct a multi-level object tree; each tree node represents the semantic mask of an object or part of an object. Figure 3 As shown, to ensure accurate spatial relationships between masks of different scales, a multi-view 2D instance tree is ultimately obtained after steps such as mask filling, edge detection, and mask cutting. Through this tree structure, the mask of a child node is completely contained within the mask of its parent node, thus maintaining the hierarchical relationship between the object and its components.

[0072] S103: Perform 3D Gaussian reconstruction, parameterize the position, scale, rotation, opacity, and color information of each Gaussian point, and configure a low-dimensional learnable instance feature vector for each Gaussian point to capture the structural information of its corresponding region; render the instance feature vectors of all Gaussian points to a 2D planar image to generate a feature map.

[0073] In the 3D Gaussian reconstruction process of this embodiment, each Gaussian point Instance feature vectors are parameterized with information such as position, scale, rotation, and color, and associated with their spatial location and semantic features. It is learned and rendered onto a 2D image. Specific parameters for each Gaussian point include: position. ,scale Rotation (Quaternion representation), Opacity and color information (Represented by spherical harmonic coefficients). Each Gaussian point also has a low-dimensional learnable instance feature vector. This feature vector is used to capture the structural information of the region corresponding to the point cloud. Using a differentiable renderer, the feature information of all Gaussian points is rendered into a 2D image.

[0074] Furthermore, the rendering process can explicitly represent 3D data and project it onto a 2D image plane to obtain image representations for each viewpoint. This rendering process generates image representations with higher semantic and structural expressive power, and the feature map F is calculated by combining the features of all Gaussian points using the following formula:

[0075] ;

[0076] in, This is represented as the instance feature vector. The visibility weight is calculated by projecting the Gaussian point. The projection rate is calculated recursively, and i represents the light transmission coefficient of each Gaussian point.

[0077] S104: Calculate feature prototypes based on large-scale instance masks of feature maps and multi-level object trees, and learn features through global contrastive loss to optimize global semantic features.

[0078] Step S104 is the global contrastive learning stage, specifically through contrastive loss. To optimize the learning of global semantic features.

[0079] Large-scale instance mask obtained from the object tree Calculate the feature prototype of each region Feature learning is performed using a global contrastive loss. Specifically:

[0080] ;

[0081] Global contrastive loss comprises pull loss and push loss. Pull loss minimizes the similarity of features within the same object, while push loss maximizes the difference in features between different objects. The formula is as follows:

[0082] ;

[0083] ;

[0084] ;

[0085] In the formula For large-scale instance masks.

[0086] The "close loss" minimizes the feature distance within the same object, while the "distance loss" improves semantic discrimination by maximizing the feature distance between different objects.

[0087] By minimizing this loss This allows for optimization of the feature space and improvement of the model's semantic discriminative ability.

[0088] S105: For each coarse-grained node in the multi-level object tree, render its corresponding Gaussian points as independent sub-scenes and calculate local feature maps; based on the local feature maps, perform feature learning through local contrast loss to optimize fine-grained features.

[0089] Step S105 is the local contrastive learning stage, the purpose of which is to optimize features at a fine-grained level to avoid feature space saturation caused by feature competition between small objects.

[0090] During the local learning phase, for each coarse-grained node, its Gaussian points are rendered as independent sub-scenes, and the corresponding local feature map is calculated. The features of the local region are then processed using a binarized region map B and a new fine-grained mask. The calculation yielded:

[0091] ;

[0092] Within local regions, close-in loss and push-out loss are applied to optimize fine-grained features. The specific formula for local contrast loss is as follows:

[0093] ;

[0094] The goal of this loss function is to minimize the feature differences within the same part of an object, making the features more compact.

[0095] ;

[0096] The zoom loss enhances the model's ability to distinguish object parts by maximizing the feature differences between different object components. The total local contrast loss is:

[0097] ;

[0098] By optimizing this loss It can gradually refine the representation of each object component during the learning process of local features, thereby improving the model's fine-grained discrimination ability.

[0099] S106: The consistent segmentation detection mechanism dynamically adjusts the contrast loss based on the number of segments across multiple views to avoid oversegmentation or undersegmentation. The consistent segmentation detection (CSD) mechanism is introduced to address the noise in the supervisory signal caused by inconsistencies in the object tree across multiple views; this mechanism adjusts the applied loss function by calculating the intersection of the fine-grained mask and the binarized region for each view.

[0100] In one specific implementation, step S106 specifically includes: calculating the number of segments for each viewpoint; when the number of segments is higher than a first threshold, only the closing loss in the contrast loss is applied for feature learning; when the number of segments is lower than a second threshold, only the pushing loss in the contrast loss is applied for feature learning; when the number of segments is between the first and second thresholds, both the closing loss and the pushing loss are applied for feature learning. That is, for each viewpoint, its segment number is calculated, and a smoothing algorithm is used to obtain a more stable segmentation judgment: such as... Figure 4 As shown, when the number of splits is high, it indicates over-splitting. In this case, only the convergence loss should be applied. To merge overly fragmented clusters; when the number of partitions is low, it indicates under-partitioning, in which case only the push-away loss should be applied. To separate over-merged regions; when the number of segments is at an intermediate value, apply the full contrast loss. Simultaneous optimization is performed.

[0101] S107: An adjacency graph of Gaussian points is constructed based on spatial location and instance feature vectors, and the connection strength of each Gaussian point in the adjacency graph is calculated; noise points are then identified and removed based on the connection strength. To further improve segmentation quality, noisy Gaussian points generated during training need to be removed. In this embodiment, a graph structure is used to identify and remove semantically unstable Gaussian points. Specifically, an adjacency graph W of Gaussian points is constructed using spatial location and feature similarity, and the connection strength of each Gaussian point is calculated. The adjacency matrix W is calculated using the following formula. ij :

[0102] ;

[0103] in, and Let i and j be the spatial locations corresponding to Gaussian points i and j, respectively. and These are the instance feature vectors corresponding to Gaussian points i and j, respectively. and These are the standard deviations calculated for the pairwise distance distributions in the location space and feature space, respectively. and Specifically, first calculate the distance (positional distance or feature distance) between all Gaussian point pairs, then group these distance values ​​into a set and calculate the standard deviation of this set; all distance calculations use L2 distance.

[0104] Then, calculate the connection strength at each point. If the connection strength at a certain point If the connection point is below the preset connection threshold, it is considered a noise point and removed, thereby improving the final segmentation quality.

[0105] S108: Output the final 3D semantic segmentation result. After optimization in the aforementioned steps, a high-quality 3D semantic segmentation result is finally obtained. Thus, this embodiment, through the Consistent Segmentation Detection (CSD) mechanism and graph-based Gaussian point denoising, can effectively eliminate noise points and achieve stable semantic segmentation in complex 3D scenes; the final result can be used for tasks such as open-vocabulary 3D object selection and scene understanding.

[0106] Figure 5 This invention illustrates a 3D Gaussian semantic segmentation and scene understanding system 100 based on tree-guided contrastive learning, provided by another embodiment of the present invention. This system implements the methods described in the above embodiments and includes a data preprocessing module 10, a tree-structured semantic construction module 20, a feature rendering and contrastive learning module 30, a consistency detection module 40, a graph denoising module 50, and a 3D semantic segmentation and understanding output module 60, wherein:

[0107] The data preprocessing module 10 is used for image acquisition and mask generation; the tree-like semantic structure construction module 20 is used to construct a multi-level object tree from multi-view images; the feature rendering and contrast learning module 30 is used for 3D Gaussian reconstruction, feature rendering, and global and local contrast learning; the consistency detection module 40 is used to implement a consistency segmentation detection mechanism and dynamically adjust the contrast loss; the graph denoising module 50 is used to remove noise points based on the graph structure; and the 3D semantic segmentation and understanding output module 60 is used to output the final 3D semantic segmentation result.

[0108] The specific functions of each functional module in the system 100 described in this embodiment correspond to the corresponding steps in the method described in the previous embodiment, and are the operations performed by the system as described above.

[0109] In summary, the 3D Gaussian semantic segmentation and scene understanding method and system based on tree-guided contrastive learning provided by this invention first inputs multi-view images and camera poses to construct a multi-level object tree to model the hierarchical semantic relationships between objects and components. Then, 3D Gaussian reconstruction is performed, parameterizing Gaussian points and associating them with instance feature vectors, generating feature maps through differentiable rendering. Next, two-stage contrastive learning is executed: global feature optimization based on the object tree and local fine-grained feature optimization for components. A consistency segmentation detection mechanism is introduced to dynamically adjust the contrast loss based on the number of multi-view segmentations, avoiding over-segmentation or under-segmentation. Finally, the connection strength of Gaussian points is calculated based on the graph structure to remove noise points, outputting high-quality 3D semantic segmentation results. Thus, this invention, through the combination of tree structure, staged contrastive learning, and graph denoising mechanisms, improves the semantic segmentation and understanding capabilities of 3D scenes, solving problems such as missing hierarchical structures, redundant contrast samples, and noise interference in existing technologies.

[0110] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0112] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A three-dimensional Gaussian semantic segmentation and scene understanding method based on tree-guided contrastive learning, characterized in that, The method comprises the steps of: inputting multi-view image data, including an image set and corresponding camera pose information; constructing a tree-shaped semantic structure, generating multi-scale object masks from the multi-view images, and constructing a multi-level object tree through a depth-first search algorithm; wherein each tree node of the multi-level object tree represents a semantic mask of an object or an object part; performing three-dimensional Gaussian reconstruction, parameterizing the position, scale, rotation, opacity and color information of each Gaussian point, and configuring a low-dimensional learnable instance feature vector for each Gaussian point to capture the structure information of the corresponding region; rendering the instance feature vectors of all Gaussian points to a two-dimensional plane image to generate a feature map; calculating feature prototypes based on the feature map and large-scale instance masks of the multi-level object tree, and performing feature learning through a global contrast loss to optimize global semantic features; for each coarse-grained node in the multi-level object tree, rendering the corresponding component Gaussian points as an independent sub-scene and calculating a local feature map; based on the local feature map, performing feature learning through a local contrast loss to optimize fine-grained features; dynamically adjusting the contrast loss according to the number of segmentations under multiple views through a consistency segmentation detection mechanism to avoid over-segmentation or under-segmentation; establishing an adjacency graph of the Gaussian points according to the spatial position and the instance feature vector, calculating the connection strength of each Gaussian point in the adjacency graph, and identifying and removing noise points according to the connection strength; outputting a final three-dimensional semantic segmentation result.

2. The method of claim 1, wherein, In the step of constructing a tree-shaped semantic structure, the multi-scale object masks are generated from the multi-view images through a SAM method.

3. The method of claim 1, wherein, The mask corresponding to each sub-node of the multi-level object tree is contained in the mask corresponding to its parent node.

4. The method of claim 1, wherein, The feature map is represented as: ； wherein, is represented as the instance feature vector, is a visibility weight calculated by the projection of the Gaussian point, is a projection rate calculated recursively, representing the light transmission coefficient of each Gaussian point, and i is the Gaussian point number.

5. The method of claim 1, wherein, The feature prototype of each region is calculated based on the following formula: ； The local feature map is calculated based on the binary region mapping and the fine-grained mask, and its calculation formula is: ； wherein, is the large-scale instance mask, is the binarized region map, is the fine-grained mask, is the feature map.

6. The method of claim 1, wherein, The global contrast loss and the local contrast loss both include a pull-in loss and a push-away loss; wherein the pull-in loss is used to minimize the feature distance within the same object, and the push-away loss is used to maximize the feature distance between different objects.

7. The method of claim 6, wherein, The dynamic adjustment of the contrast loss according to the number of segmentations under multiple views through the consistency segmentation detection mechanism specifically includes: calculating the number of segmentations under each view; when the number of segmentations is higher than a first threshold, only the pull-in loss in the contrast loss is applied for feature learning; when the number of segmentations is lower than a second threshold, only the push-away loss in the contrast loss is applied for feature learning; when the number of segmentations is between the first threshold and the second threshold, the pull-in loss and the push-away loss are simultaneously applied for feature learning.

8. The method of claim 1, wherein, The establishment of the adjacency graph of the Gaussian points according to the spatial position and the instance feature vector specifically includes: The adjacency matrix between the Gauss points i and j is calculated according to the following formula : ； wherein, and are the spatial positions corresponding to the Gaussian point i and the Gaussian point j, respectively, and are the instance feature vectors corresponding to the Gaussian point i and the Gaussian point j, respectively, and are the standard deviations computed from the pairwise distance distributions in the position space and the feature space, respectively.

9. The method of claim 8, wherein, The connection strength of the Gaussian points Based on the following formula: ； The identification and removal of noise points according to the connection strength specifically include: The connection strength A target Gaussian point lower than a preset connection threshold is determined as a noise point.

10. A tree-guided contrastive learning based 3D Gaussian semantic segmentation and scene understanding system, configured to implement the method of any one of claims 1-9. It comprises: a data preprocessing module for image acquisition and mask generation; a tree-shaped semantic structure construction module for constructing a multi-level object tree from multi-view images; The feature rendering and contrast learning module is used for three-dimensional Gaussian reconstruction, feature rendering, and global and local contrast learning. The consistency detection module is used for implementing a consistency segmentation detection mechanism and dynamically adjusting a contrast loss. The graph denoising module is used for removing noise points based on a graph structure. The three-dimensional semantic segmentation and understanding output module is used for outputting a final three-dimensional semantic segmentation result.

Images