A method for constructing a 3D Gaussian model based on semantic segmentation

Abstract

This application discloses a method for constructing a 3D Gaussian model based on semantic segmentation, relating to the fields of computer graphics and vision. The method includes: acquiring a 2D image dataset and preprocessing it; training a fully convolutional network (FCN) based on the preprocessed dataset; using the trained FCN to perform semantic segmentation on the acquired 2D images to obtain 2D semantic labels; and constructing a 3D Gaussian model with semantic segmentation labels based on the 2D semantic labels. Addressing the issue of low rendering quality caused by the lack of semantic understanding in large-scale scenes in 3D Gaussian sputtering, this application utilizes the City Gaussian algorithm to solve the memory overflow and rendering efficiency problems of traditional 3D Gaussian sputtering in large-scale infrastructure modeling. Its multi-level detail rendering strategy ensures high-quality rendering effects and accurate semantic expression at different viewing distances, avoiding the problem of inconsistent quality between near and far views in traditional rendering methods.

Description

Technical Field

[0001] This application relates to the field of image data processing, and in particular to a method for constructing a three-dimensional Gaussian model based on semantic segmentation. Background Technology

[0002] With the increasing complexity of modern engineering construction and infrastructure development, the need for accurate processing and analysis of infrastructure models is becoming more and more urgent. In the field of large-scale infrastructure scenario modeling, such as bridges and roads, traditional model processing methods face many challenges. Among them, accurately identifying and separating different components is a key issue, such as bridge piers and road surfaces, which require separate analysis and operation in subsequent engineering design, maintenance and management.

[0003] In recent years, techniques such as Neural Radiation Field (NeRF) and 3D Gaussian Splatting have demonstrated great potential in 3D model processing. 3D Gaussian Splatting, in particular, utilizes Gaussian distributions to represent objects in 3D space. Each Gaussian distribution can be seen as a description of local information about the object, and multiple Gaussian distributions combined can accurately represent complex 3D shapes and structures. This method effectively captures the geometric and appearance information of objects and achieves high-quality rendering results.

[0004] However, existing 3D Gaussian sputtering techniques face two main problems when handling large-scale infrastructure scenes: first, they lack semantic understanding capabilities, failing to automatically identify and segment different components within the scene; second, they are prone to memory overflow and slow rendering speeds in large-scale scene rendering. Especially in large-scale infrastructure scenes, which may require more than 20 million Gaussian units, this not only leads to memory overflow errors during training but also significantly increases rendering time.

[0005] While traditional semantic segmentation methods can segment different components in 2D images, they are difficult to apply directly to 3D Gaussian models. Furthermore, most existing 3D semantic segmentation methods are based on point clouds or voxel representations, failing to fully leverage the advantages of 3D Gaussian sputtering in terms of rendering quality and efficiency. Therefore, effectively combining semantic segmentation technology with 3D Gaussian sputtering to achieve accurate component identification and high-quality rendering in large-scale infrastructure scenarios has become a pressing technical challenge. Summary of the Invention

[0006] To address the issue of low rendering quality caused by the lack of semantic understanding in large-scale scenes of 3D Gaussian sputtering, this application provides a method for constructing 3D Gaussian models based on semantic segmentation. By adding semantic attribute fields to 3D Gaussian primitives and employing the City Gaussian algorithm and a multi-level detail rendering strategy, this method efficiently processes large-scale infrastructure scenes and improves the rendering quality of large-scale infrastructure.

[0007] This application provides a method for constructing a 3D Gaussian model based on semantic segmentation, including: acquiring a 2D image dataset and preprocessing it; training a fully convolutional network (FCN) based on the preprocessed dataset; using the trained FCN to perform semantic segmentation on the acquired 2D images to obtain 2D semantic labels; and constructing a 3D Gaussian model with semantic segmentation labels based on the 2D semantic labels.

[0008] Furthermore, a two-dimensional image dataset is acquired and preprocessed, including: acquiring two-dimensional images of infrastructure and performing semantic segmentation annotation on the acquired two-dimensional images to obtain semantically segmented annotated images; converting the semantically segmented annotated images into an array format, where the background category number is 0, and the component category numbers start from 1 and increment, containing a total of 69 categories. Data augmentation processing is performed on the acquired original images and semantically segmented annotated images to obtain augmented images; the acquired original images, semantically segmented annotated images, and augmented images are then mixed to obtain the preprocessed dataset.

[0009] Furthermore, data augmentation processing is performed on the acquired original images and semantically segmented labeled images, including: selecting a cropping ratio within a preset cropping ratio range. Calculate the height of the cutting area based on the cutting ratio r. and width Within the preset offset range, select the vertical offset amount. and horizontal offset According to altitude and width and vertical offset and horizontal offset Calculate the vertical boundaries of the clipping regions respectively. and horizontal boundary According to the vertical boundary and horizontal boundary The original image J and the semantic segmentation labeled image A are cropped respectively to obtain local images. and segmentation labeled images ; local image and segmentation labeled images Resampled to the original size to obtain an enhanced image. and segmentation labeled images .

[0010] Furthermore, the vertical boundary and horizontal boundary It is calculated using the following formula:

[0011] ; ;

[0012] in, and The height and width of the original image; and Define the height and width of the cropping area; and This represents the offset in the vertical and horizontal directions.

[0013] Furthermore, the trained fully convolutional network (FCN) is used to perform semantic segmentation on the acquired two-dimensional image to obtain two-dimensional semantic labels. This includes: taking the acquired two-dimensional image as input, the convolutional and pooling layers of the FCN extract image features, and the output layer generates a predicted class probability distribution for each pixel; assigning the class label with the highest probability to each pixel in the image according to the predicted class probability distribution; identifying and determining the boundaries of each class region according to the distribution of class labels, and generating a two-dimensional semantic label image with the same size as the input image, wherein the value of each pixel position corresponds to the class label at that position.

[0014] Furthermore, based on the two-dimensional semantic labels, a three-dimensional Gaussian model with semantic segmentation labels is constructed, including: performing viewpoint annotation based on the acquired original image and the semantically segmented labeled image to obtain an annotated viewpoint image; mixing the annotated viewpoint image and the two-dimensional semantic label image to obtain a training dataset; using the acquired original image, the two-dimensional semantic label image, and the camera pose during image acquisition as input triples based on the training dataset, constructing a three-dimensional Gaussian model using the CityGaussian algorithm; using the structured bundle method (SFM) to perform three-dimensional reconstruction on the acquired original image to generate an initial point cloud for initializing the three-dimensional Gaussian model; and adding... The semantic attribute field stores the category labels of the components, resulting in a semantically expanded 3D Gaussian model. During the rendering phase of the CityGaussian algorithm, different detail levels for different rendering regions are set based on the Nyquist sampling rate. Gaussian sputtering rendering is then performed on the semantically expanded 3D Gaussian model according to the set detail levels, generating a rendered image with semantic information. Based on the differences between the rendered image and the original image and semantically labeled image in the training set, the optimization gradient of the Gaussian primitive parameters is calculated using the differentiable rendering method in the CityGaussian algorithm. Based on the optimization gradient, the Gaussian primitives in the 3D Gaussian model are optimized to obtain a 3D Gaussian model with semantic segmentation labels.

[0015] Furthermore, in the rendering stage of the City Gaussian algorithm, different detail levels are set based on the Nyquist sampling rate. Gaussian sputtering rendering is then performed on the semantically expanded 3D Gaussian model according to the set detail levels, generating a rendered image with semantic information. This includes: determining the scene region to be rendered based on the current viewpoint position and orientation; setting the detail level of the scene region to be rendered based on the Nyquist sampling rate; selecting 3D Gaussian primitives of corresponding precision from the semantically expanded 3D Gaussian model according to the set detail levels; sputtering the selected 3D Gaussian primitives into screen space based on the camera intrinsics and camera pose of the acquired image, and sorting them in depth order; and projecting the sorted 3D Gaussian primitives onto the viewpoint to be rendered through alpha rendering, and calculating the color at position x in the viewpoint plane. ;Calculate the semantics of position x in the view plane from the perspective to be rendered. ; Synthesize rendering scene regions at different levels of detail to obtain the final rendered image with semantic information.

[0016] Furthermore, calculate the color at position x in the view plane. The formula is as follows: ;in, The opacity of the k-th Gaussian; Indicates the color associated with the view; To represent the projection of a Gaussian distribution onto a two-dimensional plane, the projection function is used. The calculation method is as follows: ; Let represent the opacity of the t-th Gaussian; This represents the k-th 3D Gaussian. Indicates camera intrinsic parameters; Indicates the camera pose.

[0017] Furthermore, the semantics of position x in the view plane are calculated. The formula is as follows: Where, softmax represents the normalization function, This is the prediction result of the last layer of the fully convolutional network (FCN) at pixel position x, reflecting the confidence score of each pixel belonging to different semantic categories.

[0018] Furthermore, based on the differences between the rendered image and the original and semantically labeled images in the training set, the optimization gradient of the Gaussian parameters is calculated using the differentiable rendering method in the City Gaussian algorithm. Then, based on the optimization gradient, the Gaussian parameters in the 3D Gaussian model are optimized to obtain a 3D Gaussian model with semantic segmentation labels, including: calculating color... Color loss L1 between the original image and the original image; semantic calculation The semantic loss L2 between the color loss L1 and the semantic loss L2 is calculated; the total loss function L is constructed based on the color loss L1 and the semantic loss L2; ​​the gradient of the total loss function L with respect to the three-dimensional Gaussian meta-parameters, which include three-dimensional coordinates, is calculated. Opacity Geometric structure, spherical harmonic characteristics Colors related to the view Based on the calculated gradient, Gaussian units are cloned or pruned to obtain a three-dimensional Gaussian model with semantic segmentation labels.

[0019] Compared to existing technologies, the advantages of this application are:

[0020] (1) Traditional 3D scene understanding techniques mainly employ point cloud segmentation or voxelization methods, but these methods suffer from significant drawbacks in infrastructure segmentation, such as insufficient accuracy, high computational overhead, and lack of rendering quality. This application innovatively extends 2D semantic segmentation results to 3D Gaussian functions, achieving deep integration of semantic information and geometric representation. This application does not simply segment the scene but constructs a complete 3D model with refined semantic labels, directly integrating semantic information into the Gaussian function. This method ensures that semantic features and geometric features are optimized synchronously by adding semantic attribute fields to each 3D Gaussian primitive, enabling the model to maintain high-quality visual rendering while possessing accurate semantic understanding capabilities.

[0021] (2) The City Gaussian algorithm solves the memory overflow and rendering efficiency problems of traditional 3D Gaussian sputtering in large-scale infrastructure modeling. Its multi-level detail rendering strategy ensures high-quality rendering effects and accurate semantic expression at different viewing distances, avoiding the problem of inconsistent quality between near and far views in traditional rendering methods.

[0022] (3) This application uses an α-rendering mechanism to render semantic labels and color information using the same mechanism, which solves the problem of converting semantic information from 2D to 3D and ensures the consistency of component recognition from different perspectives. This feature is particularly important for the detection and maintenance of infrastructure, enabling accurate identification of each component from any angle. Attached Figure Description

[0023] This application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0024] Figure 1 This is a schematic diagram illustrating an exemplary application scenario of a semantic segmentation-based 3D Gaussian model construction method according to some embodiments of this application;

[0025] Figure 2 This is an exemplary block diagram of a method for constructing a 3D Gaussian model based on semantic segmentation, according to some embodiments of this application;

[0026] Figure 3 These are comparative diagrams showing the effects of this application and existing models according to some embodiments of this application;

[0027] Figure 4 This is a comparison diagram of the present application and existing models, based on some embodiments of the present application. Detailed Implementation

[0028] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0030] Reference Figure 1 The application scenario shown, taking the Luoshe Bridge as an example, illustrates our proposed infrastructure segmentation and component individualization method based on a 3D Gaussian model, which includes the following steps:

[0031] Step 1: Preprocessing and enhancement of 2D image data;

[0032] Step 2: Training the Fully Convolutional Network (FCN) model;

[0033] Step 3: Perform segmentation prediction based on the trained fully convolutional network model;

[0034] Step 4: Construct a 3D Gaussian model with semantic segmentation labels based on 2D semantic labels;

[0035] Step 1 includes the following steps:

[0036] Step 1-1: Convert the semantic information of the dataset segmentation annotation into an array, where the background category is labeled as 0, and the category labels of infrastructure components start from 1 and increase. The Luoshe Bridge includes 8 categories: arch rib, hanger, cross brace, diagonal brace, expansion joint, end block, support, and pier, labeled as 1 to 8 respectively. The labels are: "arch rib": 1, "hanger": 2, "cross brace": 3, "diagonal brace": 4, "expansion joint": 5, "end block": 6, "support": 7, "pier": 8.

[0037] Steps 1-2 involve the high cost and limited quantity of professionally labeled data for infrastructure components (such as the arch ribs and hangers of the Luoshe Bridge). This method significantly expands the training data volume without changing the semantic annotations by magnifying local regions of the limited original images, thus alleviating the dependence of deep learning models on large amounts of labeled data. Specifically, for each image, a random aspect ratio is selected compared to the original... Figure 1 For regions smaller than the original image, the region is enlarged to the same size as the input of the fully convolutional network, thus expanding the original data scale without changing the annotation semantics.

[0038] In steps 1-2, the significant size differences among infrastructure components (e.g., large arch ribs versus small expansion joints) mean that direct training can easily lead to the model neglecting smaller components. This method addresses this by randomly cropping and enlarging the components, giving them more "attention" in the training samples and improving the accuracy of identifying small components such as hangers, supports, and expansion joints. Specifically, the cropping ratio is first selected. ,in, and Calculate the height of the cutting area based on the minimum and maximum cutting ratios, respectively. and width The calculation formula is: ; ;in, and Define the height and width of the original image. Then select the vertical and horizontal offsets, respectively. and They come from uniform distributions. ; ;in, and These are the maximum offsets in the vertical and horizontal directions, respectively. The vertical and horizontal boundaries of the cropping region. and It can be represented as: ; .

[0039] Apply this method to the original image and split annotations Cropping yields a partial image. and split annotations The cropped image and annotations are resampled to their original size. To obtain an enhanced image and its segmentation annotations In the formula, , , and As a hyperparameter, it needs to be specified manually. Among them, and The larger the difference, and A larger value for the hyperparameters results in richer generated samples. However, excessively large values ​​may generate data that deviates too much from the original sample distribution, interfering with model training. Therefore, the hyperparameter values ​​should be within a reasonable range. In the aerial image of the Luoshe Bridge, the bridge structure occupies a moderate proportion of the frame, with a size of 5280*3956 pixels. Therefore, the values ​​of the four hyperparameters are 0.2, 0.9, 50, and 50, respectively. Augmenting the data in this way can achieve good training results.

[0040] Steps 1-3: Mix the original image and the enhanced image as an augmented dataset. By mixing the original image and the enhanced image, the model can learn the distribution of the original data and obtain a wider range of feature changes from the enhanced data, effectively preventing overfitting to the limited original data.

[0041] In step 2, a fully convolutional network (FCN) is trained based on the preprocessed dataset. The augmented dataset is divided into training and test sets for batch training. After each training round, the segmentation accuracy is calculated on the test set, and then the model performance is optimized by adjusting the model hyperparameters based on the segmentation accuracy. Generally, the model structure is designed as a multi-layer fully connected convolutional network with a learning rate of 0.0001, and the Adam optimizer is used to dynamically adjust the learning rate during training. Specifically, in this application, FCN is used for pixel-level classification of infrastructure 2D images, preserving the spatial information of the input image and generating an output feature map with the same size as the input image.

[0042] Step 3 includes: using the trained FCN model to perform segmentation prediction on all aerial images. Based on learned features and classification rules, the model automatically assigns a category label to each pixel and completes boundary recognition. Specifically, in this application, semantic segmentation is used to identify different components (such as bridge piers, road surfaces, etc.) in infrastructure images, classifying each pixel into a specific component category or background. More specifically, in this application, the value of each pixel location corresponds to the category label at that location (the background category label is 0, and the component category labels increment from 1). These labels are stored in array form, directly reflecting the component category to which each region in the image belongs, and are key intermediate data for extending semantic information from two dimensions to a three-dimensional Gaussian model.

[0043] Step 4 includes the following steps:

[0044] Step 4-1 involves mixing manually annotated viewpoint images with unannotated viewpoint images predicted by the model, using this mixture as training data for the 3D Gaussian model. The manually annotated viewpoint images contain manually annotated semantic segmentation maps, while the semantic segmentation maps of the unannotated viewpoint images are derived from FCN predictions. This is important because, firstly, infrastructure components exhibit significantly different features under different viewpoints. For example, some components (such as connection nodes and supports) may only be fully visible from a specific viewpoint, while being occluded from other views. Manual annotation ensures that these critical components are correctly identified from the optimal viewing angle. Secondly, during the optimization of the 3D Gaussian model, manually annotated viewpoints act as "anchor points," constraining the distribution of semantic features in 3D space. When FCN predictions are incorrect, these anchor points provide correction signals, preventing error accumulation and amplification.

[0045] Step 4-2 uses the acquired original image (RGB image), semantic labels, and camera pose information as input triples to construct a 3D Gaussian model using all aerial data. Specifically, the 3D Gaussian model needs to simultaneously learn the scene's geometric structure, appearance features, and semantic attributes. The triples provide complete information across these three dimensions: the RGB image provides color, texture, and lighting information; the semantic labels provide component category information for each pixel; and the camera pose provides spatial geometric relationships and viewing angle information. Furthermore, semantic segmentation is essentially a 2D task, while the infrastructure model requires 3D semantic representation. The triples establish a precise mapping from 2D semantic labels to 3D space using camera pose. Moreover, the same 3D point should have a consistent semantic label when viewed from different perspectives. By binding semantic labels to camera pose, the system can detect and correct semantic inconsistencies across perspectives, improving the model's spatial semantic coherence. Finally, Gaussian model optimization relies on differentiable rendering techniques, requiring simultaneous comparison of the rendered results with the original RGB image and semantic labels. The triples provide all the reference information needed for this process, achieving joint optimization of geometry, appearance, and semantics.

[0046] The process of constructing a 3D Gaussian model based on the City Gaussian algorithm includes the following key steps: Processing multi-view RGB images using the Structured Bundles Method (SfM) to obtain sparse point clouds and accurate camera poses; using the SfM point cloud as the center position of the initial Gaussian primitives, initializing each point as a spherical Gaussian distribution; assigning initial attributes to each Gaussian primitive: position coordinates, opacity, geometry (scaling and rotation), and spherical harmonic features (color); adding a semantic attribute field, with initial values ​​based on the mode category projected onto the semantic labels of each viewpoint. The infrastructure scene (such as a bridge) is divided into multiple overlapping sub-blocks, and Gaussian models are pre-trained independently and in parallel for each sub-block, significantly reducing memory requirements. For each training viewpoint (triple), the following iterations are performed: projecting the current Gaussian model onto that viewpoint; calculating the color loss between the rendered RGB image and the original image; calculating the semantic loss between the rendered semantic map and the semantic labels; combining the two losses through backpropagation to update all attributes of the Gaussian primitives; and merging the Gaussian models of each sub-block.

[0047] In detail, traditional 3D Gaussian sputtering (3DGS) technology faces serious memory bottlenecks and computational efficiency issues when modeling large-scale infrastructure scenes. As scene complexity increases and the number of Gaussian units exceeds several million, the training process can lead to memory overflow, and rendering speed will drop sharply.

[0048] In step 4-2 of this application, the City Gaussian algorithm is used to model large-scale infrastructure scenes. The City Gaussian model is an innovative development based on 3D Gaussian sputtering (3DGS), effectively overcoming the limitations of 3DGS in large-scale scenes through clever algorithm design. It employs a divide-and-conquer training strategy and Level of Detail (LoD) rendering, enabling real-time rendering of large-scale scenes while maintaining high-quality rendering. Through its meticulous LoD strategy, City Gaussian ensures rich detail and strong realism in rendering effects at different view distances. Simultaneously, its block-based LoD selection and aggregation strategy significantly improves rendering efficiency, giving it real-time performance in large-scale scenes.

[0049] Furthermore, City Gaussian boasts cross-scale consistency, maintaining consistent rendering quality across different viewing distances and avoiding the abrupt changes common in traditional rendering techniques. Due to its multi-layered scene representation, City Gaussian significantly reduces memory requirements while enabling the rendering of large-scale scenes on ordinary hardware. These advantages make it a promising candidate for applications in urban planning and digital twins, supporting the creation of highly realistic urban digital models and various simulation analyses.

[0050] During the training phase, the City Gaussian model divides a large-scale scene into multiple sub-blocks, and performs independent and parallel 3D Gaussian pre-training on each sub-block. Then, using global scene priors, all sub-blocks are jointly optimized to achieve seamless fusion. Based on the fused Gaussian primitives, representations at different levels of detail are generated through compression. In the rendering phase, the scene region to be rendered is first determined based on the current viewpoint position and orientation. Then, based on the Nyquist sampling rate, an appropriate level of detail is selected for different regions, and Gaussian sputtering rendering is performed using the selected LoD level. Finally, the rendering results of different regions are seamlessly synthesized to generate the final image.

[0051] Step 4-3, in which traditional 3D Gaussian sputtering uses discrete 3D Gaussian sputtering... Each scene is represented and equipped with learnable attributes, including: 3D coordinates. Opacity Geometry (scaling and rotation), spherical harmonic features Colors related to the view This application extends the data structure by adding a semantic attribute field to each Gaussian element. The semantic attribute field has a length of 9 and corresponds to background 0 and 8 types of infrastructure components.

[0052] For a given camera intrinsic parameters , No. Camera position in Zhang's aerial photographs Gaussian data is splashed into the screen space. Among these, camera intrinsics... The intrinsic parameter matrix K describing the optical characteristics of a camera is typically a 3×3 matrix, containing the focal length. and principal point coordinates Information such as camera intrinsics. This parameter determines how 3D points are projected onto the image plane and is a fundamental parameter for establishing the mapping relationship between the world coordinate system and the pixel coordinate system. In this application, camera intrinsics are typically obtained through camera calibration or extracted from EXIF ​​data. Camera position: An external parameter describing the position and orientation of the camera in the world coordinate system, usually represented as a 4×4 transformation matrix [R|t], where R is a 3×3 rotation matrix and t is a 3×1 translation vector. Camera pose defines the rigid body transformation from the world coordinate system to the camera coordinate system, and in this application, it is estimated by an SfM algorithm (such as COLMAP).

[0053] Specifically, sputtering Gaussian to screen space refers to the process of projecting a 3D Gaussian function onto a 2D image plane. Specifically, for each 3D Gaussian primitive, its center point is projected onto screen space using a projection matrix (a combination of poses within the camera), while its 2D Gaussian distribution in screen space is calculated based on the camera viewpoint and the primitive's covariance matrix. This projection takes into account perspective effects and the spatial orientation of the primitives, resulting in distant Gaussian primitives having a smaller coverage area on the screen, while closer primitives cover a larger area, accurately simulating the real imaging process.

[0054] Sort by depth (Gaussian elements projected onto screen space according to their distance from the camera), and by... - The rendering is projected onto the new viewpoint plane, and calculations are performed on it. The location color is determined as follows:

[0055] ,in, To represent the projection of a Gaussian distribution onto a two-dimensional plane, the projection function is used. The calculation method is as follows: .

[0056] We use the method proposed in HUGS to obtain the semantic map of 3D Gaussians, similar to color rendering, to compute in the new view plane... Location semantics It uses the algorithm from HUGS, at the pixel position in the last layer of FCN. On the prediction results The softmax function is used for normalization calculation, and the calculation method is as follows: .

[0057] In particular, traditional 2D semantic segmentation often produces inconsistent results under different viewpoints. This application, however, binds semantic information to Gaussian units in 3D space, ensuring semantic consistency across different viewpoints. When a 3D point is observed from different viewpoints, its semantic label remains consistent; only the projected shape changes with the viewpoint.

[0058] During training, a typical initialization choice is a point cloud generated using Structure-from-Motion (SfM), such as COLMAP. Then, based on gradients obtained from differentiable rendering, Gaussian points are cloned, densed, pruned, and continuously optimized. However, for rendering large-scale infrastructure scenes, this may require more than 20 million primitives, which can easily lead to memory overflow errors during training and slow down rendering time. Therefore, we chose the City Gaussian model from step 4-2, which largely alleviates this problem.

[0059] Detailed Figure 3 The following diagrams sequentially illustrate the effects of the original 3D Gaussian, Mip-Splatting, Scaffold-GS, and this application. Performance metrics for each model are detailed in Table 1. Figure 3 As can be seen from Table 1:

[0060] While the original 3D Gaussian algorithm can achieve basic scene reconstruction, it completely lacks semantic information. The reconstructed bridge surface is blurry, and the road centerline is indistinguishable. Its PSNR is only 21.91 and SSIM is 0.794, indicating limited reconstruction quality. At the same time, it consumes as much as 2403.5MB of memory and requires 41 minutes and 23 seconds for training, making it inefficient for large-scale scene processing.

[0061] Mip-Splatting primarily addresses the issues of jagged edges and blurriness in rendering, resulting in improved visual quality with a PSNR of 22.65 and an SSIM of 0.804. However, it still cannot provide semantic segmentation capabilities, as the extracted boundary between the bridge surface and the water area is blurry. Furthermore, it has the longest training time (49 minutes and 27 seconds) and an FPS of only 38, leading to low rendering efficiency.

[0062] Scaffold-GS improves the processing efficiency of large-scale scenes through structured representation, reducing memory usage to 812.3MB, increasing FPS to 97, and shortening training time to 20 minutes and 48 seconds. However, it still has limitations in semantic understanding, with the extracted bridge skeleton exhibiting fuzziness, and its PSNR (20.67) and SSIM (0.778) indices being relatively low.

[0063] The method presented in this application demonstrates the best performance, achieving breakthroughs in visual quality with a PSNR of 26.13, an SSIM of 0.880, and LPIPS reduced to 0.211, indicating optimal reconstruction quality, structural preservation, and perceptual quality. Furthermore, it clearly identifies the semantic boundaries of different components, enabling precise segmentation of bridge structures, piers, railings, and other components. In terms of performance, it is also outstanding, requiring only 18 minutes and 56 seconds of training time, achieving an FPS of 102, and a minimum memory footprint of only 667.5MB, showcasing significant efficiency advantages in large-scale scene rendering.

[0064] Table 1 Comparison of Performance Indicators of Each Model

[0065] Methods PSNR SSIM LPIPS Train time FPS Mem. (MB) 3DGS 21.91 0.794 0.259 41min23s 45 2403.5 Scaffold-GS 20.67 0.778 0.281 20min48s 97 812.3 Mip-Splatting 22.65 0.804 0.249 49min27s 38 1732.1 Ours 26.13 0.880 0.211 18min56s 102 667.5

[0066] Figure 4The images sequentially showcase Omniseg3D-GS, this application, and real-world images captured on-site. As can be seen from the images, while Omniseg3D-GS, as an existing semantic 3D Gaussian method, possesses certain semantic segmentation capabilities, it suffers from semantic boundary ambiguity and inaccurate component classification in complex infrastructure scenarios, particularly at the intersection of bridges and roads where semantic confusion is prone to occur. This application's method, through improved semantic attribute field design and a multi-level detail rendering strategy, significantly enhances the accuracy of semantic segmentation. The generated rendered images are closer to real-world images in terms of component boundary clarity and semantic consistency. Especially when handling components with different materials and geometric features, such as bridge piers, road surfaces, and guardrails, it maintains stable segmentation results, achieving overall visual quality and semantic accuracy approaching the level of real-world scenes.

[0067] Specifically, differentiable rendering methods refer to rendering techniques capable of calculating the partial derivatives of the rendering process with respect to model parameters. Essentially, it constructs a differentiable mapping function from 3D Gaussian parameters to the 2D rendering result, allowing derivative calculations for each step in the rendering process (projection, sorting, compositing). This application simultaneously establishes differentiable channels for color rendering and semantic rendering, enabling backpropagation of color differences between the rendered image and the original image, as well as semantic differences between the rendered semantic map and the labeled image, to the Gaussian parameters. Furthermore, a gradient path is established between multi-level detail representations (LoD), allowing optimization of high-level (coarse) representations to guide improvements in low-level (fine) representations.

[0068] The gradient is the vector of partial derivatives of the total loss function L with respect to each Gaussian parameter, representing the direction and magnitude of parameter optimization. Cloning refers to the process of adaptively adding Gaussian elements based on rendering errors. Pruning is the process of removing redundant Gaussian elements that contribute little to rendering, aiming to control model complexity and improve rendering efficiency.

[0069] The foregoing illustrative description of the present application and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. The accompanying drawings are only one embodiment of the present application, and the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present application, such designs should fall within the scope of protection of this application. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for constructing a 3D Gaussian model based on semantic segmentation, characterized in that, include: The process involves acquiring and preprocessing a two-dimensional image dataset, including: acquiring two-dimensional images of infrastructure and performing semantic segmentation and annotation on the acquired two-dimensional images to obtain semantically segmented and annotated images; converting the semantically segmented and annotated images into an array format, where the background category is labeled as 0, and the component category labels start from 1 and increment; specifically, the arch rib is labeled as 1, the hanger as 2, the cross brace as 3, the diagonal brace as 4, the expansion joint as 5, the end block as 6, the support as 7, and the pier column as 8. Data augmentation processing is performed on the acquired original image and the semantically segmented and labeled image to obtain the augmented image; The collected original images, semantically segmented and labeled images, and enhanced images are mixed to obtain the preprocessed dataset; Train a fully convolutional semantic segmentation network (FCN) based on the preprocessed dataset. The trained fully convolutional network (FCN) is used to perform semantic segmentation on the acquired two-dimensional images to obtain two-dimensional semantic labels; Construct a 3D Gaussian model with semantic segmentation labels based on 2D semantic labels; Data augmentation processing is performed on the acquired raw images and semantically segmented and labeled images, including: Within the preset cropping range, select the cropping ratio. ; Calculate the height of the cutting area based on the cutting ratio r. and width ; Within the preset offset range, select the vertical offset amount. and horizontal offset ; According to height and width and vertical offset and horizontal offset Calculate the vertical boundaries of the clipping regions respectively. and horizontal boundary ; Based on the vertical boundary and horizontal boundary The original image J and the semantic segmentation labeled image A are cropped respectively to obtain local images. and segmentation labeled images ; local image and segmentation labeled images Resampled to the original size to obtain an enhanced image. and segmentation labeled images ; Vertical boundary and horizontal boundary It is calculated using the following formula: ； ； in, and The height and width of the original image; and Define the height and width of the cropping area; and These represent the offsets in the vertical and horizontal directions; S3, using the trained fully convolutional network FCN to perform semantic segmentation on the acquired 2D images, obtaining 2D semantic labels, including: The acquired two-dimensional image is used as input. The convolutional and pooling layers of the fully convolutional network FCN extract image features, and the output layer generates a predicted class probability distribution for each pixel. Based on the predicted class probability distribution, assign the class label with the highest probability to each pixel in the image; Based on the distribution of category labels, the boundaries of each category region are identified and determined, and a two-dimensional semantic label image with the same size as the input image is generated, wherein the value of each pixel position corresponds to the category label of the position; Based on the two-dimensional semantic labels, construct a three-dimensional Gaussian model with semantic segmentation labels, including: Based on the acquired original image and the semantic segmentation labeled image, viewpoint annotation is performed to obtain the labeled viewpoint image; The labeled viewpoint images and two-dimensional semantic label images are mixed to obtain the training dataset; Based on the training dataset, the original images, two-dimensional semantic label images, and camera poses during the image acquisition process are used as input triples to construct a three-dimensional Gaussian model using the CityGaussian algorithm. The structured beam method (SFM) was used to reconstruct the original images in three dimensions, generating an initial point cloud for initializing the three-dimensional Gaussian model. Set the semantic attribute field of each 3D Gaussian primitive that constitutes the 3D Gaussian model to store the category label of the component, and obtain the semantically expanded 3D Gaussian model. In the rendering stage of the CityGaussion algorithm, the level of detail for different rendering regions is set based on the Nyquist sampling rate. The semantically extended 3D Gaussian model is then rendered by Gaussian sputtering according to the set level of detail to generate a rendered image with semantic information. Based on the differences between the rendered image and the original image and semantically labeled image in the training set, the optimization gradient of Gaussian parameters is calculated using the differentiable rendering method in the CityGaussian algorithm; and based on the optimization gradient, the Gaussian parameters in the 3D Gaussian model are optimized to obtain a 3D Gaussian model with semantic segmentation labels. Generate rendered images with semantic information, including: Determine the area of ​​the scene to be rendered based on the current viewpoint position and direction; Set the level of detail for the scene area to be rendered based on the Nyquist sampling rate; Based on the set level of detail, select the corresponding precision 3D Gaussian primitives from the semantically expanded 3D Gaussian model; Based on the camera intrinsics and camera pose of the acquired images, the selected 3D Gaussian primitives are sputtered onto the screen space and sorted in depth order; Alpha-rendering projects the sorted 3D Gaussian primitives onto the viewpoint to be rendered, and calculates the color at position x in the viewpoint plane. ; From the viewpoint to be rendered, calculate the semantics of position x in the view plane. ; By compositing rendering scene regions at different levels of detail, a final rendered image with semantic information is obtained.

2. The method for constructing a 3D Gaussian model based on semantic segmentation according to claim 1, characterized in that: Calculate the color at position x in the view plane The formula is as follows: ;in, This represents the opacity of the k-th Gaussian term; Indicates the color associated with the view; To represent the projection of a Gaussian distribution onto a two-dimensional plane, the projection function is used. The calculation method is as follows: ; Let represent the opacity of the t-th Gaussian; This represents the k-th 3D Gaussian. Indicates camera intrinsic parameters; Indicates the camera pose.

3. The method for constructing a 3D Gaussian model based on semantic segmentation according to claim 1, characterized in that: The semantics of position x in the view plane The formula is as follows: Where, softmax represents the normalization function, This is the prediction result of the last layer of the fully convolutional network FCN at pixel position x.

4. The method for constructing a 3D Gaussian model based on semantic segmentation according to claim 1, characterized in that: Based on the differences between the rendered image and the original image and semantically labeled image in the training set, the optimized gradient of the Gaussian meta-parameters is calculated using the differentiable rendering method in the CityGausssion algorithm. Based on the optimization gradient, the Gaussian primitives in the 3D Gaussian model are optimized to obtain a 3D Gaussian model with semantic segmentation labels, including: Calculate color The color loss L1 between the original image and the original image; Computational semantics The semantic loss L2 between the image and the semantically labeled image; Construct the total loss function L based on the color loss L1 and the semantic loss L2; Calculate the gradient of the total loss function L with respect to the three-dimensional Gaussian meta-parameters, which include three-dimensional coordinates. Opacity Geometric structure, spherical harmonic characteristics Colors related to the view ; Based on the calculated gradient, Gaussian units are cloned or pruned to obtain a 3D Gaussian model with semantic segmentation labels.

Images