Three-dimensional Gaussian model construction method based on semantic segmentation

Abstract

The invention discloses a three-dimensional Gaussian model construction method based on semantic segmentation, and relates to the field of computer graphics and vision, and the method comprises the steps: collecting a two-dimensional image data set, and carrying out the preprocessing of the two-dimensional image data set; training a full convolutional network (FCN) according to the preprocessed data set; performing semantic segmentation on the acquired two-dimensional image by using the trained full convolutional network FCN to obtain a two-dimensional semantic tag; constructing a three-dimensional Gaussian model with a semantic segmentation tag according to the two-dimensional semantic tag; aiming at low rendering quality caused by lack of semantic understanding of three-dimensional Gaussian sputtering in a large-scale scene, the problem of memory overflow and rendering efficiency of traditional 3D Gaussian sputtering in large-scale infrastructure modeling is solved through a City Gaussian algorithm. The multi-level detail rendering strategy ensures that a high-quality rendering effect and accurate semantic expression can be maintained under different observation distances, and the problem of inconsistent quality of far-myopia images and myopia images in a traditional rendering method is avoided.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a method for constructing a three-dimensional Gaussian model based on semantic segmentation. Background Art

[0002] With the increasing complexity of modern engineering construction and infrastructure development, the need for accurate processing and analysis of infrastructure models is becoming increasingly urgent. Traditional model processing methods face numerous challenges in modeling large-scale infrastructure scenarios, such as bridges and roads. Accurately identifying and separating different components, such as bridge piers and pavement, is a key issue. These components require separate analysis and manipulation during subsequent engineering design, maintenance, and management.

[0003] In recent years, technologies such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting have demonstrated great potential in 3D model processing. 3D Gaussian Splatting, in particular, uses Gaussian distributions to represent objects in 3D space. Each Gaussian distribution can be viewed as a description of the object's local information, and combining multiple Gaussian distributions can accurately represent complex 3D shapes and structures. This approach effectively captures the object's geometry and appearance, achieving high-quality rendering.

[0004] However, existing 3D Gaussian sputtering technology faces two major issues when processing large-scale infrastructure scenes: first, it lacks semantic understanding capabilities and cannot automatically identify and segment different components in the scene; second, it is prone to memory overflow and slow rendering when rendering large-scale scenes. In particular, large-scale infrastructure scenes may require more than 20 million Gaussian primitives, which not only leads to memory overflow errors during training but also significantly increases rendering time.

[0005] While traditional semantic segmentation methods can segment different components in 2D images, they are difficult to directly apply to 3D Gaussian models. Existing 3D semantic segmentation methods are mostly based on point cloud or voxel representations, which fail to fully leverage the advantages of 3D Gaussian sputtering in rendering quality and efficiency. How to effectively combine semantic segmentation technology with 3D Gaussian sputtering to achieve accurate component identification and high-quality rendering in large-scale infrastructure scenarios has become a pressing technical challenge. Summary of the Invention

[0006] In response to the lack of semantic understanding of three-dimensional Gaussian splattering in large-scale scenes, which leads to low rendering quality, this application provides a three-dimensional Gaussian model construction method based on semantic segmentation. By adding semantic attribute fields to three-dimensional Gaussian primitives, adopting the City Gaussion algorithm and multi-level detail rendering strategy, it can efficiently process large-scale infrastructure scenes and improve the quality of large-scale infrastructure rendering.

[0007] The present application provides a method for constructing a three-dimensional Gaussian model based on semantic segmentation, comprising: collecting and preprocessing a two-dimensional image dataset; training a fully convolutional network (FCN) based on the preprocessed dataset; performing semantic segmentation on the collected two-dimensional images using the trained fully convolutional network (FCN) to obtain two-dimensional semantic labels; and constructing a three-dimensional Gaussian model with semantic segmentation labels based on the two-dimensional semantic labels.

[0008] Furthermore, a two-dimensional image dataset is collected and preprocessed, including: collecting two-dimensional images of the infrastructure and semantically segmenting and annotating the collected two-dimensional images to obtain semantically segmented and annotated images; converting the semantically segmented and annotated images into an array format, where the background category is labeled 0 and the component category labels start at 1 and increase in number, for a total of 69 categories. Data augmentation is performed on the collected original images and the semantically segmented and annotated images to obtain enhanced images; and the collected original images, semantically segmented and annotated images, and enhanced images are mixed to obtain a preprocessed dataset.

[0009] Furthermore, data enhancement processing is performed on the collected original image and semantic segmentation annotated image, including: selecting the cropping ratio within the preset cropping ratio range ; Calculate the height of the cropping area according to the cropping ratio r and width ; Select the vertical offset within the preset offset range and horizontal offset ; According to the height and width , and the vertical offset and horizontal offset , calculate the vertical boundaries of the clipping area respectively and horizontal boundaries ; According to the vertical boundary and horizontal boundaries , respectively crop the original image J and the semantic segmentation annotation image A to obtain the local image and segmentation annotation images ; The local image and segmentation annotation images Resample to the original size to get the enhanced image and segmentation annotation images .

[0010] Furthermore, the vertical boundary and horizontal boundaries , calculated using the following formula:

[0011] ; ;

[0012] in, and is the height and width of the original image; and is the height and width of the cropping area; and The vertical and horizontal offsets.

[0013] Furthermore, the trained fully convolutional network (FCN) is used to perform semantic segmentation on the captured two-dimensional image to obtain two-dimensional semantic labels, including: taking the captured two-dimensional image as input, the convolution layer and pooling layer of the fully convolutional network (FCN) extract image features, and the output layer generates a predicted category probability distribution for each pixel; according to the predicted category probability distribution, each pixel in the image is assigned a category label with the highest probability; according to the distribution of the category labels, the boundaries of each category area are identified and determined, and a two-dimensional semantic label image with the same size as the input image is generated, wherein the value of each pixel position corresponds to the category label of the position.

[0014] Furthermore, a three-dimensional Gaussian model with semantic segmentation labels is constructed based on the two-dimensional semantic labels, including: performing perspective annotation based on the collected original image and the semantic segmentation annotated image to obtain an annotated perspective image; mixing the annotated perspective image and the two-dimensional semantic label image to obtain a training data set; based on the training data set, the collected original image, the two-dimensional semantic label image and the camera pose during the image acquisition process are used as input triples to construct a three-dimensional Gaussian model through the CityGaussion algorithm; using the structured beam method SFM to perform three-dimensional reconstruction on the collected original image to generate an initial point cloud for initializing the three-dimensional Gaussian model; adding a three-dimensional Gaussian basis element to each three-dimensional Gaussian basis element that constitutes the three-dimensional Gaussian model. The semantic attribute field is used to store the category label of the component to obtain the semantically expanded three-dimensional Gaussian model. During the rendering phase of the CityGaussion algorithm, the level of detail of different rendering areas is set based on the Nyquist sampling rate. The semantically expanded three-dimensional Gaussian model is then Gaussian sputtered rendered according to the set level of detail to generate a rendered image with semantic information. Based on the difference between the rendered image and the original image and semantically labeled image in the training set, the optimized gradient of the Gaussian primitive parameters is calculated using the differentiable rendering method in the CityGaussion algorithm. Based on the optimized gradient, the Gaussian primitives in the three-dimensional Gaussian model are optimized to obtain a three-dimensional Gaussian model with semantic segmentation labels.

[0015] Furthermore, in the rendering stage of the City Gaussion algorithm, the detail level of different areas is set based on the Nyquist sampling rate, and the semantically expanded three-dimensional Gaussian model is Gaussian sputtering rendered according to the set detail level to generate a rendered image with semantic information, including: determining the scene area to be rendered according to the current viewpoint position and direction; setting the detail level of the scene area to be rendered based on the Nyquist sampling rate; selecting a three-dimensional Gaussian primitive of corresponding accuracy from the semantically expanded three-dimensional Gaussian model according to the set detail level; sputtering the selected three-dimensional Gaussian primitives to the screen space according to the camera intrinsic parameters and camera pose of the acquired image, and sorting them in depth order; projecting the sorted three-dimensional Gaussian primitives to the perspective to be rendered through α-rendering, and calculating the color of the position x in the perspective plane. ; In the perspective to be rendered, calculate the semantics of position x in the perspective plane ; Synthesize rendered scene areas at different levels of detail to obtain the final rendered image with semantic information.

[0016] Furthermore, the color of position x in the viewing plane is calculated , through the following formula: ;in, The opacity of the kth Gaussian; Represents the color associated with the view; It is the projection of Gaussian distribution on the two-dimensional plane, using the projection function , calculated as follows: ; represents the opacity of the t-th Gaussian; represents the kth 3D Gaussian, Represents the camera internal parameters; Represents the camera pose.

[0017] Furthermore, the semantics of position x in the viewing plane is calculated , through the following formula: ; Among them, softmax represents the normalization function, It is the prediction result of the last layer of the fully convolutional network FCN at pixel position x, reflecting the confidence score of each pixel belonging to different semantic categories.

[0018] Furthermore, based on the difference between the rendered image and the original image and semantic label image in the training set, the optimization gradient of the Gaussian primitive parameters is calculated by the differentiable rendering method in the City Gaussion algorithm; and based on the optimized gradient, the Gaussian primitives in the three-dimensional Gaussian model are optimized to obtain a three-dimensional Gaussian model with semantic segmentation labels, including: calculating color Color loss L1 between the original image; computational semantics The semantic loss L2 between the color loss L1 and the semantic label image; construct a total loss function L based on the color loss L1 and the semantic loss L2; calculate the gradient of the total loss function L with respect to the three-dimensional Gaussian primitive parameters, the three-dimensional Gaussian primitive parameters including the three-dimensional coordinates , opacity , geometry, spherical harmonic characteristics Colors associated with the view ; According to the calculated gradient, the Gaussian primitives are cloned or pruned to obtain a three-dimensional Gaussian model with semantic segmentation labels.

[0019] Compared with the existing technology, the advantages of this application are:

[0020] (1) Traditional 3D scene understanding technologies mainly use point cloud segmentation or voxelization processing methods, but they have significant defects in infrastructure segmentation, such as insufficient accuracy, high computational overhead, and poor rendering quality. This application innovatively extends the 2D semantic segmentation results to the 3D Gaussian function, achieving a deep fusion of semantic information and geometric representation. This application not only simply segments the scene, but also constructs a complete 3D model with fine semantic labels, and the semantic information is directly integrated into the Gaussian function. This method ensures the simultaneous optimization of semantic features and geometric features by adding semantic attribute fields to each 3D Gaussian basis element, so that the model has accurate semantic understanding capabilities while maintaining high-quality visual rendering.

[0021] (2) The City Gaussion algorithm solves the memory overflow and rendering efficiency issues of traditional 3D Gaussian sputtering in large-scale infrastructure modeling. Its multi-level detail rendering strategy ensures high-quality rendering effects and accurate semantic expression at different viewing distances, avoiding the inconsistent quality of near and far views in traditional rendering methods.

[0022] (3) This application uses the α-rendering mechanism to render semantic labels and color information using the same mechanism, solving the problem of converting semantic information from 2D to 3D and ensuring consistent component recognition from different perspectives. This feature is particularly important for infrastructure inspection and maintenance, allowing accurate identification of individual components from any angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present application will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0024] Figure 1 is a schematic diagram of an exemplary application scenario of a method for constructing a three-dimensional Gaussian model based on semantic segmentation according to some embodiments of the present application;

[0025] Figure 2 is an exemplary module diagram of a method for constructing a three-dimensional Gaussian model based on semantic segmentation according to some embodiments of the present application;

[0026] Figure 3 This is a comparison effect diagram of the present application and the existing model according to some embodiments of the present application;

[0027] Figure 4 This is another comparison effect diagram of the present application and the existing model shown in some embodiments of the present application. DETAILED DESCRIPTION

[0028] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] Reference Figure 1 In the application scenario shown, taking the Luoshe Bridge as an example, the infrastructure segmentation and component singulation method based on the 3D Gaussian model we proposed includes the following steps:

[0031] Step 1: 2D image data preprocessing and enhancement;

[0032] Step 2: Training of the fully convolutional network (FCN) model;

[0033] Step 3: Perform segmentation prediction based on the trained fully convolutional network model;

[0034] Step 4: Build a 3D Gaussian model with semantic segmentation labels based on the 2D semantic labels;

[0035] Step 1 includes the following steps:

[0036] Step 1-1: Convert the semantic information of the segmented and labeled dataset into an array, where the background category is labeled 0 and the category labels of infrastructure components start from 1 and increase. The Luoshe Bridge includes 8 categories, including arch ribs, hangers, cross braces, diagonal braces, expansion joints, end blocks, supports, and piers, which are labeled 1 to 8 respectively. The labels are: "arch rib": 1, "hanger": 2, "cross brace": 3, "diagonal brace": 4, "expansion joint": 5, "end block": 6, "support": 7, and "pier": 8.

[0037] In step 1-2, the acquisition cost of professional annotated data of infrastructure components (such as the arch ribs and suspenders of Luoshe Bridge) is high and the quantity is limited. This method significantly expands the amount of training data without changing the semantic annotation by zooming in on a limited original image, thus alleviating the reliance of deep learning models on large amounts of annotated data. Specifically, for each image, a random image with a length-width ratio similar to the original image is selected. Figure 1 For an area that is consistent but smaller than the original image, the area is enlarged. The enlarged size is consistent with the input size of the full convolutional network, and the original data scale is expanded without changing the annotation semantics.

[0038] In step 1-2, the sizes of different components in the infrastructure vary significantly (such as large arch ribs and small expansion joints). Direct training can easily cause the model to ignore small components. This method randomly crops and enlarges small components to gain more "attention" in the training samples, thereby improving the recognition accuracy of small components such as hangers, supports, and expansion joints. Specifically, first select the cropping ratio. ,in, and Calculate the height of the cropping area by taking the minimum and maximum values of the cropping ratio respectively and width , the calculation formula is ; ;in, and is the height and width of the original image. Then select the vertical and horizontal offsets, respectively and , which come from uniform distribution ; ;in, and The maximum offset in the vertical and horizontal directions respectively. The vertical and horizontal boundaries of the cropping area and It can be expressed as: ; .

[0039] In this way, the original image and segmentation annotations Cropping to obtain a partial image and segmentation annotations , the cropped image and annotations are resampled to their original size , and get the enhanced image and its segmentation annotations In the formula, 、 、 and Appears as a hyperparameter and needs to be given manually. and The greater the difference, and Larger values of increase the richness of the generated samples. However, excessively large values may generate data that deviates significantly from the original sample distribution, disrupting model training. Therefore, hyperparameter values should be within a reasonable range. Aerial images of the Luoshe Bridge occupy a moderate proportion of the frame, measuring 5280 x 3956 pixels. Therefore, the four hyperparameter values of 0.2, 0.9, 50, and 50, respectively, enhance the data accordingly, achieving good training results.

[0040] Steps 1-3, mixing the original image and the enhanced image as an enhanced dataset. By mixing the original image and the enhanced image, the model can not only learn the original data distribution, but also obtain a wider range of feature changes from the enhanced data, effectively preventing overfitting on the limited original data.

[0041] In step 2, a fully convolutional network (FCN) is trained based on the preprocessed dataset. The augmented dataset is divided into a training set and a test set for batch training. After each round of training, the segmentation accuracy is calculated on the test set. Based on this accuracy, model performance is optimized by adjusting model hyperparameters. Generally, the model structure is designed as a multi-layer fully connected convolutional network, with a learning rate set to 0.0001. The Adam optimizer is used to dynamically adjust the learning rate during training. Specifically, in this application, the FCN is used to perform pixel-level classification of two-dimensional infrastructure images. It can maintain the spatial information of the input image and generate an output feature map with the same size as the input image.

[0042] Step 3 includes: using the trained FCN model to perform segmentation prediction on all aerial images. The model automatically assigns a category label to each pixel based on the learned features and classification rules, and completes boundary recognition. Specifically, in this application, semantic segmentation is used to identify different components in infrastructure images (such as bridge piers, road surfaces, etc.) and classify each pixel into a specific component category or background. More specifically, in this application, the value of each pixel position corresponds to the category label of that position (the background category label is 0, and the component category label starts at 1 and increases incrementally). These labels are stored in array form, directly reflecting the component category to which each area in the image belongs, and are key intermediate data for expanding semantic information from two-dimensional to three-dimensional Gaussian models.

[0043] Step 4 includes the following steps:

[0044] In step 4-1, manually annotated view images are mixed with model-predicted unannotated view images to serve as training data for the 3D Gaussian model. The manually annotated view images contain manually annotated semantic segmentation maps, while the semantic segmentation maps of the unannotated view images come from the FCN predictions. On the one hand, the features presented by infrastructure components differ significantly from one viewpoint to another. For example, some components (such as connection nodes and supports) may only be fully visible from a specific viewpoint and obscured from other views. Manual annotation ensures that these key components are correctly identified from the optimal viewing angle. On the other hand, during the 3D Gaussian model optimization process, the manually annotated viewpoints serve as "anchor points," constraining the distribution of semantic features in 3D space. When FCN predictions are erroneous, these anchor points provide correction signals to prevent error accumulation and amplification.

[0045] In step 4-2, the captured original image (RGB image), semantic label, and camera pose information are used as input triplets to construct a 3D Gaussian model using all aerial data. Specifically, the 3D Gaussian model must simultaneously learn the scene's geometric structure, appearance characteristics, and semantic attributes. The triplets provide complete information in all three dimensions: the RGB image provides color, texture, and lighting information; the semantic label provides component category information for each pixel; and the camera pose provides spatial geometric relationships and viewing angle information. Furthermore, semantic segmentation is inherently a 2D task, while infrastructure models require 3D semantic representation. The triplets establish a precise mapping from 2D semantic labels to 3D space using the camera pose. Furthermore, the same 3D point should have consistent semantic labels when viewed from different viewpoints. By binding semantic labels to camera pose, the system can detect and correct semantic inconsistencies across viewpoints, improving the spatial semantic coherence of the model. Finally, Gaussian model optimization relies on differentiable rendering techniques, requiring a simultaneous comparison of the rendered results with the original RGB image and semantic labels. The triplets provide all the necessary reference information for this process, enabling joint optimization of geometry, appearance, and semantics.

[0046] The process of constructing a 3D Gaussian model based on the City Gaussion algorithm includes the following key steps: Using the Structured Beam Method (SfM) to process multi-view RGB images to obtain a sparse point cloud and accurate camera pose; using the SfM point cloud as the center position of the initial Gaussian basis point, each point is initialized to a spherical Gaussian distribution; assigning initial attributes to each Gaussian basis point: position coordinates, opacity, geometry (scale and rotation), and spherical harmonic features (color); adding a semantic attribute field, whose initial value is based on the majority category of the point's projection onto the semantic label for each viewpoint. Infrastructure scenes (such as bridges) are divided into multiple overlapping sub-blocks, and Gaussian models are pre-trained independently and in parallel for each sub-block, significantly reducing memory requirements. For each training view (triplet), the current Gaussian model is iterated: projecting the current Gaussian model onto that viewpoint; calculating the color loss between the rendered RGB image and the original image; calculating the semantic loss between the rendered semantic map and the semantic labels; combining these two losses and backpropagating to update all Gaussian basis point attributes. Finally, the Gaussian models of each sub-block are merged.

[0047] In detail, traditional 3D Gaussian sputtering (3DGS) technology faces serious memory bottlenecks and computational efficiency issues when modeling large-scale infrastructure scenes. When the scene complexity increases and the number of Gaussian primitives exceeds millions, the training process will lead to memory overflow and the rendering speed will drop sharply.

[0048] In step 4-2 of this application, the City Gaussian algorithm is used to model large-scale infrastructure scenarios. The City Gaussian model is an innovative development based on 3D Gaussian sputtering (3DGS). Through clever algorithm design, it effectively overcomes the limitations of 3DGS in large-scale scenarios. It adopts a divide-and-conquer training strategy and multi-level detail rendering (LoD), enabling large-scale scenes to achieve real-time rendering while ensuring high-quality rendering. City Gaussian's meticulous LoD strategy ensures rich and realistic rendering effects at different viewing distances. At the same time, the block-based LoD selection and aggregation strategy significantly improves rendering efficiency, enabling real-time performance in large-scale scenarios.

[0049] City Gaussian also boasts cross-scale consistency, maintaining consistent rendering quality at varying viewing distances and avoiding the abrupt shifts often seen in traditional rendering techniques. Thanks to its multi-layered scene representation, City Gaussian significantly reduces memory requirements while enabling the rendering of large-scale scenes on standard hardware. These advantages offer broad application prospects in fields such as urban planning and digital twins, enabling the creation of highly realistic digital city models and various simulation analyses.

[0050] During the training phase, the City Gaussion model divides a large-scale scene into multiple sub-blocks, performing independent and parallel 3D Gaussian pre-training on each sub-block. Then, leveraging a global scene prior, all sub-blocks are jointly optimized to achieve seamless fusion. Based on the fused Gaussian primitives, representations at different levels of detail are generated through compression. During the rendering phase, the scene regions to be rendered are determined based on the current viewpoint position and orientation. Based on the Nyquist sampling rate, appropriate levels of detail are selected for each region. Gaussian splatter rendering is performed using the selected LoD levels, and the rendering results from the different regions are seamlessly combined to generate the final image.

[0051] Step 4-3, wherein conventional 3D Gaussian sputtering is performed with discrete 3D Gaussian Representing scenes, each of which is equipped with learnable attributes, including: 3D coordinates , opacity , geometry (scaling and rotation), spherical harmonics Colors associated with the view This application expands the data structure and adds a semantic attribute field to each Gaussian primitive, wherein the length of the semantic attribute field is 9, corresponding to background 0 and 8 types of infrastructure components.

[0052] For a given camera intrinsic parameter , No. The camera pose of the aerial image , Gaussian is splashed to the screen space. Among them, the camera intrinsic parameter The internal parameter matrix K that describes the optical characteristics of the camera is usually a 3×3 matrix, including the focal length and principal point coordinates This parameter determines how the 3D point is projected onto the image plane, and is the basic parameter for establishing the mapping relationship between the world coordinate system and the pixel coordinate system. In this application, the camera intrinsic parameters are usually obtained through camera calibration or extracted from EXIF data. Camera position: An external parameter that describes the position and orientation of the camera in the world coordinate system, usually expressed as a 4×4 transformation matrix [R|t], where R is a 3×3 rotation matrix and t is a 3×1 translation vector. The camera pose defines the rigid body transformation from the world coordinate system to the camera coordinate system, and is estimated by the SfM algorithm (such as COLMAP) in this application.

[0053] Specifically, Gaussian sputtering to screen space refers to the process of projecting a 3D Gaussian function onto the 2D image plane. Specifically, for each 3D Gaussian primitive, its center point is projected into screen space using the projection matrix (a combination of participating poses within the camera). Simultaneously, its 2D Gaussian distribution in screen space is calculated based on the camera's view angle and the primitive's covariance matrix. This projection takes into account perspective effects and the primitive's spatial orientation, ensuring that distant Gaussian primitives have a smaller footprint on the screen, while nearby primitives cover a larger area, accurately simulating the real-world imaging process.

[0054] Sort by depth (the Gaussian primitives projected into screen space by their distance from the camera) and by - Rendering is projected onto the new viewing plane and calculated in Position colors are as follows:

[0055] ,in, It is the projection of Gaussian distribution on the two-dimensional plane, using the projection function , calculated as follows: .

[0056] We use the method proposed in HUGS to obtain the semantic map of 3D Gaussian, similar to the color rendering, and calculate the Positional semantics , the algorithm in HUGS is used, and the last layer of FCN is at pixel position The prediction results , the softmax function is used for normalization calculation, and the calculation method is as follows: .

[0057] In particular, traditional 2D semantic segmentation often produces inconsistent results from different viewpoints. This application, however, binds semantic information to Gaussian primitives in 3D space, ensuring semantic consistency across different viewpoints. When a 3D point is observed from different viewpoints, its semantic label remains consistent; only the projected shape changes with the viewpoint.

[0058] During training, a typical initialization choice is a point cloud generated using a Structure-from-Motion (SfM) method, such as COLMAP. Gaussian points are then cloned, densified, pruned, and continuously optimized based on the gradients obtained from differentiable rendering. However, large-scale infrastructure rendering applications can require over 20 million primitives, which can easily lead to out-of-memory errors during training and slow rendering times. Therefore, we chose the City Gaussion model from step 4-2, which largely alleviates this problem.

[0059] Detailed, Figure 3 The original 3D Gaussian, Mip-Splatting, Scaffold-GS and the renderings of this application are shown in turn. The performance indicators of each model are detailed in Table 1. Figure 3 As can be seen from Table 1:

[0060] Although the original 3D Gaussian model can achieve basic scene reconstruction, it completely lacks semantic information. The reconstructed bridge deck is relatively blurry, and the center line of the road is completely indistinguishable. Its PSNR is only 21.91 and SSIM is 0.794, indicating limited reconstruction quality. At the same time, the memory usage is as high as 2403.5MB, and the training time takes 41 minutes and 23 seconds, which is inefficient in large-scale scene processing.

[0061] Mip-Splatting mainly solves the problems of aliasing and blurring in rendering, and improves visual quality. The PSNR reaches 22.65 and the SSIM is 0.804. However, it still cannot provide semantic segmentation capabilities. The extracted boundary between the bridge deck and the water area is blurred. The training time is the longest (49 minutes and 27 seconds), and the FPS is only 38, indicating low rendering efficiency.

[0062] Scaffold-GS improves the processing efficiency of large-scale scenes through structured representation, reducing memory usage to 812.3MB, increasing FPS to 97, and shortening training time to 20 minutes and 48 seconds. However, it still has limitations in semantic understanding. The extracted bridge skeleton is blurred, and its PSNR (20.67) and SSIM (0.778) indicators are relatively low.

[0063] The method of this application performed the best, not only achieving breakthrough progress in visual quality, with PSNR reaching 26.13, SSIM as high as 0.880, and LPIPS reduced to 0.211, indicating that the reconstruction quality, structural preservation and perception quality are all optimal; at the same time, it can also clearly identify the semantic boundaries of different components, and achieve accurate segmentation of components such as bridge bodies, piers, and guardrails; in terms of performance, it is even more outstanding, with training time taking only 18 minutes and 56 seconds, FPS reaching 102, and memory usage as low as 667.5MB, showing significant efficiency advantages in large-scale scene rendering.

[0064] Table 1 Comparison of performance indicators of various models

[0065] Methods PSNR SSIM LPIPS Train time FPS Mem.(MB) 3DGS 21.91 0.794 0.259 41 minutes and 23 seconds 45 2403.5 Scaffold-GS 20.67 0.778 0.281 20 minutes and 48 seconds 97 812.3 Mip-Splatting 22.65 0.804 0.249 49 minutes and 27 seconds 38 1732.1 Ours 26.13 0.880 0.211 18 minutes 56 seconds 102 667.5

[0066] Figure 4 Omniseg3D-GS, this application, and real images collected on-site are shown in turn. It can be seen from the figure that Omniseg3D-GS, as an existing semantic three-dimensional Gaussian method, has certain semantic segmentation capabilities, but has problems such as blurred semantic boundaries and inaccurate component classification in complex infrastructure scenarios, especially at the intersection of bridges and roads. Semantic confusion is prone to occur; the method of this application significantly improves the accuracy of semantic segmentation through improved semantic attribute field design and multi-level detail rendering strategy. The generated rendered images are closer to real images in terms of component boundary clarity and semantic consistency. In particular, when dealing with components with different materials and geometric features such as bridge piers, road surfaces, and guardrails, it can maintain a stable segmentation effect, and the overall visual quality and semantic accuracy have reached a level close to that of real scenes.

[0067] Specifically, a differentiable rendering method refers to a rendering technique that can calculate the partial derivatives of the rendering process with respect to model parameters. Its essence is to construct a differentiable mapping function from three-dimensional Gaussian parameters to two-dimensional rendering results, so that derivatives can be calculated at each step in the rendering process (projection, sorting, and synthesis). In this application, differentiable channels for color rendering and semantic rendering are simultaneously established, so that the color difference between the rendered image and the original image, as well as the semantic difference between the rendered semantic map and the labeled image, can be back-propagated to the Gaussian parameters. In addition, a gradient path is established between multiple levels of detail (LoD) representations, so that the optimization of high-level (coarse) representations can guide the improvement of low-level (fine) representations.

[0068] Gradients are the vectors of partial derivatives of the total loss function L with respect to the parameters of each Gaussian primitive, representing the direction and magnitude of parameter optimization. Cloning is the process of adaptively adding Gaussian primitives based on rendering error. Pruning is the process of removing redundant Gaussian primitives that have little contribution to rendering, aiming to control model complexity and improve rendering efficiency.

[0069] The invention of the present application and its implementation methods are described schematically above. This description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention of the present application, and the actual structure is not limited to this. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of the present application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. Words such as first and second are used to indicate names and do not indicate any specific order.

Claims

1. A method for constructing a three-dimensional Gaussian model based on semantic segmentation, characterized in that: include: Acquire and preprocess 2D image datasets; According to the preprocessed dataset, train the fully convolutional semantic segmentation network FCN; Use the trained fully convolutional network FCN to perform semantic segmentation on the collected two-dimensional images and obtain two-dimensional semantic labels; Based on the two-dimensional semantic labels, a three-dimensional Gaussian model with semantic segmentation labels is constructed.

2. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 1, wherein: Acquire a 2D image dataset and perform preprocessing, including: Collecting two-dimensional images of the infrastructure and performing semantic segmentation and annotation on the collected two-dimensional images to obtain semantic segmentation and annotation images; Convert the semantic segmentation annotated image into an array, where the background category is labeled 0 and the component category labels start from 1 and increase in number. Perform data enhancement processing on the collected original image and semantic segmentation annotated image to obtain an enhanced image; The collected original images, semantic segmentation and annotation images, and enhanced images are mixed to obtain the preprocessed dataset.

3. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 2, wherein: Perform data enhancement on the collected original images and semantic segmentation annotated images, including: Select a crop ratio from the preset crop ratio range. ; Calculate the height of the cropping area based on the cropping ratio r and width ; Select the vertical offset within the preset offset range. and horizontal offset ; According to height and width , and the vertical offset and horizontal offset , calculate the vertical boundaries of the clipping area respectively and horizontal boundaries ; According to the vertical boundary and horizontal boundaries , respectively crop the original image J and the semantic segmentation annotation image A to obtain the local image and segmentation annotation images ; The local image and segmentation annotation images Resample to the original size to get the enhanced image and segmentation annotation images .

4. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 2, wherein: Vertical boundary and horizontal boundaries , calculated using the following formula: ； ； in, and is the height and width of the original image; and is the height and width of the cropping area; and The vertical and horizontal offsets.

5. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 2, wherein: S3, using the trained fully convolutional network FCN to perform semantic segmentation on the collected 2D images and obtain 2D semantic labels, including: The collected two-dimensional image is used as input, and the convolutional layer and pooling layer of the fully convolutional network (FCN) extract image features. The output layer generates a predicted category probability distribution for each pixel. Assign each pixel in the image the class label with the highest probability based on the predicted class probability distribution; Based on the distribution of category labels, the boundaries of each category area are identified and determined, and a two-dimensional semantic label image with the same size as the input image is generated, where the value of each pixel position corresponds to the category label of the position.

6. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 1, wherein: Based on the two-dimensional semantic labels, a three-dimensional Gaussian model with semantic segmentation labels is constructed, including: Based on the collected original image and the semantic segmentation annotation image, the perspective annotation is performed to obtain the annotated perspective image; Mix the annotated view images and the two-dimensional semantic label images to obtain the training dataset; Based on the training dataset, the collected original image, the 2D semantic label image, and the camera pose during the image acquisition process are used as input triples to construct a 3D Gaussian model using the City Gaussion algorithm. The structured beam method (SFM) is used to reconstruct the original image to generate an initial point cloud for initializing the 3D Gaussian model. Setting the semantic attribute field of each three-dimensional Gaussian basis element constituting the three-dimensional Gaussian model to store the category label of the component, thereby obtaining a semantically expanded three-dimensional Gaussian model; In the rendering phase of the City Gaussion algorithm, the level of detail (LOD) of different rendering areas is set based on the Nyquist sampling rate. Gaussian sputtering rendering is performed on the semantically expanded 3D Gaussian model according to the set LOD, generating a rendered image with semantic information. According to the difference between the rendered image and the original image and semantically labeled image in the training set, the optimization gradient of the Gaussian primitive parameters is calculated using the differentiable rendering method in the CityGaussion algorithm; and based on the optimized gradient, the Gaussian primitives in the three-dimensional Gaussian model are optimized to obtain a three-dimensional Gaussian model with semantic segmentation labels.

7. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 6, wherein: Generate rendered images with semantic information, including: Determine the scene area to be rendered based on the current viewpoint position and direction; Set the level of detail for the scene area to be rendered based on the Nyquist sampling rate; According to the set level of detail, a 3D Gaussian basis element of corresponding accuracy is selected from the semantically expanded 3D Gaussian model; According to the camera intrinsic parameters and camera pose of the acquired image, the selected 3D Gaussian primitives are splashed into the screen space and sorted in depth order; Project the sorted 3D Gaussian primitives to the perspective to be rendered through α-rendering, and calculate the color of position x in the perspective plane ; At the perspective to be rendered, calculate the semantics of position x in the perspective plane ; Rendered scene regions at different levels of detail are synthesized to obtain the final rendered image with semantic information.

8. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 7, wherein: Calculate the color at position x in the viewing plane , through the following formula: ;in, represents the opacity of the kth Gaussian; Represents the color associated with the view; It is the projection of Gaussian distribution on the two-dimensional plane, using the projection function , calculated as follows: ; Indicates the opacity of the t-th Gaussian.

9. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 7, wherein: Compute the semantics of position x in the viewing plane , through the following formula: ; Among them, softmax represents the normalization function, is the prediction result of the last layer of the fully convolutional network FCN at pixel position x.

10. The method for constructing a three-dimensional Gaussian model based on semantic segmentation according to claim 7, wherein: According to the difference between the rendered image and the original image and semantically labeled image in the training set, the optimization gradient of the Gaussian primitive parameters is calculated through the differentiable rendering method in the CityGaussion algorithm; And according to the optimization gradient, the Gaussian primitives in the three-dimensional Gaussian model are optimized to obtain a three-dimensional Gaussian model with semantic segmentation labels, including: Calculating Color Color loss L1 between the original image; Computational semantics Semantic loss L2 between the semantically labeled image; Construct the total loss function L based on the color loss L1 and semantic loss L2; Calculate the gradient of the total loss function L with respect to the three-dimensional Gaussian primitive parameters, which include the three-dimensional coordinates , opacity , geometry, spherical harmonic characteristics Colors associated with the view ; According to the calculated gradient, the Gaussian primitives are cloned or pruned to obtain a 3D Gaussian model with semantic segmentation labels.

Images