A gauss semantic segmentation method based on color and shape deformation field

Abstract

The application relates to a Gaussian semantic segmentation method based on color and shape deformation fields, which comprises the following steps: collecting image frames of a dynamic scene, establishing a static Gaussian model composed of standard Gaussian primitives, and initializing a high-dimensional semantic feature for each standard Gaussian primitive; rendering the high-dimensional semantic features contained in the static Gaussian model by using a given rasterization pipeline to obtain a rendered semantic feature map; extracting instances and instance masks contained in the image frames, and constructing a corresponding relationship vector of image pixels and instance masks; performing comparative learning based on the corresponding relationship vector and the rendered semantic feature map to predict shape change and color change, and then rendering the dynamic Gaussian model at the current moment by using the given rasterization pipeline to perform semantic segmentation. The application can realize end-to-end joint optimization of geometry, appearance and semantics in a dynamic scene, and can balance the spatiotemporal consistency, segmentation accuracy and real-time rendering capability.

Description

Technical Field

[0001] This invention relates to the field of dynamic scene semantic segmentation technology, and in particular to a Gaussian semantic segmentation method based on color and shape deformation fields. Background Technology

[0002] With the advancement of computer vision and graphics technologies, the digital reconstruction of 3D scenes has evolved from simple geometric and appearance replication to dynamic scene digitalization with semantic awareness. In core scenarios such as VR / AR interaction, service robot navigation, and autonomous driving, machines not only need to reproduce the dynamically changing 3D environment with high fidelity, but also need to accurately map high-level semantic information (instance category, individual ID, etc.) in 2D images to 3D space to achieve continuous semantic tracking of dynamic targets.

[0003] Explicit 3D representation methods, such as 3D Gaussian sputtering (3DGS), have gradually replaced neural implicit representations as the mainstream paradigm for 3D reconstruction due to their high-fidelity rendering quality in static scenes and native real-time interaction advantages. However, when extending to dynamic semantic reconstruction, existing solutions have the following problems: Existing methods that use large 2D visual models to assist 3D segmentation mostly adopt a frame-by-frame independent prediction mode, which lacks the inherent geometric constraints of 3D space and the continuity of the temporal dimension. When these unstable 2D semantic masks are directly mapped to a 3D Gaussian field that undergoes drastic deformation, changes in viewpoint, illumination fluctuations, and dynamic occlusion can easily destroy the correspondence of features, resulting in the reconstructed semantic labels drifting and high-frequency flickering in the time series, making it impossible to achieve stable dynamic semantic tracking under unsupervised conditions. In existing explicit semantic field construction, semantic attributes are usually encoded as continuous feature vectors. However, the core rendering mechanism of 3D Gaussian sputtering relies on depth-ordered alpha blending. This linear interpolation method is effective in processing color information, but when processing discrete semantic categories, it can lead to the incorrect mixing of semantic features of different objects, especially at the edges of front and back occlusion. Existing contrastive learning strategies are unable to completely correct this feature dilution caused by the rendering pipeline mechanism, resulting in a decrease in the accuracy of object edge segmentation and failing to meet the requirements of fine segmentation.

[0004] Existing dynamic scene reconstruction methods often rely excessively on complex deep neural networks to fit the nonlinear deformation of the scene. This not only leads to long model training cycles and huge consumption of computing resources, but also makes it difficult to maintain the original real-time rendering advantage of 3DGS during the inference stage. Furthermore, the lack of an efficient, component-decoupled dynamic representation architecture makes it difficult to ensure high accuracy in dynamic scene geometric reconstruction and semantic understanding while simplifying the computation process. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a Gaussian semantic segmentation method based on color and shape deformation fields, which can achieve end-to-end joint optimization of geometry, appearance and semantics in dynamic scenes, while taking into account spatiotemporal consistency, segmentation accuracy and real-time rendering capabilities.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a Gaussian semantic segmentation method based on color and shape deformation fields, comprising: Simultaneously acquire image frames of a dynamic scene from multiple given perspectives; A static Gaussian model composed of standard Gaussian elements is established based on the acquired image frames, and a high-dimensional semantic feature is initialized for each standard Gaussian element. The high-dimensional semantic features contained in the static Gaussian model are rendered using a given rasterization pipeline to obtain the rendered semantic feature map of the static Gaussian model. Extract the two-dimensional semantic mask of each image frame, obtain the instances and instance masks contained in the image frame, and construct the correspondence vector between image pixels and instance masks; Based on the corresponding relationship vector and the rendered semantic feature map, comparative learning is performed to predict the shape change and color change of each standard Gaussian element in the static Gaussian model at the current time. Based on the generated shape and color changes, a dynamic Gaussian model for the current moment is obtained by rendering using the given rasterization pipeline, and then semantic segmentation of the dynamic scene is performed based on the dynamic Gaussian model.

[0007] Furthermore, the step of establishing a static Gaussian model composed of standard Gaussian elements based on the acquired image frames includes: Extract sparse point clouds of the dynamic scene from the acquired image frames; The coordinates of each point in the sparse point cloud are used as the initial mean position of the standard Gaussian unit; Initialize the rotation, scaling, and opacity of each standard Gaussian element to obtain a static Gaussian model composed of standard Gaussian elements.

[0008] Furthermore, the step of rendering the high-dimensional semantic features contained in the static Gaussian model using a given rasterization pipeline to obtain the rendered semantic feature map of the static Gaussian model includes: The cumulative alpha blending value of each image pixel in an image frame is calculated using a given rasterization pipeline as the rendering semantic feature at the corresponding image pixel; The rendering semantic feature map of the static Gaussian model is generated based on the rendering semantic features at all image pixels.

[0009] Furthermore, the cumulative alpha value of the image pixel is calculated based on smooth Gaussian features, which are obtained by calculating the high-dimensional semantic features of the current image pixel in a set number of neighboring image pixels in three-dimensional space.

[0010] Furthermore, the two-dimensional semantic mask is generated by performing zero-sample segmentation on each image frame using a pre-trained visual base model.

[0011] Furthermore, the construction of the correspondence vector between image pixels and instance masks includes: Randomly sample the extracted instance masks to obtain a sampled mask subset that includes a set number of instance masks; The attribution of each image pixel in the sampling mask subset is encoded to obtain the corresponding relationship vector of each image pixel.

[0012] Furthermore, the comparative learning based on the corresponding relation vector and the rendered semantic feature map includes: Based on the corresponding relationship vector, two image pixel pairs with the same instance mask are extracted as positive sample pairs to form a positive sample pair set, and two image pixel pairs with conflicting instance masks are extracted as negative sample pairs to form a negative sample pair set. For each positive sample pair, the positive sample term loss is calculated based on the rendering feature similarity between the corresponding two image pixels in the rendered semantic feature map; For each negative sample pair, the negative sample term loss is calculated based on the rendering feature similarity between the corresponding two image pixels in the rendered semantic feature map; The contrast loss is calculated based on the positive sample loss and the negative sample loss, and the shape change and color change of the standard Gaussian unit are updated by backpropagation.

[0013] Furthermore, the contrast loss is expressed as: in, This represents the contrast loss. and These represent the losses for the positive sample terms and the losses for the negative sample terms, respectively. and Let them represent the set of positive sample pairs and the set of negative sample pairs, respectively. This represents the similarity of rendering features between pixel i and pixel j. and Let represent the safety boundaries for positive samples and negative samples, respectively.

[0014] Furthermore, the shape change is predicted using a shape deformation field, and the color change is predicted using a color deformation field.

[0015] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: 1. This invention, based on Gaussian sputtering, achieves strong spatiotemporal consistency of semantic tags in dynamic scenes. It employs an explicit semantic feature binding strategy based on 3D Gaussian primitives. Unlike the traditional loosely coupled approach of projecting 2D segmentation results frame-by-frame into 3D space, this invention directly binds high-dimensional semantic feature vectors as inherent attributes to each Gaussian particle. Combined with shape and color deformation fields, when an object deforms, the semantic tags move and deform synchronously with the geometric structure. This mechanism fundamentally ensures the continuity of semantic information over time, effectively eliminating semantic mask flickering and drift caused by viewpoint changes or occlusion, and significantly improving the stability of dynamic scene segmentation.

[0016] 2. This invention improves the semantic segmentation accuracy under object edge and complex occlusion conditions. Addressing the feature mixing and edge blurring issues that easily occur during 3DGS rasterization, this invention introduces a KNN-based feature smoothing mechanism and an improved boundary relaxation contrast loss function, effectively filtering noisy features in the Gaussian field and enhancing semantic consistency within the neighborhood.

[0017] 3. This invention achieves a balance between high-fidelity reconstruction and low computational consumption, possessing real-time rendering capabilities. This invention employs a decoupled multi-stage training strategy. By decoupling the dynamic scene into standard Gaussian geometry, a dual deformation field, and a semantic feature field, the converged geometric appearance parameters are frozen or fine-tuned during the semantic learning stage, optimizing only low-dimensional semantic features. Combining low-resolution rendering of semantic feature maps with a bilinear interpolation upsampling strategy, this significantly reduces memory usage and training time while maintaining the original real-time rendering advantages of 3DGS.

[0018] 4. It can provide visual reconstruction effects that are more in line with physical laws, thereby helping semantic branches to understand scene content more accurately and improving the generalization ability in practical applications such as robot navigation and VR / AR interaction. Attached Figure Description

[0019] Figure 1 This is a flowchart of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the Gaussian feature construction and rendering according to an embodiment of the present invention; Figure 3 This is a design architecture diagram of an embodiment of the present invention; Figure 4This is a schematic diagram illustrating the implementation process of an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0021] The embodiments of the present invention relate to a Gaussian semantic segmentation method based on color and shape deformation fields, such as... Figure 1 As shown, the entire process can be roughly divided into S0 standard Gaussian scene representation generation, S1 Gaussian semantic feature construction and rendering, S2 Gaussian feature semantic learning and clustering, S3 dynamic scene Gaussian appearance integration, and S4 dynamic scene semantic segmentation. Specifically, it includes: S0 Standard Gaussian Scene Representation Generation (Establishing a Static Gaussian Model): Simultaneously acquire image frames of a dynamic scene from multiple given perspectives; establish a static Gaussian model composed of standard Gaussian primitives based on the acquired image frames; S1 Gaussian Semantic Feature Construction and Rendering: Initialize a high-dimensional semantic feature for each standard Gaussian primitive; render the high-dimensional semantic features contained in the static Gaussian model using the given rasterization pipeline to obtain the rendered semantic feature map of the static Gaussian model; S2 Gaussian Feature Semantic Learning and Clustering: Extract the two-dimensional semantic mask of each image frame, obtain the instances and instance masks contained in the image frame, and construct the correspondence vector between image pixels and instance masks; S3 Dynamic Scene Gaussian Appearance Integration: Comparative learning is performed using the basis correspondence vector and the rendered semantic feature map to predict the shape change and color change of each standard Gaussian element in the static Gaussian model at the current moment; Based on the generated shape change and color change, the dynamic Gaussian model at the current moment is obtained by rendering using the rasterization pipeline in step S1. S4 Dynamic Scene Semantic Segmentation: Semantic segmentation based on dynamic Gaussian model.

[0022] Step S1, Gaussian semantic feature construction and rendering, can be implemented using the following methods.

[0023] To endow each Gaussian unit with semantic understanding capabilities, Extend a D-dimensional semantic feature vector Unlike color attributes, semantic features are typically considered view-independent intrinsic properties, therefore they are not modeled using spherical harmonic functions, but rather directly treated as optimizable parameter vectors. The entire process is detailed below. Figure 2As shown. During the rendering stage, for a given camera viewpoint, the rendering semantic features at pixel p are accumulated through alpha blending, and the calculation process is completely consistent with color rendering: in It is the set of Gaussian elements sorted on that pixel. It is the opacity of the i-th Gaussian. It refers to transmittance.

[0024] To reduce memory consumption and improve training efficiency, the semantic feature dimension is typically set to a small value (e.g., D=32) during training, and then mapped to the final metric space using a lightweight multilayer perceptron or linear projection layer. Furthermore, to accelerate rendering, the semantic feature map's rendering resolution is usually set to 1 / 2 or 1 / 4 of the original image, and then upsampled to the original size using bilinear interpolation for alignment with the ground truth mask.

[0025] Using smoothed Gaussian features helps filter out noisy Gaussian features that also have high similarity scores with unrelated objects. During rendering, a smoothed feature vector is used, and the smoothed Gaussian feature of the i-th Gaussian is calculated using the following formula, where KNN(i) represents the k nearest neighbors of the i-th Gaussian based on its 3D position: Step S2, Gaussian feature semantic learning and clustering, can be implemented using the following methods.

[0026] To learn a continuous semantic field in 3D space, a learnable D-dimensional semantic feature vector is introduced for each Gaussian point i. Due to the lack of ground truth 3D semantics, a 2D mask generated by the powerful zero-shot segmentation capability of the SAM model is used as supervision. However, the SAM mask is temporally uncorrelated and has blurred edges. Some methods introduce a hard-sample mining strategy based on pixel-mask correspondences. The pixel-mask correspondence vector construction is performed on the training view. SAM generates a set of binary masks. To handle mask overlap and segmentation ambiguity, instead of simply assigning pixels to a single mask, a pixel-mask correspondence vector is defined. For image coordinates... For each pixel, its corresponding vector encodes the pixel's affiliation within the sampling mask subset: in This represents the number of randomly sampled masks. If two pixels have similar y-vectors, it means they have a very high probability of belonging to the same semantic structure in the current 2D observation, and are classified as a positive sample pair; if the y-vectors of two pixels differ greatly, it means they are likely to belong to different objects, and are classified as a negative sample pair. Specifically, this classification can be achieved by calculating the cosine similarity of the y-vectors corresponding to the two pixels: If the similarity is greater than the set threshold, it means that the two pixels are likely to belong to the same instance in the mask and are classified as positive sample pairs. If the similarity is less than or equal to the set threshold, it means that the two pixels are likely to belong to different instances in the mask and are classified as negative sample pairs.

[0027] Step S3, Gaussian appearance integration for dynamic scenes, can be implemented using the following methods.

[0028] First, the dynamic scene is decoupled into a standard Gaussian field and a dual-deformation field. Assume the i-th Gaussian element in the standard space is represented by Gi, which contains geometric parameters (position...). Rotation Scaling At time t, through the shape deformation field and color deformation field Predict its change: The deformed Gaussian parameters are used for subsequent projection and rendering. In semantic segmentation tasks, the pre-trained Gaussian geometry and appearance parameters of the scene are frozen, or fine-tuned with a small learning rate, focusing on learning newly introduced semantic attributes. This design not only enables the model to handle complex dynamic motion with high fidelity, but more importantly, semantic features are directly bound to Gaussians. This means that regardless of how an object moves or deforms, the particles representing parts of that object always carry the same semantic labels, fundamentally solving the problem of semantic temporal consistency in dynamic scenes.

[0029] To improve 2D mask consistency to 3D feature consistency, a contrastive learning framework based on the Gram matrix is ​​constructed. Traditional hard sample mining strategies typically employ binary threshold truncation, optimizing only hard samples within a threshold range. For example, SADG's hard sample mining mechanism focuses on misclassified hard pixel pairs. This hard truncation mechanism can easily lead to insufficient compactness in the intra-class distribution of the feature space, causing the model to stop optimizing after meeting the minimum similarity threshold.

[0030] This implementation uses an improved boundary-relaxed contrastive loss to guide supervised learning. This strategy does not completely discard correctly classified samples, but introduces a stricter safety boundary, forcing positive samples to be not only similar, but also sufficiently similar; and negative samples to be not only dissimilar, but also sufficiently dissimilar.

[0031] First, calculate the rendering feature similarity matrix of the sampled pixels. , and These are the pixel features obtained after rendering pixels i and j, respectively. Based on the mask correspondence matrix, the loss function is... Decoupling into positive sample terms With negative sample items And introduce hinge functions To smooth the gradient: in, and Let represent the set of positive sample pairs with consistent masks and the set of negative sample pairs with conflicting masks, respectively. This represents the similarity of rendering features between pixel i and pixel j. This represents the safety boundary (threshold) for positive samples. It requires that positive samples (pairs of pixels with consistent masks) not only be similar, but also have similarity scores. Must be greater than or equal to If the similarity does not reach this strict threshold (i.e. ), will produce Loss. This forces the model to bring features belonging to the same object "close enough". This represents the corresponding negative sample safety boundary, which requires that the feature similarity score of the negative sample must be less than or equal to... If the similarity exceeds this threshold (i.e. ), will produce This loss forces the model to push features of different objects "far enough" to the boundary.

[0032] The invention will now be described in further detail with reference to the overall architecture and specific implementation examples shown in the accompanying drawings.

[0033] The device used in this invention is a PC computer with a high-performance GPU for computation; the operating system is Linux or Windows, and it requires the relevant CUDA environment, Python environment and PyTorch package; the device can take pictures, such as a mobile phone.

[0034] Overall design architecture as follows Figure 3 As shown, the specific steps are as follows: Step 1: Acquisition and preprocessing of dynamic scene data.

[0035] First, video of the scene containing moving objects is captured using a data acquisition device. During filming, the device must be moved smoothly, and multiple viewing angles of the scene should be covered as much as possible to ensure the integrity of the 3D information. Then, video processing tools are used to segment the video into a frame sequence and perform quality screening, removing frames with severe motion blur. Next, the filtered image set is processed using a Structure for Motion Restoration (SfM) algorithm, such as Colmap, to extract the sparse point cloud of the scene and calculate the camera intrinsic and extrinsic parameters for each image. Using a pre-trained visual model such as SAM, zero-shot segmentation is performed on each frame to generate a corresponding 2D semantic mask map. These 2D masks will serve as supervision signals for subsequent training of the 3D semantic field.

[0036] Step 2: Model initialization and parameter setting.

[0037] This step outlines the key parameter settings for the model. The coordinates of each point in the sparse point cloud are used as the initial mean position of the Gaussian elements; the rotation, scaling, and opacity of the Gaussian elements are randomly initialized. Simultaneously, a learnable 32-dimensional semantic feature vector is initialized for each Gaussian element.

[0038] Next, we discussed the parameter settings for multi-stage training. The first stage involved 3,000 iterations, optimizing only the static Gaussian parameters. The second stage optimized dynamic attributes and semantic features, with a total of 20,000 iterations. The Adam optimizer was used to update all parameters, and the learning rate for the Gaussian attributes adopted an exponential decay strategy, gradually decreasing from 1.6e-4 to 1.6e-6.

[0039] Step 3: Construction of dynamic deformation field and fine appearance.

[0040] The mean position (x, y, z) of each Gaussian and the current timestamp t are input into the encoder. First, the values ​​are mapped to a high-dimensional space using sine coding. Then, projection, interpolation, and feature concatenation are performed through a multi-resolution plane or voxel grid to obtain a feature vector containing spatiotemporal information. The spatiotemporal features are input into a multi-head decoder to predict the change in the current time t relative to the initial state. This change is superimposed on the original Gaussian parameters to obtain the deformed Gaussian at that time. To capture subtle changes in material color, the position-encoded RGB information is received, processed by a deformable MLP, and the residual term of the color is predicted and fused with the basic spherical harmonic color to achieve dynamic high-fidelity rendering.

[0041] Step 4: While the geometry and appearance gradually converge, simultaneously optimize the semantic feature vector of Gaussian binding.

[0042] First, using the same rasterization pipeline as for color rendering, the D-dimensional semantic feature vectors of each Gaussian are blended and rendered into a 2D feature map for semantic rendering. To save GPU memory, the semantic feature map is typically rendered at a low resolution and mapped to the class space through a lightweight linear layer, followed by upsampling to the original image size using bilinear interpolation.

[0043] Then, the cross-entropy loss between the rendered semantic map and the SAM mask generated in step 1 is calculated. The boundary relaxation contrastive loss proposed in this invention is adopted. A pixel-mask correspondence matrix is ​​constructed to bring pixel features belonging to the same object closer together and push features of different objects further apart in the feature space, and a safety boundary is introduced to prevent overfitting. During the optimization process, the K nearest neighbors of each Gaussian element in 3D space are calculated periodically to force Gaussians in the neighborhood to have similar semantic features, thereby eliminating isolated noise points.

[0044] Step 5: After training is complete, the system generates a dynamic Gaussian model containing geometric, appearance, and semantic attributes, which can be used for reasoning and application.

[0045] Users can load model parameters by modifying the JSON configuration file.

[0046] During the inference phase, given any viewpoint and timestamp, the model can render a high-fidelity RGB image and the corresponding pixel-level semantic segmentation map in real time.

[0047] Users can interactively click on an object, and the 3D mask of the object will be automatically extracted based on semantic features to achieve target segmentation in dynamic scenes.

[0048] Practical application solutions such as Figure 4 As shown, image frames from different times (t1, t2) and perspectives of a dynamic scene are acquired. Two types of aligned semantically relevant data are generated synchronously for each frame. A 2D semantic feature map is obtained through a rasterization pipeline consistent with color rendering. Two pixels p1 and p2 sampled from the image correspond to a rendered semantic feature vector y1 and y2, encoding the pixel's SAM mask affiliation, serving as a supervision signal for semantic learning. The total semantic loss is obtained by summing the positive and negative sample losses. Backpropagation is used to optimize the 3D Gaussian feature field, providing accurate Gaussian modeling for subsequent target segmentation in dynamic scenes.

Claims

1. A Gaussian semantic segmentation method based on color and shape deformation fields, characterized in that, include: Simultaneously acquire image frames of a dynamic scene from multiple given perspectives; A static Gaussian model composed of standard Gaussian elements is established based on the acquired image frames, and a high-dimensional semantic feature is initialized for each standard Gaussian element. The high-dimensional semantic features contained in the static Gaussian model are rendered using a given rasterization pipeline to obtain the rendered semantic feature map of the static Gaussian model. Extract the two-dimensional semantic mask of each image frame, obtain the instances and instance masks contained in the image frame, and construct the correspondence vector between image pixels and instance masks; Based on the corresponding relationship vector and the rendered semantic feature map, comparative learning is performed to predict the shape change and color change of each standard Gaussian element in the static Gaussian model at the current time. Based on the generated shape and color changes, a dynamic Gaussian model for the current moment is obtained by rendering using the given rasterization pipeline, and then semantic segmentation of the dynamic scene is performed based on the dynamic Gaussian model.

2. The semantic segmentation method according to claim 1, characterized in that, The step of establishing a static Gaussian model composed of standard Gaussian units based on the acquired image frames includes: Extract sparse point clouds of the dynamic scene from the acquired image frames; The coordinates of each point in the sparse point cloud are used as the initial mean position of the standard Gaussian unit; Initialize the rotation, scaling, and opacity of each standard Gaussian element to obtain a static Gaussian model composed of standard Gaussian elements.

3. The semantic segmentation method according to claim 1, characterized in that, The process of rendering the high-dimensional semantic features contained in the static Gaussian model using a given rasterization pipeline to obtain the rendered semantic feature map of the static Gaussian model includes: The cumulative alpha blending value of each image pixel in an image frame is calculated using a given rasterization pipeline as the rendering semantic feature at the corresponding image pixel; The rendering semantic feature map of the static Gaussian model is generated based on the rendering semantic features at all image pixels.

4. The semantic segmentation method according to claim 3, characterized in that, The cumulative alpha value of an image pixel is calculated based on smooth Gaussian features, which are obtained by calculating the high-dimensional semantic features of a set number of neighboring image pixels in three-dimensional space.

5. The semantic segmentation method according to claim 1, characterized in that, The two-dimensional semantic mask is generated by performing zero-sample segmentation on each image frame using a pre-trained visual basic model.

6. The semantic segmentation method according to claim 1, characterized in that, The construction of the mapping vector between image pixels and instance masks includes: Randomly sample the extracted instance masks to obtain a sampled mask subset that includes a set number of instance masks; The attribution of each image pixel in the sampling mask subset is encoded to obtain the corresponding relationship vector of each image pixel.

7. The semantic segmentation method according to claim 1, characterized in that, The comparative learning based on the corresponding relation vector and the rendered semantic feature map includes: Based on the corresponding relationship vector, two image pixel pairs with the same instance mask are extracted as positive sample pairs to form a positive sample pair set, and two image pixel pairs with conflicting instance masks are extracted as negative sample pairs to form a negative sample pair set. For each positive sample pair, the positive sample term loss is calculated based on the rendering feature similarity between the corresponding two image pixels in the rendered semantic feature map; For each negative sample pair, the negative sample term loss is calculated based on the rendering feature similarity between the corresponding two image pixels in the rendered semantic feature map; The contrast loss is calculated based on the positive sample loss and the negative sample loss, and the shape change and color change of the standard Gaussian unit are updated by backpropagation.

8. The semantic segmentation method according to claim 7, characterized in that, The contrast loss is expressed as: in, This represents the contrast loss. and These represent the losses for the positive sample terms and the losses for the negative sample terms, respectively. and Let them represent the set of positive sample pairs and the set of negative sample pairs, respectively. and These represent the safe boundaries for positive samples and the safe boundaries for negative samples, respectively. This represents the similarity of rendering features between pixel i and pixel j.

9. The semantic segmentation method according to claim 1, characterized in that, The shape change is predicted using a shape deformation field, and the color change is predicted using a color deformation field.

Images