Domain generalization semantic segmentation method and device based on intrinsic decomposition and neural rendering

Abstract

The application discloses a domain generalization semantic segmentation method and device based on internal parameter decomposition and neural rendering, and the method comprises the following steps: establishing a bidirectional conversion model, wherein the bidirectional conversion model comprises an internal parameter decomposition submodel and a neural rendering submodel, and the internal parameter decomposition submodel and the neural rendering submodel are trained; decoupling a source domain synthetic image through the internal parameter decomposition submodel to obtain an attribute set of the source domain synthetic image, augmenting the attribute set of the source domain synthetic image, combining a randomly selected text prompt, generating an enhanced synthetic image through the neural rendering submodel; and establishing a training data set in combination with the source domain synthetic image and the enhanced synthetic image, and training a semantic segmentation network model by using the training data set. The method and device provided by the application train the semantic segmentation network model by generating high-quality enhanced training data, and the cross-domain generalization performance of the trained semantic segmentation network model is significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a domain generalization semantic segmentation method and apparatus based on intrinsic parameter decomposition and neural rendering. Background Technology

[0002] Semantic segmentation, a key technology in computer vision, has broad application prospects in various fields such as autonomous driving, medical image analysis, and robot perception. Training high-performance semantic segmentation models typically relies on large-scale datasets with pixel-level precise annotations. However, acquiring such datasets is costly and time-consuming, limiting the scalability of training datasets and their coverage of diverse scenarios. Therefore, synthetic data generated using computer graphics techniques has become an important data source for training semantic segmentation models due to its ability to conveniently provide rich annotation information and relatively low generation cost.

[0003] While synthetic data facilitates model training, it inherently differs from real-world images in visual appearance, often exhibiting systematic biases in areas such as lighting conditions, texture details, object material properties, and the realism of physical interactions. This domain shift, caused by the distributional differences between the synthetic data domain (source domain) and the real data domain (target domain), significantly impacts the generalization performance and robustness of models trained on synthetic data when transferred to real-world scenes.

[0004] Existing technologies employ domain generalization techniques to address the aforementioned issues. However, in cross-domain semantic segmentation tasks that involve moving from rendered data to real-world data, relying solely on the pre-trained knowledge of large-scale visual foundational models is often insufficient to fully bridge the gap between domains. Rendered data still has limitations compared to real-world images in terms of the realism of material details, the complexity of lighting environments, and the realism of physical interactions. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a domain generalization semantic segmentation method and apparatus based on intrinsic parameter decomposition and neural rendering, which can generate high-quality augmented training datasets to train semantic segmentation network models, thereby significantly improving the cross-domain generalization performance of the trained semantic segmentation network models.

[0006] In a first aspect, embodiments of the present invention provide a domain-generalized semantic segmentation method based on intrinsic parameter decomposition and neural rendering, including:

[0007] S1. Establish a bidirectional conversion model, which includes an intrinsic parameter decomposition sub-model and a neural rendering sub-model;

[0008] The intrinsic parameter decomposition sub-model is trained, which can decouple the source domain synthetic image to obtain a physical intrinsic parameter image.

[0009] The neural rendering sub-model is trained to generate an enhanced synthetic image by combining the physical intrinsic parameter image with a guide word as a condition.

[0010] S2. The source domain synthesized image is decoupled using the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0011] The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate the enhanced synthesized image through the neural rendering sub-model;

[0012] S3. Combine the source domain synthesized image and the enhanced synthesized image to establish a training dataset, and use the training dataset to train a semantic segmentation network model;

[0013] When training the semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

[0014] Preferably, training the intrinsic parameter decomposition sub-model includes:

[0015] S101. Train the intrinsic parameter decomposition sub-model using a source domain synthesized image I with an input attribute set F. The encoder E maps the source domain synthesized image I and its attribute set F to the latent space, respectively, to obtain the image latent variable representation z. I The latent variable representation of attribute z F ;

[0016] S102. Generate the potential source domain synthetic image I using a diffusion model, and convert the attribute set F into a first noise latent variable representation through a diffusion process.

[0017] S103. Denoise the potential source domain synthesized image I using a denoising network, combined with the image latent variable representation z. I and the first noise latent variable representation Once the original latent representation corresponding to the attribute set F is obtained, the intrinsic parameter decomposition submodel training is complete.

[0018] Preferably, training the neural rendering sub-model includes:

[0019] S201. Train a neural rendering sub-model using a physically intrinsic parameter image I with an input attribute set F. Then, use encoder E to map the physically intrinsic parameter image I and its attribute set F into a latent space, obtaining the image latent variable representation z. I The latent variable representation of attribute z F ;

[0020] S202. Generate the potential physical intrinsic parameter image I using a diffusion model, and represent the latent variable z of the image through a diffusion process. I The latent variable representation of attribute z F Combined, a second noise latent variable representation z is generated. t ;

[0021] S203. Denoise the potential physical intrinsic parameter image I using a denoising network, combined with the noise latent variable representation z. t The original latent representation corresponding to the attribute set F is obtained by combining the attribute set F with the original latent representation of the attribute set F. At this point, the neural rendering sub-model has been trained.

[0022] Preferably, the source domain synthesized image is decoupled using the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0023] The enhanced synthetic image is generated by combining the attribute set of the un-augmented source domain synthesized image with the attribute set of the augmented source domain synthesized image, and incorporating the randomly selected text prompt, through the neural rendering sub-model. This process includes:

[0024] The source domain synthesized image I is processed using the intrinsic parameter decomposition submodel. syn Decoupling is performed to obtain the source domain synthesized image I syn The set of attributes F syn Its expression is as follows:

[0025] F syn ={A syn N syn ,R syn M syn ,L syn}

[0026] In the formula, A syn Let N be the albedo. syn R is the normal. syn For roughness, M syn For metallicity, L syn For illumination;

[0027] The source domain synthesized image I syn The attribute set Fsyn albedo A in syn and light L syn Augmentation is performed to obtain the augmented source domain synthesized image I. syn The attribute set F′ syn Its expression is as follows:

[0028] F′ syn ={N syn ,A′ syn ,R syn M syn ,L′ syn}

[0029] In the formula, A′ syn For the augmented albedo, L′ syn The augmented illumination;

[0030] The un-augmented source domain is used to synthesize the image I. syn The attribute set F syn Combined with the augmented source domain to form image I syn The attribute set F′ syn Combined, an enhanced synthetic image I′ is generated through the neural rendering sub-model. syn Its expression is as follows:

[0031] I' syn =Attr2RGB(N syn ,A′ syn ,R syn M syn ,L′ syn ,T prompt )

[0032] In the formula, Attr2RGB is the neural rendering sub-model, T prompt The randomly selected text prompt.

[0033] Preferably, the expression for the training dataset is:

[0034] I∈R H×W×3

[0035] In the formula, H and W are the image height and image width, respectively, and 3 is the number of RGB channels;

[0036] When the training dataset is At that time, the backbone network can output a set of feature maps, the expression of which is:

[0037] F = DINOv2(I) = {f1, f2, ..., f L}

[0038] In the formula, DINOv2 represents the backbone network based on the visual fundamental model DINOv2. This represents the feature map of the i-th layer;

[0039] Among them, H i and W i C represents the spatial size of the feature map. i L represents the number of channels in the feature map, and L represents the number of layers in the feature map.

[0040] Preferably, when training the semantic segmentation network model, a feature pyramid network is used to aggregate features at different levels in a bottom-up path. When the feature map of the k-th layer in the feature pyramid network is p... k When generating multi-scale feature representations, the expression is:

[0041] P = FPN(F) = {p2, p3, p4, p5}

[0042] In the formula, FPN is the feature pyramid network, and p2, p3, p4, and p5 represent the feature maps of layers 2 to 5 in the feature pyramid network, respectively, corresponding to specific layers of the feature pyramid network.

[0043] Secondly, embodiments of the present invention provide a domain generalization semantic segmentation apparatus based on intrinsic parameter decomposition and neural rendering, comprising:

[0044] The bidirectional conversion model building module includes:

[0045] The intrinsic parameter decomposition sub-model is trained, which can decouple the source domain synthetic image to obtain a physical intrinsic parameter image.

[0046] The neural rendering sub-model is trained to generate an enhanced synthetic image by combining the physical intrinsic parameter image with a guide word as a condition.

[0047] Attribute augmentation modules are used to include:

[0048] The source domain synthesized image is decoupled from the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0049] The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate the enhanced synthesized image through the neural rendering sub-model;

[0050] The semantic segmentation network model training module includes:

[0051] A training dataset is established by combining the source domain synthesized image and the enhanced synthesized image, and a semantic segmentation network model is trained using the training dataset.

[0052] When training the semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

[0053] Thirdly, embodiments of the present invention provide a computing device, including:

[0054] One or more processing units;

[0055] A storage unit is used to store one or more programs.

[0056] Wherein, when the one or more programs are executed by the one or more processing units, the one or more processing units perform the method described in the first aspect.

[0057] Fourthly, embodiments of the present invention provide a computer-readable storage medium having processor-executable non-volatile program code, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0058] The embodiments of the present invention bring the following beneficial effects:

[0059] The domain generalization semantic segmentation method and apparatus based on intrinsic parameter decomposition and neural rendering provided in this embodiment can solve the problems of insufficient physical realism, poor structure preservation, and easy misalignment of semantic information with labels in the existing technology when using synthetic data for domain generalization semantic segmentation.

[0060] The method and apparatus provided in this invention, by delving into the underlying physical principles of rendering data, first decompose the intrinsic parameters of the synthesized image using an intrinsic parameter decomposition sub-model, then perform high-quality rendering using a neural rendering sub-model, and then obtain an enhanced synthesized image by targeting the attribute set of the synthesized image. Next, a training dataset is obtained by combining the synthesized image and the enhanced synthesized image. This training dataset not only accurately preserves the original semantic structure, but also more closely approximates the distribution of the real world and is more diverse in key physical appearances such as lighting and materials. The trained semantic segmentation network model significantly improves the generalization performance and robustness from the synthesized data domain to the unknown real data domain.

[0061] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0063] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering provided in an embodiment of the present invention.

[0065] Figure 2 A visual comparison diagram of the semantic segmentation results of the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering provided in the embodiments of the present invention.

[0066] Figure 3 This is a schematic diagram of the domain generalization semantic segmentation device based on intrinsic parameter decomposition and neural rendering provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] To facilitate understanding of this embodiment, in conjunction with Figure 1 This paper provides a detailed description of the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering disclosed in the embodiments of the present invention.

[0069] Example 1

[0070] Domain-generalized semantic segmentation methods based on intrinsic parameter decomposition and neural rendering include:

[0071] S1. Establish a bidirectional conversion model, which includes an intrinsic parameter decomposition sub-model and a neural rendering sub-model.

[0072] Train the intrinsic parameter decomposition sub-model, which can decouple the source domain synthetic image to obtain the physical intrinsic parameter image;

[0073] The neural rendering sub-model is trained to generate enhanced synthetic images by combining physical intrinsic parameter images with guide words as conditions.

[0074] In this embodiment, both the intrinsic parameter decomposition sub-model and the neural rendering sub-model are established by fine-tuning the pre-trained generative model StableDiffusion 2.1, and both can utilize its diffusion process for fine-grained condition generation and inverse inference.

[0075] In S1, the preferred training intrinsic parameter decomposition sub-model includes:

[0076] S101. Train the intrinsic parameter decomposition sub-model using a source domain synthesized image I with input attribute set F. The encoder E maps the source domain synthesized image I and its attribute set F to the latent space, respectively, to obtain the image latent variable representation z. I The latent variable representation of attribute z F .

[0077] S102. A potential source domain synthetic image I is generated using a diffusion model, and the attribute set F is converted into a first noise latent variable representation through a diffusion process.

[0078] S103. Denoise the latent source domain synthesized image I using a denoising network, combined with the image latent variable representation z. I and the first noise latent variable representation Once the original latent representation corresponding to the attribute set F is obtained, the intrinsic parameter decomposition submodel training is complete.

[0079] In the above steps, in S101, the expression for the attribute set F of the source domain synthesized image I is:

[0080] F∈{A,N,R,M,L}

[0081] In the formula, A is the albedo, N is the normal, R is the roughness, M is the metallicity, and L is the lighting.

[0082] Image latent variable representation z I The latent variable representation of attribute z F The expressions are as follows:

[0083] z I =E(I)

[0084] z F =E(F)

[0085] In S102, the attribute set F is converted into a noise latent variable representation through a diffusion process. The expression is:

[0086]

[0087] In the formula, Let t be the attenuation factor during the diffusion process, and t be the diffusion step. It is Gaussian noise.

[0088] In S103, the denoising network is optimized using v-prediction loss.

[0089] The purpose of the above steps is to enable the intrinsic parameter decomposition sub-model to learn the complex mapping relationship of inferring underlying physical properties from visual appearance by training the intrinsic parameter decomposition sub-model.

[0090] After the intrinsic parameter decomposition submodel is trained, a new source domain synthetic image is input into the intrinsic parameter decomposition submodel, conditioned on text prompts with target attributes. The VAE decoder obtains the physical intrinsic parameter image, and each attribute in the attribute set F of the physical intrinsic parameter image is back-denoised and sampled to output the latent variable representation of the attributes of the physical intrinsic parameter image.

[0091] In S1, the preferred training of the neural rendering sub-model includes:

[0092] S201. Train the neural rendering sub-model using a physically intrinsic parameter image I with an input attribute set F. The encoder E maps the physically intrinsic parameter image I and its attribute set F to the latent space, respectively, to obtain the image latent variable representation z. I The latent variable representation of attribute z F .

[0093] S202. Generate a latent physical intrinsic parameter image I using a diffusion model, and represent the latent variables z of the image through a diffusion process. I The latent variable representation of attribute z F Combined, a second noise latent variable representation z is generated. t .

[0094] S203. Denoise the latent physical intrinsic parameter image I using a denoising network, combined with the noise latent variable representation z. t By combining the attribute set F, we obtain the original latent representation corresponding to the attribute set F. At this point, the neural rendering sub-model training is complete.

[0095] In the above steps, in S201, the expression for the attribute set F of the physical intrinsic parameter image I is:

[0096] F∈{A,N,R,M,L}

[0097] In the formula, A is the albedo, N is the normal, R is the roughness, M is the metallicity, and L is the illumination. The latent variable of the image represents z. I The latent variable representation of attribute z F The expressions are as follows:

[0098] z I =E(I)

[0099] z F =E(F)

[0100] In S202, the latent variable z of the image is represented by a diffusion process. I The latent variable representation of attribute z F Combined, a second noise latent variable representation z is generated. t The expression is:

[0101]

[0102] In S203, the denoising network is optimized using v-prediction loss.

[0103] The purpose of the above steps is that the trained neural rendering sub-model can generate enhanced synthetic images that can both retain the structural features of the original image and adjust the image style and environmental factors according to attributes.

[0104] Once the neural rendering sub-model is trained, it is input into the physical intrinsic parameter image using a guide word as a condition. Combined with the attribute set F, the VAE decoder can obtain an enhanced synthetic image and output the latent variable representation of the attributes of the enhanced synthetic image.

[0105] The guide words are generated based on the multimodal large model GPT-4o. For example, the guide words include, but are not limited to: "sunny day", "soft light", "city road", "rows of palm trees" and "yellow road markings".

[0106] S2. The source domain synthesized image is decoupled by the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0107] The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate an enhanced synthesized image through a neural rendering sub-model.

[0108] In S2, preferably, the source domain synthesized image is decoupled by an intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0109] The attribute set of the unaugmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate an enhanced synthesized image through a neural rendering sub-model, including:

[0110] The source domain synthesized image I is obtained through an intrinsic parameter decomposition submodel. syn Decoupling is performed to obtain the source domain synthesized image I syn The set of attributes F syn Its expression is as follows:

[0111] F syn ={A syn N syn ,R syn M syn ,L syn}

[0112] In the formula, A syn Let N be the albedo. syn R is the normal. syn For roughness, M syn For metallicity, L syn For illumination;

[0113] Synthesized image I from source domain syn The set of attributes F syn albedo A in syn and light L syn Augmentation is performed to obtain the augmented source domain synthesized image I. syn The attribute set F′ syn Its expression is as follows:

[0114] F′ syn ={N syn ,A′ syn ,R syn M syn ,L′ syn}

[0115] In the formula, A′ syn L′ represents the augmented albedo. syn For the enhanced lighting;

[0116] Synthesize the unaugmented source domain into an image I syn The set of attributes F syn Image I synthesized with the augmented source domain syn The attribute set F′ syn Combining, an enhanced synthetic image I′ is generated through a neural rendering sub-model. syn Its expression is as follows:

[0117] I' syn =Attr2RGB(N syn ,A′syn ,R syn M syn ,L′ syn ,T prompt )

[0118] In the formula, Attr2RGB is the neural rendering sub-model, T prompt This is a randomly selected text prompt.

[0119] In this embodiment, the randomly selected text prompts include: "night", "sunny day", "rainy day", "foggy day", and "snowy day".

[0120] In this embodiment, the attribute set F of the source domain synthesized image is... syn albedo A in syn and light L syn To augment, while for the normal N syn Roughness R syn and metallicity M syn The reasons for not augmenting are as follows:

[0121] Albedo A syn Differences may exist across different datasets composed of source-domain synthesized images. To account for the matching of augmented data with the target domain (e.g., Cityscapes), adjustments are made via color correction; illumination L syn This also has a significant impact on image quality, so random color shift and random gamma correction are used for adjustment.

[0122] And the normal N syn The image exhibits relatively consistent geometric detail features in both the synthetic and real images, therefore no augmentation is performed; for roughness R... syn and metallicity M syn Since there is limited relevant data for the two classes mentioned above during the decomposer training, the decomposer cannot accurately estimate the two attributes mentioned above, and therefore no augmentation is performed.

[0123] The purpose of step S2 is to simulate changes under different environments and conditions by preserving and augmenting the different physical material properties of the source domain synthetic image, thereby generating diverse enhanced synthetic images that can better reflect changes in real scenes.

[0124] S3. Combine source domain synthesized images and enhanced synthesized images to establish a training dataset, and use the training dataset to train a semantic segmentation network model.

[0125] When training a semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

[0126] In this embodiment, the source domain synthesized image carries a corresponding semantic segmentation label, and the enhanced synthesized image directly reuses the semantic segmentation label corresponding to the source domain synthesized image;

[0127] The semantic segmentation network model is a semantic segmentation network architecture based on a visual base model; the backbone network comes from the self-supervised learning-based visual base model DINOv2, which has a strong cross-domain feature representation capability; the specific decoder is the semantic segmentation decoder Mask2Former, which can efficiently process multi-scale feature information and generate high-quality segmentation masks.

[0128] In S3, the preferred expression for the training dataset is:

[0129] I∈R H×W×3

[0130] In the formula, H and W are the image height and image width, respectively, and 3 is the number of RGB channels;

[0131] When the training dataset is At that time, the backbone network can output a set of feature maps, the expression of which is:

[0132] F = DINOv2(I) = {f1, f2, ..., f L}

[0133] In the formula, DINOv2 represents the backbone network based on the visual fundamental model DINOv2. This represents the feature map of the i-th layer;

[0134] Among them, H i and W i C represents the spatial dimensions of the feature map. i L represents the number of channels in the feature map, and L represents the number of layers in the feature map.

[0135] In this embodiment, as the network depth increases, the spatial resolution of the feature map gradually decreases, while the number of channels increases.

[0136] In S3, preferably, when training the semantic segmentation network model, a feature pyramid network is used to aggregate features at different levels in a bottom-up path. When the feature map of the k-th layer in the feature pyramid network is p k When generating multi-scale feature representations, the expression is:

[0137] P = FPN(F) = {p2, p3, p4, p5}

[0138] In the formula, FPN is the feature pyramid network, and p2, p3, p4, and p5 represent the feature maps of layers 2 to 5 in the feature pyramid network, respectively, corresponding to specific layers of the feature pyramid network.

[0139] In this embodiment, the Feature Pyramid Network (FPN) is suitable for semantic segmentation tasks. Its role is to combine deep features rich in semantic information and shallow features rich in spatial details, thereby enhancing the semantic segmentation network model's ability to segment objects of different sizes.

[0140] The feature map p of the k-th layer in the feature pyramid network k This can be expressed as:

[0141] p k ∈R H / k×W / k×C

[0142] In the formula, C is the number of feature channels that are unified across all pyramid level feature maps after 1x1 convolution adjustment.

[0143] The purpose of step S3 is to combine source domain synthesized images and enhanced synthesized images to establish a high-quality training dataset for training the semantic segmentation network model. By leveraging the powerful feature extraction capabilities of the semantic segmentation network model and through a carefully designed network architecture and training strategy, the trained semantic segmentation network model can be successfully generalized to real-world data scenarios, demonstrating excellent segmentation performance and robustness.

[0144] The experimental data results of the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering provided in this embodiment are shown in Table 1.

[0145] Table 1. Experimental data results of the domain generalization semantic segmentation method.

[0146]

[0147] Referring to Table 1, on the Cityscapes dataset, the method provided in this embodiment achieves an average Mean Intersection over Union (mIoU) of 64.57%, significantly higher than existing representative domain generalization methods, such as the ResNet-based SPC method (mIoU of 44.10%) and the Mask2Former-based CMFormer method (mIoU of 55.31%). On the BDD100K dataset, the method provided in this embodiment achieves an mIoU of 56.61%, also far exceeding the SPC method (mIoU of 40.46%) and the CMFormer method (mIoU of 49.91%). On the Mapillary dataset, the method provided in this embodiment achieves an mIoU of 64.51%, also significantly better than the SPC method (mIoU of 45.51%) and the CMFormer method (mIoU of 60.09%).

[0148] Compared to state-of-the-art methods that also employ diffusion models for data augmentation, such as the DGInStyle and DIDEX methods, the method presented in this embodiment achieves superior mIoU performance on all the aforementioned target domain datasets: Cityscapes, BDD100K, and Mapillary. For example, on the Cityscapes dataset, the mIoU of the method presented in this embodiment is 2.57% higher than that of the DIDEX method; on the BDD100K dataset, the mIoU of the method presented in this embodiment is 2.31% higher than that of the DIDEX method; and on the Mapillary dataset, the mIoU of the method presented in this embodiment is 1.51% higher than that of the DIDEX method.

[0149] Based on the above data, it can be concluded that the average mIoU of the method provided in this embodiment is approximately 2.13% higher than that of the DIDEX method. These significant performance gains strongly demonstrate that the proposed strategy based on intrinsic parameter decomposition and neural rendering can more effectively bridge the gap between the synthetic and real domains, and generate higher-quality training data that is more conducive to generalization learning, compared to global style transfer methods.

[0150] In addition to mIoU as a quantitative metric, model performance and robustness can be intuitively evaluated by visualizing semantic segmentation results, especially in challenging scenarios. On images of the standard Cityscapes validation set, the method presented in this embodiment demonstrates high-quality segmentation results, such as... Figure 2 As shown.

[0151] Figure 2 This embodiment demonstrates the superior robustness of the trained semantic segmentation network model in handling complex lighting and environmental changes. For example, in situations with strong shadow occlusion (such as shadows cast by buildings or trees) or glaring sunlight directly illuminating highlight areas (such as the high contrast between the road surface in the first row, or the contrast between shadows and bright areas around pedestrians in the fifth row), many existing methods easily produce incorrect category predictions or unclear boundaries. In contrast, the trained semantic segmentation network model, i.e. Figure 2 The proposed method can reliably identify and accurately segment objects within these regions, with no obvious missegmentation observed. This is thanks to the training data generated by diverse lighting conditions with different lighting parameters, which allows the model to learn the essential invariance to changes in lighting, rather than just changes in surface brightness.

[0152] Furthermore, the trained semantic segmentation network model exhibits stronger adaptability and consistency in handling subtle differences in material textures and object occlusion. For example, in tasks requiring fine differentiation of visually similar regions (e.g., Figure 2For all the car hoods at the bottom of the image and the adjacent road surface, or the intersection of the shoulder and the road surface in the second row, the trained semantic segmentation network model can achieve more accurate boundary division. Furthermore, in scenarios where objects are partially occluded (e.g., the bench on the left side of the fourth row is partially occluded), the trained semantic segmentation network model can better maintain consistency in recognizing the overall category of the object, rather than misclassifying the occluded portion. These improvements not only benefit from the effective augmentation of intrinsic material parameters but also demonstrate the importance of structure preservation during data augmentation.

[0153] It is worth noting that the trained semantic segmentation network model also performs exceptionally well in preserving details of small objects. Even difficult-to-discern targets such as distant pedestrians, bicycles, or complex traffic signs can be clearly identified and segmented. This advantage stems from two factors: firstly, the high-quality, high-resolution augmented training data generated by the neural rendering module ensures detailed information; secondly, it is attributed to the synergistic effect of the carefully designed network architecture. The frozen backbone network DINOv2 provides powerful, cross-domain robust low-level feature representations, while the specially fine-tuned Mask2Former decoder and feature pyramid network effectively utilize these features and are optimized to adapt to various changes introduced by the augmentation of intrinsic parameters, ultimately decoding rich feature information into fine and accurate pixel-level segmentation masks.

[0154] Example 2

[0155] Combination Figure 3 The domain generalization semantic segmentation training device based on intrinsic parameter decomposition and neural rendering disclosed in this invention includes:

[0156] The bidirectional conversion model building module includes:

[0157] A bidirectional transformation model is established, which includes an intrinsic parameter decomposition sub-model and a neural rendering sub-model.

[0158] Train the intrinsic parameter decomposition sub-model, which can decouple the source domain synthetic image to obtain the physical intrinsic parameter image;

[0159] The neural rendering sub-model is trained to generate enhanced synthetic images by combining physical intrinsic parameter images with guide words as conditions.

[0160] Attribute augmentation modules are used to include:

[0161] The source domain synthesized image is decoupled by an intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented.

[0162] The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate an enhanced synthesized image through a neural rendering sub-model.

[0163] The semantic segmentation network model training module includes:

[0164] A training dataset is established by combining source domain synthetic images and enhanced synthetic images, and a semantic segmentation network model is trained using the training dataset.

[0165] When training a semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

[0166] Example 3

[0167] A computing device disclosed in an embodiment of the present invention includes:

[0168] One or more processing units;

[0169] A storage unit is used to store one or more programs.

[0170] Wherein, when one or more programs are executed by one or more processing units, the one or more processing units are able to execute all the contents of the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering disclosed in Embodiment 1 of this application.

[0171] Example 4

[0172] The present invention discloses a computer-readable storage medium having processor-executable non-volatile program code.

[0173] When the computer program is executed by the processor, it can implement all the contents of the domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering disclosed in Embodiment 1 of this application.

[0174] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0175] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A domain-generalized semantic segmentation method based on intrinsic parameter decomposition and neural rendering, characterized in that, include: S1. Establish a bidirectional conversion model, which includes an intrinsic parameter decomposition sub-model and a neural rendering sub-model; The intrinsic parameter decomposition sub-model is trained, which can decouple the source domain synthetic image to obtain a physical intrinsic parameter image. The neural rendering sub-model is trained to generate an enhanced synthetic image by combining the physical intrinsic parameter image with a guide word as a condition. The guide words are generated based on the multimodal large model GPT-4o, and the guide words include: "sunny day", "soft light", "city road", "rows of palm trees", and "yellow road markings"; S2. The source domain synthesized image is decoupled using the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented. The enhanced synthetic image is generated by combining the attribute set of the un-augmented source domain synthesized image with the attribute set of the augmented source domain synthesized image, and incorporating randomly selected text prompts, through the neural rendering sub-model. This process includes: The source domain synthesized image is obtained through the intrinsic parameter decomposition submodel. Decoupling is performed to obtain the source domain synthesized image. attribute set Its expression is as follows: ； In the formula, Albedo, For normal line, For roughness, For metallicity, For illumination; The source domain synthesized image The set of attributes albedo in and light Augmentation is performed to obtain the augmented source domain synthesized image. The set of attributes Its expression is as follows: ； In the formula, The augmented albedo, The augmented illumination; Synthesize the un-augmented source domain image The set of attributes Synthesized image with the augmented source domain attribute set Combined, enhanced synthetic images are generated through the neural rendering sub-model. Its expression is as follows: ； In the formula, For the neural rendering sub-model, The randomly selected text prompt; The randomly selected text prompts include: "night", "sunny day", "rainy day", "foggy day", and "snowy day"; The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate the enhanced synthesized image through the neural rendering sub-model; S3. Combine the source domain synthesized image and the enhanced synthesized image to establish a training dataset, and use the training dataset to train a semantic segmentation network model; When training the semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

2. The domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering according to claim 1, characterized in that, Training the intrinsic parameter decomposition sub-model includes: S101. Train the intrinsic parameter decomposition sub-model using a source domain synthesized image I with input attribute set F. The encoder E maps the source domain synthesized image I and its attribute set F to the latent space, respectively, to obtain the image latent variable representation. and attribute latent variable representation ; S102. Generate the potential source domain synthetic image I using a diffusion model, and convert the attribute set F into a first noise latent variable representation through a diffusion process. ; S103. Denoise the potential source domain synthesized image I using a denoising network, combined with the image latent variable representation. and the first noise latent variable representation The original latent representation corresponding to the attribute set F is obtained, and at this point the intrinsic parameter decomposition submodel training is complete.

3. The domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering according to claim 1, characterized in that, Training the neural rendering sub-model includes: S201. Train a neural rendering sub-model using a physically intrinsic parameter image I with an input attribute set F. Then, use encoder E to map the physically intrinsic parameter image I and its attribute set F into a latent space, respectively, to obtain the image latent variable representation. and attribute latent variable representation ; S202. Generate the latent physical intrinsic parameter image I using a diffusion model, and represent the latent variables of the image through a diffusion process. and attribute latent variable representation Combined, a second noise latent variable representation is generated. ; S203. Denoise the potential physical intrinsic parameter image I using a denoising network, combined with the second noise latent variable representation. The original latent representation corresponding to the attribute set F is obtained by combining the attribute set F with the original latent representation of the attribute set F. At this point, the neural rendering sub-model has been trained.

4. The domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering according to claim 1, characterized in that, The expression for the training dataset is: ； In the formula, H and W are the image height and image width, respectively, and 3 is the number of RGB channels; When the training dataset is At that time, the backbone network can output a set of feature maps, the expression of which is: ； In the formula, This represents the backbone network based on the visual fundamental model DINOv2. This represents the feature map of the i-th layer; in, and The spatial dimensions of the feature map. The number of feature map channels. The number of layers in the feature map.

5. The domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering according to claim 1, characterized in that, When training the semantic segmentation network model, a feature pyramid network is used to aggregate features at different levels in a bottom-up path. When the feature map of the k-th layer in the feature pyramid network is... When generating multi-scale feature representations, the expression is: ； In the formula, FPN is the Feature Pyramid Network. These represent the feature maps of layers 2 through 5 in the feature pyramid network, corresponding to specific layers of the feature pyramid network.

6. An apparatus for a domain generalization semantic segmentation method based on intrinsic parameter decomposition and neural rendering according to claim 1, characterized in that, include: The bidirectional conversion model building module includes: The intrinsic parameter decomposition sub-model is trained, which can decouple the source domain synthetic image to obtain a physical intrinsic parameter image. The neural rendering sub-model is trained to generate an enhanced synthetic image by combining the physical intrinsic parameter image with a guide word as a condition. Attribute augmentation modules are used to include: The source domain synthesized image is decoupled from the intrinsic parameter decomposition sub-model to obtain the attribute set of the source domain synthesized image, and the attribute set of the source domain synthesized image is augmented. The attribute set of the un-augmented source domain synthesized image is combined with the attribute set of the augmented source domain synthesized image, and combined with randomly selected text prompts, to generate the enhanced synthesized image through the neural rendering sub-model; The semantic segmentation network model training module includes: A training dataset is established by combining the source domain synthesized image and the enhanced synthesized image, and a semantic segmentation network model is trained using the training dataset. When training the semantic segmentation network model, the weight parameters of the backbone network in the semantic segmentation network model are frozen, and end-to-end fine-tuning is performed only on specific decoders.

7. An electronic device, characterized in that, It includes a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the steps of the method as described in any one of claims 1 to 5 by invoking the machine-readable instructions.

8. A computer-readable storage medium having processor-executable non-volatile program code, characterized in that, When the program code is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images