Domain generalization method based on self-supervised domain mapping
Through self-supervised domain mapping and encoder-decoder architecture, the domain offset problem in semantic segmentation is solved, the model's performance and generalization ability on the target domain is improved, and the dependence on the annotated data is reduced.
Patent Information
- Application Number
- CN202411797969.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-18
AI Technical Summary
The prior art has domain offset problems in semantic segmentation tasks, which leads to poor performance on unseen target domains and relying on a large amount of labeled data, making it difficult to implement in practical applications.
The domain generalization method based on self-supervised domain mapping is adopted to project the target data to the source domain through data augmentation and self-supervised learning to reduce the inter-domain gap, and style removal, style acquisition and image reconstruction are used using the encoder-decoder architecture.
Significantly improves the performance of the model on unseen target domains, especially on Cityscapes, BDD and Mapillary datasets, improves the generalization ability of the model, and reduces dependence on labeled data.
Smart Images
Figure CN119850406A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a semantic segmentation problem in domain generalization (DG), specifically a domain generalization method based on self-supervised domain mapping. Background Art
[0002] Deep learning has made breakthrough progress in the field of semantic segmentation, thanks to large-scale densely annotated training images. However, in real life, it takes a long time to obtain annotated image data for segmentation. For example, in the Cityscapes dataset provided by Cordts et al. in the work "The Cityscapes Dataset for Semantic Urban Scene Understanding.", it takes 1.5 hours to annotate an image with a resolution of 2048×1024, and even 3.3 hours to annotate images under severe weather conditions. Although synthetic data can be used for training, Richter et al. proposed in "Playing for Data: Ground Truth from Computer Games" that convolutional neural networks (CNNs) are very sensitive to domain shift.
[0003] The domain shift problem is prevalent in computer vision tasks, especially in semantic segmentation. When a model is trained on a source domain, its performance often degrades significantly if it is directly applied to an unseen target domain. To alleviate this problem, existing domain generalization methods usually learn domain-invariant features by suppressing domain-sensitive features. However, this approach may not fully utilize the data characteristics of the target domain, thus affecting the generalization ability of the model. In addition, many methods rely on a large amount of labeled data, which is often not feasible in practical applications. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a domain generalization method based on self-supervised domain mapping to reduce the gap between the source domain and the target domain, thereby improving the performance of the model in the target domain.
[0005] The technical solution of the present invention is:
[0006] A domain generalization method based on self-supervised domain mapping reduces the gap between domains by mapping data back to the source domain and improves the performance of the model on the target domain. During testing, the target data is first projected to the source domain using self-supervised source domain mapping to alleviate domain shift, and then the trained segmentation network is used to generate segmentation results.
[0007] Furthermore,
[0008] Use data augmentation strategies to generate data with different distributions, and learn a mapping function in a self-supervised manner to map the augmented data back to the original data.
[0009] The domain shift in the training phase is simulated by performing data augmentation on the source domain data.
[0010] The encoder-decoder architecture is used for image feature acquisition and image reconstruction.
[0011] in,
[0012] The encoder gradually reduces the spatial dimension of the input data through convolutional layers while increasing the depth of the feature map to capture more complex patterns;
[0013] The task of the decoder is to progressively restore the spatial dimensions of the feature maps while reducing the depth of the feature maps until the output has the same dimension as the original input or the desired dimension.
[0014] Going further,
[0015] There are three steps involved:
[0016] Step 1: Style removal: Standardize the feature maps of the image through instance normalization technology, adjust the brightness and contrast of each feature map so that their average brightness is zero and the contrast is one, thereby removing the style features in the image;
[0017] Step 2: Style acquisition: The feature map after style removal is combined with the feature map of the target style through adaptive instance normalization technology, and the brightness and contrast of the feature map are adjusted to match the characteristics of the target style, so that the image that originally only has content information acquires a new style.
[0018] Step 3: Image reconstruction, converting the processed feature map back to an image through the decoder component.
[0019] in,
[0020] The decoder receives the processed feature map as input and gradually restores the spatial details of the image through calculations, and finally outputs an image that is similar to the original image but with a new style.
[0021] The encoder can be used for downstream semantic segmentation tasks.
[0022] The beneficial effects of the present invention are
[0023] In the cross-domain semantic segmentation task, the generalization ability of the model is crucial. In order to improve the performance of the model on unseen target domains, the present invention designs a self-supervised source domain mapping (SSDP) method. The table shows the mIoU results of source domain mapping networks with different designs on tasks G to C, B and M. Specifically, the three designs of SSDP (w / AE), SSDP (w / o AdaIN) and SSDP achieved an average mIoU of 36.41%, 43.13% and 44.87%, respectively. These results show that SSDP significantly improves the average performance of the model by introducing AdaIN technology to reconstruct the original data in the feature space. The experimental results are shown in the following table. The average mIoU of the baseline model on the Cityscapes (C), BDD (B) and Mapillary (M) datasets are 28.95%, 25.12% and 28.18%, respectively, and the overall average mIoU is 27.42%. SSDP(w / AE), SSDP(w / o AdaIN) and SSDP achieved average mIoU of 36.41%, 43.13% and 44.87% respectively. These results show that SSDP significantly improves the performance of the model on unseen target domains through style removal, style acquisition and image reconstruction, especially on Cityscapes, BDD and Mapillary datasets.
[0024] The indicator used is mIoU, which is an important indicator to measure the average performance of the model on different categories:
[0025]
[0026] where TPc, FPc, and FNc represent the true positive, false positive, and false negative counts of category c, respectively, and |C| is the total number of categories.
[0027] The average mIoU of the baseline model on the Cityscapes (C), BDD (B) and Mapillary (M) datasets is 28.95%, 25.12% and 28.18%, respectively, and the overall average mIoU is 27.42%. SSDP (w / AE), SSDP (w / o AdaIN) and SSDP achieve an average mIoU of 36.41%, 43.13% and 44.87%, respectively. These results show that SSDP significantly improves the performance of the model on unseen target domains through style removal, style acquisition and image reconstruction, especially on the Cityscapes, BDD and Mapillary datasets.
[0028] BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is the overall training framework diagram of the present invention;
[0030] Figure 2 It is a schematic diagram of the self-supervised domain mapping module. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0032] The present invention proposes a method based on self-supervised source domain projection (SSDP). SSDP learns a mapping by mapping the enhanced data back to the original data, which can map the unseen target domain data to the source domain during the training phase. This process helps to reduce the data distribution differences between different domains, thereby improving the generalization ability of the model. Specifically, a data augmentation strategy is used in the training phase to generate data with different distributions, and a mapping function is learned in a self-supervised manner to map the enhanced data back to the original data. In this way, even in the absence of target domain data, the gap between domains can be effectively reduced.
[0033] The present invention designs a Self-Supervised Source Domain Projection (SSDP) module. This module learns a mapping by mapping the augmented data back to the original data, which can map the unseen target domain data to the source domain during the training phase. This process helps to reduce the data distribution differences between different domains, thereby improving the generalization ability of the model.
[0034] Specifically, this method uses a data augmentation strategy to generate data with different distributions, and learns a mapping function in a self-supervised manner to map the augmented data back to the original data. Assuming x is the original data, A(·) is the data augmentation function, and F(·) is the mapping function, this method can define the reconstruction loss as shown in formula (1):
[0035]
[0036] Among them, represents the xi-th original data sample, A(xi) represents the enhanced data sample, F(A(xi)) represents the mapped data sample, N is the number of samples, and Lrecon is the reconstruction loss, which is used to ensure that the enhanced data can be accurately mapped back to the original data.
[0037] SSDP aims to project data to the source domain to mitigate domain shift during testing. Since the target data is not available during training, this method does not adopt the paradigm of domain adaptation to directly obtain the style transfer network from the target to the source domain. This method uses a data augmentation strategy to generate data with different distributions from the source domain and projects the augmented data to its corresponding source domain original data to learn the SSDP of this method.
[0038] In this method, SSDP is described as a mapping F:T→S. Given the target data x, the goal is to make F(x) close to the source domain S. Since the target domain data is not available during training, this method simulates the domain shift in the training phase by performing data augmentation on the source domain data. This method projects the augmented data onto the original data to learn the SSDP of this method. Specifically, the SSDP design of this method is as follows: Figure 2 As shown. In this method, xa=A(x) is recorded as the enhanced data, where A(|) is the enhancement function. In this method, the original data x and the enhanced data xa are input into the encoder E of SSDP to obtain features fx and fxa.
[0039] In order to eliminate the style information in fxa, this method uses instance normalization to obtain the normalized feature f^xa, as shown in formula (2):
[0040]
[0041] Among them, μ(fxa) and σ(fxa) are the channel-level mean and standard deviation of feature fxa, respectively.
[0042] At the same time, this method calculates the channel-level standard deviation and mean of the feature fx corresponding to the input original image, which contains the style information of x as affine parameters to transform the normalized feature f^xa. This method assumes that the transformed feature f~xa contains the content information of xa and the style information of x. The specific conversion formula is as follows:
[0043]
[0044] Then, the method inputs f~xa into the decoder D to obtain the reconstructed original data ~x. Since the method only uses data augmentation to create sample pairs x and xa, the SSDP training scheme of the method can be regarded as a self-supervised approach.
[0045] Specifically, an encoder-decoder architecture is used for image feature acquisition and image reconstruction. This architecture usually consists of two parts: an encoder and a decoder. The encoder usually uses convolutional layers to extract high-level features of the input data, while the decoder uses up-convolutional layers (also called transposed convolutional layers) to reconstruct the output from these features, such as in image segmentation or image-to-image translation tasks.
[0046] The encoder gradually reduces the spatial dimension of the input data through a series of convolutional layers, while increasing the depth of the feature map to capture more complex patterns. The mathematical expression of the convolutional layer is as follows:
[0047]
[0048] Where y1 represents the output of the first layer, y L represents the output of the Lth layer, wL represents the weight, (m,n,c,f) represents the weight value of the weight matrix wL at the corresponding position, and bL represents the bias.
[0049] The task of the decoder is to gradually restore the spatial dimensions of the feature map, while possibly reducing the depth of the feature map, until the output has the same dimension as the original input or the desired dimension. The mathematical expression of the upper convolutional layer is as follows:
[0050]
[0051] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A domain generalization method based on self-supervised domain mapping, characterized in that By mapping the data back to the source domain, the gap between domains is reduced and the performance of the model on the target domain is improved. During testing, the target data is first projected to the source domain using self-supervised source domain mapping to alleviate domain shift, and then the trained segmentation network is used to generate segmentation results.
2. The method according to claim 1, characterized in that Use data augmentation strategies to generate data with different distributions, and learn a mapping function in a self-supervised manner to map the augmented data back to the original data.
3. The method according to claim 2, characterized in that The domain shift in the training phase is simulated by performing data augmentation on the source domain data.
4. The method according to claim 1, characterized in that The encoder-decoder architecture is used for image feature acquisition and image reconstruction.
5. The method according to claim 4, characterized in that The encoder gradually reduces the spatial dimension of the input data through convolutional layers while increasing the depth of the feature map to capture more complex patterns; The task of the decoder is to progressively restore the spatial dimensions of the feature maps while reducing the depth of the feature maps until the output has the same dimension as the original input or the desired dimension.
6. The method according to claim 5, characterized in that There are three steps involved: Step 1: Style removal: Standardize the feature maps of the image through instance normalization technology, adjust the brightness and contrast of each feature map so that their average brightness is zero and the contrast is one, thereby removing the style features in the image; Step 2: Style acquisition: The feature map after style removal is combined with the feature map of the target style through adaptive instance normalization technology, and the brightness and contrast of the feature map are adjusted to match the characteristics of the target style, so that the image that originally only has content information acquires a new style. Step 3: Image reconstruction, converting the processed feature map back to an image through the decoder component.
7. The method according to claim 6, characterized in that The decoder receives the processed feature map as input and gradually restores the spatial details of the image through calculations, and finally outputs an image similar to the original image but with a new style.
8. The method according to claim 7, characterized in that The encoder can be used for downstream semantic segmentation tasks.