An unsupervised domain adaptation segmentation method based on wavelet transform and context association

Abstract

The application discloses an unsupervised domain adaptation segmentation method based on wavelet transform and context connection, comprising the following steps: constructing an encoder-decoder architecture semantic segmentation model, wherein the semantic segmentation model comprises a student network and a teacher network, and the student network and the teacher network share weights; obtaining a source domain image prediction value through the student network, obtaining a target domain image prediction value through the teacher network, and preliminarily optimizing the student network; taking a class with the maximum target domain image prediction value as a pseudo label of the image, and secondarily optimizing the student network based on the target domain image prediction value and the corresponding pseudo label; obtaining a target domain occlusion image based on the target domain image prediction value, and secondarily optimizing the student network based on the target domain occlusion image; and repeating the optimization process until a preset condition is met. The application improves the generalization performance of the model in the target domain and improves the prediction ability of the model for pixels in a class dense area.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to an unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships. Background Technology

[0002] Semantic segmentation is an important task in computer vision, aiming to assign each pixel in an image to a specific semantic category. Unlike other image classification tasks, semantic segmentation requires classifying each pixel in an image to achieve pixel-level understanding of the image.

[0003] In many practical applications, we often face the problem of a mismatch between the distribution of training and testing data, which leads to a decline in the performance of segmentation models in new domains. The traditional solution is to collect more labeled data to train the model; however, acquiring labeled data is often costly or even infeasible. In contrast, unsupervised domain adaptation (UDA) methods utilize labeled data from the source domain and unlabeled data from the target domain to transfer knowledge from the source domain to the target domain, thereby solving the problem of data distribution mismatch.

[0004] Current UDA methods mainly fall into two categories: adversarial training and self-training. Compared to adversarial training-based UDA, self-training-based UDA eliminates the need for adversarial networks, reducing training complexity and computational resource requirements. Furthermore, self-training-based UDA encourages cross-domain alignment while preserving more domain-specific semantic information, leading to better model generalization and making it the mainstream UDA method. Specifically, self-training-based UDA assigns a pseudo-label to each pixel in the target domain image as additional supervision based on confidence scores, and then simultaneously optimizes the network model using the source domain image, source domain label, target domain image, and target domain pseudo-label. Existing methods, lacking explicit guidance on whether to use low-frequency or high-frequency information for training, prioritize using low-frequency components of the data to reduce loss during training. Only when loss cannot be further reduced will high-frequency components be considered for training. The drawbacks of this training approach are twofold: firstly, it lacks effective utilization of high-frequency components in the early stages of training; secondly, the available useful high-frequency components become increasingly scarce in later stages, forcing the extraction of sample-level noise and high-frequency information to reduce loss, which can lead to subsequent prediction bias. In addition, due to limitations in the receptive field and imbalance in training data, existing methods need to be improved in predicting similar-looking pixels in densely populated regions of different categories. Summary of the Invention

[0005] The purpose of this invention is to provide an unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships, so as to solve the problems existing in the prior art.

[0006] To achieve the above objectives, this invention provides an unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships, comprising:

[0007] Construct an encoder-decoder architecture semantic segmentation model, which includes a student network and a teacher network, with the student network and teacher network sharing weights;

[0008] The source domain image and the target image are acquired. The source domain image is processed by the student network to obtain the predicted value of the source domain image. The target domain image is processed by the teacher network to obtain the predicted value of the target domain image.

[0009] The student network is initially optimized based on the predicted values ​​of the source domain images and their corresponding labels. The category with the largest predicted value of the target domain image is used as the pseudo label of the image. The student network is then further optimized based on the predicted values ​​of the target domain images and their corresponding pseudo labels.

[0010] The target domain occlusion image is obtained based on the target domain image prediction value, and the student network is then optimized a second time based on the target domain occlusion image;

[0011] Repeat the optimization process until the preset conditions are met, and then use the optimized model to achieve unsupervised adaptive segmentation.

[0012] Optionally, the process of constructing the feature encoder includes adding a high-low frequency separation layer, a low-frequency perturbation layer, and a high-low frequency fusion reconstruction layer after the first convolutional layer and the residual block of the original ResNet-101 encoder to obtain the feature encoder.

[0013] Optionally, the high-low frequency separation layer uses the discrete wavelet transform method to separate and obtain low-frequency features and high-frequency features;

[0014] The low-frequency perturbation layer includes a pixel attention module, which calculates attention on a pixel-by-pixel and channel-by-channel basis based on low-frequency features. After calculating the mean and variance in the channel dimension, attention weighting is used to resample from the estimated distribution to obtain low-frequency perturbation features.

[0015] The high- and low-frequency fusion reconstruction layer uses the discrete wavelet transform method to fuse and reconstruct the high-frequency features and the low-frequency perturbation features to obtain enhanced features as the input of the next layer of the feature encoder.

[0016] Optionally, the pixel attention module consists of a linear function, a ReLU activation function, a Dropout function, and a Sigmoid activation function.

[0017] Optionally, the number of the high-low frequency separation layer, the low-frequency disturbance layer, and the high-low frequency fusion reconstruction layer is not less than 1.

[0018] Optionally, the process of obtaining the target domain occlusion image includes: obtaining the prediction probability of image blocks in the target domain image based on the prediction value of the target domain image, obtaining candidate region blocks based on the prediction probability; obtaining the occlusion mask of the candidate region blocks, and obtaining the target domain occlusion image based on the occlusion mask.

[0019] Optionally, the process of obtaining candidate region blocks includes: obtaining one-hot encoding of the predicted values ​​of the target domain image; performing convolutional downsampling on the one-hot encoding; and obtaining the prediction probability of each image block in the target domain image through softmax; calculating the information entropy of the prediction probability and sorting it in descending order; and selecting a fixed proportion of image blocks as candidate region blocks based on the information entropy sorting result.

[0020] Optionally, the process of obtaining the target domain occlusion image further includes: setting transformation coefficients, processing the candidate region block through the transformation coefficients, obtaining an occlusion mask through Bernoulli distribution transformation, and overlaying the occlusion mask onto the target domain image using bilinear interpolation to obtain the target domain occlusion image.

[0021] Optionally, the student network is initially optimized by minimizing the cross-entropy loss between the predicted values ​​of the source domain image and the corresponding labels; and then further optimized by minimizing the cross-entropy loss between the predicted values ​​of the target domain image and the pseudo-labels.

[0022] Optional,

[0023] The cross-entropy loss function is expressed as:

[0024]

[0025] Among them, the probability result of segmentation Let y represent the probability that the i-th pixel belongs to class k. i Let be the label of the i-th pixel.

[0026] The technical effects of this invention are as follows:

[0027] This invention analyzes and processes data from the frequency domain perspective. Based on a thorough learning of the domain-specific low-frequency semantics corresponding to smooth blocks in both the source and target domains, it introduces specific low-frequency perturbations using channel-wise attention weighting on pixels. This forces the model to utilize existing effective supervision signals during training to focus on and learn high-frequency semantic details with high sharing and domain invariance. Ultimately, this allows the model to better understand and represent the similarities and differences between different domains, thereby improving its generalization performance in the target domain. Furthermore, this invention combines information entropy to calculate region complexity and selects occluded reconstruction blocks based on region complexity. This encourages the model to connect with more effective different contexts, improving its prediction ability for pixels in densely populated category regions and addressing the problem of poor prediction performance for similar-looking pixels in densely populated category regions in existing technologies. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the feature encoder structure in an embodiment of the present invention. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0033] Example 1

[0034] like Figure 1 As shown, this embodiment provides an unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships, including:

[0035] Dataset preparation

[0036] This embodiment is tested on the GTA5->Cityscapes semantic segmentation benchmark. The source domain (synthetic) data used is the GTA5 dataset, which contains 24,966 images with a resolution of 1914×1052; the target domain (real) data used is the Cityscapes dataset, which contains 2,975 training images and 500 validation images with a resolution of 2048×1024. The evaluation metric used is the mean inter-pixel intersection-union ratio (mIoU).

[0037] Data preprocessing

[0038] Due to the limited training data, this embodiment introduces data augmentation operations to increase the amount of training data and allow the model to be trained sufficiently. The source domain data is randomly cropped into 1280×720 image blocks, and the target domain data is randomly cropped into 1024×512 image blocks. Data augmentation methods such as Gaussian blur, grayscale transformation, brightness change, contrast change, saturation change, hue change, and random flipping are applied to expand the data volume.

[0039] Parameter settings

[0040] This embodiment uses two NVIDIA GV100 graphics cards. The hyperparameters of the model are set as follows: batch size of 4, iterations of 125,000. The DeepLab-v2 framework with ResNet-101 encoder is used as the initial segmentation model, and the optimizer is the SGD optimizer with an initial learning rate of 0.00025, momentum of 0.9, and weight decay of 0.0005.

[0041] Based on the above embodiments, a student network for final training and a teacher network for auxiliary training are constructed. The teacher network serves as a stable, slowly updated version of the student network, and both share network weights. Collaborative training is performed on these two networks using source domain images, source domain labels, target domain images, and target domain pseudo-labels.

[0042] The first step involves constructing a feature encoder sensitive to high-frequency information based on wavelet transform. This step adds a high-low frequency separation layer, a low-frequency perturbation layer, and a high-low frequency fusion reconstruction layer after the first convolutional layer and the residual block of the original ResNet-101 encoder, respectively. The structure is as follows: Figure 2 As shown.

[0043] A high-frequency information-sensitive feature encoder is constructed based on wavelet transform, including:

[0044] The encoder is based on the ResNet-101 architecture and adds a high-low frequency separation layer, a low-frequency perturbation layer, and a high-low frequency fusion and reconstruction layer after the first convolutional layer and the residual block, respectively.

[0045] The high-frequency separation layer employs the DWTForward discrete wavelet transform method from the pytorch_wavelets library, with a decomposition and reconstruction level J of 1, a wavelet function wave of 'db1', and a mode of 'zero'. The high-frequency separation layer yields approximation coefficients and detail coefficients, hereinafter referred to as the low-frequency and high-frequency components. The low-frequency component corresponds to large-scale information with flat changes in its respective domain, typically the main parts of the image such as texture, which are easier for the model to learn. The high-frequency component corresponds to small-scale detail information with high sharing and domain invariance, typically the edges (contours) and details of the image, which are easily ignored by the model during training.

[0046] The low-frequency perturbation layer contains a pixel attention module, denoted as PAM (Pixel Attention Module), which calculates the weights of each pixel and channel in the feature map. Compared to the existing Squeeze-and-Excitation (SE) attention, PAM automatically learns the weights of each channel for each feature point. These weight parameters represent the contribution of each channel to that feature point. By adjusting the channel weights, the model can selectively emphasize or weaken the representation of a particular feature point. PAM consists of a linear function, a ReLU activation function, a Dropout function, and a Sigmoid activation function. The default input and output dimensions are 64, the dimensionality reduction factor is 16, and the dropout factor is 0.5. The input to this layer is the low-frequency part of the feature. After entering, it first passes through the pixel attention module to calculate the attention weights, then calculates the mean and variance along the channel dimension, and finally uses attention weighting to resample from the estimated distribution to obtain the final perturbed low-frequency part of the feature. This process can be expressed as:

[0047]

[0048]

[0049]

[0050]

[0051] Where α∈[0,1] represents the interference intensity, H and W represent the height and width of the feature, c is the corresponding class, and F l , These represent the low-frequency features of the original, resampled, and final perturbed samples, respectively, with W being the attention weight.

[0052] The high-low frequency fusion reconstruction layer adopts the DWTInverse discrete wavelet inverse transform method in the pytorch_wavelets library, with wavelet function wave being 'db1' and mode being 'zero'. It fuses and reconstructs the low-frequency part of the features after being disturbed by the low-frequency perturbation layer with the high-frequency part of the original features, and outputs new enhanced features as the input of the next layer of the network.

[0053] The second step involves inputting the source domain image into the student network encoder, where it undergoes convolution, high- and low-frequency feature separation (64 channels), low-frequency feature perturbation (64 channels), high- and low-frequency feature fusion and reconstruction (64 channels), residual block processing, high- and low-frequency feature separation (2048 channels), low-frequency feature perturbation (2048 channels), and high- and low-frequency feature fusion and reconstruction (2048 channels) to obtain coded features, which serve as the input to the decoder. Finally, the decoder outputs the predicted value P of the source domain image. s By minimizing the source domain image prediction value P s With source domain label Y s The cross-entropy loss function is used to optimize network parameters and achieve initial optimization. The cross-entropy loss function used in this embodiment is:

[0054]

[0055] Among them, the probability result of segmentation Let y represent the probability that the i-th pixel belongs to class k. i Let be the label of the i-th pixel.

[0056] The third step involves using the teacher network to obtain the predicted values ​​of the target domain image, and then using the class with the highest predicted value for each pixel as the pseudo-label for that pixel. Specifically:

[0057]

[0058] The fourth step involves inputting the target domain image into the student network encoder, which then undergoes convolution, high- and low-frequency feature separation (64 channels), low-frequency feature perturbation (64 channels), high- and low-frequency feature fusion and reconstruction (64 channels), residual block processing, high- and low-frequency feature separation (2048 channels), low-frequency feature perturbation (2048 channels), and high- and low-frequency feature fusion and reconstruction (2048 channels) to obtain coded features, which serve as the input to the decoder. Finally, the decoder outputs the predicted value P of the target domain image. t By minimizing the target domain image prediction value P t pseudo-tags for target domain The cross-entropy loss between nodes is used to optimize network parameters and perform secondary optimization.

[0059] Step 5: Occlusion reconstruction of the target domain image. First, the predicted value P of the target domain image obtained in step 4 is... tThe image is converted into a one-hot encoding, then convolutional downsampling is performed on the one-hot encoding, followed by softmax to obtain the predicted probability of the image patch, and the information entropy is calculated from it. Sort all entropy values ​​in descending order and select the top 50% as candidate image patches. Instantiate a... A position mask M of the same size is used, with the corresponding positions of the candidate points set to 1, and then multiplied by a transformation coefficient β (where β is 0.6). A Bernoulli distribution transformation is then performed on the processed position mask M to obtain the final occlusion mask FM. FM is upsampled to the size of the target domain image using bilinear interpolation, and then element-wise multiplied with the target domain image to obtain the target domain occlusion image based on information entropy. The fourth step described above is then performed on this image to optimize the network parameters, resulting in a secondary optimization.

[0060] Based on information entropy sorting and selection, a target domain data occlusion reconstruction module is constructed, including:

[0061] This module differs from existing random occlusion techniques. Its core principle is to calculate the regional complexity of an image by computing pixel information entropy. It then sorts the regions by complexity, selecting low-complexity regions as occlusion areas and retaining high-complexity regions. This ensures that each display region retains as many categories as possible, providing richer and more varied semantics and context for model training. The model can then acquire more comprehensive information from these different contextual cues, ultimately improving its semantic understanding and reasoning abilities and better resolving pixel prediction ambiguity. This module includes a 9×9 convolutional layer with frozen weights, a stride of 8, padding of 1, and input and output dimensions equal to the total number of categories. Its main function is to downsample the one-hot encoded target domain image prediction values ​​through a convolutional layer, followed by softmax to obtain the predicted probability of image patches, and then calculate their information entropy. A higher entropy indicates that the prediction for that image patch is less accurate and contains more categories. The purpose of sorting in descending order is to select the region block with the most categories. This region is multiplied by a transformation coefficient β and subjected to a Bernoulli distribution transformation to obtain the final mask. The mask is then upsampled using bilinear interpolation and overlaid on the original target domain image to obtain the occluded image. The cross-entropy loss between the predicted value of the occluded image and the pseudo-label is calculated to realize training scenarios with multiple contexts and multiple semantics.

[0062] Step 6: Repeat the above training steps for the preset number of iterations, and output the image semantic segmentation result.

[0063] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships, characterized in that, Includes the following steps: Construct an encoder-decoder architecture semantic segmentation model, which includes a student network and a teacher network, with the student network and teacher network sharing weights; The encoder is a feature encoder based on wavelet transform that is sensitive to high-frequency information; The process of constructing the feature encoder includes adding a high-low frequency separation layer, a low-frequency perturbation layer, and a high-low frequency fusion reconstruction layer after the first convolutional layer and the residual block of the original ResNet-101 encoder to obtain the feature encoder. The high- and low-frequency separation layer uses the discrete wavelet transform method to separate low-frequency features and high-frequency features. The low-frequency perturbation layer includes a pixel attention module, which calculates attention on a pixel-by-pixel and channel-by-channel basis based on low-frequency features. After calculating the mean and variance in the channel dimension, attention weighting is used to resample from the estimated distribution to obtain low-frequency perturbation features. The high- and low-frequency fusion reconstruction layer uses the discrete wavelet inverse transform method to fuse and reconstruct the high-frequency features and the low-frequency perturbation features to obtain enhanced features as the input of the next layer of the feature encoder. The source domain image and the target image are acquired. The source domain image is processed by the student network to obtain the predicted value of the source domain image. The target domain image is processed by the teacher network to obtain the predicted value of the target domain image. The student network is initially optimized based on the predicted values ​​of the source domain images and the corresponding labels of the source domain images. The category with the largest predicted value of the target domain image is used as the pseudo label of the image. The first loss is calculated based on the predicted values ​​of the target domain images and the corresponding pseudo labels. A target domain occlusion image is obtained based on the target domain image prediction value, and a second loss is calculated based on the target domain occlusion image; Based on the first loss and the second loss, the student network is further optimized; The process of obtaining the target domain occlusion image includes: obtaining the prediction probability of image blocks in the target domain image based on the prediction value of the target domain image, obtaining candidate region blocks based on the prediction probability; obtaining the occlusion mask of the candidate region blocks, and obtaining the target domain occlusion image based on the occlusion mask; The process of obtaining candidate region blocks includes: obtaining one-hot encoding of the predicted values ​​of the target domain image; performing convolutional downsampling on the one-hot encoding; and obtaining the prediction probability of each image block in the target domain image through softmax; calculating the information entropy of the prediction probability and sorting it in descending order; and selecting a fixed proportion of image blocks as candidate region blocks based on the information entropy sorting result. Repeat the optimization process until the preset conditions are met, and then use the optimized model to achieve unsupervised adaptive segmentation.

2. The unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships according to claim 1, characterized in that, The pixel attention module consists of a linear function, a ReLU activation function, a Dropout function, and a Sigmoid activation function.

3. The unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships according to claim 1, characterized in that, The number of high-low frequency separation layers, low-frequency disturbance layers, and high-low frequency fusion reconstruction layers is not less than 1.

4. The unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships according to claim 1, characterized in that, The process of obtaining the target domain occlusion image also includes: setting transformation coefficients, processing the candidate region block through the transformation coefficients, obtaining an occlusion mask through Bernoulli distribution transformation, and upsampling the occlusion mask through bilinear interpolation onto the target domain image to obtain the target domain occlusion image.

5. The unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships according to claim 1, characterized in that, The first loss is the cross-entropy loss between the predicted value of the target domain image and the pseudo-label, and the second loss is the cross-entropy loss between the predicted value of the occluded image of the target domain and the pseudo-label.

6. The unsupervised domain adaptive segmentation method based on wavelet transform and contextual relationships according to claim 5, characterized in that, The cross-entropy loss function is expressed as: ， Among them, the probability result of segmentation , Indicates the first Each pixel belongs to The probability of a class For the first A label of 1 pixel.

Images