Self-adaptive method, system and equipment during testing for cross-domain semantic segmentation of remote sensing image and medium

Abstract

The invention discloses a remote sensing image cross-domain semantic segmentation-oriented adaptive method, system and device during testing and a medium. The method comprises the following steps: acquiring an initial semantic segmentation model in a source domain data set and a target domain image; initializing a teacher network and a student network based on the initial semantic segmentation model; predicting the target domain image through a teacher network to generate a high-confidence pseudo tag; applying spatial disturbance to the target domain image, predicting the target domain image before and after disturbance through a student network, and calculating spatial consistency loss; updating student network parameters; and updating the teacher network through the updated student network parameters until a final semantic segmentation model is obtained. According to the method, the adaptive capability of cross-domain semantic segmentation of the remote sensing image is improved through a high-confidence false label generation mechanism and space consistency loss training.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and in particular to a test-time adaptive method, system, device and medium for cross-domain semantic segmentation of remote sensing images. Background Technology

[0002] Semantic segmentation technology is crucial in remote sensing image analysis. However, existing models trained on labeled training sets often experience significant performance degradation during practical deployment due to the distribution discrepancy between the training set (source domain) and the actual application scenario (target domain). Test-Time Adaptation (TTA) aims to improve the model's adaptability to dynamically changing environments. Unlike Unsupervised Domain Adaptation (UDA), it cannot obtain labeled source domain data when adjusting model parameters; it only allows viewing the dynamically changing target domain without labeled information, making implementation challenging.

[0003] Among existing methods, the Tent method was the first to propose TTA and apply it to image classification. It minimizes the target domain prediction entropy by adjusting the affine parameters of the Batch Normalization (BN) layer, which is effective in image classification tasks. However, its performance improvement is limited when applied to semantic segmentation of remote sensing images. The reasons include: semantic segmentation prediction is dense, and simple entropy minimization can easily lead to model collapse; fine-tuning only the BN layer results in poor adaptability to Transformer architecture models (which have fewer BN layers). Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a test-time adaptive method, system, device, and medium for cross-domain semantic segmentation of remote sensing images to address the problems of dense semantic segmentation prediction, easy model collapse due to simple entropy minimization, and poor adaptability to Transformer architecture models due to only fine-tuning the BN layer.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a test-time adaptive method for cross-domain semantic segmentation of remote sensing images, comprising the following steps: acquiring an initial semantic segmentation model in a source domain dataset and a target domain image; initializing a teacher network and a student network based on the initial semantic segmentation model; predicting the target domain image using the teacher network to generate high-confidence pseudo-labels; applying spatial perturbation to the target domain image, predicting the target domain image before and after the perturbation using the student network, and calculating the spatial consistency loss; updating the student network parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the prediction results, as well as the spatial consistency loss; updating the teacher network using the updated student network parameters until a final semantic segmentation model is obtained for semantic segmentation prediction of the target domain remote sensing image.

[0007] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the step of generating the high-confidence pseudo-label includes: the teacher network predicts each pixel in the target domain image to obtain the predicted probability of each category; for each pixel, the category with the highest probability and the category with the second highest probability are determined from the predicted probabilities of each category; the probability difference between the category with the highest probability and the category with the second highest probability is calculated; when the probability difference is greater than a preset threshold, the category with the highest probability of the pixel is taken as the pseudo-label; and the filtered pseudo-label is taken as the high-confidence pseudo-label.

[0008] The beneficial effects of this preferred technical solution are: it can select high-confidence pixels far from the decision boundary as pseudo-labels, and at the same time correctly filter points falling near the decision boundary, thereby increasing the number and accuracy of pseudo-labels and avoiding the initial semantic segmentation model learning noisy features.

[0009] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the step of applying spatial perturbation to the target domain image and calculating spatial consistency loss includes: performing at least one spatial transformation operation, such as rotation, cropping, or scaling, on the target domain image to obtain a spatially perturbated target domain image; a student network predicting the target domain image before spatial perturbation to obtain a prediction result before perturbation; a student network predicting the target domain image after spatial perturbation to obtain a prediction result after perturbation; and calculating the difference between the prediction result before perturbation and the prediction result after perturbation to obtain the spatial consistency loss.

[0010] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the step of calculating the difference between the prediction result before perturbation and the prediction result after perturbation includes: performing an inverse spatial transformation on the prediction result after perturbation to align it with the prediction result before perturbation in spatial location; calculating the prediction probability difference between the aligned prediction result after perturbation and the prediction result before perturbation pixel by pixel; and summing the prediction probability differences of all pixels to obtain the spatial consistency loss.

[0011] The beneficial effects of this preferred technical solution are: it enhances the robustness of the initial semantic segmentation model through spatial perturbation, while avoiding the negative impact that spectral perturbation may bring, ensuring the consistency of the initial semantic segmentation model's predictions before and after perturbation, and improving the generalization ability in the target domain.

[0012] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the step of updating the student network parameters includes: the student network predicting the target domain image to obtain the student network prediction result; calculating the cross-entropy loss between the student network prediction result and the high-confidence pseudo-label; adding the cross-entropy loss to the spatial consistency loss to obtain the total loss; and updating the student network parameters through backpropagation using the total loss.

[0013] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the step of updating the teacher network includes: obtaining the updated student network parameters and the teacher network; performing a weighted average of the updated student network parameters and the current teacher network according to a preset smoothing coefficient; and using the result of the weighted average as the updated teacher network.

[0014] The beneficial effects of this preferred technical solution are: by adjusting the network parameters, it is compatible with model architectures such as Transformer with few or no BN layers.

[0015] As a preferred embodiment of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images described in this invention, the steps of initializing the teacher network and student network based on the initial semantic segmentation model include: obtaining the network parameters of the initial semantic segmentation model; copying the network parameters of the initial semantic segmentation model to the teacher network to complete the initialization of the teacher network; and copying the network parameters of the initial semantic segmentation model to the student network to complete the initialization of the student network.

[0016] In a second aspect, the present invention provides a test-time adaptive system for cross-domain semantic segmentation of remote sensing images, comprising: an initial model acquisition module for acquiring an initial semantic segmentation model trained on a source domain dataset and a target domain image; The network initialization module is used to initialize the teacher network and student network based on the initial semantic segmentation model. The pseudo-label generation module is used to predict the target domain image through the teacher network and generate high-confidence pseudo-labels; The consistency loss calculation module is used to apply spatial perturbation to the target domain image, predict the target domain image before and after the perturbation through the student network, and calculate the spatial consistency loss. The student network update module is used to update the student network parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the student network prediction results, as well as the spatial consistency loss. The teacher network update module is used to update the teacher network based on the updated student network parameters. The prediction output module is used to perform semantic segmentation prediction on remote sensing images of the target domain using the final semantic segmentation model.

[0017] Thirdly, the present invention provides an electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of a test-time adaptive method for cross-domain semantic segmentation of remote sensing images.

[0018] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: By employing a high-confidence pseudo-label generation mechanism and spatial consistency loss training, this invention enhances the adaptive capability of cross-domain semantic segmentation of remote sensing images. Specifically, this invention uses the probability difference between the highest-probability category and the second-highest-probability category as the pseudo-label selection condition. This allows for the selection of high-confidence pixels far from the decision boundary while correctly selecting pixels falling near the decision boundary, avoiding the problem of misselecting boundary noise in existing fixed-threshold methods and improving the quantity and accuracy of pseudo-labels. Furthermore, this invention, tailored to the characteristics of remote sensing images, applies spatial perturbation to the target domain image for consistency training. By constraining the spatial consistency of prediction results before and after perturbation, the robustness and generalization ability of the model are enhanced, avoiding the potential negative impact of spectral perturbation.

[0020] This invention employs an adaptive framework of teacher and student networks, supporting fine-tuning of all network parameters. This overcomes the limitation of existing methods that only adjust batch normalization layer parameters, making this method compatible with advanced model architectures such as Transformer with or without batch normalization layers. By updating teacher network parameters using an exponential moving average method, smooth parameter updates are achieved, further improving the model's training stability and robustness. Furthermore, for cross-domain scenarios with band differences, this invention can combine generative adversarial networks to translate images from the source domain dataset, generating a transfer dataset with a style consistent with the target domain images. This effectively reduces inter-domain differences, thereby enhancing the model's adaptability in cross-band tasks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a diagram illustrating the overall structure of a test-time adaptive method for cross-domain semantic segmentation of remote sensing images, as described in one embodiment of the present invention.

[0023] Figure 2 This is a qualitative result diagram of cross-domain semantic segmentation of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images according to an embodiment of the present invention. Detailed Implementation

[0024] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0025] Example 1, referring to Figure 1 As an embodiment of the present invention, a test-time adaptive method for cross-domain semantic segmentation of remote sensing images is provided, comprising the following steps S100~S600: S100: Obtain the initial semantic segmentation model from the source domain dataset and the target domain image.

[0026] S200. Initialize the teacher network and student network based on the initial semantic segmentation model.

[0027] S300. The target domain image is predicted through the teacher network to generate high-confidence pseudo-labels.

[0028] S400. Apply spatial perturbation to the target domain image, predict the target domain image before and after the perturbation through the student network, and calculate the spatial consistency loss.

[0029] S500: The student network updates its parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the prediction results, as well as the spatial consistency loss.

[0030] S600: Update the teacher network by updating the student network parameters until the final semantic segmentation model is obtained, which is used to perform semantic segmentation prediction on remote sensing images of the target domain.

[0031] It should be noted that in practical deployments, the performance of remote sensing image semantic segmentation models often degrades due to the distribution differences between the source domain dataset and the target domain application scenario. During testing, adaptive methods cannot obtain labeled source domain datasets when adjusting model parameters, and can only optimize the model based on unlabeled target domain images, which is difficult to implement. Existing Tent methods minimize prediction entropy by adjusting batch normalization layer parameters, but when applied to remote sensing image semantic segmentation, due to dense predictions and only fine-tuning the batch normalization layer, they have poor adaptability to models with fewer batch normalization layers, such as Transformers, resulting in limited performance improvement. In traditional teacher-student frameworks, the fixed-threshold pseudo-label filtering mechanism is prone to misselecting noisy category boundary points and does not differentiate and optimize for the spectral and spatial characteristics of remote sensing images. Furthermore, the inter-domain differences caused by different band combinations in remote sensing images have not yet been effectively addressed.

[0032] Therefore, to address the aforementioned issues of cross-domain adaptation and pseudo-label quality control, the following steps (S100-S600) are employed to construct an adaptive framework for the teacher and student networks. A probability difference filtering mechanism is used to generate high-confidence pseudo-labels, avoiding noisy data at decision boundary points. Consistency training is performed through spatial perturbation to enhance the model's generalization ability to target domain images. The teacher network parameters are updated using an exponential moving average method, supporting full parameter fine-tuning and compatibility with various advanced model architectures. This achieves accurate semantic segmentation prediction of remote sensing images in the target domain, thereby improving the final semantic segmentation model's adaptability and prediction accuracy in cross-domain scenarios.

[0033] Example 2, refer to Figures 1-2 As an embodiment of the present invention, based on the above embodiment, a test-time adaptive method for cross-domain semantic segmentation of remote sensing images is provided.

[0034] In this embodiment of the application, step S100 obtains the initial semantic segmentation model in the source domain dataset and the target domain image.

[0035] Specifically, the source domain dataset is a remote sensing image dataset with annotation information, such as the Potsdam RGB dataset or the Vaihingen dataset. The initial semantic segmentation model... The model is trained using supervised learning on the source domain dataset, and then optimized using semantically labeled source domain data until it achieves the expected segmentation accuracy on the source domain dataset. After training, the resulting initial semantic segmentation model contains a complete network structure and optimized network parameters. .

[0036] The target domain images are unlabeled remote sensing images, such as those in the BC403 dataset. There are distributional differences between the target domain images and the source domain dataset. These differences may stem from different geographical regions, imaging conditions, band combinations, or image resolutions. For example, when the source domain dataset uses an RGB band combination, the target domain image might use an IR-RG band combination, or the source and target domains might be aerial images from different cities. During the adaptive testing process, only target domain images can be acquired. ,in, The target domain images are limited by the number of images, but the annotation information of the source domain dataset cannot be obtained.

[0037] Step S200, which initializes the teacher network and student network based on the initial semantic segmentation model, specifically includes A1~A3: A1. Obtain the network parameters of the initial semantic segmentation model.

[0038] Specifically, from the initial semantic segmentation model after training... Extract all network parameters These parameters include the weights and biases of convolutional layers, the parameters of batch normalization layers, and the parameters of fully connected layers. For Transformer-based models, these also include the weight parameters of the self-attention mechanism and the parameters of the multilayer perceptron. Network parameters This represents the feature representation capability that the model has learned on the source domain dataset.

[0039] A2. Copy the network parameters of the initial semantic segmentation model to the teacher network to complete the initialization of the teacher network.

[0040] Specifically, a teacher network with the same architecture as the initial semantic segmentation model is constructed. The extracted network parameters The entire network is copied to the teacher network, so that the initial parameters of the teacher network are... The teacher network is used to generate high-confidence pseudo-labels, providing supervision signals for the training of the student network. During subsequent testing and adaptation, the parameters of the teacher network are updated using an exponential moving average method.

[0041] A3. Copy the network parameters of the initial semantic segmentation model to the student network to complete the initialization of the student network.

[0042] Specifically, a student network with the same architecture as the initial semantic segmentation model is constructed. Similarly, the extracted network parameters The entire network is copied to the student network, so that the initial parameters of the student network are also set. The student network is trained using high-confidence pseudo-labels and spatial consistency loss, and its parameters are continuously updated through backpropagation to adapt to the distribution characteristics of the target domain image. After initialization, the teacher and student networks have the same network structure and parameters, but in subsequent iterations, their parameter update methods differ. The student network updates directly using gradient descent, while the teacher network updates smoothly based on the student network parameters using an exponential moving average method.

[0043] In this embodiment, MiT-b0 is preferably used as the encoder architecture for the basic semantic segmentation model. Of course, other advanced semantic segmentation model architectures, such as SegFormer and DeepLabV3+, can also be used. This invention does not limit the specific model architecture; any model capable of performing semantic segmentation tasks is applicable.

[0044] In this embodiment of the application, step S300 predicts the target domain image through the teacher network to generate high-confidence pseudo-labels, specifically including steps B1 to B5: B1. The teacher network predicts each pixel in the target domain image and obtains the predicted probability for each category.

[0045] Specifically, the target domain image Input to teacher network The teacher network performs pixel-by-pixel semantic segmentation prediction for each pixel in the image. For a pixel at position (m,n) in the image, the teacher network outputs the predicted probability distribution of the pixel belonging to each category. Where m represents the cell location index. Let C represent the category index, and C be the total number of categories in the semantic segmentation task. The predicted probability is obtained by normalization using the Softmax function, satisfying the following condition: ,and .

[0046] Taking remote sensing image semantic segmentation as an example, the total number of categories C includes, but is not limited to, multiple land cover categories such as buildings, roads, vegetation, water bodies, and bare soil. For the PotsdamRGB dataset, the total number of categories C=6, including impermeable surfaces, buildings, low vegetation, trees, cars, and background debris. For each pixel in the image, the teacher network outputs the probability value of that pixel belonging to these 6 categories.

[0047] B2. For each pixel, determine the category with the highest probability and the category with the second highest probability from the predicted probabilities of the categories.

[0048] Specifically, for the target domain image For a pixel at position m, predict its probability by class m. In this process, the category with the highest probability is determined by maximizing the value: ; in, This indicates the category index with the highest predicted probability for that pixel. The operation returns the category index that maximizes the predicted probability.

[0049] At the same time, determine the category with the second highest probability from the remaining C−1 categories: ; in, This represents the category index with the second highest predicted probability for this pixel. The corresponding probability values ​​are denoted as follows: and .

[0050] For example, for a given pixel, if its predicted probabilities of belonging to the 6 categories are respectively The category with the highest probability is category 2 (buildings), with a corresponding probability value of The category with the second highest probability is category 3 (low vegetation), with a corresponding probability value of .

[0051] B3. Calculate the probability difference between the category with the highest probability and the category with the second highest probability.

[0052] Specifically, for each pixel, calculate the probability difference \Delta p between the category with the highest probability and the category with the second highest probability: ; in, This represents the probability difference of the pixel at position m. This represents the probability value of the class with the highest probability for that pixel. This represents the probability value of the second-highest probability category for that pixel. Probability difference. This reflects the confidence level of the teacher network in classifying the pixel; the larger the difference, the more certain the teacher network is in classifying the pixel.

[0053] For example, the probability difference of the pixel is When the probability difference is large, it indicates that the pixel is clearly biased towards a certain category and far from the category decision boundary; when the probability difference is small, it indicates that the pixel may be located near the decision boundary between the two categories, and the classification uncertainty is high.

[0054] This invention uses probability difference as a selection criterion, unlike existing CBST methods which use the absolute probability value of the highest-probability class. In existing methods, if a pixel has probabilities of 0.51 and 0.49 in two classes, although the highest probability value is low, the pixel is likely located on the decision boundary and misclassified. This invention, by calculating the probability difference (0.02 in this example), can effectively identify and exclude such uncertain boundary points.

[0055] B4. When the probability difference is greater than a preset threshold, the class with the highest probability of the pixel is taken as the pseudo-label.

[0056] Specifically, setting a preset threshold For each pixel, determine its probability difference. Is it greater than the preset threshold? The rules for generating pseudo-tags are expressed as follows: ; in, This represents the pseudo-label value of the pixel at position m in category c. When the probability difference of this pixel exceeds a preset threshold... When, choose the category with the highest probability. The pixel is marked as 1 (selected as a pseudo-label), and all other categories are marked as 0; otherwise, the pixel is not selected as a pseudo-label, and all categories are marked as 0.

[0057] Preset threshold The value range is preferably 0.3 to 0.7, more preferably 0.4 to 0.6. In this embodiment, a preset threshold is used. Set to 0.5. When If the highest probability category of a pixel is at least 0.5 higher than the second highest probability category, then the pixel is considered to have high confidence, its prediction result is reliable, and it can be used as a pseudo-label for training student networks.

[0058] For example, for probability difference The pixels, if a preset threshold is set If the pixel does not meet the filtering criteria, then the pixel does not meet the filtering criteria (because...). ), will not be selected as a pseudo-label; if the probability difference If the filtering conditions are met, the pixel with the highest probability category (building) is marked as a pseudo-label.

[0059] B5. Use the filtered pseudo-labels as high-confidence pseudo-labels.

[0060] Specifically, for the target domain image All pixels undergo the filtering process described in steps B1 to B4 above. Pixels that meet the condition of a probability difference greater than a preset threshold and their corresponding highest probability categories are summarized to form high-confidence pseudo-labels for the image. The high-confidence pseudo-labels exist in the form of sparse labeling, that is, pseudo-labels are provided only for a portion of the high-confidence pixels, while no labels are provided for the low-confidence pixels.

[0061] The generation of high-confidence pseudo-labels can be represented as: ; in, This represents the total number of pixels in the target domain image. and These are the height and width of the image, respectively (in pixels). This represents the parameters of the teacher network at the t-th iteration.

[0062] The high-confidence pseudo-labels generated through the above steps have high accuracy and can provide reliable supervision signals for the training of student networks. Compared with the traditional method of using a fixed threshold to select the class with the highest probability, the probability difference screening mechanism of this invention can more accurately select high-confidence pixels far from the decision boundary, while correctly selecting pixels falling near the decision boundary. This increases the number and accuracy of pseudo-labels and avoids the problem of performance degradation caused by the model learning noisy features.

[0063] In one optional implementation, the preset threshold Adjust according to the training process. In the early stages of training, a higher threshold can be set (e.g., ...). To ensure the accuracy of pseudo-labels; as the model gradually adapts to the target domain image, the threshold can be gradually reduced (e.g., ...). This increases the number of pseudo-labels and improves the utilization of training data. The dynamic threshold adjustment strategy can be expressed as: ,in, As the initial threshold, t is the decay coefficient, and t is the current iteration number.

[0064] In another alternative implementation, different preset thresholds can be set for different categories for certain specific application scenarios. For example, for categories that appear less frequently in the target domain image (such as cars and debris), a lower threshold can be set to increase the number of pseudo-labels for that category; for categories that appear more frequently (such as buildings and roads), a higher threshold can be set to ensure the quality of the pseudo-labels. Category-specific thresholds can be expressed as... Adjustments are made based on the category prior distribution and the teacher network's prediction confidence for that category.

[0065] In this embodiment, step S400 applies a spatial perturbation to the target domain image, and uses a student network to predict the target domain images before and after the perturbation, calculating the spatial consistency loss. Specifically, this includes C1~C4: C1. Perform at least one spatial transformation operation, such as rotation, cropping, or scaling, on the target domain image to obtain a spatially perturbed target domain image.

[0066] Specifically, for the target domain image Applying spatial perturbation operations Generate the target domain map after spatial perturbation. The spatial perturbation operation It includes at least one of rotation transformation, cropping transformation, and scaling transformation, used to change the spatial geometric properties of an image.

[0067] For rotation transformations, the rotation angle is randomly selected. The target domain image is rotated around its center. In this embodiment, the rotation angle is... The angle is preferably set to 15° to 30°, more preferably 20°. The rotation transformation is achieved using a rotation matrix: ; in, Indicates the rotation angle as The rotation matrix is ​​used to transform the coordinates of each pixel in the image, resulting in a rotated image.

[0068] For cropping transformation, a cropping region is randomly selected from the target domain image, and the size of the cropping region is the same as the original image size. Among them, the cutting ratio The cropped image is adjusted to the original image size using interpolation methods. For example, for a target domain image of size 512×512 pixels, when the cropping ratio r=0.8, the cropped area size is 410×410 pixels. After cropping, it is adjusted back to 512×512 pixels using bilinear interpolation or bicubic interpolation.

[0069] For scaling transformations, the scaling factor is randomly selected. The target domain image is scaled. In this embodiment, the scaling factor is preferably set to [0.8, 1.2], meaning the image can be reduced to 80% of its original size or enlarged to 120% of its original size. The scaling operation is expressed as: ; in, and These are the height and width of the original image, respectively. This represents an image scaling function that resizes the image back to its original size. .

[0070] In this embodiment, any of the above spatial perturbation operations can be used alone, or multiple spatial perturbation operations can be used in combination. For example, the target domain image can be rotated by 15° first, and then a cropping operation with a cropping ratio of 0.85 can be performed to generate a target domain image with comprehensive spatial perturbation. Combining multiple spatial perturbations can more fully enhance the spatial robustness of the model.

[0071] It should be noted that this invention only employs spatial perturbation operations and does not use spectral perturbation operations (such as color jitter, brightness adjustment, contrast adjustment, etc.). According to the disclosure document, for adaptive tasks during remote sensing image testing, spatial perturbation can effectively improve the model's generalization ability, while spectral perturbation may introduce features inconsistent with the target domain image distribution, thus having a negative impact. Therefore, this invention is specifically designed for the characteristics of remote sensing images, selectively using only spatial perturbation for consistency training.

[0072] C2. The student network predicts the target domain image before spatial perturbation and obtains the prediction result before perturbation.

[0073] Specifically, the original target domain image Enter student network The student network performs semantic segmentation prediction for each pixel in the image and outputs the prediction result before perturbation. Where m represents the cell location index and c represents the category index, the prediction result before perturbation can be expressed as: ; in, This indicates that the student network represents the target domain image. The predicted output is a three-dimensional tensor with dimension . , which correspond to the image height, width, and number of categories, respectively. For each pixel location (m,n), the prediction result is... This represents the probability that the pixel belongs to category c, and the sum of the probabilities of all categories is 1.

[0074] For example, for a 512×512 pixel remote sensing image with 6 categories, the dimension of the prediction result before perturbation is 512×512×6, containing prediction results for 262,144 pixels, with each pixel having a probability value for 6 categories.

[0075] C3. The student network predicts the target domain image after spatial perturbation and obtains the prediction result after perturbation.

[0076] Specifically, the spatially perturbed target domain image generated in step C1 Enter student network The student network performs semantic segmentation prediction on each pixel in the perturbed image and outputs the prediction result after perturbing. The prediction result after perturbation can be expressed as: ; in, The student network's predicted output for the spatially perturbed target domain image is also a three-dimensional tensor with dimension 1. Because of the spatial perturbation operation, the content of the image changes in spatial location after perturbation, but the semantic information contained in the image (such as the types of land features such as buildings, roads, and vegetation) remains unchanged.

[0077] The core idea of ​​consistency training is that, for the same target domain image, the prediction results of the student network should remain consistent at the semantic level, regardless of whether spatial perturbations are applied. By minimizing the difference in prediction results before and after perturbation, the student network is encouraged to learn feature representations that are invariant to spatial transformations, thereby improving the model's generalization ability and robustness in the target domain.

[0078] C4. Calculate the difference between the prediction results before and after the disturbance to obtain the spatial consistency loss, specifically including C4.1 to C4.3: C4.1 Perform an inverse spatial transformation on the post-disturbance prediction result to align it spatially with the pre-disturbance prediction result.

[0079] Specifically, because a spatial perturbation operation is applied to the target domain image in step C1... This leads to changes in the prediction results after the disturbance. Compared with the prediction results before the disturbance They are spatially misaligned. To calculate the pixel-level difference between them, an inverse spatial transformation is needed on the perturbed prediction. Transform it back to the same spatial coordinate system as the original image.

[0080] For rotational transformations, the inverse transformation is the reverse rotation, i.e., the rotation angle is... : ; For cropping transformations, the inverse transform maps the cropped and adjusted prediction results back to the corresponding positions in the original image by recording the original position and size of the cropped region. For scaling transformations, the inverse transform is reverse scaling, i.e., a scaling factor of 1 / s.

[0081] The aligned, perturbed prediction result is expressed as follows: ; in, This represents the perturbed prediction result after inverse spatial transformation and alignment, and its spatial location compared to the prediction result before perturbation. Complete correspondence, allowing for pixel-by-pixel comparison.

[0082] C4.2 Calculate the difference in prediction probability pixel by pixel between the aligned, perturbated prediction result and the unperturbed prediction result.

[0083] Specifically, for each pixel location m∈{1,2,...,M}, the difference in prediction probability between the aligned, perturbated prediction result and the unperturbated prediction result at that pixel is calculated. The difference in prediction probability can be measured in various ways. In this embodiment, the L1 norm (absolute difference) or L2 norm (squared difference) is preferably used for calculation.

[0084] When using the L1 norm, the difference in prediction probability for a single pixel is calculated as follows: ; in, Let represent the L1 norm predicted probability difference of the cell at position m, where C is the total number of categories, and the summation of the absolute values ​​of the probability differences over all categories.

[0085] When using the L2 norm, the difference in prediction probability for a single pixel is calculated as follows: ; in, Let m be the L2 norm predicted probability difference for the cell at position m, summed by the squared probability differences for all classes.

[0086] In another implementation, cross-entropy or KL divergence can be used to calculate the prediction probability difference. When using KL divergence, the prediction probability difference for a single pixel is calculated as follows: ; in, It represents the KL divergence of a pixel at position m, which measures the difference in the predicted probability distribution before and after the perturbation.

[0087] C4.3. Sum the differences in prediction probabilities for all pixels to obtain the spatial consistency loss.

[0088] Specifically, all in the target domain image The spatial consistency loss of the entire image is obtained by summing the predicted probability differences of each pixel. It can be represented as: ; in, Represents the target domain image Spatial consistency loss, where M is the total number of pixels in the image, and d(m) represents the difference in predicted probability for pixel m (which can be...). , or The normalized spatial consistency loss is obtained by averaging the differences of all pixels.

[0089] When using the L1 norm, the complete expression for the spatial consistency loss is: ; When using the L2 norm, the complete expression for the spatial consistency loss is: ; In this calculation, M×C in the denominator is used to normalize the differences between all pixels and all categories, making images of different sizes and tasks with different numbers of categories comparable.

[0090] Spatial consistency loss This study measures the consistency of the student network's predictions for the same target domain image before and after applying spatial perturbations. A smaller loss value indicates stronger robustness to spatial transformations and more stable prediction results. By minimizing the spatial consistency loss during training, the student network learns feature representations that are invariant to spatial transformations such as rotation, cropping, and scaling, thereby improving the model's generalization ability in the target domain.

[0091] For example, for a 512×512 pixel remote sensing image with C=6 categories, if the predicted probabilities before and after the perturbation are very similar in most pixels (e.g., the probability difference is less than 0.01), then spatial consistency loss occurs. The value is small (e.g., 0.005); if the prediction results before and after the perturbation differ significantly (e.g., the category predictions for some pixels are completely different), then the spatial consistency loss is small. The value is relatively large (e.g., above 0.5).

[0092] In one alternative implementation, a weighting coefficient can be applied to the spatial consistency loss. This is used to adjust the relative importance of spatial consistency loss in the total loss. The weighted spatial consistency loss is expressed as... The weighting coefficient The preferred value range is 0.1 to 2.0, more preferably 0.5 to 1.5. In this embodiment, the weighting coefficient... Set to 1.0, meaning that spatial consistency loss and cross-entropy loss have the same weight.

[0093] In another alternative implementation, independent spatial consistency losses can be calculated for different spatial perturbation operations, and multiple consistency losses can be weighted and combined. For example, the consistency loss for rotational perturbations can be calculated separately. Consistency loss due to trimming disturbance Consistency loss with scaling perturbation The total spatial consistency loss is expressed as: ; in, These are the weighting coefficients corresponding to each spatial perturbation operation. The spatial robustness of the model is further enhanced through joint constraints of multiple spatial perturbations.

[0094] In this embodiment of the application, step S500 involves the student network updating its parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the prediction results, as well as the spatial consistency loss. Specifically, this includes steps D1 to D4: D1. The student network makes predictions on the target domain image, and the prediction results of the student network are obtained.

[0095] Specifically, the target domain image Enter student network The student network performs semantic segmentation prediction for each pixel in the image and outputs the student network prediction results. Where \theta represents the current parameters of the student network, Indicates the cell's location index. Let c represent the total number of pixels, c∈{1,2,...,C} represent the category index, and C represent the total number of categories in the semantic segmentation task.

[0096] The student network prediction results can be represented as: ; in, It is a three-dimensional tensor with dimension . For each pixel location m, the student network outputs the predicted probability that the pixel belongs to each category. The predicted probability is obtained by normalization using the Softmax function, satisfying... ,and .

[0097] For example, for a 512×512 pixel remote sensing image with 6 categories (C=6), the student network's prediction results will have a dimension of 512×512×6. For a pixel at position (100, 200) in the image, assume the predicted probabilities for the 6 categories output by the student network are... This indicates that the probability of the pixel belonging to the building category (Category 2) is 0.55, the probability of it belonging to the low vegetation category (Category 3) is 0.20, and so on.

[0098] It should be noted that the student network prediction here is for the original target domain image (without spatial perturbation) and is used for supervised learning with the high-confidence pseudo-labels generated by the teacher network in step S300.

[0099] D2. Calculate the cross-entropy loss between the student network prediction results and the high-confidence pseudo-labels.

[0100] Specifically, high-confidence pseudo-labels generated by the teacher network in step S300 are utilized. As a supervisory signal, the student network prediction results are calculated. Cross-entropy loss between high-confidence pseudo-labels .

[0101] Since high-confidence pseudo-labels are sparse (labels are provided only for a subset of high-confidence pixels), the cross-entropy loss is computed only on pixels with pseudo-labels. Let... For a set of pixels with high-confidence pseudo-labels, the cross-entropy loss can be expressed as: ; in, Represents the target domain image Cross-entropy loss, This indicates the number of pixels with high-confidence pseudo-labels. This indicates that the teacher network generates pseudo-labels (one-hot encoded, with values ​​of 0 or 1) for pixel m in category c. This represents the student network's prediction probability for pixel m in category c.

[0102] Taking remote sensing image semantic segmentation as an example, assuming that in a 512×512 pixel image, 150,000 pixels are marked as high-confidence pseudo-labels through the filtering in step S300, then... For a cell with a pseudo-label Assuming its pseudo-label is a building category (Category 2), that is... The pseudo-label values ​​for the other categories are 0. The student network's predicted probability for this pixel is... Then the cross-entropy loss contribution of this pixel is: ; Repeat the above calculation for all pixels with high-confidence pseudo-labels and calculate the average value to obtain the cross-entropy loss of the entire image. .

[0103] Cross-entropy loss measures the difference between the student network's predictions and the high-confidence pseudo-labels generated by the teacher network. A smaller loss value indicates that the student network's predictions are closer to the pseudo-labels, and the better the learning performance. By minimizing the cross-entropy loss, the student network can learn feature representations of the target domain image from the high-quality supervision signals provided by the teacher network.

[0104] In one alternative implementation, different weights can be applied to the cross-entropy loss for different categories to address the class imbalance problem. The class-weighted cross-entropy loss can be expressed as: ; in, The weight coefficient for category c can be set based on the frequency of each category in the target domain image or the prediction confidence of the teacher network. Larger weights can be assigned to categories with low frequency (such as cars and objects), while smaller weights can be assigned to categories with high frequency (such as buildings and roads), thus balancing the contributions of each category during training.

[0105] D3. Add the cross-entropy loss to the spatial consistency loss to obtain the total loss.

[0106] Specifically, the cross-entropy loss calculated in step D2 and the spatial consistency loss calculated in step S400 are weighted and summed to obtain the total loss of the student network. The total loss can be expressed as: ; in, Represents the target domain image Total loss, This is the weighting coefficient for spatial consistency loss, used to adjust the relative importance of spatial consistency loss in the total loss.

[0107] D4. Update the student network parameters by backpropagating the total loss.

[0108] Specifically, the gradient descent optimization algorithm is used to calculate the gradient of the student network parameters θ based on the total loss, and the student network parameters are updated through backpropagation. The parameter update rule can be expressed as: ; in, This represents the parameters of the student network at the t-th iteration. This indicates the updated student network parameters. Indicates the learning rate. Indicates total loss The gradient of the student network parameter θ.

[0109] In one alternative implementation, a batch update strategy can be employed, where the total loss is calculated and the parameters are updated for multiple target domain images in a batch. Batch updates can reduce the variance of parameter updates and improve training stability.

[0110] In this embodiment of the application, step S600 updates the teacher network with the updated student network parameters until a final semantic segmentation model is obtained, which is used to perform semantic segmentation prediction on remote sensing images in the target domain. The step of updating the teacher network includes E1~E3: E1. Obtain the updated student network parameters and teacher network.

[0111] Specifically, in step S500, after the student network updates its parameters through backpropagation, the updated student network parameters are obtained. The subscript t+1 represents the parameters after the (t+1)th iteration. Simultaneously, the current teacher network is obtained. and its corresponding teacher network parameters , where the subscript t represents the teacher network parameters after the t-th iteration.

[0112] In the initial phase (t=0), both the teacher network and the student network are initialized using an initial semantic segmentation model pre-trained on the source domain dataset. As the iteration process progresses, the student network θ is continuously updated through supervised learning and consistency constraints, while the teacher network parameters... The parameters of the student network are smoothly inherited using the exponential moving average method.

[0113] The teacher and student networks have the same network architecture but different parameters. During the test-time adaptation process, the student network acts as the primary learning network, directly receiving gradient updates; while the teacher network, as an auxiliary network, updates indirectly by smoothing the student network parameters, thereby maintaining the stability of predictions and the ability to generate high-quality pseudo-labels.

[0114] For example, after the 100th iteration, the student network parameters have been updated to... The current teacher network parameters are (Obtained by smoothing the student network parameters from the 99th iteration). At this point, it is necessary to utilize... and To calculate the updated teacher network parameters.

[0115] E2. Perform a weighted average of the updated student network parameters and the current teacher network parameters according to a preset smoothing coefficient.

[0116] Specifically, an exponential moving average method is used to calculate a weighted average of the updated student network parameters and the current teacher network parameters. The exponential moving average is a parameter smoothing technique that maintains the stability of the teacher network parameters while gradually incorporating new knowledge learned by the student network. E3. The weighted average result is used as the updated teacher network.

[0117] The parameters obtained in step E2 using the exponential moving average Assigned to the teacher network as the updated teacher network The updated teacher network will be used in the next iteration to generate new high-confidence pseudo-labels, providing supervisory signals for the student network.

[0118] Table 1: Cross-domain semantic segmentation results from Vaihingen to BC403 Different regions within the same wavelength band (RGB→RGB): Potsdam RGB to BC403 Table 1 and Figure 2 The quantitative and qualitative results of single-stage TTA are presented separately. The CRITERIA method effectively improves the model's performance on the test dataset, increasing the overall mIoU and F1 score from 45.24% and 58.25% to 50.12% and 62.23%, respectively. Figure 2 The qualitative results shown demonstrate that the CRITERIA method improves the TTA semantic segmentation results from Potsdam-RGB to BC403.

[0119] In summary, this invention improves the adaptive capability of cross-domain semantic segmentation of remote sensing images by employing a high-confidence pseudo-label generation mechanism and spatial consistency loss training. Specifically, this invention uses the probability difference between the highest-probability category and the second-highest-probability category as the pseudo-label selection condition. This allows for the selection of high-confidence pixels far from the decision boundary while correctly selecting pixels falling near the decision boundary, avoiding the problem of misselecting boundary noise in existing fixed-threshold methods and improving the quantity and accuracy of pseudo-labels. Furthermore, this invention, tailored to the characteristics of remote sensing images, applies spatial perturbation to the target domain image for consistency training. By constraining the spatial consistency of the prediction results before and after perturbation, the robustness and generalization ability of the model are enhanced, avoiding the negative impact that spectral perturbation may bring.

[0120] This invention employs an adaptive framework of teacher and student networks, supporting fine-tuning of all network parameters. This overcomes the limitation of existing methods that only adjust batch normalization layer parameters, making this method compatible with advanced model architectures such as Transformer with or without batch normalization layers. By updating teacher network parameters using an exponential moving average method, smooth parameter updates are achieved, further improving the model's training stability and robustness. Furthermore, for cross-domain scenarios with band differences, this invention can combine generative adversarial networks to translate images from the source domain dataset, generating a transfer dataset with a style consistent with the target domain images. This effectively reduces inter-domain differences, thereby enhancing the model's adaptability in cross-band tasks.

[0121] Example 3 illustrates a test-time adaptive method for cross-domain semantic segmentation of remote sensing images. It should be noted that the technical solution of this test-time adaptive system for cross-domain semantic segmentation of remote sensing images is based on the same concept as the aforementioned test-time adaptive method for cross-domain semantic segmentation of remote sensing images. Details not described in detail in the test-time adaptive system for cross-domain semantic segmentation of remote sensing images in this embodiment can be found in the description of the aforementioned test-time adaptive method for cross-domain semantic segmentation of remote sensing images.

[0122] This embodiment also provides a test-time adaptive system for cross-domain semantic segmentation of remote sensing images, including: The initial model acquisition module is used to acquire the initial semantic segmentation model trained on the source domain dataset, as well as the target domain image; The network initialization module is used to initialize the teacher network and student network based on the initial semantic segmentation model. The pseudo-label generation module is used to predict the target domain image through the teacher network and generate high-confidence pseudo-labels; The consistency loss calculation module is used to apply spatial perturbation to the target domain image, predict the target domain image before and after the perturbation through the student network, and calculate the spatial consistency loss. The student network update module is used to update the student network parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the student network prediction results, as well as the spatial consistency loss. The teacher network update module is used to update the teacher network based on the updated student network parameters. The prediction output module is used to perform semantic segmentation prediction on remote sensing images of the target domain using the final semantic segmentation model.

[0123] This embodiment also provides an electronic device suitable for test-time adaptive methods for cross-domain semantic segmentation of remote sensing images, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the test-time adaptive method for cross-domain semantic segmentation of remote sensing images as proposed in the above embodiment.

[0124] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the test-time adaptive method for cross-domain semantic segmentation of remote sensing images as proposed in the above embodiments.

[0125] The storage medium proposed in this embodiment and the test-time adaptive method for cross-domain semantic segmentation of remote sensing images proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0126] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0127] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A test-time adaptive method for cross-domain semantic segmentation of remote sensing images, characterized in that, Includes the following steps: Obtain the initial semantic segmentation model from the source domain dataset, and the target domain image; The teacher network and student network are initialized based on the initial semantic segmentation model. The target domain image is predicted using a teacher network to generate high-confidence pseudo-labels; Spatial perturbation is applied to the target domain image, and the target domain image before and after perturbation is predicted through the student network. Spatial consistency loss is calculated. The student network updates its parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the prediction results, as well as the spatial consistency loss. The teacher network is updated by updating the student network parameters until the final semantic segmentation model is obtained.

2. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 1, characterized in that, The steps for generating the high-confidence pseudo-labels include: The teacher network predicts each pixel in the target domain image and obtains the predicted probability for each category; For each pixel, determine the category with the highest probability and the category with the second highest probability from the predicted probabilities of the said categories; Calculate the probability difference between the category with the highest probability and the category with the second highest probability; When the probability difference is greater than a preset threshold, the class with the highest probability of that pixel is taken as a pseudo-label. The filtered pseudo-labels are used as high-confidence pseudo-labels.

3. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 2, characterized in that, The steps of applying spatial perturbation to the target domain image and calculating spatial consistency loss include: Perform at least one spatial transformation operation, such as rotation, cropping, or scaling, on the target domain image to obtain a spatially perturbed target domain image. The student network predicts the target domain image before spatial perturbation and obtains the prediction result before perturbation; The student network predicts the target domain image after spatial perturbation and obtains the prediction result after perturbation. The spatial consistency loss is obtained by calculating the difference between the prediction results before and after the disturbance.

4. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 3, characterized in that, The steps for calculating the difference between the prediction results before and after the disturbance include: Perform an inverse spatial transformation on the post-disturbance prediction result to align it spatially with the pre-disturbance prediction result; The difference in prediction probability is calculated pixel by pixel between the aligned, perturbed prediction result and the original prediction result. The spatial consistency loss is obtained by summing the differences in predicted probabilities for all pixels.

5. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 4, characterized in that, The steps for updating student network parameters include: The student network makes predictions on the target domain image, and the student network prediction results are obtained. Calculate the cross-entropy loss between the student network prediction results and the high-confidence pseudo-labels; The total loss is obtained by adding the cross-entropy loss to the spatial consistency loss. The total loss is then used to update the student network parameters via backpropagation.

6. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 5, characterized in that, The steps to update the teacher network include: Obtain updated student network parameters and teacher network parameters; The updated student network parameters and the current teacher network are weighted averaged according to a preset smoothing coefficient. The weighted average result is used as the updated teacher network.

7. The test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in claim 6, characterized in that, The steps for initializing the teacher network and student network based on the initial semantic segmentation model include: Obtain the network parameters of the initial semantic segmentation model; The network parameters of the initial semantic segmentation model are copied to the teacher network to complete the initialization of the teacher network; The network parameters of the initial semantic segmentation model are copied to the student network to complete the initialization of the student network.

8. A test-time adaptive system for cross-domain semantic segmentation of remote sensing images, employing the method described in any one of claims 1-7, characterized in that, include: The initial model acquisition module is used to acquire the initial semantic segmentation model trained on the source domain dataset, as well as the target domain image; The network initialization module is used to initialize the teacher network and student network based on the initial semantic segmentation model. The pseudo-label generation module is used to predict the target domain image through the teacher network and generate high-confidence pseudo-labels; The consistency loss calculation module is used to apply spatial perturbation to the target domain image, predict the target domain image before and after the perturbation through the student network, and calculate the spatial consistency loss. The student network update module is used to update the student network parameters by combining the cross-entropy loss between the high-confidence pseudo-labels and the student network prediction results, as well as the spatial consistency loss. The teacher network update module is used to update the teacher network based on the updated student network parameters. The prediction output module is used to perform semantic segmentation prediction on remote sensing images of the target domain using the final semantic segmentation model.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the test-time adaptive method for cross-domain semantic segmentation of remote sensing images as described in any one of claims 1 to 7.

Images