A remote sensing image semantic segmentation method based on joint saliency and domain adaptation

Abstract

The application discloses a remote sensing image semantic segmentation method based on joint saliency and domain adaptation. The implementation process comprises the following steps: 1) training a classification network by using a remote sensing image dataset with image-level labels; 2) calculating and fusing multi-level class activation maps to obtain an initial saliency map for the foreground class remote sensing image; 3) extracting multi-dimensional local features of the image to realize image group feature correlation sorting; 4) combining the feature correlation sorting and the initial saliency map to calculate a joint saliency map and use the joint saliency map as a pixel-level pseudo label; 5) training a semantic segmentation network, improving the generalization ability of the network through a domain adaptation module, and completing semantic segmentation of the remote sensing image; 6) inputting the image into the trained semantic segmentation network to obtain a final saliency map; and 7) performing threshold segmentation on the final saliency map to extract a region of interest. The method disclosed by the application can mine feature correlation in an image group, suppress background interference, improve the generalization ability of a model, and realize accurate semantic segmentation of remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a semantic segmentation method for remote sensing images based on joint saliency and domain adaptation. Background Technology

[0002] With the rapid development of remote sensing technology, an increasing number of satellites and airborne sensors are providing us with high-resolution optical remote sensing images. The information contained in these images is becoming increasingly richer with the improvement of resolution, resulting in a massive amount of remote sensing data that urgently needs processing. Traditional methods of manually interpreting useful information are inefficient and can no longer meet current needs. In the information age, using computers to digitally process remote sensing images has become a major trend in remote sensing image processing. In recent years, artificial intelligence technology, led by deep learning, has developed rapidly, and using deep neural networks for remote sensing image processing has become the mainstream research method. This allows for processing of remote sensing visualization data, such as scene classification, target detection, semantic segmentation, and change detection.

[0003] Compared to traditional methods, deep learning-based semantic segmentation of remote sensing images is a data-driven feature learning process. Deep neural networks offer numerous advantages, including strong data representation capabilities, outstanding generalization ability, and the ability to handle complex scene information. Based on the accuracy of training data annotation, the training conditions of these methods can be categorized into strong annotation conditions and weak annotation conditions.

[0004] For semantic segmentation tasks, strong labeling refers to training data with accurate, large-scale pixel-level labels. Combining multi-scale feature fusion and using network forward propagation, semantic segmentation is achieved. For example, Li et al. proposed a two-stream pyramid nested network in their paper "Nested Network with Two-Stream Pyramid for Salient Object Detection in Optical Remote Sensing Images". This network uses a two-stream pyramid module and an encoder-decoder module with nested connections to accurately locate salient targets at different scales, effectively suppress cluttered information in the background area of ​​remote sensing images, and achieve target semantic segmentation of optical remote sensing images.

[0005] Deep learning models trained on strongly labeled datasets have powerful expressive capabilities, but their performance is heavily dependent on the scale and accuracy of the labeling, requiring significant human resources. When faced with scenarios where a large number of accurately labeled samples cannot be produced, the network performance will rapidly decline.

[0006] Compared with semantic segmentation methods for remote sensing images under strong annotation conditions, training models under weak annotation conditions only utilize image-level labels with low annotation accuracy to complete network training. The intermediate layers of the network simulate the hierarchical information processing mechanism of the human brain for images, and combine post-processing strategies to generate pixel-level saliency maps. For example, Chokkal et al. proposed a top-down weakly supervised saliency analysis framework in the paper "Backtrackingspatial pyramid pooling-based image classifier for weakly supervised top–down salient object detection". This framework first uses a backtracking strategy to complete top-down saliency analysis, and then combines it with a bottom-up saliency analysis method to complete saliency analysis and salient object segmentation under image-level annotation only.

[0007] While deep learning models trained on weakly labeled datasets effectively reduce labor costs, their accuracy still lags behind those trained under strongly labeled conditions due to insufficient labeling precision. This results in higher computational complexity, reduced testing and output efficiency, and the problem of local focusing, making it difficult to uncover the complete target region.

[0008] In addition, remote sensing images are characterized by a wide variety of ground features, complex structural information, and varying target scales. They are also susceptible to adverse weather conditions such as fog and haze. These image characteristics make accurate labeling of remote sensing images more difficult, and noise interference such as fog and haze can also occur in the data set.

[0009] In summary, this invention proposes a semantic segmentation method for remote sensing images based on joint saliency and domain adaptation. It aims to generate an initial pseudo-label saliency map using an image-level labeled dataset, analyze and rank features for contrast correlation, uncover common salient targets in image groups, suppress haze interference, and improve the model's generalization ability for remote sensing images through a domain adaptation module, thereby achieving accurate semantic segmentation of remote sensing images. In the field of remote sensing image processing, domain adaptation refers to the ability to transfer a model from the source domain to the target domain by learning the differences between source and target domain data; this is often simply referred to as domain adaptation. Summary of the Invention

[0010] The purpose of this invention is to provide a semantic segmentation method for remote sensing images based on joint saliency and domain adaptation. This method is used to accurately detect and segment target regions in massive amounts of remote sensing images. Traditional remote sensing image segmentation algorithms often require expert knowledge for feature extraction, are highly dependent on prior knowledge, and have poor generalization ability. In recent years, semantic segmentation methods represented by deep learning have developed rapidly. However, segmentation methods under strong supervision require a large number of pixel-level labeled training samples, consuming high human resources. Target segmentation under weak supervision is affected by the accuracy of labeling, resulting in poor extraction accuracy. In addition, remote sensing images are easily affected by severe weather such as haze, which increases the difficulty of accurate labeling and precise segmentation. Therefore, the method of this invention mainly focuses on three aspects:

[0011] 1) Reduce the model's dependence on annotation accuracy and improve target segmentation accuracy;

[0012] 2) Use image-level annotation to reduce labor costs;

[0013] 3) Improve the model's generalization ability.

[0014] This method targets a dataset of remote sensing images with image-level labels. First, a classification convolutional neural network is trained, fusing multi-level class activation maps to generate an initial saliency map. Then, superpixel segmentation is performed on the remote sensing images, extracting texture, spectral contrast, and hue contrast features as multi-dimensional local features. The correlation of superpixels in the image set is calculated and ranked. The initial saliency map is then refined using the correlation ranking list to obtain a joint saliency map. Finally, the joint saliency map is used as pixel-level pseudo-labels to train a domain-adaptive remote sensing image semantic segmentation network, calculating the final saliency map of the remote sensing images. After thresholding, the semantic segmentation result is obtained. The specific steps include:

[0015] Step 1: Using a remote sensing image dataset with image-level labels, train a classification convolutional neural network to build a saliency analysis model for weakly labeled remote sensing images. This classification convolutional neural network is used to calculate the probability that the input remote sensing image belongs to the foreground class and the background class. The foreground class remote sensing image is a remote sensing image that contains the target area, and the background class remote sensing image is a remote sensing image that does not contain the target area.

[0016] Step 2: Calculate the class activation maps of the multi-level foreground remote sensing images and perform weighted fusion to obtain the initial saliency map. That is, first, input the foreground remote sensing images into the classification convolutional neural network trained in Step 1. The network outputs multi-channel feature maps from multiple convolutional layers. Then, calculate the gradient response of each channel of the multi-channel feature map and use the gradient response as an adaptive weight to perform weighted averaging of the multi-channel feature maps to obtain the class activation map of the convolutional layer. The class activation maps of multiple levels are weighted and fused to obtain the initial saliency map.

[0017] Step 3: Extract multidimensional local features of foreground remote sensing images and calculate the correlation ranking list. That is, firstly, perform superpixel segmentation on the foreground remote sensing images. The superpixels are described by multidimensional features composed of texture features, spectral contrast features, and hue contrast features. Then, for each superpixel in the foreground remote sensing images, perform correlation calculation with other superpixels and rank them according to the order of correlation from strong to weak to obtain the correlation ranking list.

[0018] Step 4: Using the correlation ranking list and the initial saliency map obtained in Step 2, calculate the joint saliency map of the foreground remote sensing image. That is, firstly, for each superpixel in the foreground remote sensing image, calculate the saliency value of the superpixel based on the initial saliency map. Then, select the N superpixels with the highest correlation from the correlation ranking list, normalize the correlation of these N superpixels and use them as weights, and perform a weighted summation of the saliency values ​​of these N superpixels to obtain the saliency value of the superpixel. After the saliency values ​​of all superpixels in an image are calculated, the joint saliency map of the image is obtained.

[0019] Step 5: Train a domain-adaptive remote sensing image semantic segmentation network. First, construct a remote sensing image semantic segmentation network. Then, use the joint saliency map generated in Step 4 as pixel-level pseudo-labels, define different image groups as the source and target domains, and input them into the network for training. At the same time, improve the network's generalization ability by aligning the entropy distribution of the target domain with the entropy distribution of the source domain.

[0020] Step 6: After the domain-adaptive remote sensing image semantic segmentation network is trained, the remote sensing image is input into the trained network during the testing phase to obtain the final saliency map of the remote sensing image.

[0021] Step 7: Using the Otsu's method, calculate the segmentation threshold of the final saliency map of a set of remote sensing images. Use this threshold to segment the final saliency map of the remote sensing images into a binary image template, with "1" representing residential areas and "0" representing non-residential areas. Finally, multiply the binary image template with the remote sensing images to obtain the final semantic segmentation result of the remote sensing images. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention.

[0023] Figure 2 This is an example of remote sensing imagery used in this invention, along with the corresponding ground truth. The first row is the original imagery from the GeoEye-1 satellite, and the second row is the ground truth.

[0024] Figure 3The example image shows the multi-level feature maps and initial saliency map. From top to bottom, they are the feature maps output by the 4th, 16th, and last convolutional layers, the initial saliency map, and the joint saliency map.

[0025] Figure 4 This section compares the saliency maps generated using the method of this invention and other methods for the test set. From left to right, they represent the original image, the ground truth, and the results using MFF, SACH, Layer-CAM, and the method of this invention, respectively. MFF stands for Multiscale Feature Fusion (MFF), SACH for Saliency Analysis of Co-Occurrence Histogram (SACH), and Layer-CAM for Remote Sensing Image Saliency Analysis using Hierarchical Class Activation Maps (Exploring Hierarchical Class Activation Maps for Localization).

[0026] Figure 5 The example image is compared with the semantic segmentation results of the method of this invention and other methods. From left to right, the images represent the original image, the ground truth, and the results using MFF, GLSA, Layer-CAM, and the method of this invention, respectively. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings. The overall framework of the present invention is as follows: Figure 1 As shown, the implementation details of each step will now be introduced.

[0028] Step 1: Train a classification convolutional neural network using a remote sensing image dataset with image-level labels;

[0029] The optical remote sensing image training dataset used in this invention is I = {I1, I2, ..., I...} M}, Let M be a remote sensing image of size m×n with h channels, containing a total of M images, where m represents the length of the image, n represents the width of the image, and i = 1, 2, ..., M. The training dataset contains the i-th image I. i The tag is If y i = [1,0], then the image belongs to the foreground class. If y i If the value is [0,1], then the image belongs to the background category.

[0030] The constructed classification convolutional neural network architecture is VGGNet-16, which contains 13 convolutional layers and 5 max-pooling layers. To complete the classification task, two fully connected layers with 256 neurons each are added after the fifth pooling layer to increase the network's non-linearity. The classification layer contains two neurons and a normalized exponential function to calculate the probability that the input image belongs to the foreground and background classes. Here, the trained classification network is denoted as the mapping network. Where W and b represent the trained weights and biases, respectively.

[0031] Step 2: Calculate the initial saliency map of the foreground remote sensing images in the training dataset;

[0032] Since the initial saliency map generated in this step will be used for subsequent joint saliency analysis to extract residential areas from remote sensing images, only the multi-level gradient-weighted class activation maps (Grad-CAM) of the foreground remote sensing images in the training dataset are calculated and weighted and fused as the initial saliency map. Figure 3 The pseudo-label saliency map of the example image is shown.

[0033] First, I i The input is fed into the classification convolutional neural network trained in step one to determine I. i Is it a foreground type of remote sensing image? If I i For foreground remote sensing images, the output value of the neuron before the activation function in the decision layer is recorded as... The multi-channel feature map output by the l-th convolutional layer of the network is A. l Secondly, the gradient response of the output layer neuron to the k-th channel of the multi-channel feature map is calculated, and then the mean of the gradient response is used as the adaptive weight.

[0034]

[0035] in, Represents multi-channel feature map A l The gray level at position (s,t) in the k-th channel, where Z represents the multi-channel feature map A. l The number of pixels.

[0036] The remote sensing image I is obtained by weighted averaging of the multi-channel feature maps output from the l-th convolutional layer. i Hierarchical feature map Fmap l .

[0037]

[0038] in, A representsl The k-th channel, ReLU is a linear rectified unit (ReLU).

[0039] Finally, the feature maps of each convolutional layer are weighted and fused to obtain the remote sensing image I. i Initial saliency map Smap i .

[0040]

[0041] Where, ω l The weights are assigned to the feature maps at each level.

[0042] Step 3: Extract multidimensional local features from foreground remote sensing images and calculate a correlation ranking list;

[0043] In this step, we use the Simple Linear Iterative Cluster (SLIC) superpixel segmentation method to segment the image and extract superpixel features as local features.

[0044] For foreground remote sensing images I i Superpixel segmentation is performed to obtain superpixel groups. in, Indicate I i The kth superpixel, I i There are N superpixels in total.

[0045] For each superpixel, extract its texture features, spectral contrast features, and hue contrast features, and use the resulting multidimensional feature vector. To describe superpixels That is I i Multidimensional local features.

[0046] After extracting multidimensional local features from each image in a set of images, the feature distance between any two superpixels is calculated. This feature distance is defined as local correlation. The closer the feature distance, the higher the correlation, and vice versa.

[0047]

[0048] in, Image I M1 The k-th superpixel in Image I M2 The t-th superpixel in the array, φ i This represents the weight of the i-th feature.

[0049] Finally, the superpixel feature distances are sorted from smallest to largest, that is, the local correlations are sorted from high to low, to obtain image I. i Relevance sorting list

[0050] Step 4: Combine the correlation ranking list with the initial saliency map obtained in Step 2 to calculate the joint saliency map of the foreground remote sensing images;

[0051] First, using the initial saliency map obtained in step two, calculate the average saliency value of each pixel in each superpixel as the initial saliency value S of the superpixel. i ' nitial Then, the weights w are calculated based on the feature distances between superpixels. c If the correlation between two superpixels is too low, adding that superpixel to the calculation will reduce the accuracy. Therefore, only the top C superpixels in the correlation ranking list are selected for calculation, and the values ​​of the top C superpixels in the correlation ranking list are...

[0052]

[0053] Finally, the superpixel saliency values ​​with high similarity are fused to obtain the final joint superpixel saliency value.

[0054]

[0055] Where S is image I i The joint saliency map, P c Represents a relevance-sorted list The c-th superpixel in the array, S' initial (P c ) represents superpixel P c The initial significance value.

[0056] Step 5: Construct a domain-adaptive semantic segmentation network framework. By making the entropy distribution of the target domain similar to that of the source domain, the entropy is indirectly minimized, thereby improving the model's generalization ability and achieving semantic segmentation of remote sensing images.

[0057] A domain-adaptive remote sensing image semantic segmentation network is constructed, consisting of two parts: a semantic segmentation network and a fully convolutional discriminator network. The semantic segmentation network performs semantic segmentation on images from both the source and target domains simultaneously, deceiving the discriminator network, which distinguishes the outputs from the source and target domain images.

[0058] For the source domain training sample x s The corresponding pseudo-label saliency map is S. s The semantic segmentation network outputs F(x)s When training only on source domain data without domain adaptation, the loss function is L. seg .

[0059] L seg (x s ,S s )=-S s ·logF(x s )

[0060] For a training sample x, the network predicts a value F(x), and its weighted self-information is defined as I. x .

[0061] I x = -F(x)·logF(x)

[0062] Then, construct a fully convolutional discriminator D,I x As input, θ D For training parameters, the class label "1" represents the source domain and "0" represents the target domain, L D Let L be the cross-entropy loss. Then the overall network loss is L.

[0063]

[0064] Where, χ s For the source domain training dataset, χ t For the training dataset of the target domain, λ adv The weights are for opposing terms.

[0065] Step 6: Using the remote sensing image semantic segmentation network trained in Step 5, input the remote sensing images from the test set into the trained network to obtain the final saliency map;

[0066] Step 7: Calculate the joint segmentation threshold of the joint saliency map using the Otsu's method to achieve accurate semantic segmentation of the remote sensing image;

[0067] A set of joint saliency map segmentation thresholds is obtained by using the Otsu's method. The joint saliency map of the remote sensing image is then segmented into a binary image template using this threshold. "1" represents residential areas and "0" represents non-residential areas. Finally, the binary image template is multiplied by the corresponding remote sensing image to obtain the final semantic segmentation result of the remote sensing image.

[0068] The effectiveness of this invention can be further illustrated by the following experimental results and analysis:

[0069] 1. Experimental Data

[0070] This invention selects a set of visible light remote sensing images of residential areas from GeoEye-1 satellite source images, and extracts a set of images with a size of 256×256 from them as the experimental source images for this paper, such as... Figure 2 As shown.

[0071] 2. Comparative Experiment

[0072] To evaluate the performance of the method of this invention, we designed the following comparative experiments, selecting representative existing visual attention methods: Multi-Feature Fusion (MFF), Satiety Analysis Based on Co-occurrence Histogram (SACH), and Weakly Supervised Satiety Analysis (Layer-CAM) to compare their performance with that of the method of this invention. Subjectively, we compared the saliency maps and semantic segmentation results generated by different methods, such as... Figure 4 and Figure 5 As shown.

[0073] The results show that, apart from the method of this invention, other methods all produced serious misclassifications in areas with significant background interference. SACH calculates salient regions of the image by computing co-occurrence histograms, thus producing a large response to non-salient regions with prominent colors and brightness, but performing poorly in remote sensing images with complex background interference. Compared to SACH, MFF achieves better results, but because this method uses wavelet transform in the frequency domain, it produces a high response to fine textures. Layer-CAM integrates multi-level features to improve localization accuracy, but due to the high resolution of GeoEye-1 images, there are too many shallow feature points, causing the generated salient map to respond to regions of other categories, thus reducing segmentation accuracy. The method of this invention has better boundary preservation, a more uniform gray-level distribution within salient regions, and can better eliminate interference from complex backgrounds. For foggy images, such as in rows 2 and 3, our method also has better segmentation results, demonstrating its stronger generalization ability.

Claims

1. A semantic segmentation method for remote sensing images based on joint saliency and domain adaptation. This method targets a dataset of remote sensing images with image-level labels. First, a classification convolutional neural network is trained, fusing multi-level class activation maps to generate an initial saliency map. Then, superpixel segmentation is performed on the remote sensing images, extracting texture features, spectral contrast features, and hue contrast features of the superpixels as multi-dimensional local features. The correlation of superpixels in the image set is calculated and ranked. The initial saliency map is corrected using the correlation ranking list to obtain a joint saliency map. Finally, the joint saliency map is used as pixel-level pseudo-labels to train a domain-adaptive semantic segmentation network for remote sensing images, calculating the final saliency map of the remote sensing images. After thresholding, the semantic segmentation result is obtained. The method is characterized by… Includes the following steps: Step 1: Using a remote sensing image dataset with image-level labels, train a classification convolutional neural network to build a saliency analysis model for weakly labeled remote sensing images. This classification convolutional neural network is used to calculate the probability that the input remote sensing image belongs to the foreground class and the background class. The foreground class remote sensing image is a remote sensing image that contains the target area, and the background class remote sensing image is a remote sensing image that does not contain the target area. Step 2: Calculate the class activation maps of the multi-level foreground remote sensing images and perform weighted fusion to obtain the initial saliency map. That is, first, input the foreground remote sensing images into the classification convolutional neural network trained in Step 1. The network outputs multi-channel feature maps from multiple convolutional layers. Then, calculate the gradient response of each channel of the multi-channel feature map and use the gradient response as an adaptive weight to perform weighted averaging of the multi-channel feature maps to obtain the class activation map of the convolutional layer. The class activation maps of multiple levels are weighted and fused to obtain the initial saliency map. Step 3: Extract multidimensional local features of foreground remote sensing images and calculate the correlation ranking list. That is, firstly, perform superpixel segmentation on the foreground remote sensing images. The superpixels are described by multidimensional local features composed of texture features, spectral contrast features and hue contrast features. Then, for each superpixel in the foreground remote sensing images, perform correlation calculation with other superpixels and rank them according to the order of correlation from strong to weak to obtain the correlation ranking list. Step 4: Using the correlation ranking list and the initial saliency map obtained in Step 2, calculate the joint saliency map of the foreground remote sensing image. That is, firstly, for each superpixel in the foreground remote sensing image, calculate the saliency value of the superpixel based on the initial saliency map. Then, select the N superpixels with the highest correlation from the correlation ranking list, normalize the correlation of these N superpixels and use them as weights, and perform a weighted summation of the saliency values ​​of these N superpixels to obtain the saliency value of the superpixel. After the saliency values ​​of all superpixels in an image are calculated, the joint saliency map of the image is obtained. Step 5: Train a domain-adaptive remote sensing image semantic segmentation network. First, construct a remote sensing image semantic segmentation network. Then, use the joint saliency map generated in Step 4 as pixel-level pseudo-labels, define different image groups as the source and target domains, and input them into the network for training. At the same time, improve the network's generalization ability by aligning the entropy distribution of the target domain with the entropy distribution of the source domain. Step 6: After the domain-adaptive remote sensing image semantic segmentation network is trained, the remote sensing image is input into the trained network during the testing phase to obtain the final saliency map of the remote sensing image. Step 7: Using the Otsu's method, calculate the segmentation threshold of the final saliency map of a set of remote sensing images. Use this threshold to segment the final saliency map of the remote sensing images into a binary image template, with "1" representing residential areas and "0" representing non-residential areas. Finally, multiply the binary image template with the remote sensing images to obtain the final semantic segmentation result of the remote sensing images.

Images