An unsupervised land cover classification method based on class-aware domain adaptation

Abstract

An unsupervised land cover classification method based on class-aware domain adaptation, 1): the data used for domain adaptation land cover classification model training consists of a labeled source domain data set and an unsupervised target domain data set; 2): mean subtraction and normalization of the input image are performed to accelerate the convergence of weights and bias parameters in the network: 3): convolution operation of the convolution kernel is constructed to perform the initial downsampling operation; 4): a pre-trained residual network is constructed; 5): a category-level discriminator is established to form a local domain adaptation module; 6): global adversarial domain adaptation is used to solve the edge distribution offset; 7): an upsampling layer is constructed to restore the original size and generate the final pixel-level classification prediction. The present invention can extract and utilize the potential category information contained in high-resolution optical remote sensing images.

Description

Technical Field

[0001] The present invention provides an unsupervised domain-adaptive land cover classification model for high-resolution optical remote sensing images, which relates to the fields of land surveying and mapping, environmental remote sensing, resource investigation, etc., and belongs to the field of model algorithm development. Background Art

[0002] With the rapid improvement of computer processing and storage capabilities and the development of artificial intelligence, the traditional manually designed feature method has gradually been upgraded to an automatically learned feature extraction method, which has had a profound impact on remote sensing fields such as land cover mapping and achieved excellent classification performance. These semantic segmentation models mainly learn knowledge from training samples and then classify test data in a supervised manner, including automatic encoder-decoders, multi-level architectures, multi-scale structures, attention mechanisms, etc. However, deep supervision models are highly dependent on labeled data. When optical remote sensing images are affected by some irresistible factors, such as acquisition time, lighting conditions, sensor type, etc., significant data offsets are likely to occur between data sets obtained from a specific area or other different areas. In this case, when the segmentation model that has been well trained on the source domain data is generalized to other scenes (target domains) with different conditions, the performance of the model will face a serious decline.

[0003] To solve the problem of lack of target domain data labels, domain adaptation methods have been proposed one after another. The essence of domain adaptation is to match different data spaces by learning domain-invariant features, so that the model learned from labeled source data can be generalized to target domain data without any reference annotations in an unsupervised manner. Traditional domain adaptation methods, such as first-order moments and second-order moments, seek to minimize some metrics of the distance between the source distribution and the target distribution, but perform poorly in pixel-level segmentation tasks. Given the ability to learn abstract feature representations, deep learning technology has gradually expanded to domain adaptation research, with maximum mean deviation and adversarial learning as the two main branches. The former usually uses some statistical information of feature distribution (such as mean or covariance) to match different domains, but when the distribution does not conform to the Gaussian rule, its effect will be poor, while adversarial-based methods can reduce domain shift by generating false features or samples deceiving the discriminator, and have become popular in image processing.

[0004] In the unsupervised domain adaptation land cover classification task, generative adversarial learning methods mainly focus on three representation levels, including feature level, pixel level, and category level. These related methods usually use simple binary discriminators to perform global adversarial domain adaptation, while ignoring the latent category information of the target domain. On the other hand, there is a significant low inter-class variance in high-resolution remote sensing images, which usually leads to the negative impact of dense prediction. These adverse effects will be particularly serious when performing domain adaptation.

[0005] Therefore, based on an in-depth investigation and understanding of the characteristics of land cover types, it is of great application value to reasonably introduce an unsupervised land cover classification model driven by deep learning that takes into account the latent category information of the target domain. Summary of the invention

[0006] The present invention proposes a land cover classification method based on deep learning and class-aware domain adaptation. The classification algorithm can extract and utilize the potential category information contained in high-resolution optical remote sensing images. The method first performs normalization processing by counting the statistical distribution characteristics of the original remote sensing images on the RGB bands of the source domain and the target domain. Secondly, a strong baseline network is designed according to the input image size to extract category-related high-level semantic features. Then, the category-level discriminator and the binary discriminator are integrated in the feature space and the output space to perform local and global adversarial domain adaptation respectively.

[0007] To achieve the above objectives, the specific steps of the present invention are as follows:

[0008] 1): The data used for domain-adaptive land cover classification model training consists of a labeled source domain dataset and an unsupervised target domain dataset. The source domain dataset includes the original remote sensing image and the ground truth, and is cropped into many small blocks by a sliding window:

[0009]

[0010] Among them, x and y are the remote sensing image and the corresponding ground truth, h and w are the input height and width, c is the number of surface cover categories, and n is the number of training samples.

[0011] The target domain dataset used as model input only contains cropped remote sensing image patches:

[0012]

[0013] 2): n b As the number of source or target domain data sample inputs for each iterative training process, mean subtraction and normalization are performed on the input images to accelerate the convergence of weights and bias parameters in the network:

[0014] x i =(x i,j -m j ) / d j (3)

[0015] Among them, x i,j is the pixel value of each pixel of input image i in band j, m j and d j are the mean and variance of all image data in this band respectively.

[0016] 3): According to the size of the above input image data, a convolution operation with a convolution kernel size of 7×7 is constructed to perform the initial downsampling operation:

[0017] f(x i ) = conv(x i )(4)

[0018] Among them, x i is the input data of the source domain or target domain, conv is a two-dimensional convolution operation, f(x i ) are output features.

[0019] 4): Based on the above output features, a pre-trained residual network is constructed to extract effective semantic features of source or target domain data. The baseline network consists of multiple residual blocks with the same structure as the feature extraction path for network inference. Each residual block can be expressed as:

[0020] H=R(F(x))+F(x)(5)

[0021] Among them, F(x) is the input feature of the residual block, R(.) and H are the superimposed nonlinear convolution operation and mapped output features respectively.

[0022] 5): According to the baseline network, the high-level semantic features F of the source domain and the target domain can be obtained s ∈R c×h×w and F t ∈R c×h×w , a novel category-level discriminator is developed to constitute the local domain adaptation module.

[0023] First, the discriminator labels belonging to training samples in different data spaces are defined as vectors [0; l] or [l; 0], and two specific strategies are designed to construct their non-zero elements l. The design scheme of the source domain discriminator hard label hl is:

[0024]

[0025] in, Represents the softmax prediction of the kth category. For the target domain data, a soft strategy is adopted to generate the discriminator label of the target domain to solve the domain adaptation problem of all object categories:

[0026]

[0027] in, represents the logits prediction for the k-th class.

[0028] According to the above predefined domain labels, our local domain adaptation module also follows the adversarial process in adversarial learning, and its adversarial loss for updating the baseline network can be expressed as:

[0029]

[0030] Among them, D l is a category-level local discriminator.

[0031] 6): Predict P based on the source domain generated by the baseline network and the classifier s and target domain prediction P t , global adversarial domain adaptation is used to solve the marginal distribution offset. This module also follows the adversarial process in adversarial learning. The adversarial loss of updating the baseline network and classifier can be expressed as:

[0032]

[0033] 7): For the target domain output P generated by the above hybrid framework t , construct an upsampling layer to restore the original size and generate the final pixel-level classification prediction:

[0034] P = upsample(P t )(10)

[0035] Among them, the unsample upsampling adopts the bilinear method.

[0036] The present invention adopting the above technical solution can extract and utilize the potential category information contained in high-resolution optical remote sensing images. The method first performs normalization processing by counting the statistical distribution characteristics of the original remote sensing images on the RGB bands of the source domain and the target domain, and then designs a strong baseline network according to the input image size to extract high-level semantic features related to the category, and then integrates the category-level discriminator and the binary discriminator in the feature space and the output space to perform local and global adversarial domain adaptation respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an example of a domain-adapted land cover classification dataset of the present invention;

[0038] Figure 2 The residual network block of the present invention;

[0039] Figure 3 It is the local adversarial domain adaptation module of the present invention;

[0040] Figure 4 It is the global adversarial domain adaptation module of the present invention. Specific implementation plan

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation described is only a part of the implementation of the present invention, not all embodiments. Based on the implementation of the present invention, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention proposes a land cover classification method based on deep learning and class-aware domain adaptation. The classification algorithm can extract and utilize the potential category information contained in high-resolution optical remote sensing images. The method first performs normalization processing by counting the statistical distribution characteristics of the original remote sensing images on the RGB bands of the source domain and the target domain. Secondly, a strong baseline network is designed according to the input image size to extract category-related high-level semantic features. Then, the category-level discriminator and the binary discriminator are integrated in the feature space and the output space to perform local and global adversarial domain adaptation respectively.

[0043] To achieve the above objectives, the specific steps of the present invention are as follows:

[0044] 1): The data used for domain-adaptive land cover classification model training consists of a labeled source domain dataset and an unsupervised target domain dataset ( Figure 1 ). The source domain dataset includes the original remote sensing images and ground truth values, which are cropped into many small blocks by sliding windows:

[0045]

[0046] Among them, x and y are the remote sensing image and the corresponding ground truth, h and w are the height and width of the input, c is the number of surface cover categories, and n is the number of training samples.

[0047] The target domain dataset used as model input only contains cropped remote sensing image patches:

[0048]

[0049] 2): n b As a batch, the number of source or target domain data samples is used as the input for each iterative training process. The input images are subjected to mean subtraction and normalization to accelerate the convergence of the weights and bias parameters in the network:

[0050] x i =(x i,j -m j ) / d j (13)

[0051] Among them, x i,j is the pixel value of each pixel in the input image i in band j, m j and dj are the mean and variance of all image data in this band respectively.

[0052] 3): According to the size of the above input image data, a convolution operation with a convolution kernel size of 7×7 is constructed to perform the initial downsampling operation:

[0053] f(x i ) = conv(x i )(14)

[0054] Among them, x i is the input data of the source domain or target domain, conv is a two-dimensional convolution operation, f(x i ) are output features.

[0055] 4): Based on the above output features, a pre-trained residual network is constructed to extract effective semantic features of source or target domain data. The baseline network consists of multiple residual blocks with the same structure ( Figure 2 ) as the feature extraction path for network reasoning, each residual block can be expressed as:

[0056] H=R(F(x))+F(x)(15)

[0057] Among them, F(x) is the input feature of the residual block, R(·) and H are the superimposed nonlinear convolution operation and the mapped output feature, respectively.

[0058] 5): High-level semantic features of the source and target domains obtained by the baseline network constructed based on the baseline network F s ∈R c×h×w and F t ∈R c×h×w , develop a novel category-level discriminator to form a local domain adaptation module ( Figure 3 ).

[0059] First, the discriminator labels belonging to training samples in different data spaces are defined as vectors [0; l] or [l; 0], and two specific strategies are designed to construct their non-zero elements l. The design scheme of the source domain discriminator hard label hl is:

[0060]

[0061] in, Represents the softmax prediction of the kth category. For the target domain data, a soft strategy is adopted to generate the discriminator label of the target domain to solve the domain adaptation problem of all object categories:

[0062]

[0063] in, represents the logits prediction for the k-th class.

[0064] According to the above predefined domain labels, our local domain adaptation module also follows the adversarial process in adversarial learning, and its adversarial loss for updating the baseline network can be expressed as:

[0065]

[0066] Among them, D l is a category-level local discriminator.

[0067] 6): Predict P based on the source domain generated by the baseline network and the classifier s and target domain prediction P t , using the global adversarial domain adaptation module ( Figure 4 ) solves the marginal distribution offset. This module also follows the adversarial process in adversarial learning. The adversarial loss of updating the baseline network and classifier can be expressed as:

[0068]

[0069] 7): For the target domain output P generated by the above hybrid framework t , construct an upsampling layer to restore the original size and generate the final pixel-level classification prediction:

[0070] P = upsample(P t ) (20)

[0071] Among them, the unsample upsampling adopts the bilinear method.

Claims

1. An unsupervised land cover classification method based on class-aware domain adaptation, characterized in that: The steps include: 1): The data used for domain-adaptive land cover classification model training consists of a labeled source domain dataset and an unsupervised target domain dataset; the source domain dataset includes the original remote sensing image and the ground truth, and is cropped into many small blocks by a sliding window: Among them, x and y are the remote sensing image and the corresponding ground truth, h and w are the input height and width, c is the number of land cover categories, and n is the number of training samples; The target domain dataset used as model input only contains cropped remote sensing image patches: 2): n b As the number of source or target domain data sample inputs for each iterative training process, mean subtraction and normalization are performed on the input images to accelerate the convergence of weights and bias parameters in the network: x i =(x i,j -m j ) / d j (3) Among them, x i,j is the pixel value of each pixel of input image i in band j, m j and d j are the mean and variance of all image data in this band respectively; 3): According to the size of the above input image data, a convolution operation with a convolution kernel size of 7×7 is constructed to perform the initial downsampling operation: f(x i )=conv(x i ) (4) Among them, x i is the input data of the source domain or target domain, conv is a two-dimensional convolution operation, f(x i ) is the output feature; 4): Based on the above output features, a pre-trained residual network is constructed to extract effective semantic features of the source domain or target domain data; the residual network is composed of multiple residual blocks with the same structure as the feature extraction path for network inference, and each residual block can be expressed as: H=R(F(x))+F(x) (5) Among them, F(x) is the input feature of the residual block, R(.) and H are the superimposed nonlinear convolution operation and the mapped output feature respectively; 5): High-level semantic features F of the source and target domains obtained by the baseline network composed of residual networks s ∈R c×h×w and F t ∈R c×h×w , establish a category-level discriminator to form a local domain adaptation module; 6): Predict P based on the source domain generated by the baseline network and the classifier s and target domain prediction P t , global adversarial domain adaptation is used to solve the marginal distribution offset. This module also follows the adversarial process in adversarial learning. The adversarial loss of updating the baseline network and classifier can be expressed as: 7): For the target domain output P generated by the hybrid framework t , construct an upsampling layer to restore the original size and generate the final pixel-level classification prediction: P = upsample(P t )(7) Among them, the unsample upsampling adopts the bilinear method.

2. The unsupervised land cover classification method based on class perception domain adaptation according to claim 1 is characterized in that: In step 5), the establishment process of the category-level discriminator is as follows: First, the discriminator labels of training samples in different data spaces are defined as vectors [0; l] or [l; 0], and two specific strategies are used to construct their non-zero elements l. The design scheme of the source domain discriminator hard label hl is as follows: in, represents the softmax prediction of the kth category; for the target domain data, a soft strategy is adopted to generate the discriminator label of the target domain to solve the domain adaptation problem of all object categories: in, represents the logits prediction of the kth category; According to the predefined domain labels, the local domain adaptation module also follows the adversarial process in adversarial learning, and its adversarial loss for updating the baseline network can be expressed as: Among them, D l is a category-level local discriminator.

Images