Farthest point sampling method for semantic segmentation of remote sensing image style imbalance dataset

Abstract

The application discloses a farthest point sampling method for a remote sensing image semantic segmentation style unbalanced dataset, which distinguishes the styles of different images by constructing the RGB space probability distribution of each image, calculates the distance between the RGB space probability distributions of each two samples to measure the similarity of the styles, and constructs a distance matrix, and finally adopts a farthest point sampling algorithm to sample the dataset, aiming to select a representative data subset with a smaller scale and a balanced and diverse style from a large-scale dataset, so as to reduce the scale of the dataset and the calculation resource consumption of training a deep model while improving the robustness and generalization ability of the model. In the face of a large-scale redundant remote sensing image dataset and a calculation resource limited scene, the dataset sampling method is effective for improving the generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image data sampling technology, specifically relating to a farthest point sampling method for a style imbalanced dataset for semantic segmentation of remote sensing images. Background Technology

[0002] In recent years, with the development of remote sensing technology, remote sensing images have been increasingly widely used in agricultural management, environmental protection, disaster monitoring, urban planning, and other fields. Semantic segmentation of remote sensing images aims to label each pixel in raster-format geospatial data with a set of semantic labels, such as buildings, bare land, rivers, and roads, and is an important component of many applications. With the development of convolutional neural networks and powerful feature extraction methods, semantic segmentation algorithms have been greatly improved. Today, methods based on fully convolutional networks, such as FCN, U-Net, and DeepLab [Yuan X, Shi J, Gu LA review of deep learning methods for semantic segmentation of remote sensing imagery[J]. Expert Systems with Applications, 2021, 169: 114417], almost dominate the field of semantic segmentation. However, semantic segmentation of remote sensing images remains a challenge. One reason is that, in addition to differences in satellite sensors, variations in atmospheric conditions, panchromatic and multispectral fusion algorithms, atmospheric and radiometric corrections, seasons, weather, regions, and landforms result in diverse styles of remote sensing images, which places higher demands on the generalization ability of the model.

[0003] Thanks to the development of the internet and the spirit of open source, we can easily obtain many publicly available remote sensing image datasets. To improve generalization ability, we often merge multiple datasets to train a model with stronger generalization capabilities. However, limited by computing resources, we cannot merge as many datasets as possible for training. Real-world large-scale remote sensing image datasets are often redundant and follow a long-tailed distribution [Zhang Y, Kang B, Hooi B, et al. Deep long-tailed learning: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023], meaning that a small number of styles have a large number of training samples, while a large number of styles have only a small number of training samples. This means that models trained using style-imbalanced training sets typically perform well only in head styles but lack generalization ability, making it difficult to generalize to other styles. This is essentially a manifestation of head style overfitting. Generally, we expect the training set to have balanced and diverse styles to avoid overfitting the model to a single head style, thus ensuring the model's generalization ability. This involves dataset sampling techniques.

[0004] In practice, a common sampling method is a technique widely used in class-imbalanced datasets [Haixiang G, Yijing L, Shang J, et al. Learning from class-imbalanced data: Review of methods and applications[J]. Expert systems with applications, 2017, 73: 220-239]. Its goal is to undersample the head classes and oversample the tail classes based on the class label distribution, thereby balancing the inter-class distribution. However, it is not suitable for the style imbalance problem in remote sensing semantic segmentation datasets because style imbalance differs from class imbalance. The key issue is that in reality, when dealing with style-imbalanced datasets, we cannot know the style of each sample, and therefore cannot oversample or undersample based on the sample's style. Summary of the Invention

[0005] In view of the above, the present invention provides a farthest point sampling method for style-imbalanced datasets in semantic segmentation of remote sensing images. It distinguishes the styles of different images by calculating the color distribution of each image and calculates the distance between the color distributions of different images as the style difference. Then, the farthest point sampling algorithm is used to sample the original dataset. The aim is to select a representative subset of data with diverse styles from a large-scale dataset, thereby reducing the size of the dataset and the computational resource consumption for training deep models while improving the robustness and generalization ability of the model.

[0006] A method for sampling the farthest point of a style-imbalanced dataset for semantic segmentation of remote sensing images includes the following steps:

[0007] (1) Obtain the original redundant and style-unbalanced remote sensing image dataset. For the image samples in the dataset, calculate their RGB color space distribution to represent the style of the samples.

[0008] (2) Perform Gaussian smoothing on the RGB color space distribution of the image samples and convert it into a spatial probability distribution;

[0009] (3) Calculate the distance between the spatial probability distributions of each pair of samples in the dataset, and construct a distance matrix to quantify the style differences between samples;

[0010] (4) Based on the distance matrix, the image samples in the dataset are sampled using the farthest point sampling algorithm to obtain a smaller and style-balanced subset of data for downstream model training tasks.

[0011] Further, the specific implementation of step (1) is as follows: First, the RGB color space is discretized, that is, the range of 0 to 255 of each axis of RGB is divided into 32 parts, each part contains 8 gray levels, and the range of each axis of the discretized RGB color space is 0 to 31. Then, the distribution frequency of all pixels in the image sample in the discretized RGB color space is counted. For the RGB color space distribution f(i,j,k) of the image sample, it is represented as the number of pixels in the image sample whose RGB three-channel gray values ​​correspond to i,j,k, where i,j,k are natural numbers and 0≤i,j,k≤31.

[0012] Furthermore, the specific implementation of Gaussian smoothing in step (2) is as follows: For the RGB color space distribution f(i,j,k) of the image sample, firstly, edge values ​​need to be filled in f(i,j,k) before convolution, and then a Gaussian convolution kernel with a size of 3×3×3 and a standard deviation of 1.0 is used to smooth f(i,j,k), as shown in the following expression:

[0013]

[0014] Where g(i,j,k) is the smoothed RGB color space distribution.

[0015] Furthermore, in step (2), the smoothed RGB color space distribution g(i,j,k) is divided by the total number of pixels in the image samples, thereby transforming it into a spatial probability distribution with a sum of 1.0.

[0016] Furthermore, in step (3), the distance between the pairwise sample spatial probability distributions is calculated using the Bach distance, and the specific calculation expression is as follows:

[0017]

[0018] Where p(i,j,k) and q(i,j,k) are the spatial probability distributions of any two image samples in the dataset, respectively, and D B (p(i,j,k),q(i,j,k)) is the Bach distance between p(i,j,k) and q(i,j,k).

[0019] Furthermore, the specific implementation of step (4) is as follows:

[0020] 4.1 Set the size and number of data subsets, and initially select a sample from the dataset to include in the data subset;

[0021] 4.2 For any sample in the dataset, retrieve its distance to all samples in the data subset based on the distance matrix, and take the minimum distance as d;

[0022] 4.3 According to step 4.2, traverse all samples in the dataset and select the sample with the largest corresponding d value to include in the data subset;

[0023] 4.4 Repeat steps 4.2 to 4.3 until the data subset reaches the set size.

[0024] Furthermore, the initialization process in step 4.1 is as follows: first, extract the pair of samples with the largest distance value from the dataset according to the distance matrix, and then arbitrarily select one sample from this pair of samples to include in the data subset.

[0025] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0026] 1. This invention expresses the abstract concept of style using a color space probability distribution, and the differences between styles are expressed by the distance between color distributions, making it feasible to sample remote sensing image datasets with style imbalance.

[0027] 2. The data subset sampled by the farthest point sampling algorithm of this invention takes into account the style of the tail (sparse sample size) samples in the long-tail distribution, so that the deep learning model trained on this data subset has strong generalization ability and will not overfit to the head style due to the long-tail distribution of styles.

[0028] 3. In the face of redundant remote sensing image segmentation datasets with large data scales and limited computing resources, this invention can adjust the number of samples according to the size of computing resources, save computing resources during model training, and make the trained model have better generalization ability as much as possible. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the sampling method for the farthest point of remote sensing image datasets according to the present invention. Detailed Implementation

[0030] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] This invention relates to a sampling method for constructing a smaller, style-balanced subset of data from a raw, redundant, and style-imbalanced remote sensing image dataset. The specific implementation steps are as follows: Figure 1 As shown, the detailed implementation process is as follows:

[0032] (1) Color space discretization and distribution statistics: In order to simplify the computational cost, the RGB color three-dimensional space is first discretized. Each axis range of 0 to 255 is divided into 32 bins, and each bin contains 8 gray levels. The range of each axis of the simplified RGB color space is 0 to 31. Then, the distribution of all pixels of each sample image in the discrete color space is statistically analyzed.

[0033] (2) Gaussian smoothing and probability distribution transformation: The color space distribution is represented by f(i,j,k), where For any 0 ≤ i, j, k ≤ 31, to ensure the size remains unchanged after Gaussian convolution (32 × 32 × 32), we need to pad the 3D probability distribution with edge values ​​before convolution. Then, we use a Gaussian convolution kernel with size = 3 × 3 × 3 and σ = 1.0 to smooth the color space distribution f(i, j, k). Assuming g(i, j, k) is the smoothed color space distribution, the calculation formula is as follows:

[0034]

[0035] To better calculate the Bach distance, the smoothed color space distribution of each sample is divided by the sum of the values ​​of that color space distribution, thus transforming it into a three-dimensional probability distribution with a total sum of 1.0.

[0036] (3) Constructing the distance matrix between samples: Calculate the distance between each pair of RGB spatial probability distributions to obtain a two-dimensional distance matrix recording the distance between each pair of samples. Obviously, this distance matrix is ​​symmetric about the diagonal, and the smaller the distance, the higher the style similarity (the diagonal distance value is 0). There are many methods to calculate the two probability distributions: such as Bach's distance, Pearson similarity coefficient, cosine similarity, etc. In this implementation, we use Bach's distance D. B The classic formula for measuring the similarity of two three-dimensional probability distributions p(i,j,k) and q(i,j,k) is expressed in the following form:

[0037]

[0038] (4) Farthest Point Sampling Algorithm: Based on the two-dimensional distance matrix, the farthest point sampling algorithm is used to select samples. Assuming the original dataset has N samples, the algorithm... This means that the target sample consists of M samples forming a data subset. Each time, the sample point with the largest minimum distance to all samples in the currently selected sample set S = {S0, S1…} is selected and added to set S. The specific steps are as follows:

[0039] 4.1 Initial Sample Selection: For the input distance matrix, select any one of the sample points from the pair with the largest distance value as the starting point S0, obtaining the initial sampling set S = {S0}. Simultaneously, use an array... Record the minimum distance from all N samples to all samples in the sampling set S (initially, the sampling set contains only one sample point S0), where L i Let L represent the minimum distance from the i-th sample point to all samples in the sampling set S. Obviously, if the i-th sample point has already been added to the sampling set, then L... i =0;

[0040] 4.2 Sample Addition: Given an array L recording the minimum distances from all N samples to all samples in the sampling set S, select the sample point P corresponding to the maximum distance from this array. x Add to the sampling set S;

[0041] 4.3 Update the minimum distance record: Take all samples from the constructed distance matrix and add them to P. x Distance array (Equivalent to the distance data in the x-th row or column of the distance matrix constructed in step (3)), obviously arrays D and L have the same length, for 1≤i≤N, if D i Less than L i Then update L i =D i ,Right now In this way, the updated array L will always keep track of the minimum distance from all N samples to all samples in the sampling point set S;

[0042] 4.4 Iteratively add sample points: Repeat steps 4.2 to 4.3 until sampling of M samples is completed.

[0043] (5) Use the selected subset of data to train the model to save computing resources while improving the model’s generalization ability.

[0044] 5.1 Regarding the evaluation of the model's generalization ability, we used the publicly available dataset Satellitedataset I (global cities) from Wuhan University. This dataset was collected from cities and various remote sensing resources worldwide and contains 204 images (512×512 3-channel remote sensing images). Besides differences in satellite sensors, atmospheric conditions, panchromatic and multispectral fusion algorithms, atmospheric and radiometric corrections, and seasonal variations make these samples suitable for testing the robustness and generalization ability of the building extraction model. We used the Intersection over Union (IOU) metric to represent the model's generalization ability on this dataset.

[0045] 5.2 The original large redundant dataset used in this implementation method is a dataset of approximately 5,000 512×512 3-channel remote sensing images.

[0046] 5.3 The parameters of the training model used in this implementation are as follows: In the experiment, a 5-layer UNet model was built in PyTorch on an NVIDIA GTX2060 GPU, with a learning rate of 1e-3, a cosine annealing strategy, and iterative training for 30 epochs with a batch size of 10. For data augmentation, we used random cropping and resizing to the original size, random flipping, random rotation, and random Gaussian blurring.

[0047] 5.4 This implementation method sets up a comparative experiment with random sampling under the same conditions. The number of samples is the same as that of the algorithm in this invention. Models with the same randomly initialized parameters are trained for 30 epochs on both the dataset sampled by the algorithm of this invention and the randomly sampled dataset. The IOU metric that performs best on the Satellite dataset I (global cities) is selected as its generalization ability. Based on the characteristics of the original dataset and the size of the computing resources, three comparative experiments were conducted with sampling sample sizes of 1000, 1500, and 2000. It is obvious that the reduction in the training set after sampling will greatly reduce the training time and the consumption of training resources compared with the original dataset. The results of the three comparative experiments are shown in Table 1:

[0048] Table 1

[0049]

[0050]

[0051] The experimental results show that the IOU index of the three experimental schemes based on the sampling algorithm of this invention and the random sampling algorithm in this embodiment is improved. This indicates that the sampling algorithm of this invention can reduce the size of the original dataset and save computational resources while maintaining the style diversity of the samples as much as possible to train a model with better generalization ability. Therefore, this invention is effective in improving the generalization ability of the model by sampling the dataset and maintaining the style diversity of the samples as much as possible in the face of large redundant remote sensing datasets and limited computational resources.

[0052] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for sampling the farthest point of a style-imbalanced dataset for semantic segmentation of remote sensing images, comprising the following steps: (1) Obtain the original redundant and style-unbalanced remote sensing image dataset. For the image samples in the dataset, calculate their RGB color space distribution to represent the style of the samples. (2) Perform Gaussian smoothing on the RGB color space distribution of the image samples and convert it into a spatial probability distribution; (3) Calculate the distance between the spatial probability distributions of each pair of samples in the dataset, and construct a distance matrix to quantify the style differences between samples; (4) Based on the distance matrix, the image samples in the dataset are sampled using the farthest point sampling algorithm to obtain a style-balanced subset of data for downstream model training tasks. The specific implementation method is as follows: 4.1 Set the size and number of data subsets, and initially select a sample from the dataset to include in the data subset; 4.2 For any sample in the dataset, retrieve its distance to all samples in the data subset based on the distance matrix, and take the minimum distance as d; 4.3 According to step 4.2, traverse all samples in the dataset and select the sample with the largest corresponding d value to include in the data subset; 4.4 Repeat steps 4.2 to 4.3 until the data subset reaches the set size.

2. The farthest point sampling method of claim 1, wherein: The specific implementation of step (1) is as follows: First, the RGB color space is discretized, that is, the range of 0 to 255 of each axis of RGB is divided into 32 parts, each part contains 8 gray levels. The range of each axis of the discretized RGB color space is 0 to 31. Then, the distribution frequency of all pixels in the image sample in the discretized RGB color space is counted. For the RGB color space distribution f(i,j,k) of the image sample, it is represented as the number of pixels in the image sample whose gray values ​​of the RGB three channels correspond to i,j,k, where i,j,k are natural numbers and 0≤i,j,k≤31.

3. The farthest point sampling method of claim 1, wherein: The specific implementation of Gaussian smoothing in step (2) is as follows: For the RGB color space distribution f(i,j,k) of the image sample, firstly, edge values ​​need to be filled in f(i,j,k) before convolution, and then a Gaussian convolution kernel with a size of 3×3×3 and a standard deviation of 1.0 is used to smooth f(i,j,k). The specific expression is as follows: Where g(i,j,k) is the smoothed RGB color space distribution.

4. The farthest point sampling method of claim 1, wherein: In step (2), the smoothed RGB color space distribution g(i,j,k) is divided by the total number of pixels in the image samples, thus transforming it into a spatial probability distribution with a sum of 1.

0.

5. The farthest point sampling method of claim 1, wherein: In step (3), the distance between the pairwise spatial probability distributions of samples is calculated using the Bach distance, and the specific calculation expression is as follows: where: p(i,j,k) and q(i,j,k) are the spatial probability distributions of any two image samples in the dataset, D B (p(i,j,k),q(i,j,k)) is the Bhattacharyya distance of p(i,j,k) and q(i,j,k).

6. The farthest point sampling method of claim 1, wherein: The initialization process in step 4.1 is as follows: First, extract the pair of samples with the largest distance value from the dataset according to the distance matrix, and then arbitrarily select one sample from this pair of samples to include in the data subset.

Images