Difficult Sample Mining Method for Remote Sensing Images

Abstract

The present invention discloses a method for mining difficult samples of remote sensing images. The method pre-trains a semantic segmentation module including two semantic segmentation models, inputs the image to be interpreted into the semantic segmentation module, obtains a first semantic segmentation result and a second semantic segmentation result, and then obtains a common semantic segmentation result, a full semantic segmentation result and an intersection-and-union ratio through intersection calculation, union calculation and intersection-and-union ratio calculation. Finally, by comparing the intersection-and-union ratio and a preset IOU threshold, the samples with an intersection-and-union ratio less than the preset IOU threshold are recorded as difficult samples, and the samples with an intersection-and-union ratio greater than the preset IOU threshold are recorded as simple samples. The difficult samples are manually labeled, the simple samples are manually quality-checked, and added to a sample library. The method obtains different semantic segmentation results for the same image through two different semantic segmentation models, and distinguishes simple samples from difficult samples by calculating the intersection-and-union ratio of the segmentation results. Simple samples and manually labeled difficult samples can expand the sample library.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a method for mining difficult samples of remote sensing images. Background Art

[0002] In recent years, the development of aerospace technology, sensor technology, control technology, electronic technology, computer technology and communication technology has greatly promoted the development of remote sensing technology. Various remote sensing platforms operating in space continuously observe the earth at multiple scales, multiple levels, multiple angles and multiple spectrums, and various advanced earth observation systems continuously provide rich data sources to the ground. How to obtain the required information from massive remote sensing data in a timely and accurate manner and make use of it is one of the current problems that need to be solved urgently. Remote sensing image interpretation technology has emerged with the development of remote sensing technology. The data obtained by the sensor must be processed and interpreted to become useful information. Faced with the ever-increasing variety of massive remote sensing image data, how to achieve rapid and accurate interpretation of remote sensing images has become the key to the application of remote sensing technology. Compared with ordinary images, remote sensing images have problems such as complex backgrounds and incomplete target features, which can easily interfere with interpretation.

[0003] Remote sensing interpretation has gone through a process from manual interpretation to semi-automatic interpretation, and is developing towards fully intelligent interpretation. The interpretation method based on deep learning has great advantages. It can automatically perform end-to-end feature extraction and interpretation without manual intervention. Interpretation based on deep learning has become one of the key development directions in the current remote sensing field. Deep learning requires a large number of training samples to improve the generalization ability of the model and alleviate the overfitting phenomenon. However, according to a number of experiments, under the support of a certain base number of training samples, simply continuing to increase the number of training samples will not improve the performance of the model, and sometimes it will lead to a decrease in model performance. When selecting training samples, it is necessary to consider the balance and diversity of samples. Summary of the invention

[0004] The present invention proposes a method for mining difficult samples of remote sensing images. Different semantic segmentation results for the same image are obtained through two different semantic segmentation models. The intersection-over-union ratio of the segmentation results is calculated, and the IOU threshold is used to distinguish simple samples from difficult samples to achieve mining of difficult samples. Simple samples and difficult samples that have been manually annotated can be used to expand the sample library. In addition, one of the segmentation models adopted by the present invention uses frame field technology to improve the contour accuracy of the segmentation results, which can greatly improve the interpretation accuracy of buildings and structures, thereby ensuring that the simple samples finally obtained can be directly stored after manual quality inspection. The difficult samples that have been manually annotated can effectively improve the balance and diversity of the sample library and solve the problem of incomplete target features.

[0005] To achieve the above technical objectives, the technical solution of the present invention is as follows:

[0006] A remote sensing image difficult sample mining method, the method comprising the following steps:

[0007] S1 samples the remote sensing sample library and divides the sampled samples into a training set and a test set;

[0008] S2 uses the training set and the test set to train a semantic segmentation module, where the semantic segmentation module includes a first semantic segmentation model and a second semantic segmentation model;

[0009] S3: inputting the image to be interpreted into the semantic segmentation module, the first semantic segmentation model outputs a first semantic segmentation result, and the second semantic segmentation model outputs a second semantic segmentation result;

[0010] S4 performs intersection calculation, union calculation and intersection-over-union ratio calculation on the first semantic segmentation result and the second semantic segmentation result to obtain a common segmentation result, a full segmentation result and an intersection-over-union ratio;

[0011] S5 compares the intersection-over-union ratio with a preset IOU threshold, and determines a simple sample and a difficult sample according to the comparison result;

[0012] S6 manually labels the difficult samples, performs manual quality inspection on the simple samples, and adds the difficult samples and the simple samples to the remote sensing sample library.

[0013] Furthermore, in step S1, sampling the remote sensing sample library includes:

[0014] Set the sample base, and set the similarity threshold according to the sample base;

[0015] Calculating the similarity between the image to be interpreted and the samples in the remote sensing sample library, wherein the similarity includes texture structure similarity, gradient structure similarity and edge structure similarity;

[0016] Selecting samples whose similarity is greater than a similarity threshold in the remote sensing sample library;

[0017] A sample base number of samples is selected from the samples whose similarity is greater than the similarity threshold.

[0018] Furthermore, the first semantic segmentation model includes:

[0019] A first semantic segmentation network outputs a pixel segmentation map, wherein the pixel segmentation map includes two channels, corresponding to internal labels and boundary labels, respectively, and the first semantic segmentation network is U-Net or D-LinkNet;

[0020] A frame field network, which outputs a frame field according to the pixel segmentation map, wherein the frame field includes four channels corresponding to two complex coefficients and a complex number respectively;

[0021] The frame field post-processing network optimizes the contour of the pixel segmentation map according to the active contour model method to make it consistent with the frame field.

[0022] Furthermore, the second semantic segmentation model includes:

[0023] The second semantic segmentation network is a variational autoencoder VAE or VQ-VAE or VQ-VAE-2 or β-VAE.

[0024] Furthermore, in step S2, using the training set and the test set to train a semantic segmentation module includes:

[0025] The supervision information of the training set includes the target internal label and the target boundary label;

[0026] The loss functions used in training include segmentation loss, frame field loss, output coupling loss, and final loss.

[0027] Optionally, in step S3, the first semantic segmentation result is a pixel segmentation map with optimized contours, which is saved in a vector form; the second semantic segmentation result is a pixel segmentation map, which is saved in a vector form.

[0028] Optionally, step S3 further includes: cropping the image to be interpreted.

[0029] Optionally, step S4 includes:

[0030]

[0031]

[0032]

[0033] Among them, i is the sequence number of the semantic segmentation result, M 1 is the first semantic segmentation result, M 2 is the second semantic segmentation result, S inter is the public segmentation result, S union The full segmentation result.

[0034] Optionally, in step S5, the preset IOU threshold includes an IOU upper threshold and an IOU lower threshold:

[0035] The second semantic segmentation result whose intersection-over-union ratio is less than the IOU lower threshold is recorded as a difficult sample;

[0036] The first semantic segmentation result whose IOU is greater than the IOU upper threshold is recorded as a simple sample.

[0037] The present invention proposes a method for mining difficult samples of remote sensing images. First, the remote sensing sample library is sampled to obtain the best training sample data. The training sample data is used to train a semantic segmentation module. The semantic segmentation module includes two semantic segmentation models, one of which uses frame field learning technology to improve the accuracy of the semantic segmentation result contour. Then, the trained semantic segmentation module is used to segment the image to be interpreted to obtain two semantic segmentation results. The intersection, union and intersection-over-union ratio between the two semantic segmentation results are calculated. The similarity of the two semantic segmentation results is obtained according to the size of the intersection-over-union ratio. Finally, by comparing the intersection-over-union ratio and the preset IOU threshold, simple samples and difficult samples are obtained, simple samples are manually quality inspected, and difficult samples are manually labeled. The simple samples and difficult samples finally obtained are placed in the remote sensing sample library. The beneficial effects of the present invention are:

[0038] (1) A new method for mining difficult samples in remote sensing images by calculating the intersection-over-union ratio of semantic segmentation results is proposed;

[0039] (2) By expanding the sample library through the mined difficult samples, the problem of sample imbalance in the training stage is improved. The number of difficult samples is increased during the training process so that difficult samples with fewer target features and incomplete features can be fully trained;

[0040] (3) Frame field learning is used to effectively improve the segmentation accuracy of building contours in semantic segmentation, thereby improving the accuracy of mining difficult samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a process flow of an embodiment of a method for mining difficult samples in remote sensing images according to the present invention;

[0043] Figure 2 This is a flow chart of an embodiment of a method for mining difficult samples in remote sensing images according to the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. It should be noted that as long as there is no conflict, the various embodiments and various features of the embodiments in the present invention can be combined with each other, and the technical solutions formed are all within the protection scope of the present invention.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0046] The technical ideas of the present invention are as follows: (1) Different semantic segmentation models have different semantic understanding capabilities for the target image. By using a semantic segmentation module including two different semantic segmentation models to perform semantic segmentation on the same image, the similarity of the semantic segmentation results can be regarded as the difficulty of understanding the image. The two semantic segmentation results with high similarity are recorded as simple samples, and the ones with low similarity are recorded as difficult samples after manual annotation, so as to realize the mining of difficult samples; (2) Difficult samples can improve the balance and diversity of the sample library and improve the learning effect of the subsequent interpretation model; (3) Using frame field learning technology, by adding frame fields to the convolutional network, useful information is provided for segmentation vectorization and regularization. Frame field learning technology can handle complex scenes such as large buildings and buildings with internal holes by fitting a skeleton graph in which each edge connects two nodes with a vertex. It improves the interpretation ability of a single model for buildings and thus improves the accuracy of mining difficult samples.

[0047] Please refer to Figure 1 , Figure 1 The figure is a flow chart of the method for mining difficult samples of remote sensing images according to the present invention. Compared with the existing method, the method comprises the following steps:

[0048] S1 samples the remote sensing sample library and divides the sampled samples into a training set and a test set;

[0049] S2 uses the training set and the test set to train a semantic segmentation module, where the semantic segmentation module includes a first semantic segmentation model and a second semantic segmentation model;

[0050] S3: inputting the image to be interpreted into the semantic segmentation module, the first semantic segmentation model outputs a first semantic segmentation result, and the second semantic segmentation model outputs a second semantic segmentation result;

[0051] S4 performs intersection calculation, union calculation and intersection-over-union ratio calculation on the first semantic segmentation result and the second semantic segmentation result to obtain a common segmentation result, a full segmentation result and an intersection-over-union ratio;

[0052] S5 compares the intersection-over-union ratio with a preset IOU threshold, and determines a simple sample and a difficult sample according to the comparison result;

[0053] S6 manually labels the difficult samples, performs manual quality inspection on the simple samples, and adds the difficult samples and the simple samples to the remote sensing sample library.

[0054] Example

[0055] S1 samples the remote sensing sample library and divides the sampled samples into training sets and test sets.

[0056] The samples of the remote sensing sample library used in this embodiment are pictures, which can be in different formats and sizes. Pictures of different sizes and formats need to be preprocessed and adjusted to a uniform format and size. Sample annotation is pixel-level annotation, which can be completed by manual annotation or machine annotation. The annotation content includes the type of the feature and the internal label and boundary label of the feature. The feature type targeted by this embodiment is a building.

[0057] The purpose of sampling includes selecting an appropriate number of sample bases and selecting samples with high correlation with the image to be interpreted. The specific sampling steps are: setting the sample base, setting the similarity threshold according to the sample base; calculating the similarity between the image to be interpreted and the samples in the remote sensing sample library, the similarity includes texture structure similarity, gradient structure similarity and edge structure similarity; selecting samples in the remote sensing sample library whose similarity is greater than the similarity threshold; selecting sample base samples from the samples whose similarity is greater than the similarity threshold.

[0058] The samples obtained by sampling can be divided into a training set and a test set according to a ratio. In this embodiment, the samples obtained by sampling are divided into a training set and a test set according to a ratio of 8:2.

[0059] S2 uses the training set and the test set to train a semantic segmentation module, where the semantic segmentation module includes a first semantic segmentation model and a second semantic segmentation model.

[0060] In this embodiment, the first semantic segmentation model includes a semantic segmentation network, a frame field network, and a frame field post-processing network. The semantic segmentation network outputs a pixel segmentation map, and the pixel segmentation map includes two channels, corresponding to internal labels and boundary labels respectively. The frame field network outputs a frame field according to the pixel segmentation map, and the frame field includes four channels, corresponding to two complex coefficients and complex numbers respectively. The frame field post-processing network optimizes the contour of the pixel segmentation map according to the active contour model method to make it consistent with the frame field.

[0061] In this embodiment, an improved U-Net for building extraction is used as a semantic segmentation network. U-Net uses a fully convolutional network for semantic segmentation and a U-shaped network structure. The downsampling process in this structure can capture semantic information, and the corresponding upsampling process can be accurately positioned, so that end-to-end training can be achieved using fewer training samples and better training results can be obtained. In this embodiment, the input feature dimension is 128*128*3, corresponding to the three bands of the training sample image, and the output feature dimension is 128*128*2, corresponding to the internal label and boundary label of the object. The improved U-Net network adopts a four-layer network structure, including four downsampling and four upsampling. The input sample image is first subjected to a multi-scale maximum pooling operation with pooling sizes of 2, 3, 5, and 7, respectively. The four feature maps obtained are cascaded and spliced, and then restored to their original size through 1*1 convolution and the Relu activation function using the merged feature map. Then, through three convolution operations of the residual module, the input first undergoes two convolution operations with a kernel size of 3*3, batch normalization, and Relu activation function operations to obtain features, and then the input is added to the features, and finally a convolution operation with a kernel size of 3, batch normalization, and Relu activation function is performed. Internal loss and boundary loss are defined for internal labels and boundary labels:

[0062]

[0063]

[0064]

[0065]

[0066] in, is the real pixel, y is the predicted pixel, and are the internal labels and boundary labels of the input image, y int and edge are the predicted internal pixels and boundary pixels, 0<α<1 is a hyperparameter, which is set to 0.2 in this embodiment, L BCE is the cross entropy loss function, L Dice is the Dice loss function, H and W are the height and width of the input image, both of which are 128 in this embodiment, and L int is the internal loss, L edge is the boundary loss.

[0067] The structure of the frame field network includes a convolution layer, a batch normalization layer, an ELU activation function, a convolution layer and a tanh activation function. The input of the frame field network is a cascade of the pixel segmentation map output by the semantic segmentation network and the feature map of the semantic segmentation network. The output frame field includes four channels, corresponding to two complex coefficients and complex numbers respectively. In this embodiment, the building extraction algorithm ASM (flexible skeleton model) is used to add a frame field to the full convolution network to provide useful information for segmentation vectorization and regularization. ASM fits a skeleton graph in which each edge connects two nodes with a vertex, which can handle complex scenes such as large buildings and buildings with holes.

[0068] The frame field is a 4-vector field that specifies four vectors for each point in the plane, with the first two vectors constrained to be relative to the other two, i.e., each point is assigned a set of vectors {u, -u, v, -v}. Define the frame field as two complex numbers u, v∈C. Because buildings, unlike organic shapes, are regular structures with sharp angles, capturing directionality at these sharp angles requires two directions. The direction is encoded in a way that is independent of re-signing and sign changes, as follows:

[0069] f(z)=(z 2 -u 2 )(z 2 -v 2 )=z 4 +c2z 2 +c0=f(z;c0,c2)

[0070] c0,c2∈C are constants. Given a set (c0,c2), we can restore a pair of directions of the corresponding frame:

[0071]

[0072] The losses for training the frame field network include:

[0073]

[0074]

[0075]

[0076] Among them, θ τ ∈[0,π) is the boundary angle label, indicating the direction of the vector τ, L align and L align90 Ensure the consistency between the frame field and the real image, L smooth Enhance the smoothness of frame fields.

[0077] The frame field post-processing network uses the active skeleton model (ASM) to optimize the contours of the pixel segmentation map to align with the frame field, and uses the frame field to detect corners, thereby simplifying non-corner vertices:

[0078]

[0079]

[0080]

[0081] Among them, E is the set of edges, V is the set of vertices, e dir is the direction vector of edge e, e center is the center point of edge e, |e| is the length of edge e, l = 0.5 is the contour line used to initialize the contour by the marching squares method, E prob Make sure the contour is close to the contour line of the pixel segmentation map, E align Make all edges of the outline consistent with the frame field, E length Normalize contour lengths to avoid local overlap.

[0082] Frame field learning can effectively improve the segmentation accuracy of building contours in semantic segmentation.

[0083] The second semantic segmentation model includes a second semantic segmentation network. In this embodiment, VQ-VAE is used as the second semantic segmentation network. When VQ-VAE is used for semantic segmentation, the loss function used in training is the cross entropy loss function and the L2 loss:

[0084]

[0085] where y k is the kth output of the model, t k represents the kth label, t k In , only the label of the correct solution is 1, and the rest are 0, and k is the output node of the kth deep neural network model.

[0086] When training the second semantic segmentation network, the internal labels and boundary labels are merged, and a single label of the target object is used for training.

[0087] VQ-VAE is a generative model based on discrete latent variables and is also an unsupervised representation learning model. The learned discrete coding has strong representation ability and can be used for semantic segmentation to provide good representation.

[0088] S3 inputs the image to be interpreted into the semantic segmentation module, the first semantic segmentation model outputs a first semantic segmentation result, and the second semantic segmentation model outputs a second semantic segmentation result.

[0089] In this embodiment, the image to be interpreted is firstly cut into sample images of 128*128 size uniformly and then input into the semantic segmentation module. The output semantic segmentation results are all saved in vector form.

[0090] S4 performs intersection calculation, union calculation and intersection-over-union ratio calculation on the first semantic segmentation result and the second semantic segmentation result to obtain a common segmentation result, a full segmentation result and an intersection-over-union ratio;

[0091] S5 compares the intersection-over-union ratio with a preset IOU threshold, and determines a simple sample and a difficult sample according to the comparison result;

[0092] The similarity between the two semantic segmentation results can be quantified by calculating the intersection-union ratio. Due to the use of frame field learning technology, the segmentation result of the first semantic segmentation model is a strong semantic segmentation result, while the segmentation result of the second semantic segmentation model can be regarded as a weak semantic segmentation result. The two semantic segmentation results have different levels of semantic understanding of the image, so the two images with small differences in understanding are easy to understand and can be directly recorded as simple samples, while the images with large differences in understanding are difficult to understand and are recorded as difficult samples. The method for calculating the intersection, union, and intersection-union ratio of the semantic segmentation results in this embodiment is:

[0093]

[0094]

[0095]

[0096] Among them, i is the sequence number of the semantic segmentation result, M 1 is the first semantic segmentation result, M 2 is the second semantic segmentation result, S inter is the public segmentation result, S union The full segmentation result.

[0097] The preset IOU threshold used in this embodiment includes an IOU upper threshold and an IOU lower threshold. The IOU upper threshold is set to 0.8, and the IOU lower threshold is set to 0.3.

[0098] The first semantic segmentation result is a more accurate segmentation result, so the first semantic segmentation result can be directly used as a simple sample. The second semantic segmentation result whose intersection-over-union ratio is less than the IOU lower threshold is recorded as a difficult sample; the first semantic segmentation result whose intersection-over-union ratio is greater than the IOU upper threshold is recorded as a simple sample.

[0099] S6 manually labels the difficult samples, performs manual quality inspection on the simple samples, and adds the difficult samples and the simple samples to the remote sensing sample library.

[0100] For difficult sample images, we can quickly perform manual annotation based on the second semantic segmentation results to obtain accurate labeled difficult samples. For simple samples, we can add them to the remote sensing sample library after necessary quality inspection. Figure 2 The figure shows a block diagram of the entire process of the present invention. After adding difficult samples and simple samples to the remote sensing sample library, the balance and diversity of the sample library can be filled. The entire process can be repeated to iteratively optimize the model and sample library to achieve continuous learning.

[0101] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for mining difficult samples of remote sensing images, characterized in that: The steps include: S1 samples the remote sensing sample library and divides the sampled samples into a training set and a test set; S2 uses the training set and the test set to train a semantic segmentation module, where the semantic segmentation module includes a first semantic segmentation model and a second semantic segmentation model; S3: inputting the image to be interpreted into the semantic segmentation module, the first semantic segmentation model outputs a first semantic segmentation result, and the second semantic segmentation model outputs a second semantic segmentation result; S4 performs intersection calculation, union calculation and intersection-over-union ratio calculation on the first semantic segmentation result and the second semantic segmentation result to obtain a common segmentation result, a full segmentation result and an intersection-over-union ratio; S5 compares the intersection-over-union ratio with a preset IOU threshold, and determines a simple sample and a difficult sample according to the comparison result; S6 manually annotates the difficult samples, performs manual quality inspection on the simple samples, and adds the difficult samples and the simple samples to the remote sensing sample library; The first semantic segmentation model includes: A first semantic segmentation network outputs a pixel segmentation map, wherein the pixel segmentation map includes two channels, corresponding to internal labels and boundary labels, respectively, and the first semantic segmentation network is U-Net or D-LinkNet; A frame field network, which outputs a frame field according to the pixel segmentation map, wherein the frame field includes four channels corresponding to two complex coefficients and a complex number respectively; The frame field post-processing network optimizes the contour of the pixel segmentation map according to the active contour model method to make it consistent with the frame field.

2. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: In step S1, sampling the remote sensing sample library includes: Set the sample base, and set the similarity threshold according to the sample base; Calculating the similarity between the image to be interpreted and the samples in the remote sensing sample library, wherein the similarity includes texture structure similarity, gradient structure similarity and edge structure similarity; Selecting samples whose similarity is greater than a similarity threshold in the remote sensing sample library; A sample base number of samples is selected from the samples whose similarity is greater than the similarity threshold.

3. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: The second semantic segmentation model comprises: The second semantic segmentation network is a variational autoencoder VAE or VQ-VAE or VQ-VAE-2 or β-VAE.

4. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: In step S2, training a semantic segmentation module using the training set and the test set includes: The supervision information of the training set includes the target internal label and the target boundary label; The loss functions used in training include segmentation loss, frame field loss, output coupling loss, and final loss.

5. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: In step S3, the first semantic segmentation result is a pixel segmentation map with optimized contours, which is saved in a vector form; the second semantic segmentation result is a pixel segmentation map, which is saved in a vector form.

6. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: Step S3 also includes: cutting the image to be interpreted.

7. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: The step S4 comprises: Among them, i is the sequence number of the semantic segmentation result, M 1 is the first semantic segmentation result, M 2 is the second semantic segmentation result, S inter is the public segmentation result, S union The full segmentation result.

8. The method for mining difficult samples of remote sensing images according to claim 1, characterized in that: In step S5, the preset IOU threshold includes an IOU upper threshold and an IOU lower threshold: The second semantic segmentation result whose intersection-over-union ratio is less than the IOU lower threshold is recorded as a difficult sample; The first semantic segmentation result whose IOU is greater than the IOU upper threshold is recorded as a simple sample.

Images