Remote sensing image intelligent segmentation method based on multi-scale image content complementation

Abstract

The invention provides a remote sensing image intelligent segmentation method based on multi-scale image content complementation. The method comprises the following steps: uniformly cutting a high-resolution remote sensing digital image into a plurality of small-scale slice images easy to segment and large-scale content supplement images surrounding the slice images in a low-overlapping manner; inputting the slice image and the scaled content supplement image into an encoder based on a convolutional neural network to extract features of different scales; and carrying out weighted fusion on the features of different scales through self-attention, carrying out weighted fusion on the fused features and cross attention, sending the fused features to a decoder and a segmentation network for calculation, and finally outputting a segmentation result of the image. The method can effectively alleviate the problems that the remote sensing image is lack of information under high resolution and is lack of details under low resolution, and effectively improves the segmentation precision at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an intelligent segmentation method for ultra-high resolution remote sensing images. Background Art

[0002] With the continuous development of my country's economy and society, the level of satellite remote sensing technology has also been continuously improved. At present, the Gaofen-2 satellite independently launched by my country can collect surface information with a resolution of 2m, which can provide a large amount of effective data for urban planning, environmental pollution and other issues. By obtaining high-resolution image data in real time through satellite remote sensing technology, we can fully grasp the changes in various places, monitor the expansion of illegal buildings, illegal emissions, etc., and help relevant departments to discover problems in a timely manner and take corresponding measures to solve them.

[0003] However, for massive satellite remote sensing data, the current mainstream manual analysis is time-consuming and laborious, and it is difficult to match the massive remote sensing data, which greatly reduces the utilization efficiency of remote sensing images. Therefore, it is imperative to use computer vision technology to accurately and quickly analyze remote sensing data. In recent years, computer vision methods represented by deep neural networks have gradually been widely used in the field of image analysis, providing possibilities for intelligent segmentation of remote sensing images.

[0004] Relevant facts in multiple fields of remote sensing images can prove that there is a large gap between traditional manual analysis methods and artificial intelligence analysis methods based on deep learning. Most of the existing computer intelligent analysis methods for images are based on the feature design of natural images. When used directly on remote sensing images, the size selection of image cropping becomes a big problem. Due to the high resolution of remote sensing images, when the image cropping size is small, a single image contains less information and it is difficult to classify the target objects. If the cropping size of the image is increased, due to limited computing resources, the image is often scaled before entering the neural network, which will cause the problem of loss of details.

[0005] The present invention proposes an intelligent semantic segmentation method for remote sensing images based on multi-scale image content complementarity. Aiming at the problem that it is difficult to select the size of remote sensing images at high resolution, the present invention redesigns a semantic segmentation framework capable of inputting multi-scale images. It can simultaneously input a small-scale slice image and a large-scale content supplement image surrounding the slice image, which not only utilizes the detail information contained in the slice image, but also utilizes the additional information provided by the content supplement image, thereby solving the problem that it is difficult to select the size of remote sensing images at high resolution. Summary of the invention

[0006] The present invention provides a method for semantic segmentation of ultra-high resolution remote sensing images, characterized in that the method comprises:

[0007] Step 1: evenly and with low overlap, the high-resolution remote sensing image is cropped into a plurality of small-scale slice images that are easy to segment and a large-scale content supplementary image surrounding the slice images;

[0008] Step 2: Input the slice image and the scaled content supplement image into the encoder based on the convolutional neural network to extract features of different scales;

[0009] Step 3: Perform self-attention weighted fusion and cross-attention weighted fusion on features of different scales;

[0010] Step 4, the fusion feature is input into a decoder and a segmentation network to obtain a semantic segmentation result of the remote sensing image;

[0011] Specifically, in step 1, the ultra-high resolution remote sensing image is cropped into slice images with a side length of x pixels (the value of x is the side length of the commonly used image in the general remote sensing image segmentation network, and is usually 512 according to the actual usage) with an overlap rate of s, where s is x / 4; at the same time, a content supplementary image surrounding the slice image is cropped for each slice image, the center point of which coincides with the center point of the above slice image, and the side length is x*y, where y is a positive integer greater than or equal to 2, which needs to be adjusted autonomously according to prior knowledge;

[0012] Specifically, in step 2, according to the decoder depth, the features of the encoder at different resolutions are extracted. When a four-layer Unet decoder is used, the encoder's reciprocal outputs Fp1, Fp2, Fp3, and Fp4 at different resolutions are extracted as slice image features of four different scales, and the corresponding Fc1, Fc2, Fc3, and Fc4, four different scales of content supplementary image features; the encoder is a ResNet network, and its 4x, 8x, 16x, and 32x downsampled outputs are selected as image features corresponding to four different scales;

[0013] Specifically, in step 3, Fp4 and Fc4 are multiplied by a dimensionality reduction matrix D respectively to obtain features Fp4′ and Fc4′ with the same dimension as the attention calculation module; Fp4′ and Fc4′ are respectively passed through a self-attention module to complete the activation of the region of interest and the suppression of the non-interest region based on themselves, and obtain Fp4″ and Fc4″; Fp4″ and Fc4″ are passed through a cross attention module, and Fp4″ is used to activate the region of interest in Fc4″ and suppress the non-interest region in Fc4″, and then connected with Fp4″ through the residual structure, so as to complete the information exchange and information fusion between different pictures, and obtain the adaptive final weighted fusion feature Fp4″′; Fp4″′ and Fc4″ are respectively passed through a dimensionality increase matrix to make them the same dimension as Fp4 and Fc4, and replace Fp4 and Fc4 as the features of the next step input decoder;

[0014] Specifically, in step 4, the above image features are input into the decoder and the segmentation network to obtain the semantic segmentation result of the remote sensing image; the final output result of the segmentation network is the possible probability of each pixel belonging to each category, that is, the probability distribution map corresponding to the picture; a threshold τ is set to determine whether the maximum possible probability of each pixel is greater than τ. If so, the pixel is marked as belonging to the class corresponding to the maximum probability; otherwise, the pixel is marked as belonging to the meaningless background class. The obtained classification map is dyed according to the pre-set color table to obtain a classification result that is easy to observe;

[0015] Specifically, the encoder, decoder and segmentation head described in steps 1-4 together constitute a semantic segmentation network based on deep learning. The semantic segmentation network model needs to be optimized through training to accurately segment the target category. Specifically, based on the training sample data set, the slice image and the content supplementary image surrounding the slice image are used as input, and the labels corresponding to the slice image and the content supplementary image are used as supervision. A supervised learning method is adopted to train and optimize the semantic segmentation network model through a preset loss function to obtain a network model capable of intelligent segmentation;

[0016] Specifically, the preset loss function includes a classification loss function and a contrast loss function. The classification loss function includes the classification cross entropy loss of the slice image and the classification cross entropy loss of the content supplement image, which are weighted at a ratio of 1 to 0.25. The contrast loss function describes the similarity of the segmentation results of the overlapping areas of the slice image and the content supplement image. Specifically, the overlapping area of ​​the content supplement image with the slice image is recorded as R, and after scaling the probability distribution map corresponding to the slice image to the same size as R, the KL divergence is used to calculate the similarity between the probability distribution map corresponding to R and the probability distribution map of the scaled slice image;

[0017] Aiming at the situation that low-resolution remote sensing images lack details and high-resolution remote sensing images lack information in existing remote sensing image segmentation methods, the present invention takes high-resolution remote sensing images as segmentation objects and proposes an intelligent remote sensing image segmentation method based on multi-scale image content complementarity, thereby greatly improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0019] Figure 1 It is a flowchart of the steps of the remote sensing image intelligent segmentation method based on multi-scale image content complementarity;

[0020] Figure 2 is an example image of the high-resolution remote sensing image;

[0021] Figure 3 The left column shows sample slice images;

[0022] Figure 3 The right column contains sample images of the content supplementary images corresponding to the slice images;

[0023] Figure 4 This is an example diagram of the semantic segmentation results of the high-resolution remote sensing image. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Relevant facts in multiple fields of remote sensing images can prove that there is a large gap between traditional analysis methods and artificial intelligence analysis methods based on deep learning. In the existing methods that directly perform intelligent analysis on remote sensing images, low-resolution remote sensing images will have problems such as lack of details and inaccurate segmentation; while high-precision remote sensing images may have classification errors due to lack of image information. The present invention takes high-resolution remote sensing images as the segmentation target and proposes a remote sensing image intelligent segmentation method based on multi-scale image content complementarity, thereby greatly improving the segmentation accuracy. The specific steps are as follows:

[0026] S101: Crop the ultra-high-resolution remote sensing image into multiple large-size overview images evenly and with low overlap.

[0027] Specifically, high-resolution remote sensing images I (such as Figure 2 As shown in the figure, the slices I1, I2, ..., I2 are cut into slices I1, I2, ..., I2 with a side length of x pixels (x is an integer, such as x = 512) with an overlap rate of s. n (Slice images such as Figure 3 As shown in the left column, n is the number of slice images cut out), where the value of x is the commonly used image side length of the general remote sensing image segmentation network, and is usually 512 according to actual usage. At the same time, for each slice image I i (i ranges from [1,n]) Crop a content supplement image C around the slice image i (Content supplementary images such as Figure 3As shown in the right column, the value range of i is [1, n]), its center point coincides with the center point of the above slice image, and the side length is x*y, where y is a positive integer greater than or equal to 2, which needs to be adjusted autonomously based on prior knowledge.

[0028] S102: Input the slice image and the content supplement image into an encoder to extract features of different scales.

[0029] Specifically, for the overview image I i (i ranges from [1, n]), input it into any classification convolutional neural network, and obtain the output of the convolution module located before the classification structure in the convolutional neural network, referred to as the encoder. The encoder result is used as input, and through a combination of various convolution and attention modules, the encoder result is processed into a network model part that can be used by the segmentation head, called a decoder. According to the depth of the decoder, the output of the encoder with different resolutions is extracted. The decoder is a 4-layer UNet decoder, so the outputs Fp1, Fp2, Fp3, and Fp4 of the encoder with four different resolutions are extracted as slice image features of four different scales. For the content supplementary image C j (i ranges from [1, n]), scale it to the same size as the slice image and then input it into the encoder to extract four outputs Fc1, Fc2, Fc3, Fc4 with the same features as the slice image as the content supplementary image features. The encoder is a ResNet network, and its 4x, 8x, 16x and 32x downsampled outputs are selected as image features corresponding to four different scales.

[0030] S103: Perform self-attention and cross-attention weighted fusion on image features of different scales.

[0031] Specifically, Fp4 and Fc4 are multiplied by a dimensionality reduction matrix D respectively to obtain features Fp4′ and Fc4′ with the same dimension as the attention calculation module. Fp4′ and Fc4′ are respectively passed through a self-attention module to complete the activation of the region of interest based on themselves and the suppression of the non-interested region, and obtain Fp4″ and Fc4″; Fp4″ and Fc4″ are then passed through a cross attention module, and Fp4″ is used to activate the region of interest in Fc4″ and suppress the non-interested region in Fc4″, and then connected with Fp4″ through the residual structure, so as to complete the information exchange and information fusion between different pictures, and obtain the adaptive final weighted fusion feature Fp4″′; finally, Fp4″′ and Fc4″ are respectively passed through a dimensionality increase matrix to make them compatible with Fp4 and Fc4″. c4 dimensions are the same, and replace Fp4 and Fc4 as the features of the next input decoder. In particular, the attention module refers to a multi-head attention calculation model based on the transformer model, which can enhance the expressiveness and generalization ability of the model, focus attention on more interesting areas, and suppress non-interesting areas. The self-attention module refers to a feature that uses itself as a weight and calculates the similarity between itself and other elements, thereby completing the activation of the region of interest and the suppression of the region of non-interest based on itself. Cross attention uses the similarity between the features of the slice image and the features of the content supplement image as the weight, calculates the importance of the content supplement image features to the slice image and weights them. The residual structure is to retain the features of the slice image and fuse the features of the content supplement image.

[0032] S104: Input the image features into the decoder and the segmentation head, and output the final semantic segmentation result.

[0033] Specifically, the above image features are input into the decoder and the segmentation network to obtain the semantic segmentation results of the remote sensing image; the decoder is UNet, and the final output result of the segmentation network is the possible probability of each pixel belonging to each category. Set a threshold τ to determine whether the maximum possible probability of each pixel is greater than τ. If so, mark the pixel as belonging to the class corresponding to the maximum probability; otherwise, mark the pixel as belonging to the meaningless background class. Among them, the recommended value of τ is 0.5. Finally, the obtained classification map is dyed according to the pre-set color table to obtain a segmentation result that is easy to observe.

[0034] The above encoder, decoder and segmentation head together constitute a semantic segmentation network based on deep learning. The semantic segmentation network model needs to be optimized through training to accurately segment the target category, specifically including: based on the training sample data set, using the slice image and the content supplement image surrounding the slice image as input, using the slice image and the corresponding labels of the content supplement image as supervision, using a supervised learning method, and training and optimizing the semantic segmentation network model through a preset loss function to obtain a network model capable of intelligent segmentation. The preset loss function includes a classification loss function and a contrast loss function. The classification loss function includes the classification cross entropy loss of the slice image and the classification cross entropy loss of the content supplement image, which are weighted at a ratio of 1 to 0.25. The contrast loss function describes the similarity of the segmentation results of the overlapping area of ​​the slice image and the content supplement image. Specifically, the overlapping area of ​​the content supplement image with the slice image is recorded as R, and after scaling the probability distribution map corresponding to the slice image to the same size as R, the KL divergence is used to calculate the similarity between the probability distribution map corresponding to R and the probability distribution map of the scaled slice image.

[0035] The present invention takes high-resolution remote sensing images as segmentation objects and proposes a remote sensing image intelligent segmentation method based on multi-scale image content complementarity. At the same time, the detail information provided by small-scale slice images and the context information provided by large-scale content supplementary images are utilized to alleviate the problem of lack of details in low-resolution remote sensing images and lack of information in high-resolution remote sensing images, thereby greatly improving the segmentation accuracy.

Claims

1. A remote sensing image intelligent segmentation method based on multi-scale image content complementarity, characterized in that: The method comprises the following steps: Step 1, cropping the high-resolution remote sensing image into multiple slice images evenly and with low overlap, specifically, cropping the high-resolution remote sensing image into slice images with a side length of x pixels (the value of x is the side length of the commonly used image in the general remote sensing image segmentation network, and is usually 512 according to actual usage) with an overlap rate of s, where s is x / 4; at the same time, cropping a content supplement image surrounding the slice image for each slice image, whose center point coincides with the center point of the above slice image, and whose side length is x*y, where y is a positive integer greater than or equal to 2, which needs to be adjusted autonomously based on prior knowledge. Step 2, input the slice image and the reduced content supplement image into a convolutional neural network to extract features of different scales. Specifically, the content supplement image is reduced to the same size as the slice image, and then input into an encoder based on a convolutional neural network, and according to the depth of the decoder, the features of the encoder corresponding to different resolutions are extracted. The decoder is a four-layer Unet decoder, so the encoder outputs Fp1, Fp2, Fp3, and Fp4 of four different resolutions are extracted as slice image features of four different scales, and the corresponding Fc1, Fc2, Fc3, and Fc4, four different scales of content supplement image features; Step 3, weighted fusion of the features of different scales through self-attention and cross-attention in turn to complete the information exchange between image slices of different scales; Step 4: Input the fused features into the decoder and segmentation network to obtain the semantic segmentation result of the remote sensing image.

2. The remote sensing image intelligent segmentation method according to claim 1, characterized in that: In step 1, the ultra-high resolution remote sensing image is cropped into slice images with a side length of x pixels (the value of x is the side length of an image commonly used in a general remote sensing image segmentation network, and is usually 512 according to actual usage) with an overlap rate of s, where s is x / 4; at the same time, a content supplementary image surrounding the slice image is cropped for each slice image, whose center point coincides with the center point of the above slice image, and whose side length is x*y, where y is a positive integer greater than or equal to 2, and needs to be adjusted autonomously based on prior knowledge.

3. The remote sensing image intelligent segmentation method according to claim 1, characterized in that: In step 2, the encoder's four reciprocal outputs Fp1, Fp2, Fp3, and Fp4 of different resolutions are extracted as slice image features of four different scales, and the corresponding Fc1, Fc2, Fc3′Fc4, four different scales of content supplement image features; the encoder is a ResNet network, and its 4x, 8x, 16x, and 32x downsampled outputs are selected as image features corresponding to four different scales.

4. The remote sensing image intelligent segmentation method according to claim 3, characterized in that: In step 3, specifically: Step 31, multiply Fp4 and Fc4 by a dimension reduction matrix D respectively to obtain features Fp4′ and Fc4′ with the same dimension as the attention calculation module; Step 32, pass Fp4′ and Fc4′ through a self-attention module respectively to complete the activation of the region of interest and the inhibition of the non-region of interest based on itself, and obtain Fp4″ and Fc4″. Step 33, pass Fp4″ and Fc4″ through a cross attention module, use Fp4″ to activate the region of interest in Fc4″, suppress the region of no interest in Fc4″, and then connect it with Fp4″ through the residual structure, so as to complete the information exchange and information fusion between different slices, and obtain the adaptive final weighted fusion feature Fp4″′; Step 34, pass Fp4″′ and Fc4″ through a dimension-raising matrix respectively to make them have the same dimension as Fp4 and Fc4, and replace Fp4 and Fc4 as the features of the next step input decoder.

5. The remote sensing image intelligent segmentation method according to claim 1, characterized in that: In step 4, specifically: Step 41, input the above image features into the decoder and the segmentation network to obtain the semantic segmentation result of the remote sensing image; the decoder is a four-layer Unet network, and the output result of the final segmentation network is the probability that each pixel belongs to each category. Step 42, setting a threshold τ, determining whether the maximum value of the category probability of each pixel is greater than τ, if so, marking the pixel as belonging to the corresponding category; otherwise, marking the pixel as belonging to the meaningless background category. Step 43, coloring the obtained classification graph according to a preset color table to obtain a classification result that is easy to observe.

6. The remote sensing image intelligent segmentation method according to claim 5, characterized in that: The value of τ is 0.

5.

7. The remote sensing image intelligent segmentation method according to any one of claims 1 to 6, characterized in that: The above-mentioned encoder, decoder and segmentation head together constitute a semantic segmentation network based on deep learning. The semantic segmentation network model needs to be optimized through training to accurately segment the target category. Specifically, based on the training sample data set, the slice image and the content supplement image surrounding the slice image are used as input, and the labels corresponding to the slice image and the content supplement image are used as supervision. A supervised learning method is adopted to train and optimize the semantic segmentation network model through a preset loss function to obtain a network model capable of intelligent segmentation.

8. The remote sensing image intelligent segmentation method according to claim 7, characterized in that: The preset loss function includes a classification loss function based on slice image labels and content supplement image labels, and a comparison loss function based on the corresponding output of the slice image and the corresponding output of the content supplement image.

Images