A semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture

Abstract

In response to the problems of small single-time processing area and limited segmentation performance of high-resolution remote sensing images, the present invention designs a semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture. The architecture is a hierarchical image pyramid structure. The bottom layer is a smaller image block, referred to as "Patch", and the upper layer is a larger image area, referred to as "Region". The object processed by the "Patch" part is a smaller image block. It adopts a U-shaped structure semantic segmentation network fused with Transformer, and outputs pixel-level semantic features. The "Region" part processes a larger image area. In order to reduce the amount of calculated parameters, the global average pooling algorithm is used to convert the pixel-level semantic features output by the "Patch" block into "Patch" level semantic features. After that, the global context information at the "Region" level is extracted through the Transformer module. Finally, the global context information and pixel-level semantic features are aggregated to obtain a high-quality segmentation map. This method has broad application prospects in the field of high-resolution remote sensing image processing.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and mainly relates to a semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture. Background Art

[0002] In recent years, the use of high-resolution satellite remote sensing imagery has become increasingly widespread, with both quality and resolution increasing. This has led to its widespread application in areas such as updating basic geographic information, urban planning and design, national defense, and military security. Deep learning-based semantic segmentation of remote sensing images, a key technology in remote sensing image processing, plays a vital role in various tasks.

[0003] Remote sensing image semantic segmentation algorithms based on deep learning often draw on image semantic segmentation algorithms from the field of computer vision. For this task, Long et al. first proposed the fully convolutional network (FCN). By removing the fully connected layers from the classification network and restoring the feature map resolution through deconvolution, they successfully transformed the image classification network into an image segmentation network. Subsequently, many researchers built upon this foundation and designed a series of FCN variants, such as the DeepLab series and the Unet series, which have achieved promising results in remote sensing image semantic segmentation. The DeepLab network effectively addresses the spatial resolution degradation associated with downsampling in traditional CNN networks. Subsequently, improvements were made to the DeepLab network by incorporating depthwise separable convolutions, parallel dilated separable convolutions, and a symmetric encoder-decoder architecture, resulting in the design of the more accurate DeepLabv2, DeepLabv3, and DeepLabv3+ series, which have significantly advanced the field of image semantic segmentation. The UNet network utilizes a decoder structure that is completely symmetric to the encoder structure to gradually restore the resolution of the feature map. Using skip connections, it fully utilizes the encoder's features at all scales, significantly improving the accuracy of remote sensing image semantic segmentation.

[0004] However, due to the inherent limitations of convolutional operations, network structures based on convolutional operations have a limited receptive field, lack a global understanding of the image itself, and cannot fully utilize the image's context. Meanwhile, the Transformer architecture, with its global modeling capabilities, has been widely used in natural language processing. The Vision Transformer, developed on this basis, has also seen rapid progress in image semantic segmentation. The Vision Transformer processes the entire image in blocks and serializes it, using a self-attention mechanism to capture global contextual information, extract more powerful features, and improve image semantic segmentation accuracy. However, the Transformer also suffers from issues such as feature resolution loss and high computational power consumption for model training. While many researchers have made improvements and optimizations based on this architecture, such as the Swin Transformer, which has reduced sequence length and computational complexity to some extent, several challenges remain regarding segmentation details, training dataset size, and computational power requirements. To address this, some researchers have designed a new network, TransUNet, that fuses Transformers and CNNs. This network, while maintaining global attention, effectively preserves local image features, significantly improving image segmentation accuracy.

[0005] Another problem with semantic segmentation of remote sensing images is that the image size is too small. Since the current mainstream deep learning algorithms are mainly used for image data with smaller formats in the field of computer vision, the processed size is mostly within 512×512. This size has a limited regional range in high-resolution remote sensing images. In weak texture areas, such as large-scale buildings, water systems, vegetation, etc., the image size within 512×512 can only use too little information, resulting in unsatisfactory segmentation results. At the same time, segmentation on scaled low-resolution remote sensing images will lose a lot of detail information, especially for the detection and segmentation of small objects. To this end, the present invention proposes a large-scale semantic segmentation architecture from Patch to Region. This architecture first uses a semantic segmentation network to implement patch-level image processing and output pixel-level semantic features. Then, a global average pooling algorithm is used to convert the pixel-level semantic features into "patch"-level semantic features. After that, the global context information is extracted through the Transformer module. Finally, the global context information and pixel-level semantic features are aggregated to obtain a high-quality segmentation map. This method not only takes into account the detailed features of high-resolution remote sensing imagery but also the global information at the region level, effectively improving the accuracy of semantic segmentation of high-resolution remote sensing imagery. Furthermore, since only sparse "patch"-level semantic features are processed at the region level, the requirements for computing power and video memory are relatively low. Summary of the Invention

[0006] In response to the problems of small single-processing area and limited segmentation performance of high-resolution remote sensing images, the present invention designs a semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture. The architecture is a hierarchical image pyramid structure. The bottom layer is a smaller image block, referred to as "Patch", and the upper layer is a larger image area, referred to as "Region". The "Patch" part processes smaller image blocks, such as 256×256 size, and adopts a U-shaped structure semantic segmentation network fused with Transformer, which outputs pixel-level semantic features. The "Region" part processes larger image areas. In order to reduce the number of calculated parameters, the global average pooling algorithm is used to convert the pixel-level semantic features output by the "Patch" block into "Patch" level semantic features. The global context information at the "Region" level is then extracted through the transformer module. Finally, the global context information and pixel-level semantic features are aggregated to obtain a high-quality segmentation map. This method has broad application prospects in the field of high-resolution remote sensing image processing.

[0007] The technical solution adopted by the present invention is as follows: a semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture, comprising the following steps:

[0008] Step 1: Semantic segmentation dataset construction. Semantic segmentation datasets are constructed using high-resolution remote sensing images and corresponding vector files. These datasets include patch-level datasets and region-level datasets.

[0009] Step 2: Patch-level remote sensing image semantic segmentation network training. The patch-level semantic segmentation network adopts a U-shaped encoding and decoding structure. An attention module is added at the end of the decoder to improve feature expression capabilities. Multi-scale features are integrated through jump connections. The training set uses relatively small images. The output of the network training phase is the predicted image semantic label. The loss function is calculated by comparing the predicted label with the true value.

[0010] Step 3: Construction and training of region-level remote sensing image network. Based on the patch-level network, a region-level remote sensing image semantic segmentation network for semantic segmentation is constructed. In the patch-level network, the region-level image is divided into several patches. The patch-level semantic segmentation network is used to extract the pixel-level semantic features of each patch. Then, the global average pooling algorithm is used to convert the pixel-level semantic features into "patch" level semantic features. The global context information at the region level is extracted through the Transformer module. Finally, the global context information and pixel-level semantic features are aggregated to obtain a high-quality segmentation map. During the entire training process, the patch-level semantic segmentation network has fixed model parameters and is only used as a feature extraction network.

[0011] Furthermore, the Patch-level dataset and Region-level dataset in step 1 are constructed as follows;

[0012] First, we use relatively large images to crop the region-level dataset, which includes high-resolution remote sensing images and corresponding labels, and divide it into training, validation, and test sets. Then, we use relatively small images to crop the region-level dataset into patch-level datasets. During the cropping process, the training, validation, and test sets are kept consistent to facilitate subsequent model quality evaluation.

[0013] Furthermore, the encoding and decoding structure of the U-shaped structure in step 2 includes a convolutional network feature extraction part, a Transformer feature map encoding part, and a cascade decoding module;

[0014] The convolutional network feature extraction part consists of a feature information compression block and an information extraction block. The feature information compression module consists of a convolutional layer, a normalization layer, an activation function, and a pooling layer. The information extraction module is divided into three submodules, each of which includes three convolutional units and a residual jump connection. For each convolutional unit, it consists of a convolutional layer, a normalization layer, and an activation function. After passing through each submodule, the height and width of the compressed feature map are reduced to half of the previous ones. At the same time, feature maps of different sizes are retained as input features of the subsequent cascade decoder.

[0015] The Transformer feature map encoding part consists of the position embedding module and the Transformer encoding module;

[0016] The cascade decoding module consists of three parts: a convolutional upsampling module, a feature splicing module, and a semantic result prediction module. The convolutional upsampling decoding block consists of three serially connected blocks, each of which contains two convolutional layers and one upsampling layer. The feature splicing module is connected after each block and fuses the feature maps of the corresponding scale output by the convolutional feature extraction module through jump connections. After a series of convolutional upsampling and feature splicing operations, high-dimensional image features that fuse multi-scale information with the same size as the original image are obtained, and then the semantic result prediction module outputs the semantic segmentation results of the remote sensing image.

[0017] Furthermore, the length and width of the output feature map of the convolutional network feature extraction part are reduced to 1 / 16 of the original image. The feature calculation process of this part is as follows:

[0018] S1(x)=elu(GN(Conv 7×7 (x)))

[0019] F1(x)=elu(BN(Conv 1×1 (S1(x))))

[0020] R1(x)=1(x)+1(x)

[0021] F2(x)=elu(BN(Conv 3×3 (R1(x))))

[0022] R2(x)=2(x)+1(x)

[0023] F3(x)=Relu(BN(Conv 1×1 (R2(x))))

[0024] R3(x)=F3(x)+R2(x)

[0025] Where x is the original input image, S1(x) is the feature compression processing of the image, and F k (x) is the feature of the extracted image after compression by the kth submodule, R k (x) is the residual calculation unit of the kth submodule, and the value of k is 1, 2, 3. Conv n×n It represents a convolution operation with a convolution kernel of n, where n can be 1 or 7. BN represents normalization of batch samples, and GN represents grouping samples and calculating the normalized mean and variance in groups.

[0026] Furthermore, the Embedding module first flattens the input feature map into a 2D patch sequence through a convolution operation. The convolution kernel size and stride are both 1. The size of each patch is the same as the feature map of the upper input, 16×16, and the number of patches is 768. Through a trainable linear projection, the vectorized patches are mapped into a 196-dimensional space, and positional encoding is added to preserve position information.

[0027]

[0028] Where E is the embedded projection of Patch, E pos represents position embedding, The feature item of the nth Patch, M is the feature vector after adding position encoding;

[0029] The Transformer encoding module consists of 12 stacked Transformer units. Each Transformer unit contains a multi-head attention mechanism (MSA) and a multi-layer perceptron (MLP). The multi-head attention mechanism (MSA) uses multiple sets of different linear projections to enable the model to capture richer feature information at different positions. After the feature sequence embedded with positional information passes through the Transformer encoding module, it fully integrates the attention information, making the feature map produce a more discriminative feature representation. Finally, it passes through the Layer layer to extract global information and output a sequence with a length of 256 and a dimension of 768.

[0030] The calculation process of each Transformer unit is as follows:

[0031] x1=norm(x)

[0032] x2=x+MultiHead(x1)

[0033] y=x²+MLP(norm(x²))

[0034] Where x1 represents the input sequence, x2 represents the sequence after integrating attention information, represents the output sequence with the same size as x, y represents the final output sequence, norm refers to the normalization of the input sequence, MultiHead refers to adding attention information through multiple sets of linear projections, and MLP stands for multi-layer perceptron.

[0035] Furthermore, the convolution kernel size of the convolution layer in the block is 3, and the stride is 1; the scaling factor used in the upsampling layer is 2, that is, the size of the feature map is expanded to twice the original size.

[0036] Furthermore, the specific implementation of step 3 includes the following sub-steps:

[0037] Step 3.1, Patch-level image semantic feature extraction. In this step, the region-level image is first cropped into patch-level images. Then, the trained patch-level remote sensing image semantic segmentation network is used to process all the cropped patch-level images and output high-dimensional image features.

[0038] Step 3.2: Extracting Patch-level Features with Contextual Information. In this step, the global average pooling algorithm is first used to sparse pixel-by-pixel semantic features, that is, to convert pixel-by-pixel semantic features into Patch-level semantic features, thereby significantly reducing the amount of computation and the number of parameters in the global context information extraction process. During the semantic feature sparsification process, the Norm Layer counts the values ​​of all dimensions on each Patch-level feature map, calculates the mean and variance, and performs normalization operations to obtain multiple pixel feature maps of size 1×1. Then, the Transformer architecture is used to extract Region-level contextual information. In this process, the Patch-level semantic features within the Region are flattened into a 2D sequence, combined with the position information, and input into a Transformer module consisting of 6 series-connected Transformer units to obtain Patch-level features with Region-level contextual information.

[0039] Step 3.3: Extract pixel-level features with contextual information. Use a vector broadcasting strategy to convert the patch-level features with contextual information into pixel-level features. Combine these features with the pixel-level features obtained by the patch-level image semantic segmentation network to obtain pixel-level features with region-level contextual information.

[0040] Step 3.4: Decoding network. The decoding network consists of three lightweight convolutional network units. The pixel-level high-dimensional features are converted into one-dimensional features with a channel of 1 after passing through three convolutional units with a convolution kernel size of 3. The pixel-level features with region-level context information are then converted into region-level image size using feature dimension transformation. The features are then input into the decoding network to obtain the final semantic segmentation results at the region-level image size.

[0041] In step 3.5, network training and verification, the above model is iteratively trained using the training dataset in the region-level dataset until the model converges, and the model accuracy is verified on the region-level dataset.

[0042] Furthermore, the Region-level dataset is composed of remote sensing images with an image size greater than or equal to 2048×2048 pixels; the Patch-level dataset is composed of remote sensing images with an image size less than or equal to 512×512 pixels.

[0043] Furthermore, in step 2, in order to make the Patch model have stronger generalization ability, random flipping and color transformation are used to enhance the image samples on the Patch-level dataset.

[0044] Furthermore, in step 3, in order to improve the accuracy and robustness of the network, the spatial data augmentation method is used on the region-level dataset to improve the diversity of samples by randomly rotating, mirroring, and adjusting the color, saturation, and contrast of the image.

[0045] Compared with existing technologies, the advantages and benefits of this invention are as follows: This invention proposes a large-scale remote sensing image processing architecture from patch to region, achieving more accurate segmentation of large-area features. Experiments on a high-resolution remote sensing image building extraction dataset demonstrate that this invention can effectively improve the semantic segmentation performance of high-resolution remote sensing imagery, particularly in weakly textured areas such as large buildings. This method has broad application prospects in the field of high-resolution remote sensing image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention.

[0047] Figure 2 It is the process framework of the Region-level remote sensing image network.

[0048] Figure 3 It is a comparison diagram of the effects before and after the use of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be further described in detail below by taking remote sensing images as an example and combining with the accompanying drawings.

[0050] like Figure 2 As shown, the specific implementation of the method of the present invention includes the following steps:

[0051] Step 1: Construct a semantic segmentation dataset. Construct a semantic segmentation dataset using high-resolution remote sensing images and corresponding vector files, including two types of datasets: patch-level datasets and region-level datasets. Step 2: Patch-level remote sensing image semantic segmentation network training. The patch-level semantic segmentation network adopts a U-shaped encoding and decoding structure. An attention module is added at the end of the decoder to improve the feature expression capability, and multi-scale features are fused through jump connections. The training set uses images with a small image size (such as 256×256). The output result of the network training phase is the predicted image semantic label. The loss function is calculated by predicting the label and the true value, and the pixel-by-pixel semantic features are output in the application phase at the Region level. Step 3: Construct and train the Region-level remote sensing image network. Based on the patch-level network, a region-level remote sensing image semantic segmentation network is constructed for semantic segmentation. In this network, the region-level image is divided into several patches. The patch-level semantic segmentation network extracts pixel-level semantic features for each patch. A global average pooling algorithm is then used to convert pixel-level semantic features into patch-level semantic features. The Transformer module extracts region-level global context information. Finally, the global context information and pixel-level semantic features are aggregated to obtain high-quality segmentation maps. Throughout the training process, the patch-level semantic segmentation network maintains fixed model parameters and serves only as a feature extraction network.

[0052] The specific implementation of step 1 includes the following sub-steps:

[0053] Step 1.1: Convert vector data to raster tags. The vector data in this invention is in Shapefile format, which can capture the location and related attributes of spatial objects such as buildings in remote sensing images. Using the geographic information software ArcGIS, the shp vector tags are converted into tif raster tags, generating a binary grayscale image corresponding to the high-resolution remote sensing image.

[0054] Step 1.2: Construct a semantic segmentation dataset. First, create a region-level dataset using a larger image size (2048 or larger), including high-resolution remote sensing images and corresponding labels, and divide it into training, validation, and test sets. Then, create a patch-level dataset using a smaller image size (such as 256 or 512). During this process, the training, validation, and test sets should remain consistent to facilitate subsequent model quality assessment.

[0055] The specific implementation of step 2 includes the following sub-steps:

[0056] Step 2.1, convolutional network feature extraction. The convolutional network feature extraction part consists of a feature information compression block and an information extraction block. The feature information compression module consists of a convolution layer, a normalization layer, an activation function and a pooling layer. The convolution kernel size of the convolution layer is 7×7, the step size is 2, and the convolution kernel size of the pooling layer is 3, the step size is 2; the information extraction module is divided into 3 submodules, each submodule includes three convolution units and residual jump connections. For each convolution unit, it consists of a convolution layer, a normalization layer and an activation function. After the compressed feature map passes through each submodule, the height and width of the feature map are reduced to half of the previous one. At the same time, feature maps of different sizes are retained as input features of the subsequent cascade decoder. Finally, the length and width of the output feature map of the convolutional network feature extraction part become 1 / 16 of the original image. The feature calculation flow chart of this part is shown below:

[0057] S1(x)=elu(GN(Conv 7×7 (x)))

[0058] F1(x)=elu(BN(Conv 1×1 (S1(x))))

[0059] R1(x)=1(x)+1(x)

[0060] F2(x)=elu(BN(Conv 3×3 (R1(x))))

[0061] R2(x)=2(x)+1(x)

[0062] F3(x)=elu(BN(Conv 1×1 (R2(x))))

[0063] R3(x)=3(x)+2(x)

[0064] Where x is the original input image, S1(x) is the feature compression processing of the image, and F n (x) is the feature of the extracted image after compression by the nth submodule, R n (x) is the residual calculation unit of the nth submodule, Conv n×n It represents a convolution operation with a convolution kernel of n, BN represents the normalization of batch samples, and GN represents grouping samples and calculating the normalized mean and variance in groups.

[0065] Step 2.2: Transformer feature map encoding. The Transformer feature map encoding consists of a position embedding module and a Transformer encoding module. The embedding module first flattens the input feature map into a 2D patch sequence through a convolution operation. The convolution kernel size and stride are both 1. The size of each patch is the same as the input feature map, 16×16, and the number of patches is 768. A trainable linear projection is used to map the vectorized patches into a 196-dimensional space, and positional encoding is added to preserve position information.

[0066]

[0067] Where E is the embedded projection of Patch, E pos represents position embedding, The feature item of the nth Patch, M is the feature vector after adding position encoding.

[0068] The Transformer encoding module consists of 12 stacked Transformer units, each of which incorporates a multi-head attention mechanism (MSA) and a multi-layer perceptron (MLP). The MSA uses multiple sets of linear projections to enable the model to capture richer feature information at different positions. After passing through the Transformer encoding module, the feature sequence embedded with positional information is fully integrated with the attention information, resulting in a more discriminative feature representation in the feature map. Finally, a layer layer extracts global information, outputting a sequence with a length of 256 and a dimension of 768.

[0069] The calculation process of each Transformer unit is as follows:

[0070] x1=norm(x)

[0071] x2=x+MultiHead(x1)

[0072] y=x²+MLP(norm(x²))

[0073] Where x1 represents the input sequence, x2 represents the sequence after integrating attention information, represents the output sequence with the same size as x, y represents the final output sequence, norm refers to the normalization of the input sequence, MultiHead refers to adding attention information through multiple sets of linear projections, and MLP stands for multi-layer perceptron.

[0074] Step 2.3, cascade decoding module. The cascade decoding module consists of three parts: convolution upsampling module, feature splicing module, and semantic result prediction module. The convolution upsampling decoding block consists of three blocks connected in series. Each block contains two convolution layers and one upsampling layer. The convolution kernel size of the convolution layer is 3, the stride is 1, and the scaling factor of the upsampling layer is 2, which means that the size of the feature map is doubled. The feature splicing module is connected after each block and fuses the feature maps of the corresponding scale output by the convolution feature extraction module through jump connections. After a series of convolution upsampling and feature splicing operations, high-dimensional image features that fuse multi-scale information with the same size as the original image are obtained, and then the semantic result prediction module outputs the semantic segmentation results of the remote sensing image.

[0075] The specific implementation of step 3 includes the following sub-steps:

[0076] Step 3.1, Patch-level image semantic feature extraction. In this step, the Region-level image is first cropped into a Patch-level image. For example, if the Region-level image size is 2048×2048 and the Patch-level image size is 256×256, the Region-level image is evenly divided into 64 blocks according to an 8×8 grid. Then, the Patch-level remote sensing image semantic segmentation network trained in step 2.3 is used to process all the cropped Patch-level images. It should be pointed out that what is output here in the present invention is high-dimensional image features, not the final semantic segmentation results. At the same time, in the Patch-level image semantic feature extraction step, the parameters of the Patch-level remote sensing image semantic segmentation network are fixed and do not participate in the subsequent network training process.

[0077] Step 3.2, Patch-level feature extraction with contextual information. In this step, the global average pooling algorithm is first used to sparse the pixel-by-pixel semantic features, that is, to convert the pixel-by-pixel semantic features into patch-level semantic features, thereby significantly reducing the amount of computation and the number of parameters in the global context information extraction process. During the semantic feature sparsification process, the Norm Layer counts the values ​​of all dimensions on each patch-level feature map, calculates the mean and variance, and performs normalization operations to obtain multiple pixel feature maps of size 1×1. The Transformer architecture is then used to extract region-level contextual information. In this process, the patch-level semantic features within the region are flattened into a 2D sequence, combined with the position information, and input into a Transformer module consisting of 6 series-connected Transformer units to obtain patch-level features with region-level contextual information.

[0078] Step 3.3: Extract pixel-level features with contextual information. A vector broadcasting strategy is used to convert the patch-level features with contextual information into pixel-level features. These features are then combined with the pixel-level features obtained by the patch-level image semantic segmentation network to obtain pixel-level features with region-level contextual information.

[0079] Step 3.4: Decoding network. The decoding network consists of three lightweight convolutional units. After passing through three convolutional units connected in series, each with a kernel size of 3, the pixel-level high-dimensional features are converted into one-dimensional features with a channel size of 1. Feature dimensionality transformation is then used to convert the pixel-level one-dimensional features containing region-level context information into region-level image dimensions. This is then input into the decoding network to obtain the final semantic segmentation results at the region-level image dimension.

[0080] Step 3.5: Network training and validation. Use the training dataset in the region-level dataset to iteratively train the above model until the model converges, and verify the model accuracy on the region-level dataset.

[0081] The technical solution of the present invention will be further described in detail below by taking remote sensing images as an example and combining with the accompanying drawings.

[0082] The implementation of the present invention is divided into three stages.

[0083] The first stage is the construction of semantic segmentation datasets. In this stage, two sets of segmentation datasets of different sizes need to be constructed. The ultra-high-scale satellite remote sensing images and related shapefile vector data are converted into grayscale binary labels through Arcgis, and the entire image is divided into several non-overlapping Region-level images of size 2048×2048 by data slicing. The edge areas with invalid data and the geographical areas that do not contain spatial objects such as buildings are removed for training the Region-level network. The present invention uses a total of three sets of high-scale official images (AIDS, WBDS, Vaihingen) and a set of custom images (WHDS) to construct the Region-level dataset to evaluate the quality of the Region network model. For the AIDS large-format image (443347×336181), 7764 images of 2048 size were cut for Region network training, with 2588 images in the validation set and test set respectively. For the WBDS dataset, two large-format images (32507×15345) of different time periods were provided, and a total of 378 images of 2048 size were cut for Region network training, with 126 images in the validation set and test set respectively. For the Vaihingen large-format image, 84 images of 2048 size were cut for Region network training, with 28 images in the validation set and test set respectively. For the custom WHDS dataset, the present invention collected and annotated two large-format images (32507×15345) of different time periods and related shapefile vector data, and a total of 126 images of 2048 size were cut for Region network training, with 42 images in the validation set and test set respectively. Based on the filtered and divided Region-level data, the data is divided into blocks in sequence to construct a Patch-level remote sensing image dataset, where the size of each block is 256×256.

[0084] The second stage is the construction and training of the Patch-level remote sensing image network. In order to verify the effectiveness of the present invention, in the Patch-level remote sensing image feature extraction stage, two Patch-level semantic segmentation networks are designed based on the U-type encoding-decoding architecture, namely PatchSegNet1 and PatchSegNet2. In order to facilitate the output of pixel-level semantic features, high-dimensional feature extraction branches are added to all network ends. The decoding end of the PatchSegNet1 network only uses convolutional down (up) sampling units to gradually reduce (restore) the resolution of the feature map; the PatchSegNet2 network introduces a self-attention mechanism on the basis of the PatchSegNet1 network to process the deepest features of the encoder, thereby improving the global modeling capability of feature expression. In order to make the Patch model have stronger generalization ability, the present invention enhances the image samples from the perspectives of random flipping and color transformation during the training process.

[0085] The third stage involves building and training a regional remote sensing imagery network. Before model training, to improve network accuracy and robustness, this paper applies spatial data augmentation methods to the regional dataset. This involves random rotation, mirroring, and adjusting image color, saturation, and contrast to enhance sample diversity.

[0086] In the patent implementation, according to step 3.1, a patch-level image semantic feature extraction module is constructed. In this module, the region-level image (2048×2048) is sequentially divided into 64 stacked patch blocks (256×256). Then, the PatchSegNet1 and PatchSegNet12 training models of stage two are loaded respectively to extract the high-dimensional semantic features of the patch blocks. After the pooled high-dimensional features are integrated with the global attention mechanism, the local and global information are fused according to step 3.3.

[0087] During training, the region-level dataset partitioned in stage 1 is used. The region network is initialized with pre-trained weights and all parameters of the patch-level model are frozen. This patch module does not participate in subsequent backpropagation and gradient updates. This paper compares the performance of region networks based on different patch-level network models for building segmentation in high-scale remote sensing imagery by directly using the patch model to predict segmentation maps and then stitching them into region images.

[0088] Table 2 IOU indicators of the present invention on different data sets

[0089]

[0090] As shown in Table 1, the pyramid architecture based on the Patch+Region architecture of the present invention is significantly superior to the method of simply synthesizing patch images when processing region-level images. On the AIDS, WBDS, and Vaihingen datasets, the two patch-level semantic segmentation networks achieved good segmentation accuracy, and the accuracy was further improved after using the region-level image processing method of the present invention.

[0091] Due to the poor quality of remote sensing images in the WHDS dataset, the segmentation accuracy of the two patch-level semantic segmentation networks was poor, at 51.06% and 55.27% respectively. After using the region-level image processing method of the present invention, the accuracy was greatly improved, reaching 60.71% and 67.34% respectively.

Claims

1. A semantic segmentation method for large-scale remote sensing images from PATCH to REGION architecture, characterized by: The steps include: Step 1: Semantic segmentation dataset construction. Semantic segmentation datasets are constructed using high-resolution remote sensing images and corresponding vector files. These datasets include patch-level datasets and region-level datasets. Step 2: Patch-level remote sensing image semantic segmentation network training. The patch-level semantic segmentation network adopts a U-shaped encoding and decoding structure. An attention module is added at the end of the decoder to improve feature expression capabilities. Multi-scale features are integrated through jump connections. The training set uses relatively small images. The output of the network training phase is the predicted image semantic label. The loss function is calculated by comparing the predicted label with the true value. Step 3: Construction and training of region-level remote sensing image network. Based on the patch-level network, a region-level remote sensing image semantic segmentation network for semantic segmentation is constructed. In the patch-level network, the region-level image is divided into several patches. The patch-level semantic segmentation network is used to extract the pixel-level semantic features of each patch. Then, the global average pooling algorithm is used to convert the pixel-level semantic features into "patch"-level semantic features. The global context information at the region level is extracted through the Transformer module. Finally, the global context information and pixel-level semantic features are aggregated to obtain a high-quality segmentation map. During the entire training process, the patch-level semantic segmentation network has fixed model parameters and is only used as a feature extraction network.

2. The large-scale remote sensing image semantic segmentation method from PATCH to REGION architecture according to claim 1, characterized in that: The construction of the patch-level dataset and region-level dataset in step 1 is as follows; First, we use relatively large images to crop the region-level dataset, which includes high-resolution remote sensing images and corresponding labels, and divide it into training, validation, and test sets. Then, we use relatively small images to crop the region-level dataset into patch-level datasets. During the cropping process, the training, validation, and test sets are kept consistent to facilitate subsequent model quality evaluation.

3. The large-scale remote sensing image semantic segmentation method from PATCH to REGION architecture according to claim 1, characterized in that: The encoding and decoding structure of the U-shaped structure in step 2 includes the convolutional network feature extraction part, the Transformer feature map encoding part and the cascade decoding module; The convolutional network feature extraction part consists of a feature information compression block and an information extraction block. The feature information compression module consists of a convolutional layer, a normalization layer, an activation function, and a pooling layer. The information extraction module is divided into three submodules, each of which includes three convolutional units and a residual jump connection. For each convolutional unit, it consists of a convolutional layer, a normalization layer, and an activation function. After passing through each submodule, the height and width of the compressed feature map are reduced to half of the previous ones. At the same time, feature maps of different sizes are retained as input features of the subsequent cascade decoder. The Transformer feature map encoding part consists of the position embedding module and the Transformer encoding module; The cascade decoding module consists of three parts: a convolutional upsampling module, a feature splicing module, and a semantic result prediction module. The convolutional upsampling decoding block consists of three serially connected blocks, each of which contains two convolutional layers and one upsampling layer. The feature splicing module is connected after each block and fuses the feature maps of the corresponding scale output by the convolutional feature extraction module through jump connections. After a series of convolutional upsampling and feature splicing operations, high-dimensional image features that fuse multi-scale information with the same size as the original image are obtained, and then the semantic result prediction module outputs the semantic segmentation results of the remote sensing image.

4. The method for semantic segmentation of large-scale remote sensing images from a PATCH to REGION architecture according to claim 3, characterized in that: The length and width of the output feature map of the convolutional network feature extraction part are reduced to 1 / 16 of the original image. The feature calculation process of this part is as follows: S1(x)=Relu(GN(Conv 7×7 (x))) F1(x)=Relu(BN(Conv 1×1 (S1(x)))) R1(x)=F1(x)+S1(x) F2(x)=Relu(BN(Conv 3×3 (R1(x)))) R2(x)=F2(x)+R1(x) x3(x)=Relu(BN(Conv 1×1 (R2(x)))) R3(x)=F3(x)+R2(x) Where x is the original input image, S1(x) is the feature compression processing of the image, and F k (x) is the feature of the extracted image after compression by the kth submodule, R k (x) is the residual calculation unit of the kth submodule, and the value of k is 1, 2, 3. Conv n×n It represents a convolution operation with a convolution kernel of n, where n can be 1 or 7. BN represents normalization of batch samples, and GN represents grouping samples and calculating the normalized mean and variance in groups.

5. The method for semantic segmentation of large-scale remote sensing images from a PATCH to REGION architecture according to claim 3, characterized in that: The Embedding module first flattens the input feature map into a 2D patch sequence through a convolution operation. The convolution kernel size and stride are both 1. The size of each patch is the same as the feature map of the upper input, 16×16, and the number of patches is 768. A trainable linear projection is used to map the vectorized patches into a 196-dimensional space, and positional encoding is added to preserve position information. Where E is the embedded projection of Patch, E pos represents position embedding, The feature item of the nth Patch, M is the feature vector after adding position encoding; The Transformer encoding module consists of 12 stacked Transformer units. Each Transformer unit contains a multi-head attention mechanism (MSA) and a multi-layer perceptron (MLP). The multi-head attention mechanism (MSA) uses multiple sets of different linear projections to enable the model to capture richer feature information at different positions. After the feature sequence embedded with positional information passes through the Transformer encoding module, it fully integrates the attention information, making the feature map produce a more discriminative feature representation. Finally, it passes through the Layer layer to extract global information and output a sequence with a length of 256 and a dimension of 768. The calculation process of each Transformer unit is as follows: x1=norm(x) x2=x+MultiHead(x1) y=x²+MLP(norm(x²)) Where x1 represents the input sequence, x2 represents the sequence after integrating attention information, represents the output sequence with the same size as x, y represents the final output sequence, norm refers to the normalization of the input sequence, MultiHead refers to adding attention information through multiple sets of linear projections, and MLP stands for multi-layer perceptron.

6. The method for semantic segmentation of large-scale remote sensing images from a PATCH to REGION architecture according to claim 3, characterized in that: The convolution kernel size of the convolution layer in the block is 3, and the stride is 1; the upsampling layer uses a scaling factor of 2, which means that the size of the feature map is expanded to twice the original size.

7. The method for semantic segmentation of large-scale remote sensing images from a PATCH to REGION architecture according to claim 1, characterized in that: The specific implementation of step 3 includes the following sub-steps: Step 3.1, Patch-level image semantic feature extraction. In this step, the region-level image is first cropped into patch-level images. Then, the trained patch-level remote sensing image semantic segmentation network is used to process all the cropped patch-level images and output high-dimensional image features. Step 3.2: Extracting Patch-level Features with Contextual Information. In this step, the global average pooling algorithm is first used to sparse pixel-by-pixel semantic features, that is, to convert pixel-by-pixel semantic features into Patch-level semantic features, thereby significantly reducing the amount of computation and the number of parameters in the global context information extraction process. During the semantic feature sparsification process, the Norm Layer counts the values ​​of all dimensions on each Patch-level feature map, calculates the mean and variance, and performs normalization operations to obtain multiple pixel feature maps of size 1×1. Then, the Transformer architecture is used to extract Region-level contextual information. In this process, the Patch-level semantic features within the Region are flattened into a 2D sequence, combined with the position information, and input into a Transformer module consisting of 6 series-connected Transformer units to obtain Patch-level features with Region-level contextual information. Step 3.3: Extract pixel-level features with contextual information. Use a vector broadcasting strategy to convert the patch-level features with contextual information into pixel-level features. Combine these features with the pixel-level features obtained by the patch-level image semantic segmentation network to obtain pixel-level features with region-level contextual information. Step 3.4: Decoding network. The decoding network consists of three lightweight convolutional network units. The pixel-level high-dimensional features are converted into one-dimensional features with a channel of 1 after passing through three convolutional units with a convolution kernel size of 3. The pixel-level features with region-level context information are then converted into region-level image size using feature dimension transformation. The features are then input into the decoding network to obtain the final semantic segmentation results at the region-level image size. In step 3.5, network training and verification, the above model is iteratively trained using the training dataset in the region-level dataset until the model converges, and the model accuracy is verified on the region-level dataset.

8. The large-scale remote sensing image semantic segmentation method from PATCH to REGION architecture according to claim 1, characterized in that: Region-level datasets are composed of remote sensing images with image sizes greater than or equal to 2048×2048 pixels; patch-level datasets are composed of remote sensing images with image sizes less than or equal to 512×512 pixels.

9. The large-scale remote sensing image semantic segmentation method from PATCH to REGION architecture according to claim 1, characterized in that: In step 2, in order to make the Patch model have stronger generalization ability, random flipping and color transformation are used to enhance the image samples on the Patch-level dataset.

10. The large-scale remote sensing image semantic segmentation method from PATCH to REGION architecture according to claim 1, characterized in that: In step 3, in order to improve the accuracy and robustness of the network, the spatial data augmentation method is used on the region-level dataset to increase the diversity of samples by randomly rotating, mirroring, and adjusting the color, saturation, and contrast of the image.

Images