Multi-scale double-branch semantic segmentation method for high-resolution remote sensing image

Abstract

The invention relates to a multi-scale double-branch semantic segmentation method for a high-resolution remote sensing image, and belongs to the technical field of computers. The method comprises the steps that a data set is collected and preprocessed; a shallow feature extraction and fusion module is constructed, the module processes VIS and DSM data through ResNet of a double-branch structure, features are extracted through a convolutional layer and then fused through a feature fusion module, and different data source features can be fully mined and fused; constructing a deep feature extraction and fusion module, wherein an SCSA layer of the module combines a channel and a space attention processing feature map, and a cross-modal self-attention layer calculates cross-modal attention and fuses a result; and then constructing a cascade decoder, training the network by using the training set, and finally testing the network by using the test set. The method solves the problems that in semantic segmentation of the high-resolution remote sensing image, ground feature features are complex and diversified, spatial distribution is uneven, VIS and DSM data information fusion is insufficient and the like.

Description

Technical Field

[0001] The invention belongs to the field of computer technology and relates to a multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images. Background Art

[0002] In recent years, with the rapid progress of aviation and satellite sensor technology, high-resolution remote sensing images have been widely used in many fields. These fields cover many aspects such as urban planning, land use classification, agricultural monitoring, environmental assessment and disaster management. In urban planning, high-resolution remote sensing images can clearly present information such as the building layout and road network of the city, providing an accurate basis for the rational planning of the city; in agricultural monitoring, it can accurately monitor the growth status and planting area of ​​crops, and help agricultural production decision-making. As one of the key technologies for remote sensing image analysis, semantic segmentation has the core task of accurately classifying each pixel in the remote sensing image into a specific semantic category. This technology is of irreplaceable importance for extracting valuable information from remote sensing images and can provide basic data support for various subsequent applications. For example, in land use classification, semantic segmentation can clearly distinguish different types of land, such as cultivated land, construction land, forest land, etc., so as to achieve effective management and reasonable planning of land resources.

[0003] Classic deep learning networks such as FCN (Fully Convolutional Network) UNet and DeepLab have performed well in remote sensing image segmentation tasks. However, there are still some challenges, such as accurate segmentation of small targets, processing of complex background interference, and capturing of target edge details. Since high-resolution remote sensing images contain rich spectral, texture and spatial structure information, it has become an inevitable trend to apply the strategy of multi-scale feature extraction and multi-source data fusion to the field of remote sensing image semantic segmentation. Multi-scale and multi-modal segmentation methods have gradually become an emerging technology that has attracted much attention in this field. Therefore, how to fully explore the complementary information of different data sources, how to effectively extract multi-scale features, and more accurately perform semantic segmentation of various targets in remote sensing images, so as to improve the accuracy and efficiency of high-resolution remote sensing image semantic segmentation and meet the urgent needs of many practical application scenarios for high-precision remote sensing image analysis have become key issues. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images, which processes VIS (visible light data) and DSM (digital surface model) data respectively through a dual-branch structure, and fuses the features of the two in a shallow feature extraction and fusion module, so that the network can simultaneously learn spectral information (VIS) and terrain height information (DSM), and fully explore the complementary information of different data sources. The features of data of different scales are extracted through shallow and deep feature extraction and fusion modules, so as to better meet the segmentation requirements of targets of different scales in high-resolution remote sensing images.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images, comprising the following steps:

[0007] Step 1) Collect data sets. Download the public data sets Potsdam and Vaihingen from the Internet. Potsdam comes from aerial images of Potsdam, Germany. The spatial resolution is 5 cm per pixel. The image type is multispectral (RGB+near infrared) and digital surface model (DSM) generated by lidar. The image size is 6000×6000 pixels. The data set contains 38 images in total. There are 6 types of ground feature labels: building, ground, trees, shrubs, cars, and backgrounds. The Vaihingen data set comes from aerial images of the town of Vaihingen, Germany. The spatial resolution is 9 cm per pixel. The labels and image types are the same as those of the Potsdam data set. The image size is 2000×2000 pixels. The data set contains 33 images in total.

[0008] Step 2) preprocess the original data and expand the number of data sets using overlapping cropping;

[0009] Step 3) dividing the processed visible light data (VIS), digital surface model data (DSM) and label data into a training set and a test set;

[0010] Step 4) building a multi-scale dual-branch semantic segmentation network, wherein the network model includes: a shallow feature extraction fusion module consisting of a main branch VIS branch and an auxiliary branch DSM branch, a deep feature extraction fusion module based on a spatial and channel collaborative attention layer (SCSA) and a cross-modal self-attention layer (CMA), and a cascade encoder;

[0011] Step 5) input the VIS data and DSM of the training set into the VIS branch and the DSM branch respectively;

[0012] Step 6) inputting the VIS data into the VIS branch, and extracting multi-scale features of the visible light image by multiple convolutional layers;

[0013] Step 7) Input the DSM data into the DSM branch, and extract the features of terrain height information by independent multi-layer convolutional layers;

[0014] Step 8) feature fusion is performed on the features of each layer in the dual-branch convolution, specifically including: splicing the VIS and DSM feature maps in the channel dimension; adjusting the feature weights using the spatial attention mechanism; generating the fused feature map as the input of the next part, and the fused feature map of each layer is retained as the input of the jump connection to the cascade encoder;

[0015] Step 9) The final output features in the dual branches are input into the deep feature extraction and fusion module, which includes two attention layers, specifically: spatial and channel collaborative attention layer (SCSA) and cross-modal self-attention layer (CMA). The SCSA layer divides sub-features through sliding windows; uses deep separable 1D convolution to extract multi-semantic spatial information. Generate a spatial attention map and adjust the VIS and DSM feature maps; the CMA layer calculates the relationship between the VIS and DSM features through the self-attention mechanism; the attention results are fused to generate a cross-modal feature map.

[0016] Step 10) The cross-modal feature map is gradually restored to its spatial resolution through a cascaded decoder; specifically, the following steps are performed: using a jump connection to fuse the feature maps of the corresponding layers of the decoder and the encoder, and after multiple upsampling, generating a final fused feature map; and using a segmentation head to generate a per-pixel category prediction.

[0017] The beneficial effects of the present invention are:

[0018] (1) Multimodal fusion capability: The present invention uses a dual-branch structure to process visible light (VIS) images and digital surface model (DSM) data respectively, and extracts the features of the two modalities in a targeted manner. Compared with the traditional method of using only single-modal data (such as VIS), it can capture the rich information in remote sensing images more comprehensively. The present invention effectively combines VIS and DSM features on the basis of spatial attention through a feature fusion module, significantly improving the synergy of multimodal features.

[0019] (2) The present invention has higher semantic resolution ability: the spatial and channel collaborative attention layer is introduced to adaptively adjust the importance of different positions and feature channels. The cross-modal self-attention layer further strengthens the mutual connection between VIS and DSM features, and can more accurately distinguish different ground object categories in complex scenes.

[0020] (3) Support multi-scale feature extraction: From shallow to deep layers, the network adopts a multi-scale feature extraction mechanism at different stages. The shallow layer uses multi-layer convolution to extract detail features, and the deep layer combines SCSA and CMA to extract global semantic features.

[0021] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 It is a structural diagram of the present invention;

[0024] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0027] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0028] The present invention discloses a multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images, and the detailed steps are as follows:

[0029] Step 1) Download the public data sets Potsdam and Vaihingen from the Internet, normalize the original data, and convert the tif format image to jpg format. The value of tif data is different from RGB, which is between [0-255] and ranges from hundreds to more than a thousand. The formula for converting tif to jpg is:

[0030]

[0031] In the formula, X represents the original data value of tif, Minvalue is the minimum value of the original data value, Maxvalue is the maximum value of the original data value, and Y represents the data value after mapping, ranging from [0-255];

[0032] Step 2) Due to the preciousness of high-resolution remote sensing image datasets, the number of datasets is expanded by overlapping cropping. The cropping size is fixed to 512×512; the processed visible light data, digital surface model data, and label data are divided into training sets and test sets;

[0033] Step 3) Construct a shallow feature extraction and fusion module of the multi-scale dual-branch segmentation network. The shallow feature extraction and fusion module is specifically divided into three parts: VIS branch, DSM branch and feature fusion module; the VIS image enters the VIS branch, which is composed of multiple convolutional layers. Each convolutional layer is used to extract visible light image features at different scales. The DSM image enters the DSM branch, which is also composed of multiple convolutional layers. These convolutional layers extract features from the DSM data. Since DSM data usually reflects the height information of the terrain, which is different from the characteristics of VIS data, independent convolutional layers are required to mine its features. The feature map after each layer of convolution of the dual branches is input into the feature fusion module for fusion. In the feature fusion module, the VIS feature map F V(Dimension is C V ×H×W) and DSM feature map F D (Dimension is C D ×H×W) are concatenated in the channel dimension to obtain F concat :

[0034] F concat =[F V ,F D ] (2)

[0035] Where [*,*] represents the concatenation operation in the channel dimension, F concat The dimension is (C V +C D )×H×W, for F concat Perform a convolution operation to generate a spatial attention map M S Next, for M S Normalize the attention map and use the Sigmoid function to map the value of the attention map to between 0 and 1, indicating the importance weight of each position:

[0036] M S =δ(M S ) (3)

[0037] Where δ is the Sigmoid function. S Respectively with F V and F D Multiply them together to get F after attention adjustment in the spatial dimension V ′ and F D ′:

[0038] F V ′=M S ×F V (4)

[0039] F D ′=M S ×F D (5)

[0040] Then, the adjusted VIS feature map F V ′ and DSM characteristic graph F D ′Fuse and adjust the number of channels to obtain a new feature map F fused , F fused It is used as the input of the next VIS branch and also input to the cascade decoder through skip connection for splicing. Each layer obtains a different fused feature map.

[0041] Step 4) Construct a deep feature extraction and fusion module of the multi-scale dual-branch segmentation network. The deep feature extraction and fusion module is divided into two layers: spatial and channel collaborative attention layer (SCSA) and cross-modal self-attention layer (CMA). The SCSA layer closely combines spatial attention with channel attention, and enhances the extraction and fusion capabilities of multi-semantic information through sliding window technology. This layer uses spatial attention to capture the global and local structural information in the feature map and uses it as a guide for channel attention, thereby enhancing the ability to model the relationship between channels. For the input VIS feature map X V and DSM feature map X D , decomposed along the height and width dimensions respectively, and created two unidirectional 1D sequence structures through global average pooling and (For VIS feature maps, B is the batch size), and and (For DSM feature graphs). Then divide the feature set into K independent sub-features of the same size (here K=4), for example:

[0042]

[0043] and Similarly, a depth-wise separable 1D convolution (DWConv1d) with different kernel sizes is applied to each sub-feature to extract multi-semantic spatial information. For the i-th sub-feature of the VIS feature map Its features after convolution are:

[0044]

[0045] (k i is the convolution kernel applied to the i-th sub-feature), and Similarly, in order to calculate the spatial attention map of each feature, different semantic sub-features are aggregated and normalized using group normalization. For example, the height-direction spatial attention of the VIS feature map is calculated as:

[0046]

[0047] (σ is the Sigmoid function)Attn VH With Attn VH Similarly, further generate the spatial attention feature map:

[0048] X Vs =Attn VH ×Attn VW ×X V (9)

[0049] X Ds =Attn DH ×Attn DW ×X D (10)

[0050] Furthermore, the spatial attention feature map obtained in the previous step is input into the cross-modal self-attention layer: Vs Transform to Q V , K V 、V V , X Ds Transform to Q D , K D 、V D Make it suitable for self-attention calculation. Then, calculate cross-modal attention. Calculate the attention from VIS to DSM modality:

[0051]

[0052] Compute the attention from DSM to VIS modality:

[0053]

[0054] The attention results of the two directions are fused by weighted summation. Let α and 1-α be weights (0≤α≤1), and the fused feature map X is obtained. fused :

[0055] X fused =αAttn VD +(1-α)Attn DV (13)

[0056] Step 5) Construct a cascaded decoder. Each decoder layer upsamples the current feature map and gradually restores the spatial resolution. Each decoder layer is connected to the corresponding feature map in the encoder through a jump connection. fused (Dimension is C fused ×H fused ×W fused ) to perform initial upsampling, restore some spatial resolution, use bilinear interpolation upsampling method, and set the upsampling multiple to s. For any pixel (x, y) in the feature map (x∈[0,W fused ×s-1],y∈0,Hfused×s-1, whose corresponding coordinates in the original feature map Xfused are i,j,

[0057] i=floor(x / s) (14)

[0058] j=floor(y / s) (15)

[0059] Floor is the rounding function.

[0060] α=x / si (16)

[0061] β=y / sj (17)Then the bilinear interpolation formula is:

[0062] X unsampled (x,y)=(1-α)(1-β)X fused (i,j)+α(1-β)X fused (i+1,j)+(1-

[0063] αβXfusedi,j+1+αβi+1,j+1 (18)

[0064] The feature map of the lth layer of the fusion feature in the two branches of the encoder is F l fused , through convolution, F l fused With X unsampled After the number of channels is aligned, we get F l ' fused . l ' fused With X unsampled Add the elements to get the fused feature map.

[0065] After repeated upsampling and skip connections, the final fused feature map X is obtained final , then processed through a segmentation head to convert it into a per-pixel class prediction:

[0066] P = SoftMax(Conv 1×1 (X final )) (19)

[0067] Step 6) Use the training set in the data set to train the model and use the cross entropy loss function to optimize the model;

[0068] Step 7) Use the trained model to perform segmentation prediction on the input data in the test set, and the evaluation indicators include IoU and F1-score.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images, characterized by: The method comprises the following steps: Step 1) Collecting a data set and preprocessing the data in the data set; Step 2) construct a multi-scale dual-branch segmentation network, the network architecture is a Unet-like codec structure, and the main body is composed of a shallow feature extraction and fusion module, a deep feature extraction and fusion module, and a cascade decoder; wherein the shallow feature extraction and fusion part is a dual-branch structure, with two ResNet branches, which are used to process visible light image VIS and digital surface model DSM data respectively; after each branch extracts features through its own convolutional layer, the features of the dual branches are fused through the feature fusion module; the feature map extracted by the above dual branches is processed through the deep feature extraction and fusion part, that is, the spatial and channel collaborative attention layer SCSA and the cross-modal self-attention layer CMA to obtain the feature map X final ; The feature map X final Input cascade decoder, multiple cascade decoder blocks restore spatial resolution through jump connections of corresponding encoder layers; finally get the segmentation result; Step 3) Using the training set data to train a multi-scale dual-branch segmentation network; Step 4) Use the test set data to test the multi-scale two-branch segmentation network.

2. The multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images according to claim 1, characterized in that: The step 1) further comprises: Step 1-1, download the public data sets Potsdam and Vaihingen from the Internet, normalize the original data, and convert the tif format image to jpg format. The value of tif data is different from RGB, which is between [0-255] and ranges from hundreds to more than a thousand. The formula for converting tif to jpg is: In the formula, X represents the original data value of tif, Minvalue is the minimum value of the original data value; Maxvalue is the maximum value of the original data value; Y represents the data value after mapping, ranging from [0-255]; Step 1-2: Due to the scarcity of high-resolution remote sensing image datasets, overlapping cropping is used to expand the dataset size; and the cropping size is fixed to 512×512; Step 1-3: Divide the processed visible light data, digital surface model data and label data into a training set and a test set.

3. The multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images according to claim 1, characterized in that: Step 2) The shallow feature extraction and fusion module is specifically as follows: Step 2-1, the VIS image enters the VIS branch, which is composed of multiple convolutional layers; each convolutional layer is used to extract visible light image features of different scales; Step 2-2, the DSM image enters the DSM branch, which is also composed of multiple convolutional layers; these convolutional layers extract features from the DSM data; since DSM data usually reflects the height information of the terrain, which is different from the characteristics of VIS data, an independent convolutional layer is required to mine its features; Step 2-3: The feature map after each layer of convolution of the dual branches is used as input to the feature fusion module for fusion. In the feature fusion module, the dimension C V ×H×W VIS characteristic graph F V and dimension C D ×H×W DSM feature map F D Concatenate in the channel dimension to get F concat : F concat =[F V ,F D ] (2) Where [*, *] represents the concatenation operation in the channel dimension, F concat The dimension is (C V +C D )×H×W, for F concat Perform a convolution operation to generate a spatial attention map M S ; Then, to M S Normalize the attention map and use the Sigmoid function to map the value of the attention map to between 0 and 1, indicating the importance weight of each position: M S =δ(M S ) (3) Where δ is the Sigmoid function; the spatial attention map M S Respectively with F V and F D Multiply them together to get F after attention adjustment in the spatial dimension V ′ and F D ′: F V ′=M S ×F V (4) F D ′=M S ×F D (5) Finally, the adjusted VIS feature map F V ′ and DSM characteristic graph F D ′Fuse and adjust the number of channels to obtain a new feature map F fused , F fused It serves as the input of the next VIS branch and is also input to the cascade decoder through a skip connection for splicing.

4. The multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images according to claim 1, characterized in that: The deep feature extraction and fusion module in step 2) is specifically: Step 2-4, SCSA layer processing: The SCSA layer is an attention mechanism based on a sliding window, which combines channel attention and spatial attention; for the input VIS feature map X V and DSM feature map X D , decomposed along the height and width dimensions respectively, and created two unidirectional 1D sequence structures through global average pooling and For VIS feature maps, B is the batch size; as well as and For the DSM feature map; then divide the feature set into K independent sub-features of the same size, set K = 4: and Similarly, deep separation 1D convolution DWConv1d with different kernel sizes is applied on each sub-feature to extract multi-semantic spatial information. For the i-th sub-feature of the VIS feature map Its features after convolution are: ki is the convolution kernel applied to the i-th sub-feature, and Similarly, in order to calculate the spatial attention map of each feature, different semantic sub-features are aggregated and normalized using group normalization. For example, the height-direction spatial attention of the VIS feature map is calculated as: σ is the Sigmoid function, Attn VH With Attn VH Similarly; finally, generate the spatial attention feature map: X Vs =Attn VH ×Attn VW ×X V (9) X Ds =Attn DH ×Attn DW ×X D (10) Step 2-5, cross-modal self-attention layer processing: X Vs Transform to Q V , K V 、V V , X Ds Transform to Q D , K D 、V D Make it suitable for self-attention calculation; then, calculate cross-modal attention; calculate the attention from VIS to DSM modality: Similarly, calculate the attention Attn from DSM to VIS mode DV ; Fuse the attention results of the two directions to obtain the fused feature map X fused .

5. The multi-scale dual-branch semantic segmentation method for high-resolution remote sensing images according to claim 1, characterized in that: In step 2), the decoder finally decodes the feature map X fused Upsampling is performed to gradually restore the resolution of the original image; in the encoder, the feature map extracted by each convolution layer retains information at different levels; in the decoder, the upsampled feature map is fused with the feature map of the corresponding level in the encoder through jump connections; after multiple upsampling and jump connections, the final fused feature map X is obtained final ; Finally, it is processed through a segmentation head to convert it into a class prediction for each pixel.

Images