Large-format remote sensing image semantic segmentation method for super-long context modeling

Abstract

The invention discloses a large-format remote sensing image semantic segmentation method for super-long context modeling. The method comprises the following steps: acquiring remote sensing data; constructing a basic network comprising a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance network and a decoder based on hybrid experts; performing hierarchical pre-training on the backbone network and the global feature guidance network; performing end-to-end training and fine tuning on the basic network; and obtaining a final network model to realize large-format remote sensing image semantic segmentation. By means of the method, the visual context with the hundred million-level sequence length can be processed on a single device, and meanwhile it is guaranteed that calculation and memory overhead are within a reasonable range. Based on the backbone network architecture, an efficient expert hybrid computing strategy is adopted, and the decoder part of the model is further optimized, so that the decoder part can effectively cope with semantic segmentation tasks of different data environments.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition and segmentation, and in particular relates to a large-format remote sensing image semantic segmentation method suitable for ultra-long context modeling of large-format remote sensing images. Background Art

[0002] With the development of technologies such as big data and artificial intelligence, remote sensing image intelligent interpretation and monitoring technology based on deep learning has shown certain advantages. Land use / land cover (LULC) classification is an important task of remote sensing intelligent interpretation. It has played an important role in land resources planning, natural environment monitoring, smart city construction and national defense. The semantic segmentation model currently used in LULC tasks is usually based on the model transformation in the field of computer vision. In the existing semantic segmentation process of large-format remote sensing images, due to the limitations of hardware memory and computing power, it is usually necessary to cut large-format images with billions of pixels into small tiles of 512×512 pixels and input them into the model for learning. In this process, the scene range covered by the small tiles limits the model's mining of spatial context information of large objects such as long rivers and vast farmland, which in turn affects the accuracy and generalization ability of the semantic segmentation model in object mapping.

[0003] At present, the semantic segmentation technology for large-format images is mainly divided into three directions:

[0004] Post-processing strategy processing on the inference side. A common approach for large-format images is to cut them into smaller tiles for analysis and use overlapping and averaging strategies in the segmentation model inference phase to reduce the blocking effect. For example, Khen et al. proposed a technique to average the prediction scores in 50% overlapping areas to achieve a balance between the inference tile size and computational efficiency and accuracy. Another strategy is to increase the input tile size and trim error-prone edges from the segmentation output, or use smoother stitching techniques such as second-order spline window function weighting [1]. However, due to the limited receptive field of small tiles, these methods have difficulty in fully perceiving the contextual features of large objects, limiting the segmentation accuracy.

[0005] Improvements in the semantic segmentation model architecture. The first is improvements to the CNN-based semantic segmentation network. In order to increase the receptive field of the network, researchers have explored methods of fusing global and local features in CNN-based segmentation models. For example, GLNet[2] and ISD-Net[3] obtain global features of large-format images through downsampling and combine them with local features of tiles to improve segmentation accuracy and save memory. However, the downsampling step may lose spatial details and affect accuracy. Other studies[4,5] explored methods of fusing multi-scale contextual features in a multi-layer encoder-decoder architecture to enhance the accuracy of local segmentation. The large convolution kernels used in [6,7] attempted to introduce long-range dependencies into the CNN architecture. However, these methods are still limited in capturing large-format global contextual information because the input image size and receptive field of the model are always limited.

[0006] The second is the semantic segmentation network based on the Transformer architecture. Networks based on the Transformer architecture have attracted widespread attention due to their potential in capturing long-range contextual relationships. We call the basic Transformer architecture applied to computer vision tasks Vision Transformer (ViT). Inspired by multi-scale, pyramidal image processing, the latest progress in the development of ViT architecture focuses on efficiency and integration of multi-scale information, such as Swin[8], ViL, TNT, VPT, MViT[9], etc., through different network configuration designs to adapt to different scales or aspect ratios of visual tasks. However, when directly processing large-format image inputs with a size of billions of pixels, due to device computing and memory limitations, networks based on the ViT architecture still find it difficult to fully perceive their long sequence visual context information.

[0007] Parallel computing strategies. With the development of large models, the use of distributed deep learning systems to process these models has become a new research trend. Among these parallel systems, three main strategies are adopted: data parallelism

[10] , model parallelism

[11] , and hybrid parallelism

[12] . These methods each target specific challenges and computing requirements, and together promote the effective application of large models in complex, resource-intensive tasks. Although distributed deep learning systems have made progress in processing large-scale data, most systems do not fully consider the need to directly process large-scale imagery. This limitation makes it difficult for these systems to be directly applied to semantic segmentation tasks of ultra-large-scale remote sensing images, and the specific challenges involved have not been effectively addressed.

[0008] References:

[0009] [Reference 1] G. Chevalier, "Make smooth predictions by blending image patches, such as for image segmentation," 2020.

[0010] [Reference 2] W. Chen, Z. Jiang, Z. Wang, K. Cui, and X. Qian, "Collaborative global-local networks for memory-efficient segmentation of ultra-high resolution images," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 8924-8933.

[0011] [Reference 3] S. Guo, L. Liu, Z. Gan, Y. Wang, W. Zhang, C. Wang, G. Jiang, W. Zhang, R. Yi, and L. Ma, "Isdnet: Integrating shallow and deep networks for efficient ultra-high resolution segmentation," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4361-4370.

[0012] [Reference 4] Q. Li, W. Yang, W. Liu, Y. Yu, and S. He, "From contexts to locality: Ultra-high resolution image segmentation via locality-aware contextual correlation," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 7252-7261.

[0013] [Reference 5] Y. Li, W. Chen, X. Huang, Z. Gao, S. Li, T. He, and Y. Zhang, "MFVNet: a deep adaptive fusion network with multiple field-of-views for remote sensing image semantic segmentation," Science China Information Sciences, vol. 66, p. 140305, 2023.

[0014] [Reference 6] W. Chen, Y. Li, B. Dang, and Y. Zhang, "ElegantSeg: End-to-End Holistic Learning for Extra-Large Image Semantic Segmentation," arXiv preprint arXiv:2211.11316, 2022.

[0015] [Reference 7] S. Wang, D. M. Yang, R. Rong, X. Zhan, and G. Xiao, "Pathology image analysis using segmentation deep learning algorithms," The American journal of pathology, vol. 189, pp. 1686-1698, 2019.

[0016] [Reference 8] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 10012-10022.

[0017] [Document 9] H.Fan, B.Xiong, K.Mangalam, Y.Li, Z.Yan, J.Malik, andC.Feichtenhofer, "Multiscale vision transformers," in Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp.6824-6835.

[0018] [Document 10] A. Krizhevsky, "One weird trick for parallelizing convolutional neural networks," arXiv preprint arXiv:1404.5997, 2014.

[0019] [Literature 11] J.Dean, G.Corrado, R.Monga, K.Chen, M.Devin, M.Mao, M.Ranzato, A.Senior, P.Tucker, and K.Yang, "Large scaledistributed deep networks," Advances in neural information processing systems, vol.25, 2012.

[0020] [Document 12] V. Elango, "Pase: Parallelization strategies for efficient dnntraining," in 2021IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2021, pp.1025-1034. Summary of the invention

[0021] In view of the shortcomings of existing large-format remote sensing image semantic segmentation methods, the present invention provides a large-format remote sensing image semantic segmentation method with ultra-long context modeling. The method adopts a Transformer-based, global semantics-guided semantic segmentation architecture, which is designed to enable it to process visual contexts with a sequence length of hundreds of millions on a single device while ensuring that the computation and memory overhead are within a reasonable range. Based on the above backbone network architecture, an efficient expert hybrid computing strategy is adopted to further optimize the decoder part of the model, so that it can effectively cope with semantic segmentation tasks in different data environments.

[0022] The technical solution provided by the present invention is as follows:

[0023] In a first aspect, a large-format remote sensing image semantic segmentation method for ultra-long context modeling is provided, comprising the following steps:

[0024] Acquisition of remote sensing data;

[0025] Constructing a basic network including a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance, and a decoder based on mixed experts; the receptive field expansion layer includes a spatial aggregation layer, an embedding layer, and a global guidance network;

[0026] Hierarchical pre-training of the backbone network and the global guidance network;

[0027] End-to-end training and fine-tuning of the base network;

[0028] Obtain the final network model to achieve semantic segmentation of large-scale remote sensing images.

[0029] In a possible implementation, the backbone network based on the visual self-attention model includes input embedding and position encoding, a multi-head self-attention mechanism, a feedforward neural network, a skip connection, a normalization layer, and a global feature representation;

[0030] Methods for introducing global feature representation include:

[0031] When the backbone network embeds overlapping tiles, the global features are initialized and concatenated, and the global features and local features are passed through N self-attention modules together;

[0032] After the self-attention module, the global features are removed, and the global features are upgraded through the multi-layer perceptron and activated with the activation function to obtain the upgraded global features;

[0033] The local features are fused through the overlapping tile fusion layer to obtain fused local features;

[0034] The dimension-upgraded global features are concatenated onto the fused local feature heads.

[0035] In a possible implementation, in the receptive field expansion layer, the spatial aggregation layer is used to arrange the global feature vectors corresponding to a number of tiles in two-dimensional space to obtain a feature representation matrix of a large tile, thereby aggregating the feature vector of a single tile;

[0036] The embedding layer includes a linear layer, an activation function, and a position encoding, which is used to aggregate input features, retain spatial information through position encoding, and introduce a global feature representation of the layer by adding a classification head;

[0037] The global guidance network includes a ViT network of multiple Transformer layers for feature extraction of the receptive field.

[0038] In a possible implementation, the feature aggregation layer includes: a multi-layer perceptron, an activation function, and a convolution layer; the multi-layer perceptron and the activation function are used to connect the feature vectors collected from different levels of the backbone network through the multi-layer perceptron; then, through a layer of activation function, multiply the original input feature vector to obtain a first output result; then, the first output result is added to the original input feature to obtain a second output result; the convolution layer is used to reduce the dimension of the feature vector matrix obtained by the second result after stacking in the dimensional direction, so as to fuse the feature representations of different levels;

[0039] The global feature guidance includes a fully connected layer, which is used to perform outer product on the local features extracted by the feature aggregation layer and the global features extracted by the receptive field expansion layer, and then add the outer product result to the original local features to obtain a final fused feature vector;

[0040] The hybrid expert decoder comprises: a gating network and a multi-expert decoder; wherein the gating network comprises a weight matrix W g , maintain the TopK optimization function and the Softmax activation function to solve the decoder experts so that the fused feature vector can be assigned to the corresponding category; each expert in the multi-expert decoder restores the segmentation result of each category.

[0041] In a possible implementation, the method of performing hierarchical independent pre-training on the backbone network and the global guidance network includes:

[0042] The method of pre-training the backbone network includes: firstly, using a mask self-supervision method to improve the local feature accuracy of the backbone network; then, using a contrastive learning self-supervision method to improve its global feature accuracy;

[0043] Methods for pre-training the global guidance network include: contrastive learning self-supervised pre-training.

[0044] In a possible implementation, the method for performing end-to-end training and fine-tuning of the basic network includes:

[0045] Count the number of all training set categories and write the corresponding one-hot code label according to the category G , calculate the loss according to the category solved by the expert gate control, and get the first category loss value;

[0046] The total global features of the receptive field expansion layer are used to predict the total number of object categories in the entire image, and the loss function is set to obtain the second type of loss value;

[0047] Calculate the loss value of the semantic segmentation result to obtain the third type of loss value;

[0048] Perform weighted summation, backpropagation, and network update on the first, second, and third category loss values;

[0049] Use the weights obtained from layer-by-layer pre-training for end-to-end training, calculate the total training loss, and fine-tune the weight ratio through hyperparameters;

[0050] The total training loss is calculated as follows:

[0051] Loss=Loss seg +μLoss G +σLoss Classification

[0052] Among them, Loss seg Indicates the loss of the segmentation result compared with the true value, Loss G Represents the loss between the distribution of the gated network solution and the true category distribution, Loss Classification Represents the loss between the classification result of the number of categories and the true value, μ and σ are two hyperparameters;

[0053] Loss seg =CrossEntropy(label truth , result seg ); CrossEntropy() represents the cross entropy loss function, label truth is the true value of the feature classification; result seg It represents the final segmentation result formed by stacking several types of segmentation results obtained by the hybrid expert decoder.

[0054] In a second aspect, a large-format remote sensing image semantic segmentation device with ultra-long context modeling is provided, comprising:

[0055] A data acquisition module, used for acquiring remote sensing data;

[0056] A basic network building module, used to build a basic network including a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance, and a decoder based on a mixture of experts; the receptive field expansion layer includes a spatial aggregation layer, an embedding layer, and a global guidance network;

[0057] A pre-training module, used to perform hierarchical pre-training of the backbone network and the global guidance network;

[0058] A training module for end-to-end training and fine-tuning of the base network;

[0059] The model acquisition module is used to obtain the final network model to achieve semantic segmentation of large-scale remote sensing images.

[0060] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the large-format remote sensing image semantic segmentation method based on the ultra-long context modeling as described above is implemented.

[0061] In a fourth aspect, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the large-format remote sensing image semantic segmentation method with the aforementioned ultra-long context modeling is implemented.

[0062] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the large-format remote sensing image semantic segmentation method based on the aforementioned ultra-long context modeling.

[0063] The invention provides a large-scale remote sensing image semantic segmentation method with ultra-long context modeling, which makes full use of the characteristics of global features and local features, and effectively associates the features of the whole image without excessively increasing the memory overhead. The beneficial effects of this method are:

[0064] (1) The method effectively considers a large number of backbone network models based on visual self-attention models and corresponding pre-trained weights in the field of deep learning, and can directly select the model suitable for the relevant task as the backbone network and corresponding weights, which greatly saves the training cost and can be used to complete the semantic segmentation task of large-format images.

[0065] (2) The method introduces a spatial aggregation method of the receptive field expansion layer, which can effectively associate visual contexts with billions of sequences and make the model sparse; it effectively expands the receptive field of the semantic segmentation network, suppresses the blocking effect, and keeps the memory overhead within a reasonable range, thereby achieving a faster reasoning speed than the dense model.

[0066] (3) The method adopts a decoder based on hybrid experts to simplify the segmentation results into a set of two categories. It can make more accurate category positioning in the face of different semantic segmentation data sets and is more scalable in semantic understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the overall process of the method described in the embodiment of the present invention;

[0068] Figure 2 A schematic diagram of a neural network model for a large-format remote sensing image semantic segmentation method for ultra-long context modeling according to an embodiment of the present invention;

[0069] Figure 3Schematic diagram of the encoder structure of the backbone network that introduces global feature representation;

[0070] Figure 4 A schematic diagram of the structure of a large-format remote sensing image semantic segmentation device for ultra-long context modeling provided by the present invention;

[0071] Figure 5 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0072] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0073] Please see Figure 1 , Figure 1 A large-format remote sensing image semantic segmentation method for ultra-long context modeling of billion-pixel images is provided in an embodiment of the present application. The method comprises the following steps:

[0074] Step 1: Obtain remote sensing data.

[0075] For example, the dataset uses a dataset of a certain area in Guangdong Province. It comes from: 9 images of the "Beijing-2" satellite (resolution of 0.8 meters, size of 38045x36939 pixels); 9 images of the "Gaofen-2" satellite (resolution of 0.8 meters, size of 36252x35338 pixels); 6 images of the "Gaojing-1" satellite (resolution of 0.5 meters, size of 29552x32053 pixels); 2 images of the "Gaojing-1" satellite (resolution of 0.5 meters, size of 31668x31899 pixels) were launched from KOMPSAT-3 at a distance of 0.5 meters. The dataset is labeled into 11 categories, of which 20 images are used for training and 6 images are used for verification. The RGB data of remote sensing images can be obtained from the dataset. Since there are parts that require data preprocessing in the subsequent process, no additional preprocessing is done here.

[0076] Step 2: Construct a basic network including a backbone network based on a visual self-attention model (Vision Transformer, ViT), a receptive field expansion layer (RFEL), a feature aggregation layer (Feature Aggregation, FA), a global feature guidance network (Global Feature Institution, GFI) and a MoE-based decoder. The entire network structure is as follows: Figure 2 shown.

[0077] It should be noted that the backbone network can use SegFormer, ViT-Base, ViT-Small, ResNet and other networks.

[0078] The network structure of each layer is described in detail below.

[0079] 2.1 Backbone Network

[0080] Exemplarily, the backbone network and the corresponding decoder are only described by taking SegFormer as an example.

[0081] It should be noted that SegFormer has the following characteristics for the basic network described in the embodiment of this application: First, the backbone network of SegFormer itself has a feature aggregation part; second, due to the layered tower structure of SegFormer, the entire backbone network does not add global features. If the backbone network is not modified, the lack of global features described in this embodiment will lead to the inability to implement. Therefore, it is necessary to modify the backbone network and pre-train the modified model so that it can be applied to the method described in the embodiment of this application.

[0082] Specifically, Figure 2 As shown, the backbone network introduces global feature representation to improve SegFormer. The SegFormer includes input embedding and position encoding, multi-head self-attention mechanism, feedforward neural network, skip connection and normalization layer.

[0083] Specifically, the method of introducing global feature representation is as follows:

[0084] When the backbone network overlaps the image blocks and embeds them, the global features are initialized and spliced ​​to be added, and the global features and local features are passed through N self-attention modules together;

[0085] After the self-attention module, before the overlap patch merging, the global features are removed and the global features are upgraded through the multilayer perceptron (MLP) and activated with the activation function to obtain the upgraded global features.

[0086] After the local features are fused through the overlapping tile fusion layer, the fused local features are obtained;

[0087] The dimension-upgraded global features are concatenated onto the fused local feature heads.

[0088] 2.2, the receptive field expansion layer RFEL is composed as follows:

[0089] The receptive field expansion layer RFEL includes the spatial aggregation (SA) layer, the embedding layer and the global guidance network (GGN).

[0090] The spatial aggregation layer is used to arrange the global feature vectors corresponding to several tiles in two-dimensional space to obtain the feature representation matrix of the large tile, so as to aggregate the feature vector of a single tile;

[0091] The embedding layer includes a linear layer, an activation function, and a position encoding, which is used to aggregate input features, retain spatial information through position encoding, and introduce a global feature representation of the layer by adding a classification head;

[0092] The global guidance network includes a ViT network of multiple Transformer layers for feature extraction of a larger receptive field. Specifically, the embodiment of the present application includes a ViT network of 6 Transformer layers, a block size of 16, an embedding dimension of 384, and a self-attention mechanism using 6 attention heads.

[0093] It should be noted that each layer in the Transformer layer structure includes a multi-head self-attention mechanism, a feedforward network and a normalization layer.

[0094] The data flow in the receptive field expansion layer RFEL is as follows:

[0095] First, the global features of several tiles are aggregated through the spatial aggregation layer; then they are embedded and positionally encoded through the embedding layer; finally, the global feature association is performed through the global guidance network to output features with a larger receptive field.

[0096] For example, in order to better illustrate the RFEL structure, the following naming convention is used: the following symbols are used to distinguish the "image" and its corresponding "input token" size. For an image x with a resolution of L×L L , will be in x L The token sequence extracted from the specified block (size is l×l) is represented as Where M is the sequence length and d is the embedding dimension extracted from images of size [l×l].

[0097] Due to memory overhead, large-format images need to be divided into blocks, but there is a lack of effective correlation between different blocks, which limits the receptive field. This requires RFEL to expand the receptive field of the blocks. 30000 , divide it into 13924 x 256 image, and for x 256Image, ViT-based methods usually use l = L 1 / 2 = 16, resulting in a sequence length of M = 256. The following is based on this setting. This means that for a size of x 256 For an image of , 256 local features and 1 global feature will be obtained. In the following, l1, l2, l3, and l4 are used to refer to the scales of 16, 256, 4096, and 30000.

[0098] For x l3 The image will be The task of the RFEL layer is to associate these 256 features. It represents the global feature vector obtained by the attention head corresponding to an l×l tokens in the sequence, and i represents the sequence number of the feature in the sequence. Indicates a length of M At the same time, each The corresponding local features are called n represents the sequence length, d l Represents the embedding dimension, and (n-1) means that the global features are removed in this part.

[0099] When entering RFEL, first Spatial aggregation is performed in the spatial aggregation layer, that is, the vectors are rearranged according to the order of embedding so that the vectors have prior spatial information. The embedding layer in RFEL does not have a convolutional layer, which mainly provides position encoding. Then, semantic association is performed through the global guidance network. For feature extraction tasks, such as x 256 Image, the first fine-grained feature extraction network is directly called the backbone network, and the higher than x 256 The global feature extraction network on the image with the resolution of [L×L] is called the Global Guidance Network (GGN), which is expressed as GGN L , and its corresponding RFEL layer is called RFEL L .GGN L All use the standard bucket-structured ViT network (such as ViT-B), so the length and dimension of the input and output feature sequences are consistent. This facilitates the guidance of local features of small receptive fields in GFI. l2 For images, M1 = 256; similarly for x 4096 For images, M2 = 256; and for x l4 , where M3=64, l4 cannot divide l3, so the corresponding blocks overlap.

[0100] So, for x l4 For images, in order to capture the dependencies that may exist at each image resolution and provide guidance, the embodiment of the present application uses multiple RFEL networks to form a multi-layer recursive structure, and regards the full-size image as a hierarchical feature aggregation similar to a long document. These features are recursively aggregated into a larger tile range, gradually expanding the receptive field.

[0101] The overall process is as follows:

[0102]

[0103] Among them, {} represents a feature set consisting of global features and local features, CLS l4 represents the global features at scale l4, u l4 It represents the local features at the l4 scale, M1, M2 and M3 are the sequence lengths, which are 256, 256 and 64 respectively.

[0104] It should be noted that the number of RFEL layers is not limited to RFELl3 and RFEL l4 These two layers are sufficient for general segmentation scenarios of general billion-pixel images. In the face of different network settings, the number of RFEL layers can be flexibly increased or decreased to achieve a suitable receptive field range.

[0105] 2.3, Feature Aggregation Layer FA

[0106] The structure of the feature aggregation layer FA includes: multi-layer perceptron, activation function and convolution layer. The multi-layer perceptron and activation function are used to perform an operation similar to self-attention, that is, for the features collected from different levels of the backbone network, they are first connected through the multi-layer perceptron, and then multiplied with the original input features through a layer of activation function to obtain the first output result; then this result is added to the original input feature to obtain the second output result, which can learn the complementary representations between different features to perform learnable feature optimization. The convolution layer is used to reduce the dimension of the feature vector matrix obtained by the second result after stacking in the dimensional direction, so as to effectively fuse the feature representations of different levels.

[0107] The data flow in the feature aggregation layer FA is as follows:

[0108] The feature vectors sampled from the four different levels of the backbone network are self-multiplied and self-added by the multi-layer perceptron and activation function. The obtained feature vectors are formally reconstructed and stacked, and then integrated into the original size using the convolutional layer for dimensionality reduction.

[0109] It should be noted that the feature information output by the last layer of the ViT architecture is not necessarily the most informative among all levels, and FA can express the coupling relationship between local features at each level, so that the attention mechanism can better capture the main body of the image and utilize more fine-grained information, thereby effectively improving the accuracy of the features.

[0110] Similarly, in order to better explain the FA structure, a naming convention is used: Since FA only involves the intermediate vectors in the backbone network operation process, the local feature vector output by the penultimate h layer is called b represents the backbone network, n represents the sequence length, d l represents the embedding dimension extracted from an image of size [l×l], and (n-1) means that this module does not need to use global features.

[0111] Take four levels of local features, such as u b-1 、u b-2 、u b-3 and u b-4 In this embodiment, since SegFormer has a "variable depth access" structure, it can access u in the smallest 16 layers of the network. b-1 、u b-4 、u b-10 、u b-14 Local features of these four levels.

[0112] It is understandable that if other types of backbone networks do not have similar structures, they can adopt the strategies of "using the last few layers", "using equal depths", and "using variable depths". Which strategy is applicable and the optimal number of layers to use depends on the characteristics of the selected backbone network.

[0113] The method described in the embodiment of the present application uses SegFormer as the backbone network, which requires replacing the multi-scale feature aggregation part in the decoder with the feature aggregation layer FA. In terms of structure, the local features will first enter a two-layer MLP and an activation function (AF) to calculate a mask matrix M b-h , as shown below:

[0114] M b-h =AF((u b-h ; ×W 1 )×W 2 )

[0115] Among them, AF() represents the activation function, and is a learnable parameter, γ is the decay rate, and this design can reduce memory overhead. Sigmoid can be selected as the activation function.

[0116] Use mask matrix to enhance local features:

[0117]

[0118] After reshaping, The vector is reconstructed into In the form of, under the conditions set in 2.2, H=16, W=16.

[0119] Use concatenation operations and convolutional layers to combine several local results:

[0120]

[0121] in, represents the p-layer aggregated features obtained by the backbone network, where p is the number of layers using local features. The feature form after the connection operation is The convolution kernel of appropriate size is used to obtain For example, when p=4, we can set k=1, s=1, p=0, g=1, c1=4d l ,c2=d l The convolutional layer.

[0122] 2.4, Global Feature Guidance GFI

[0123] RFEL gives the basic network global features with global relevance, FA gives local features with richer information and higher granularity, and the purpose of GFI is to let global features guide local features and effectively correct some context understanding errors caused by global semantics.

[0124] In 2.2, different levels of global features are defined and GFI needs to be used in conjunction with RFEL. It guides the global features obtained by the next level of global features. And the local features obtained by this level of feature aggregation layer or receptive field expansion layer Perform a global guided solution.

[0125] The present application embodiment uses the three-layer structure described in 2.2 to illustrate this process, and the global guidance process for this specific situation is as follows:

[0126] First, set RFEL l4 The local feature results Decompose into RFEL 4096 The global feature results Reconstructed into in the form of.

[0127] For the convenience of correspondence and calculation, it can be written as Then respectively and Input the MLP layer to calculate the mask matrix. The calculation process is as follows:

[0128]

[0129] in, and are learnable parameters, represents the global guidance vector at scale l3, represents the local vector of at scale l3, represents the global vector enhanced by the receptive field expansion layer at scale l4, for Mask matrix for sequences of length 256.

[0130] Finally, the original matrix Combined with the mask matrix, the global feature guidance result is obtained:

[0131]

[0132] Finally in x l3 The global guidance feature obtained by GFI and RFEL at the scale is

[0133] Similarly, the first layer of GFI also needs to be coordinated with the corresponding RFEL l4 The full local feature results of and the local feature results of FA can then be fused using the same module as above, such as:

[0134]

[0135] The combination of two layers of GFI and RFEL is an example under the current setting. This structure ensures that the model of the present invention has unlimited expansion potential for larger-format images. Through layer-by-layer semantic guidance, Will have strong contextual understanding ability.

[0136] 2.5. Decoder based on hybrid experts

[0137] In one possible implementation, the decoder of the method described in the embodiment of the present application is optimized using the MoE concept, and multi-label classification is replaced by binary classification, and the binary classification results obtained by each expert are synthesized into the final semantic segmentation result.

[0138] The gated distribution of tokens is the key to the mixture of experts because the gating network consists of learnable parameters.

[0139] In a general MoE network, the output can be expressed as:

[0140]

[0141] G σ (x) = Soft max (x·W g )

[0142] A typical gating function is usually a simple network with a softmax function, which will learn which expert to send the input to, while experts with a gating calculation of 0 will not perform any calculation.

[0143] However, the number of experts activated each time in the above network is still different. The present invention expects the expert network to be stable and efficient. By introducing the Noisy Top-K Gating network and using a lower k value, training and reasoning can be performed faster than when random experts are activated, further saving computing resources.

[0144] The hybrid expert-based decoder described in the present invention is based on this gating network. As mentioned above, the present invention replaces multi-label classification with binary classification, which means that when N categories are set as the current, N decoder experts are required to play a role. Unlike the general MoE, the present invention does not use the input itself for gating operations, but directly adopts The represented regional semantics performs a class classification task and selects the appropriate decoder expert for classification calculation, as follows:

[0145] Will After reconstruction, we get Then each can represent an x l2 area, use A classification task can be performed.

[0146] So the whole gating process can be described by the following formula:

[0147] H(x)=x·W g

[0148]

[0149] Among them, H(x) represents the gating weight, W g represents the transformation matrix, KeepTopK(ν, k) i Indicates the TopK optimization function, k indicates the number of optimizations, i indicates the number of experts, represents the gated network. k = N is the number of categories that need to be classified. As for the expert itself, it is a decoder that matches the backbone network, which depends on the user's own choice. For example, the expert corresponding to SegFormer here is a simple MLP layer.

[0150] Step 3. Pre-train the backbone network and the global guidance network hierarchically.

[0151] When building a semantic segmentation framework using only Transformer blocks, a self-supervised pre-training method based on ViT is used to extract feature representations of each level of the backbone network for downstream semantic segmentation tasks, which can effectively save training costs and effectively suppress overfitting due to over-parameterization and lack of labels.

[0152] The basic network pre-trained includes two parts: a backbone network backbone and a global guidance network GGN.

[0153] In a possible implementation, the method of performing hierarchical pre-training on the backbone network and the global feature guidance network includes:

[0154] For the backbone network, the masked self-supervision method is first used to improve the local feature accuracy, such as MAE (Masked Autoencoders); then the global feature accuracy is improved by contrastive learning self-supervision methods, such as DINO (self-distillation with no labels). In the embodiment of the present application, the batch size is set to 64, and MAE and DINO are used for training alternately for 100 epochs each, and training is performed for 12 rounds.

[0155] For each independent global guidance network in RFEL, it is also necessary to pre-train independently. Because its network size is small and the information is focused on global features, only contrastive learning self-supervised pre-training can be performed. For each global guidance network, the embodiment of the present application trains 100 epochs under the condition of 64 batches.

[0156] Step 4. Train and fine-tune the network end-to-end.

[0157] In one possible implementation, step 4 includes the following steps:

[0158] (1) Count the number of unions of all training set categories and write corresponding one-hot code labels according to the categories G , calculate the loss according to the category solved by the expert gate control, and get the first category loss value;

[0159] (2) The total global features of the last layer of RFEL are used to predict the total number of object categories in the entire image, and the corresponding loss function is set to obtain the second type of loss value.

[0160] (3) Calculate the loss value of the semantic segmentation result and obtain the third type of loss value.

[0161] (4) Perform weighted summation, back propagation, and network update on the first, second, and third category loss values.

[0162] (5) Perform end-to-end training using the weights obtained from layered pre-training. It should be noted that the number of decoders N should be set to be greater than or equal to the number of all training set categories, and the corresponding one-hot code label should be written according to the category. G , and the loss is calculated based on the category solved by the expert gating.

[0163]

[0164] Among them, CrossEntropy(y, x) is the cross entropy loss function, y represents the predicted value, Represents the true value, Loss G Represents the loss between the distribution solved by the gating network and the true category distribution, G() is the gating network, is the final global feature representation of an l2 region.

[0165] In addition, although N can be set manually, this embodiment adds a classification head to enable the model to automatically calculate the accurate N value in the inference stage, and uses the total global feature of the last layer RFEL to predict the entire image. This global feature collects information about the entire image and can effectively calculate N:

[0166] Loss Classificction =CrossEntropy(label N , Classification(CLS l4 ))

[0167] Among them, Loss Classification Indicates the loss between the classification result of the number of categories and the true value, label N represents the number of N labels, and Classification() represents the classification head for the number of categories. So adding the loss value of the semantic segmentation result, the total end-to-end training loss can be expressed as:

[0168]

[0169] Loss seg=CrossEntropy(label truth , result seg )

[0170] Loss=Loss seg +μLoss G +σLoss Classification

[0171] Among them, result seg It represents the final segmentation result formed by stacking several types of segmentation results obtained by the hybrid expert decoder. N decoder experts selected for the gating network, Loss seg Indicates the loss of the segmentation result compared with the true value, label truth is the true value of the object classification, Loss represents the total loss value, and μ and σ are two hyperparameters used to adjust the weight ratio of each subtask during the training process.

[0172] The following describes a large-format remote sensing image semantic segmentation device with ultra-long context modeling provided by the present invention. The large-format remote sensing image semantic segmentation device with ultra-long context modeling described below and the large-format remote sensing image semantic segmentation method with ultra-long context modeling described above can be referenced to each other.

[0173] Figure 4 is a structural schematic diagram of a large-format remote sensing image semantic segmentation device for ultra-long context modeling provided by an embodiment of the present invention, such as Figure 4 As shown, it includes: a data acquisition module 41, a basic network construction module 42, a pre-training module 43, a training module 44 and a model acquisition module 45, wherein:

[0174] A data acquisition module 41 is used to acquire remote sensing data;

[0175] A basic network construction module 42, used to construct a basic network including a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance, and a decoder based on a mixture of experts; the receptive field expansion layer includes a spatial aggregation layer, an embedding layer, and a global guidance network;

[0176] A pre-training module 43, used for performing hierarchical pre-training on the backbone network and the global guidance network;

[0177] A training module 44, for end-to-end training and fine-tuning of the basic network;

[0178] The model acquisition module 45 is used to obtain the final network model to achieve semantic segmentation of large-scale remote sensing images.

[0179] Figure 5An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the aforementioned large-format remote sensing image semantic segmentation method with ultra-long context modeling.

[0180] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0181] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large-format remote sensing image semantic segmentation method with ultra-long context modeling provided by the above-mentioned methods.

[0182] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the large-format remote sensing image semantic segmentation method with ultra-long context modeling provided by the above-mentioned methods.

[0183] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for large-format remote sensing images based on ultra-long context modeling, characterized in that: The following steps are involved: Acquisition of remote sensing data; Constructing a basic network including a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance, and a decoder based on mixed experts; the receptive field expansion layer includes a spatial aggregation layer, an embedding layer, and a global guidance network; Hierarchical pre-training of the backbone network and the global guidance network; End-to-end training and fine-tuning of the base network; Obtain the final network model to achieve semantic segmentation of large-scale remote sensing images.

2. The large-format remote sensing image semantic segmentation method based on ultra-long context modeling according to claim 1 is characterized in that: The backbone network based on the visual self-attention model includes input embedding and position encoding, multi-head self-attention mechanism, feedforward neural network, skip connection, normalization layer and global feature representation; Methods for introducing global feature representation include: When the backbone network embeds overlapping tiles, the global features are initialized and concatenated, and the global features and local features are passed through N self-attention modules together; After the self-attention module, the global features are removed, and the global features are upgraded through the multi-layer perceptron and activated with the activation function to obtain the upgraded global features; The local features are fused through the overlapping tile fusion layer to obtain fused local features; The dimension-upgraded global features are concatenated onto the fused local feature heads.

3. The large-format remote sensing image semantic segmentation method based on ultra-long context modeling according to claim 1 is characterized in that: In the receptive field expansion layer, the spatial aggregation layer is used to arrange the global feature vectors corresponding to several blocks in two-dimensional space, obtain the feature representation matrix of the large block, and realize the aggregation of the feature vector of a single block; The embedding layer includes a linear layer, an activation function, and a position encoding, which is used to aggregate input features, retain spatial information through position encoding, and introduce a global feature representation of the layer by adding a classification head; The global guidance network includes a ViT network of multiple Transformer layers for feature extraction of the receptive field.

4. The large-format remote sensing image semantic segmentation method based on ultra-long context modeling according to claim 1 is characterized in that: The feature aggregation layer includes: a multi-layer perceptron, an activation function and a convolution layer; the multi-layer perceptron and the activation function are used to connect the feature vectors collected from different levels of the backbone network through the multi-layer perceptron; then, through a layer of activation function, multiply the original input feature vector to obtain a first output result; then, the first output result is added to the original input feature to obtain a second output result; the convolution layer is used to reduce the dimension of the feature vector matrix obtained by the second result after stacking in the dimensional direction, so as to fuse the feature representations of different levels; The global feature guidance includes a fully connected layer, which is used to perform outer product on the local features extracted by the feature aggregation layer and the global features extracted by the receptive field expansion layer, and then add the outer product result to the original local features to obtain a final fused feature vector; The hybrid expert decoder comprises: a gating network and a multi-expert decoder; wherein the gating network comprises a weight matrix W g , maintain the TopK optimization function and the Softmax activation function to solve the decoder experts so that the fused feature vector can be assigned to the corresponding category; each expert in the multi-expert decoder restores the segmentation result of each category.

5. The large-format remote sensing image semantic segmentation method based on ultra-long context modeling according to claim 1, characterized in that: The method for performing layered independent pre-training of the backbone network and the global guidance network comprises: The method of pre-training the backbone network includes: firstly, using a mask self-supervision method to improve the local feature accuracy of the backbone network; then, using a contrastive learning self-supervision method to improve its global feature accuracy; Methods for pre-training the global guidance network include: contrastive learning self-supervised pre-training.

6. The large-format remote sensing image semantic segmentation method based on ultra-long context modeling according to claim 1, characterized in that: The method for end-to-end training and fine-tuning of the basic network includes: Count the number of all training set categories and write the corresponding one-hot code label according to the category G , calculate the loss according to the category solved by the expert gate control, and get the first category loss value; The total global features of the receptive field expansion layer are used to predict the total number of object categories in the entire image, and the loss function is set to obtain the second type of loss value; Calculate the loss value of the semantic segmentation result to obtain the third type of loss value; Perform weighted summation, backpropagation, and network update on the first, second, and third category loss values; Use the weights obtained from layer-by-layer pre-training for end-to-end training, calculate the total training loss, and fine-tune the weight ratio through hyperparameters; The total training loss is calculated as follows: Loss=Loss seg +μLoss G +σLoss Classifiation Among them, Loss seg Indicates the loss of the segmentation result compared with the true value, Loss G Represents the loss between the distribution of the gated network solution and the true category distribution, Loss Classification Represents the loss between the classification result of the number of categories and the true value, μ and σ are two hyperparameters; Loss seg =CrossEntropy(label truth , result seg ); CrossEntropy() represents the cross entropy loss function, label truth is the true value of the feature classification; result seg It represents the final segmentation result formed by stacking several types of segmentation results obtained by the hybrid expert decoder.

7. A large-format remote sensing image semantic segmentation device with ultra-long context modeling, characterized in that: include: A data acquisition module, used for acquiring remote sensing data; A basic network building module, used to build a basic network including a backbone network based on a visual self-attention model, a receptive field expansion layer, a feature aggregation layer, a global feature guidance, and a decoder based on a mixture of experts; the receptive field expansion layer includes a spatial aggregation layer, an embedding layer, and a global guidance network; A pre-training module, used to perform hierarchical pre-training of the backbone network and the global guidance network; A training module for end-to-end training and fine-tuning of the base network; The model acquisition module is used to obtain the final network model to achieve semantic segmentation of large-scale remote sensing images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the large-format remote sensing image semantic segmentation method with ultra-long context modeling as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large-format remote sensing image semantic segmentation method with ultra-long context modeling as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the large-format remote sensing image semantic segmentation method with ultra-long context modeling as described in any one of claims 1 to 6 is implemented.

Images