A Semantic Segmentation Method for Remote Sensing Images Based on Multi-Granularity Context Modeling

Abstract

This application relates to the field of image processing technology, and particularly to a remote sensing image semantic segmentation method based on multi-granularity context modeling. The method includes: acquiring sample data and dividing it into a training set, a validation set, and a test set; constructing a remote sensing image semantic segmentation model based on multi-granularity context modeling, the model consisting of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention mechanism, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature concatenation and aggregation module, and a convolution classifier; iteratively training the model on the training set until convergence, and then fine-tuning the parameters on the validation set; after confirming and fixing the parameters, obtaining the optimal model; and inputting the test set into the optimal model to obtain the semantic segmentation result of the target remote sensing image. This method improves the perception capability of the remote sensing image semantic segmentation model, thereby significantly improving the accuracy of semantic segmentation.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image processing technology, and in particular to a method for semantic segmentation of remote sensing images based on multi-granularity context modeling. Background Technology

[0002] Semantic segmentation of remote sensing images refers to the precise division of each pixel in a remote sensing image into a specific semantic category. Each pixel is assigned a category label, such as water body, vegetation, building, road, bridge, etc. The final output is a segmentation result image of the same size as the original image, with different colors representing different categories. This technology has multiple functions, including automated information extraction, refined environmental monitoring, assisting urban planning and development, and realizing disaster management and emergency response. It can provide important data support for technologies such as smart cities, disaster monitoring, environmental protection, and agricultural assessment, demonstrating significant economic and social benefits. Its research significance is profound, and its application prospects are broad.

[0003] Currently, the vast majority of remote sensing images are acquired using satellite-based space platforms and UAV or aerial camera-based air platforms. This gives remote sensing images a unique characteristic of wide coverage, resulting in targets in these images exhibiting multi-scale and uneven distribution. In recent years, with the continuous maturation of deep learning technology, two mainstream frameworks have gradually emerged in this field, represented by CNN (Convolutional Neural Networks) and VIT (Vision Transformer). CNN excels at extracting local features but lacks long-range representation capabilities, which limits its ability to perceive unevenly distributed ground features in images. VIT has unique advantages in long-range dependency mining but is slightly insufficient in extracting features from local multi-scale ground features.

[0004] Chen Songyu's team improved the traditional U-Net semantic segmentation network by replacing the convolutional layers in the encoding and decoding layers with residual convolutional units to achieve gradient vanishing. However, this method still failed to mine long-range dependencies and its ability to perceive long distances in remote sensing images remained insufficient, resulting in limited accuracy improvement. Summary of the Invention

[0005] To address the aforementioned technical issues, embodiments of this application propose a remote sensing image semantic segmentation method based on multi-granularity context modeling. This method reduces computational complexity, enables long-range contextual awareness of remote sensing images, and achieves local multi-scale ground feature representation of remote sensing images. It effectively enhances the perception capability of the remote sensing image semantic segmentation model, thereby significantly improving the segmentation accuracy of the remote sensing image semantic segmentation model.

[0006] To achieve the above objectives, embodiments of this application propose a remote sensing image semantic segmentation method based on multi-granularity context modeling, comprising: acquiring sample data labeled with real semantic labels, and dividing all sample data into training sample datasets, validation sample datasets, and test sample datasets according to a preset ratio; constructing a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention mechanism, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature concatenation and aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to perform feature extraction at different scales; the long-distance dependency mining module based on linear self-attention mechanism is used to perform spatial domain self-attention calculation and channel domain self-attention calculation. The system achieves long-distance dependency mining; a local context-aware module based on multi-scale asymmetric convolution is used to extract local context features and perform feature selection using a joint gating mechanism guided by max pooling and average pooling strategies; a multi-level feature cascading aggregation module is used to cascade and aggregate local context features at different scales to achieve information fusion and obtain the final fused features; a convolution classifier is used to output semantic segmentation results based on the final fused features; sample data from the training sample dataset is input into the remote sensing image semantic segmentation model, and the remote sensing image semantic segmentation model is iteratively trained until convergence, and the trained remote sensing image semantic segmentation model is optimized based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model; the test target sample dataset is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation results output by the optimal remote sensing image semantic segmentation model.

[0007] To achieve the above objectives, embodiments of this application also propose a remote sensing image semantic segmentation system based on multi-granularity context modeling, comprising: a sample acquisition module, a model construction module, a model training module, and a model usage module; the sample acquisition module is used to acquire sample data labeled with real semantic labels, and divide all sample data into training sample datasets, validation sample datasets, and test sample datasets according to a preset ratio; the model construction module is used to construct a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention mechanism, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature concatenation and aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to perform feature extraction at different scales; the long-distance dependency mining module based on linear self-attention mechanism is used to perform spatial domain self-attention. Force computation and channel domain self-attention computation enable long-distance dependency mining; a local context awareness module based on multi-scale asymmetric convolution is used to extract local context features and perform feature selection using max pooling and average pooling strategies and a jointly guided gating mechanism; a multi-level feature cascade aggregation module is used to cascade and aggregate local context features at different scales to achieve information fusion and obtain the final fused features; a convolution classifier is used to output semantic segmentation results based on the final fused features; a model training module is used to input sample data from the training sample dataset into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and optimize the trained remote sensing image semantic segmentation model based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model; a model usage module is used to input the test target sample dataset into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation results output by the optimal remote sensing image semantic segmentation model.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a remote sensing image semantic segmentation method based on multi-granularity context modeling as described above.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a remote sensing image semantic segmentation method based on multi-granularity context modeling as described above.

[0010] This application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling. It constructs, trains, and uses this multi-granularity context modeling-based remote sensing image semantic segmentation model for semantic segmentation of remote sensing images. The model incorporates a long-distance dependency mining module based on a linear self-attention mechanism. This module addresses the challenge of low segmentation accuracy caused by uneven target distribution in remote sensing images, enhancing the model's long-distance perception capability and improving semantic segmentation accuracy. The model also includes a local context-aware module based on multi-scale asymmetric convolution. This module solves the problem of local multi-scale representation caused by multi-scale targets in remote sensing images, enhancing the model's ability to represent targets at multiple local scales and further improving segmentation accuracy. Finally, the model incorporates a multi-level feature concatenation and aggregation module, which addresses the information gap problem caused by feature layer sampling. After feature extraction by the ResNet-34 backbone feature extraction network, long-distance dependency mining by the long-distance dependency mining module, local context feature extraction by the local context awareness module, and cascade aggregation by the multi-level feature cascade aggregation module, the semantic segmentation result is output by the convolution classifier. This reduces computational complexity, realizes long-distance context awareness of remote sensing images, and realizes local multi-scale ground feature representation of remote sensing images. This effectively improves the perception ability of the remote sensing image semantic segmentation model and thus significantly improves the segmentation accuracy of the remote sensing image semantic segmentation model.

[0011] In some optional embodiments, when dividing all sample data into training sample datasets, validation sample datasets, and test sample datasets according to a preset ratio, the sample data needs to be preprocessed by random rotation, random scaling transformation, and random scaling cropping to obtain sample data with a size of 512×512×3. This data is then divided into training sample datasets, validation sample datasets, and test sample datasets according to a preset ratio, where the three datasets have no overlap in the sample space. When inputting the sample data from the training sample dataset into the remote sensing image semantic segmentation model for iterative training, the sample data from the training sample dataset needs to be input into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network extracts features from the input sample data at different scales, obtaining feature maps with sizes of 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively denoted as e1, e2, e3, and e4.

[0012] In some optional embodiments, the long-distance dependency mining module performs spatial domain self-attention computation and channel domain self-attention computation to achieve long-distance dependency mining, including: in the spatial domain, the long-distance dependency mining module uses a linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial domain self-attention computation on feature maps of different scales to obtain spatial domain feature maps of different scales; in the channel domain, the long-distance dependency mining module uses a classical self-attention mechanism to perform channel domain self-attention computation on feature maps of different scales to obtain channel domain feature maps of different scales; and the spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which are denoted as a1, a2, a3 and a4 respectively.

[0013] In some optional embodiments, a linear self-attention mechanism based on a generalization of Bernoulli's inequality is used to perform spatial domain self-attention computation on feature maps of different scales, resulting in spatial domain feature maps of different scales, as expressed by the formula:

[0014]

[0015] Where N represents the number of pixels in each channel of the feature map after it is flattened along the spatial dimension, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and D spat,z (Q,K,V) i This represents the calculation result of each sample row vector in the spatial domain feature map at the z-th scale;

[0016] Using the classic self-attention mechanism, channel-domain self-attention computation is performed on feature maps of different scales to obtain channel-domain feature maps of different scales, which can be expressed by the following formula:

[0017]

[0018] Where C represents the number of pixels in each space after the feature map is flattened along the channel dimension, and D chan,z (Q,K,V) i This represents the calculation result of each sample row vector in the channel domain feature map at the z-th scale;

[0019] By fusing spatial domain feature maps of different scales with channel domain feature maps of different scales, long-range fused feature maps of different scales are obtained, which can be expressed by the formula:

[0020] D chan,z (Q,K,V) i This represents the calculation result of each sample row vector in the channel domain feature map at the z-th scale;

[0021] Among them, D spat,z (Q,K,V)(ez D represents the result of the residual concatenation between the spatial domain feature map at scale z and the feature map at scale z. chan,z (Q,K,V)(e z ) represents the result after the residual concatenation of the channel domain feature map at the z-th scale and the feature map at the z-th scale.

[0022] In some optional embodiments, the local context-aware module consists of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a max-pooling branch, an average-pooling branch, and a gated concatenation unit. The first extraction branch consists of a 1×1 convolution, the second extraction branch consists of a 1×3 convolution and a 3×1 convolution, the third extraction branch consists of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch consists of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features from long-range fused feature maps of different scales through four stages. Simultaneously, the four extraction branches perform local context features extraction on long-range fused feature maps of the same scale in the same stage. Contextual feature extraction: The max pooling branch consists of a max pooling layer, a fully connected layer, and a ReLU pooling layer, used to perform max pooling on long-range fused feature maps at different scales respectively; the average pooling branch consists of an average pooling layer, a fully connected layer, and a ReLU pooling layer, used to perform average pooling on long-range fused feature maps at different scales respectively; the gated concatenation unit is used to select features from the outputs of each extraction branch, finally obtaining local contextual features at different scales, denoted as h1, h2, h3, and h4 respectively, with sizes of 128×128×128, 64×64×256, 32×32×512, and 16×16×1024 respectively.

[0023] In some optional embodiments, the gated stitching unit performs feature selection on the output of each extraction branch, ultimately obtaining local context features at different scales, achieved through the following formula:

[0024]

[0025] g r,z =ReLU{MLP[Maxpool(a z )]};

[0026] p r,z =ReLU{MLP[Avgpool(a z )]};

[0027] f 1,z =ReLU{BatchNorm[Conv1×1(a z )]};

[0028] f2,z =ReLU{BatchNorm[Conv3×1 <Conx1×3(a z )>]};

[0029] f 3,z =ReLU{BatchNorm[Conv5×1 <Conx1×5(a z )>]};

[0030] f 4,z =ReLU{BatchNorm[Conv7×1 <Conx1×7(a z )>]};

[0031] Wherein, Conv1×1(·) represents 1×1 convolution, BatchNorm(·) represents batch normalization, ReLU(·) represents ReLU pooling, Conv1×3(·) represents 1×3 convolution, Conv3×1(·) represents 3×1 convolution, Conv1×5(·) represents 1×5 convolution, Conv5×1(·) represents 5×1 convolution, Conv1×7(·) represents 1×7 convolution, Conv7×1(·) represents 7×1 convolution, Avgpool(·) represents average pooling, MLP(·) represents multilayer perceptron, Maxpool(·) represents max pooling, h z This represents the local contextual features at the z-th scale, and the subscript r indicates the r-th extraction branch.

[0032] In some optional embodiments, the multi-level feature cascade aggregation module performs progressive upsampling on local context features at different scales to achieve information fusion of local context features at different scales, thereby obtaining the final fused features;

[0033] The final fusion feature is expressed by the formula:

[0034] final = FC[Upsample(d1)];

[0035] d1=Upsample[d2+Upsample(h1)];

[0036] d2=Upsample[d3+Upsample(h2)];

[0037] d3=Upsample[d4+Upsample(h3)];

[0038] d4 = Upsample(h4);

[0039] Here, Upsample(·) represents upsampling, FC(·) represents convolutional classification layer, and final represents final fused features. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a remote sensing image semantic segmentation method based on multi-granularity context modeling provided in one embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the structure of a remote sensing image semantic segmentation model based on multi-granularity context modeling provided in one embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the structure of a long-distance dependency mining module based on a linear self-attention mechanism provided in one embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the structure of a local context-aware module based on multi-scale asymmetric convolution provided in one embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the structure of a multi-level feature cascading aggregation module provided in one embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of a remote sensing image semantic segmentation system based on multi-granularity context modeling provided in another embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. In the various embodiments of this application, many technical details are presented to enable the reader to better understand this application. However, even without these technical details and various variations and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented. The division of the following embodiments is only for convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0049] One embodiment of this application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling, which is applied to an electronic device. The electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the remote sensing image semantic segmentation method based on multi-granularity context modeling proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0050] The specific process of the remote sensing image semantic segmentation method based on multi-granularity context modeling proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0051] Step 101: Obtain sample data labeled with real semantic tags, and divide all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio.

[0052] In the specific implementation, the server first needs to obtain a number of sample data labeled with real semantic labels, and then divide all the sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio.

[0053] In one example, when the server divides all sample data into training, validation, and test datasets according to a preset ratio, it needs to preprocess the sample data through random rotation, random scaling, and random scaling cropping to obtain sample data of size 512×512×3 (represented as...). The dataset is then divided into training, validation, and test datasets according to a preset ratio. It's important to note that the three datasets built by the server have no overlap in their sample spaces; therefore, during sample data preprocessing, it's crucial to ensure that the corresponding real semantic labels change synchronously.

[0054] In one example, the server selected 33 city images from the Vaihingen dataset as sample data labeled with real semantic labels. 15 of these images formed the training sample dataset, 1 was the validation sample dataset, and the remaining 17 were the test sample dataset.

[0055] Step 102: Construct a remote sensing image semantic segmentation model based on multi-granularity context modeling. The remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention mechanism, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier.

[0056] Step 103: Input the sample data from the training sample dataset into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and optimize the trained remote sensing image semantic segmentation model based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model.

[0057] In its implementation, after constructing the training, validation, and test datasets, the server needs to build a remote sensing image semantic segmentation model based on multi-granularity context modeling. This model consists of a ResNet-34 backbone feature extraction network, a long-range dependency mining module based on linear self-attention, a local context-aware module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier. The ResNet-34 backbone feature extraction network is used for feature extraction at different scales. The long-range dependency mining module based on linear self-attention performs spatial and channel-domain self-attention calculations to achieve long-range dependency mining. The local context-aware module based on multi-scale asymmetric convolution utilizes max pooling and average pooling strategies, along with a jointly guided gating mechanism, to extract local context features and perform feature selection. The multi-level feature cascade aggregation module aggregates local context features at different scales to achieve information fusion, obtaining the final fused features. The convolution classifier outputs the semantic segmentation result based on the final fused features.

[0058] In the specific implementation, after the server completes the construction of the remote sensing image semantic segmentation model, it needs to input the sample data in the training sample dataset into the remote sensing image semantic segmentation model, and iteratively train the remote sensing image semantic segmentation model until convergence (training based on cross-entropy loss). Then, the trained remote sensing image semantic segmentation model is optimized based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model.

[0059] In one example, the specific structure of the remote sensing image semantic segmentation model is as follows: Figure 2As shown, the ResNet-34 backbone feature extraction network uses ResNet-34 as the backbone network for feature extraction, mainly divided into four parts, extracting feature maps at four different scales. When the server inputs sample data from the training dataset into the remote sensing image semantic segmentation model for iterative training, it needs to input the sample data from the training dataset into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network extracts features from the input sample data at different scales, obtaining feature maps with sizes of 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively denoted as e1, e3, e4, and e5.

[0060] In one example, the specific structure of the remote sensing image semantic segmentation model is as follows: Figure 2 As shown, the specific structure of the long-distance dependency mining module is as follows: Figure 3 As shown. In the spatial domain, the long-range dependency mining module utilizes a linear self-attention mechanism based on a generalization of Bernoulli's inequality to perform spatial domain self-attention calculations on feature maps of different scales, resulting in spatial domain feature maps of different scales. In the channel domain, the long-range dependency mining module utilizes a classical self-attention mechanism to perform channel domain self-attention calculations on feature maps of different scales, resulting in channel domain feature maps of different scales. Finally, the spatial domain feature maps and channel domain feature maps of different scales are fused to obtain long-range fused feature maps of different scales, denoted as a1, a2, a3, and a4, respectively. The sizes of a1, a2, a3, and a4 are 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively.

[0061] In the spatial domain, given a remote sensing image Where H, W, and C represent the height, width, and number of channels of the remote sensing image, respectively. Flattening each channel of the remote sensing image along the spatial dimension yields the reduced-order data representation as follows: Where N = H × W, representing the number of pixels in each channel. In traditional self-attention mechanisms, the computational complexity for calculating the weighted sum of the query matrix Q, key matrix K, and value matrix V is O(N). 2 The high number of parameters and computational complexity severely reduces computational efficiency. To address this issue, a linear self-attention mechanism based on a generalization of Bernoulli's inequality is designed for the server and applied to self-attention computation in the spatial domain.

[0062] The generalized form of Bernoulli's inequality is as follows:

[0063] If x > -1 and 0 ≤ r ≤ 1, then (1 + x) rIf ≤1+rx, then according to the Law of Cosines, the following inequality holds:

[0064]

[0065] Substituting into the generalized form of Bernoulli's inequality, we get:

[0066]

[0067] Simplifying the terms on the right-hand side of the inequality, the similarity calculation mechanism between the query matrix Q and the key matrix K is as follows:

[0068]

[0069] Based on the above equation, the linear self-attention mechanism generalized from Bernoulli's inequality can be obtained as follows:

[0070]

[0071] z = 1, 2, 3, 4;

[0072] Where N represents the number of pixels in each channel of the feature map after it is flattened along the spatial dimension, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and D spat,z (Q,K,V) i This represents the calculation result of each sample row vector in the spatial domain feature map at the z-th scale.

[0073] Based on this, the computational complexity of self-attention in the spatial domain is only O(N), which is a significant improvement in computational efficiency compared to traditional self-attention mechanisms. For self-attention computation in the channel domain, since the number of feature channels is much smaller than the number of pixels (C << N), the computational complexity of the softmax regularization function in the channel domain is only O(C). 2 Furthermore, the softmax regularization function amplifies differences, thus offering significant advantages for feature capture and extraction from discrete data. Therefore, the server still employs the traditional calculation method (classical self-attention mechanism) for self-attention computation in the channel domain.

[0074] The server utilizes the classic self-attention mechanism to perform channel-domain self-attention computation on feature maps of different scales, resulting in channel-domain feature maps of different scales, as expressed by the formula:

[0075]

[0076] Where C represents the number of pixels in each space after the feature map is flattened along the channel dimension, and D chan,z (Q,K,V) i This represents the calculation result of each sample row vector in the channel domain feature map at the z-th scale.

[0077] Finally, the server fuses spatial domain feature maps of different scales with channel domain feature maps of different scales to obtain long-range fused feature maps of different scales, which can be expressed by the formula:

[0078]

[0079] Among them, D spat,z (Q,K,V)(e z D represents the result of the residual concatenation between the spatial domain feature map at scale z and the feature map at scale z. chan,z (Q,K,V)(e z ) represents the result after the residual concatenation of the channel domain feature map at the z-th scale and the feature map at the z-th scale.

[0080] In one example, the specific structure of the remote sensing image semantic segmentation model is as follows: Figure 2 As shown, the specific structure of the local context-aware module is as follows: Figure 4 As shown, the local context-aware module consists of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a max pooling branch, an average pooling branch, and a gated splicing unit.

[0081] The first extraction branch consists of a 1×1 convolution, the second extraction branch consists of a 1×3 convolution and a 3×1 convolution, the third extraction branch consists of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch consists of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features of long-range fused feature maps at different scales through four stages. The four extraction branches simultaneously extract local context features of long-range fused feature maps at the same scale in the same stage.

[0082] The max-pooling branch consists of a max-pooling layer, a fully connected layer, and a ReLU pooling layer, used to perform max-pooling on long-range fused feature maps at different scales. The max-pooling results will participate in the local context feature extraction of the four extraction branches.

[0083] The average pooling branch consists of an average pooling layer, a fully connected layer, and a ReLU pooling layer, used to perform average pooling on long-range fused feature maps at different scales. The average pooling result will participate in the local context feature extraction of the four extraction branches.

[0084] The gated splicing unit is used to select features from the output of each extraction branch, and finally obtains local context features at different scales, which are denoted as h1, h2, h3 and h4 respectively, with sizes of 128×128×128, 64×64×256, 32×32×512 and 16×16×1024 respectively.

[0085] In one example, the gated concatenation unit performs feature selection on the output of each extraction branch, ultimately obtaining local context features at different scales, achieved through the following formula:

[0086]

[0087] g r,z =ReLU{MLP[MaxPool(a z )]};

[0088] p r,z =ReLU{MLP[Avgpool(a z )]};

[0089] f 1,z =ReLU{BatchNorm[Conv1×1(a z )]};

[0090] f 2,z =ReLU{BatchNorm[Conv3×1 <Conx1×3(a z )>]};

[0091] f 3,z =ReLU{BatchNorm[Conv5×1 <Conx1×5(a z )>]};

[0092] f 4,z =ReLU{BatchNorm[Conv7×1 <Conx1×7(a z )>]};

[0093] Wherein, Conv1×1(·) represents 1×1 convolution, BatchNorm(·) represents batch normalization, ReLU(·) represents ReLU pooling, Conv1×3(·) represents 1×3 convolution, Conv3×1(·) represents 3×1 convolution, Conv1×5(·) represents 1×5 convolution, Conv5×1(·) represents 5×1 convolution, Conv1×7(·) represents 1×7 convolution, Conv7×1(·) represents 7×1 convolution, Avgpool(·) represents average pooling, MLP(·) represents multilayer perceptron, Maxpool(·) represents max pooling, h zThis represents the local contextual features at the z-th scale, and the subscript r indicates the r-th extraction branch.

[0094] In one example, the specific structure of the remote sensing image semantic segmentation model is as follows: Figure 2 As shown, the specific structure of the multi-level feature cascade aggregation module is as follows: Figure 5 As shown, the multi-level feature cascading aggregation module performs progressive upsampling on local context features at different scales, thereby achieving information fusion of local context features at different scales and obtaining the final fused features.

[0095] The final fusion feature is expressed by the formula:

[0096] final = FC[Upsample(d1)];

[0097] d1=Upsample[dd2+Upsample(h1)];

[0098] d2=Upsample[d3+Upsample(h2)];

[0099] d3=Upsample[d4+Upsample(h3)];

[0100] d4 = Upsample(h4);

[0101] Here, Upsample(·) represents upsampling, FC(·) represents the convolutional classification layer, and final represents the final fused feature. The spatial scale of final is consistent with the input remote sensing image (sample image), the number of channels is consistent with the number of image categories, and the size is 512×512×6. This is because the Vaihingen dataset contains a total of 6 land cover categories, so the final output contains 6 channels.

[0102] Step 104: Input the test target sample dataset into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0103] In practical implementation, after obtaining the optimal remote sensing image semantic segmentation model, the server can deploy it in the required application scenario. When using the model, the test target sample dataset is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0104] In this embodiment, a remote sensing image semantic segmentation model based on multi-granularity context modeling is constructed, trained, and used to perform semantic segmentation of remote sensing images. The model incorporates a long-distance dependency mining module based on a linear self-attention mechanism. This module addresses the challenge of low segmentation accuracy caused by uneven target distribution in remote sensing images, enhancing the model's long-distance perception capability and improving semantic segmentation accuracy. A local context-aware module based on multi-scale asymmetric convolution is also included. This module solves the problem of local multi-scale representation caused by multi-scale targets in remote sensing images, enhancing the model's ability to represent targets at local multi-scale levels and further improving segmentation accuracy. Finally, a multi-level feature concatenation and aggregation module is designed to address the information gap problem caused by feature layer sampling. After feature extraction by the ResNet-34 backbone feature extraction network, long-distance dependency mining by the long-distance dependency mining module, local context feature extraction by the local context awareness module, and cascade aggregation by the multi-level feature cascade aggregation module, the semantic segmentation result is output by the convolution classifier. This reduces computational complexity, realizes long-distance context awareness of remote sensing images, and realizes local multi-scale ground feature representation of remote sensing images. This effectively improves the perception ability of the remote sensing image semantic segmentation model and thus significantly improves the segmentation accuracy of the remote sensing image semantic segmentation model.

[0105] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0106] Another embodiment of this application proposes a remote sensing image semantic segmentation system based on multi-granularity context modeling. The implementation details of this system are described below. These details are provided for ease of understanding and are not essential for implementing this example. Figure 6 This is a schematic diagram of the structure of a remote sensing image semantic segmentation system based on multi-granularity context modeling proposed in this embodiment, including: a sample acquisition module 201, a model construction module 202, a model training module 203, and a model usage module 204.

[0107] The sample acquisition module 201 is used to acquire sample data labeled with real semantic labels and divide all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio.

[0108] The model building module 202 is used to construct a remote sensing image semantic segmentation model based on multi-granularity context modeling. This model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolutional layer classifier. The ResNet-34 backbone feature extraction network is used for feature extraction at different scales. The long-distance dependency mining module based on linear self-attention performs spatial domain self-attention and channel domain self-attention calculations to achieve long-distance dependency mining. The local context awareness module based on multi-scale asymmetric convolution uses max pooling and average pooling strategies, along with a jointly guided gating mechanism, to extract local context features and perform feature selection. The multi-level feature cascade aggregation module aggregates local context features at different scales to achieve information fusion and obtain the final fused features. The convolutional classifier outputs the semantic segmentation result based on the final fused features.

[0109] The model training module 203 is used to input sample data from the training sample dataset into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and optimize the trained remote sensing image semantic segmentation model based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model.

[0110] The model uses module 204 to input the test sample dataset into the optimal remote sensing image semantic segmentation model and obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0111] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0112] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0113] Another embodiment of this application proposes an electronic device, the specific structure of which is as follows: Figure 7As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to perform a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in the above method embodiments.

[0114] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0115] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0116] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in the above method embodiments.

[0117] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, ROM (Read-Only Memory), RAM (Random Access Memory), a magnetic disk, or an optical disk.

[0118] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A semantic segmentation method for remote sensing images based on multi-granularity context modeling, characterized in that, include: Obtain sample data labeled with real semantic tags, and divide all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio; A semantic segmentation model for remote sensing images based on multi-granularity context modeling is constructed. This model consists of a ResNet-34 backbone feature extraction network, a long-range dependency mining module based on linear self-attention, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier. The ResNet-34 backbone feature extraction network is used for feature extraction at different scales. The long-range dependency mining module based on linear self-attention performs spatial and channel-domain self-attention calculations to achieve long-range dependency mining. The local context awareness module based on multi-scale asymmetric convolution utilizes max pooling and average pooling strategies, along with a jointly guided gating mechanism, to extract local context features and perform feature selection. The multi-level feature cascade aggregation module aggregates local context features at different scales to achieve information fusion and obtain the final fused features. The convolution classifier outputs the semantic segmentation result based on the final fused features. The sample data in the training sample dataset is input into the remote sensing image semantic segmentation model, and the remote sensing image semantic segmentation model is iteratively trained until convergence. The trained remote sensing image semantic segmentation model is then optimized based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model. The test target sample dataset is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model; When dividing all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio, it is necessary to preprocess the sample data by random rotation, random scaling transformation and random scaling cropping to obtain sample data with a size of 512×512×3. Then, it is divided into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio. The three datasets have no overlap in the sample space. When inputting sample data from the training dataset into the remote sensing image semantic segmentation model for iterative training, the sample data from the training dataset needs to be input into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network extracts features from the input sample data at different scales, obtaining feature maps with sizes of 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively, denoted as _____. , , and The long-distance dependency mining module performs spatial domain self-attention computation and channel domain self-attention computation to achieve long-distance dependency mining, including: In the spatial domain, the long-distance dependency mining module utilizes a linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial domain self-attention calculations on feature maps of different scales, thereby obtaining spatial domain feature maps of different scales. In the channel domain, the long-distance dependency mining module uses the classic self-attention mechanism to perform channel domain self-attention calculation on feature maps of different scales to obtain channel domain feature maps of different scales. By fusing spatial domain feature maps of different scales with channel domain feature maps of different scales, long-range fused feature maps of different scales are obtained, denoted as follows: , , and ; Using a linear self-attention mechanism based on a generalization of Bernoulli's inequality, spatial domain self-attention computation is performed on feature maps of different scales to obtain spatial domain feature maps of different scales, which can be expressed by the following formula: ； ； in, This represents the number of pixels in each channel of the feature map after it has been flattened along the spatial dimension. , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates the first The calculation results of each sample row vector in the spatial domain feature map at each scale; Using the classic self-attention mechanism, channel-domain self-attention computation is performed on feature maps of different scales to obtain channel-domain feature maps of different scales, which can be expressed by the following formula: ； in, This represents the number of pixels in each space after the feature map is flattened along the channel dimension. Indicates the first The calculation results of each sample row vector in the channel domain feature map at each scale; By fusing spatial domain feature maps of different scales with channel domain feature maps of different scales, long-range fused feature maps of different scales are obtained, which can be expressed by the formula: ； in, Indicates the first Spatial domain feature map at each scale and the first The result after connecting the residuals of feature maps at each scale. Indicates the first The channel domain feature map at the first scale and the first scale The result after connecting the residuals of feature maps at each scale.

2. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 1, characterized in that, The local context-aware module consists of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a max pooling branch, an average pooling branch, and a gated splicing unit. The first extraction branch consists of a 1×1 convolution, the second extraction branch consists of a 1×3 convolution and a 3×1 convolution, the third extraction branch consists of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch consists of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features of long-range fused feature maps at different scales through four stages. The four extraction branches simultaneously extract local context features of long-range fused feature maps at the same scale in the same stage. The max pooling branch consists of a max pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to perform max pooling on long-range fused feature maps at different scales, respectively. The average pooling branch consists of an average pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to perform average pooling on long-range fused feature maps at different scales, respectively. The gated concatenation unit is used to select features from the outputs of each extraction branch, ultimately obtaining local context features at different scales, denoted as follows: , , and The dimensions are 128×128×128, 64×64×256, 32×32×512, and 16×16×1024 respectively.

3. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 2, characterized in that, The gated concatenation unit performs feature selection on the output of each extraction branch, ultimately obtaining local context features at different scales, which is achieved through the following formula: ； ； ； ； ； ； ； in, Represents a 1×1 convolution. Indicates batch normalization, Indicates ReLU pooling. This represents a 1×3 convolution. This represents a 3×1 convolution. This represents a 1×5 convolution. This represents a 5×1 convolution. This represents a 1×7 convolution. This represents a 7×1 convolution. Indicates average pooling. This represents a multilayer perceptron. This indicates max pooling. Indicates the first Local contextual features at various scales, subscripts Indicates the first Extraction branches.

4. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 3, characterized in that, The multi-level feature cascade aggregation module performs progressive upsampling on local context features at different scales to achieve information fusion of local context features at different scales, resulting in the final fused features. final The fusion feature is expressed by the formula: ； ； ； ； ； in, Indicates upsampling, Indicates the convolution class layer, This indicates the final fusion feature.

5. A remote sensing image semantic segmentation system based on multi-granularity context modeling, characterized in that, include: The sample acquisition module is used to acquire sample data labeled with real semantic labels and divide all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio. The model building module is used to construct a remote sensing image semantic segmentation model based on multi-granularity context modeling. This model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on linear self-attention, a local context awareness module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier. The ResNet-34 backbone feature extraction network performs feature extraction at different scales. The long-distance dependency mining module performs spatial domain self-attention and channel domain self-attention calculations to achieve long-distance dependency mining. The local context awareness module uses max pooling and average pooling strategies, along with a jointly guided gating mechanism, to extract local context features and perform feature selection. The multi-level feature cascade aggregation module aggregates local context features at different scales to achieve information fusion and obtain the final fused features. The convolution classifier outputs the semantic segmentation result based on the final fused features. The model training module is used to input sample data from the training sample dataset into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and optimize the trained remote sensing image semantic segmentation model based on the validation sample dataset to obtain the optimal remote sensing image semantic segmentation model. The model uses a module to input the test sample dataset into the optimal remote sensing image semantic segmentation model and obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model. When dividing all sample data into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio, it is necessary to preprocess the sample data by random rotation, random scaling transformation and random scaling cropping to obtain sample data with a size of 512×512×3. Then, it is divided into training sample dataset, validation sample dataset and test sample dataset according to a preset ratio. The three datasets have no overlap in the sample space. When inputting sample data from the training dataset into the remote sensing image semantic segmentation model for iterative training, the sample data from the training dataset needs to be input into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network extracts features from the input sample data at different scales, obtaining feature maps with sizes of 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively, denoted as _____. , , and The long-distance dependency mining module performs spatial domain self-attention computation and channel domain self-attention computation to achieve long-distance dependency mining, including: In the spatial domain, the long-distance dependency mining module utilizes a linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial domain self-attention calculations on feature maps of different scales, thereby obtaining spatial domain feature maps of different scales. In the channel domain, the long-distance dependency mining module uses the classic self-attention mechanism to perform channel domain self-attention calculation on feature maps of different scales to obtain channel domain feature maps of different scales. By fusing spatial domain feature maps of different scales with channel domain feature maps of different scales, long-range fused feature maps of different scales are obtained, denoted as follows: , , and ; Using a linear self-attention mechanism based on a generalization of Bernoulli's inequality, spatial domain self-attention computation is performed on feature maps of different scales to obtain spatial domain feature maps of different scales, which can be expressed by the following formula: ； ； in, This represents the number of pixels in each channel of the feature map after it has been flattened along the spatial dimension. , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates the first The calculation results of each sample row vector in the spatial domain feature map at each scale; Using the classic self-attention mechanism, channel-domain self-attention computation is performed on feature maps of different scales to obtain channel-domain feature maps of different scales, which can be expressed by the following formula: ； in, This represents the number of pixels in each space after the feature map is flattened along the channel dimension. Indicates the first The calculation results of each sample row vector in the channel domain feature map at each scale; By fusing spatial domain feature maps of different scales with channel domain feature maps of different scales, long-range fused feature maps of different scales are obtained, which can be expressed by the formula: ； in, Indicates the first Spatial domain feature map at each scale and the first The result after connecting the residuals of feature maps at each scale. Indicates the first The channel domain feature map at the first scale and the first scale The result after connecting the residuals of feature maps at each scale.

6. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in any one of claims 1 to 4.

Images