Remote sensing image semantic segmentation method based on multi-granularity context modeling

Abstract

The invention relates to the technical field of image processing, in particular to a remote sensing image semantic segmentation method based on multi-granularity context modeling, and the method comprises the steps: obtaining sample data, and dividing the sample data into a training set, a verification set and a test set; constructing a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the model is composed of a ResNet-34 backbone feature extraction network, a long-distance dependency relationship mining module based on a linear self-attention mechanism, a local context sensing module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module and a convolution classifier; performing iterative training on the model based on the training set until convergence, performing parameter tuning on the verification set, and obtaining an optimal model after determining and fixing parameters; and inputting the test set into the optimal model to obtain a semantic segmentation result of the target remote sensing image. According to the method, the perception capability of the remote sensing image semantic segmentation model on the remote sensing image is improved, and then the semantic segmentation precision is greatly improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a remote sensing image semantic segmentation method based on multi-granularity context modeling. Background Art

[0002] Remote sensing image semantic segmentation refers to the precise division of each pixel in the remote sensing image into a specific semantic category. Each pixel will be assigned a category label, such as water bodies, vegetation, buildings, roads, bridges, etc., and the final output is a segmentation result map with the same size as the original image and different colors to represent different categories. This technology has multiple functions such as automated information extraction, refined environmental monitoring, assisting urban planning and development, and realizing disaster management and emergency response. It can provide important data support for technical fields such as smart cities, disaster monitoring, environmental protection, and agricultural assessment, reflecting significant economic benefits and social value. The research is of far-reaching significance and has broad application prospects.

[0003] At present, the acquisition of most remote sensing images is completed on satellite-based space-based platforms and air-based platforms based on drones or aerial cameras, which makes remote sensing images have the unique property of wide coverage. Therefore, the targets in remote sensing images show multi-scale and uneven distribution characteristics. In recent years, with the continuous maturity of deep learning technology, two mainstream frameworks represented by CNN (Convolutional Neural Networks) and VIT (Vision Transformer) have gradually formed in this field. CNN is very good at extracting local features, but its long-distance representation ability is insufficient, which leads to CNN's limited perception of unevenly distributed objects in images. VIT has unique advantages in long-distance dependency mining, but it is slightly insufficient in feature extraction of local multi-scale objects.

[0004] Chen Songyu's team improved the traditional U-Net semantic segmentation network by replacing the convolutional layers in the encoding and decoding layers with residual convolutional units to achieve the vanishing of network gradients. However, this method still fails to mine long-distance dependencies and lacks the ability to perceive remote sensing images over long distances, which results in limited accuracy improvement of this method. Summary of the invention

[0005] In order to solve the above technical problems, an embodiment of the present application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling, which reduces the computational complexity, realizes long-distance context perception of remote sensing images, realizes local multi-scale ground object representation of remote sensing images, and effectively improves the perception ability of the remote sensing image semantic segmentation model for remote sensing images, thereby greatly improving the segmentation accuracy of the remote sensing image semantic segmentation model.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling, including: obtaining sample data marked with real semantic labels, dividing all sample data into training sample data sets, verification sample data sets and test sample data sets according to preset proportions; constructing a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model is composed of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to extract features of different scales; the long-distance dependency mining module based on the linear self-attention mechanism is used to perform spatial domain self-attention calculation and channel domain self-attention calculation, Realize long-distance dependency mining; the local context perception module based on multi-scale asymmetric convolution is used to utilize the maximum pooling and average pooling strategies, and the jointly guided gating mechanism to extract local context features and realize feature selection; the multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to realize information fusion and obtain the final fusion feature; the convolution classifier is used to output the semantic segmentation result based on the final fusion feature; the sample data in the training sample data set is input into the remote sensing image semantic segmentation model, the remote sensing image semantic segmentation model is iteratively trained until convergence, and the trained remote sensing image semantic segmentation model is tuned based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model; the test target sample data set is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0007] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes a remote sensing image semantic segmentation system based on multi-granularity context modeling, including: a sample acquisition module, a model construction module, a model training module and a model usage module; the sample acquisition module is used to obtain sample data marked with real semantic labels, and divide all sample data into training sample data sets, verification sample data sets and test sample data sets according to a preset ratio; the model construction module is used to construct a remote sensing image semantic segmentation model based on multi-granularity context modeling, and the remote sensing image semantic segmentation model is composed of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to extract features of different scales; the long-distance dependency mining module based on the linear self-attention mechanism is used to perform spatial domain self-attention Force calculation and channel domain self-attention calculation are used to realize long-distance dependency mining; the local context perception module based on multi-scale asymmetric convolution is used to utilize the maximum pooling and average pooling strategies and the jointly guided gating mechanism to extract local context features and realize feature selection; the multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to realize information fusion and obtain the final fusion feature; the convolution classifier is used to output the semantic segmentation result based on the final fusion feature; the model training module is used to input the sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and tune the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model; the model use module is used to input the test target sample data set into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0008] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a remote sensing image semantic segmentation method based on multi-granularity context modeling as described above.

[0009] In order to achieve the above objectives, an embodiment of the present application also proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a remote sensing image semantic segmentation method based on multi-granularity context modeling as described above.

[0010] The embodiment of the present application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling, which constructs, trains and uses a remote sensing image semantic segmentation model based on multi-granularity context modeling to perform semantic segmentation of remote sensing images. A long-distance dependency mining module based on a linear self-attention mechanism is designed in the model, which solves the problem of low segmentation accuracy caused by uneven target distribution in remote sensing images, enhances the model's long-distance perception ability of remote sensing images, and improves the model's semantic segmentation accuracy. A local context perception module based on multi-scale asymmetric convolution is designed in the model, which solves the problem of local multi-scale representation caused by multi-scale targets in remote sensing images, enhances the model's local multi-scale representation ability of targets in remote sensing images, and further improves the model's segmentation accuracy. A multi-level feature cascade aggregation module is also designed in the model, which can solve the information gap problem caused by feature layer sampling. After feature extraction by the ResNet-34 backbone feature extraction network in the model, long-distance dependency mining by the long-distance dependency mining module, local context feature extraction by the local context perception module, and cascade aggregation by the multi-level feature cascade aggregation module, the convolution classifier outputs the semantic segmentation result, which reduces the computational complexity, realizes long-distance context perception of remote sensing images, realizes local multi-scale ground object representation of remote sensing images, effectively improves the perception ability of remote sensing image semantic segmentation model for remote sensing images, and thus greatly improves the segmentation accuracy of remote sensing image semantic segmentation model.

[0011] In some optional embodiments, when all sample data are divided into training sample data sets, verification sample data sets and test sample data sets according to a preset ratio, it is necessary to preprocess the sample data through random rotation, random scale transformation, and random scale cropping to obtain sample data of a size of 512×512×3, and then divide them into training sample data sets, verification sample data sets and test sample data sets according to a preset ratio, wherein the three sets of data sets do not have any intersection in the sample space; when the sample data in the training sample data set is input into the remote sensing image semantic segmentation model and the remote sensing image semantic segmentation model is iteratively trained, it is necessary to input the sample data in the training sample data set into the ResNet-34 backbone feature extraction network, and the ResNet-34 backbone feature extraction network performs feature extraction of different scales on the input sample data to obtain feature maps of sizes of 128×128×64, 64×64×128, 32×32×256 and 16×16×512, respectively, which are denoted as e1, e2, e3 and e4.

[0012] In some optional embodiments, the long-distance dependency mining module performs spatial domain self-attention calculations and channel domain self-attention calculations to achieve long-distance dependency mining, including: in the spatial domain, the long-distance dependency mining module uses a linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial domain self-attention calculations on feature maps of different scales to obtain spatial domain feature maps of different scales; in the channel domain, the long-distance dependency mining module uses a classical self-attention mechanism to perform channel domain self-attention calculations on feature maps of different scales to obtain channel domain feature maps of different scales; the spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which are respectively denoted as a1, a2, a3 and a4.

[0013] In some optional embodiments, a linear self-attention mechanism based on the generalization of Bernoulli inequality is used to perform spatial domain self-attention calculation on feature maps of different scales to obtain spatial domain feature maps of different scales, which are expressed by the formula:

[0014]

[0015] Among them, N represents the number of pixels on each channel after the feature map is flattened along the spatial dimension, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and D spat,z (Q,K,V) i Represents the calculation result of each sample row vector in the spatial domain feature map of the z-th scale;

[0016] Using the classic self-attention mechanism, channel-domain self-attention calculation is performed on feature maps of different scales to obtain channel-domain feature maps of different scales, which are expressed by the formula:

[0017]

[0018] Among them, C represents the number of pixels in each space after the feature map is flattened along the channel dimension, and D chan,z (Q,K,V) i Represents the calculation result of each sample row vector in the channel domain feature map of the z-th scale;

[0019] The spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which can be expressed as follows:

[0020] D chan,z (Q,K,V) i Represents the calculation result of each sample row vector in the channel domain feature map of the z-th scale;

[0021] Among them, D spat,z (Q,K,V)(ez ) represents the result of the residual connection between the spatial domain feature map of the z-th scale and the feature map of the z-th scale, D chan,z (Q,K,V)(e z ) represents the result of the residual connection between the channel domain feature map of the z-th scale and the feature map of the z-th scale.

[0022] In some optional embodiments, the local context perception module is composed of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a maximum pooling branch, an average pooling branch and a gated splicing unit; the first extraction branch is composed of a 1×1 convolution, the second extraction branch is composed of a 1×3 convolution and a 3×1 convolution, the third extraction branch is composed of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch is composed of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features of long-distance fusion feature maps of different scales through four stages, and the four extraction branches perform local context features extraction on the long-distance fusion feature map of the same scale at the same stage. Extraction of contextual features; the maximum pooling branch consists of a maximum pooling layer, a fully connected layer and a ReLU pooling layer, which are used to perform maximum pooling on long-distance fusion feature maps of different scales respectively; the average pooling branch consists of an average pooling layer, a fully connected layer and a ReLU pooling layer, which are used to perform average pooling on long-distance fusion feature maps of different scales respectively; the gated splicing unit is used to perform feature selection on the output of each extraction branch, and finally obtain local contextual features of different scales, which are denoted as h1, h2, h3 and h4, with sizes of 128×128×128, 64×64×256, 32×32×512, and 16×16×1024 respectively.

[0023] In some optional embodiments, the gated concatenation unit performs feature selection on the output of each extraction branch, and finally obtains local context features of different scales, which is implemented by the following formula:

[0024]

[0025] g r,z =ReLU{MLP[Maxpool(a z )]};

[0026] p r,z =ReLU{MLP[Avgpool(a z )]};

[0027] f 1,z =ReLU{BatchNorm[Conv1×1(a z )]};

[0028] f2,z =ReLU{BatchNorm[Conv3×1 <Conx1×3(a z )>]};

[0029] f 3,z =ReLU{BatchNorm[Conv5×1 <Conx1×5(a z )>]};

[0030] f 4,z =ReLU{BatchNorm[Conv7×1 <Conx1×7(a z )>]};

[0031] Among them, Conv1×1(·) represents 1×1 convolution, BatchNorm(·) represents batch normalization, ReLU(·) represents ReLU pooling, Conv1×3(·) represents 1×3 convolution, Conv3×1(·) represents 3×1 convolution, Conv1×5(·) represents 1×5 convolution, Conv5×1(·) represents 5×1 convolution, Conv1×7(·) represents 1×7 convolution, Conv7×1(·) represents 7×1 convolution, Avgpool(·) represents average pooling, MLP(·) represents multi-layer perceptron, Maxpool(·) represents maximum pooling, h z It represents the local context feature of the z-th scale, and the subscript r represents the r-th extraction branch.

[0032] In some optional embodiments, the multi-level feature cascade aggregation module performs progressive upsampling of local context features of different scales layer by layer to achieve information fusion of local context features of different scales to obtain a final fusion feature;

[0033] The final fusion feature is expressed by the formula:

[0034] final = FC[Upsample(d1)];

[0035] d1=Upsample[d2+Upsample(h1)];

[0036] d2=Upsample[d3+Upsample(h2)];

[0037] d3=Upsample[d4+Upsample(h3)];

[0038] d4 = Upsample(h4);

[0039] Among them, Upsample(·) represents upsampling, FC(·) represents the convolution classification layer, and final represents the final fusion feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flowchart of a remote sensing image semantic segmentation method based on multi-granularity context modeling provided in one embodiment of the present application;

[0042] Figure 2 It is a structural schematic diagram of a remote sensing image semantic segmentation model based on multi-granularity context modeling provided in one embodiment of the present application;

[0043] Figure 3 It is a structural diagram of a long-distance dependency mining module based on a linear self-attention mechanism provided in one embodiment of the present application;

[0044] Figure 4 It is a structural schematic diagram of a local context perception module based on multi-scale asymmetric convolution provided in an embodiment of the present application;

[0045] Figure 5 It is a structural schematic diagram of a multi-level feature cascade aggregation module provided in one embodiment of the present application;

[0046] Figure 6 is a structural schematic diagram of a remote sensing image semantic segmentation system based on multi-granularity context modeling provided in another embodiment of the present application;

[0047] Figure 7 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0049] An embodiment of the present application proposes a remote sensing image semantic segmentation method based on multi-granularity context modeling, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the remote sensing image semantic segmentation method based on multi-granularity context modeling proposed in this embodiment are specifically described below. The following content is only the implementation details provided for ease of understanding and is not necessary for the implementation of this solution.

[0050] The specific process of the remote sensing image semantic segmentation method based on multi-granularity context modeling proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0051] Step 101: obtain sample data annotated with real semantic labels, and divide all sample data into a training sample data set, a verification sample data set, and a test sample data set according to a preset ratio.

[0052] In the specific implementation, the server first needs to obtain a number of sample data marked with real semantic labels, and then divide all the sample data into a training sample data set, a verification sample data set and a test sample data set according to a preset ratio.

[0053] In one example, when the server divides all sample data into a training sample data set, a verification sample data set, and a test sample data set according to a preset ratio, it needs to pre-process the sample data by random rotation, random scale transformation, and random scale cropping to obtain sample data of size 512×512×3 (expressed as ), and then divided into training sample data set, verification sample data set and test sample data set according to the preset ratio. It should be noted that the three sets of data sets constructed by the server do not have any intersection in the sample space. When preprocessing the sample data, it is necessary to ensure that the corresponding real semantic labels change synchronously.

[0054] In one example, the server selects 33 city images from the Vaihingen dataset as sample data annotated with real semantic labels, of which 15 images constitute the training sample dataset, 1 image is the verification sample dataset, and the remaining 17 images are the test sample datasets.

[0055] Step 102, construct a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model is composed of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier.

[0056] Step 103, input the sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and tune the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model.

[0057] In a specific implementation, after the server completes the construction of the training sample data set, the verification sample data set and the test sample data set, it is necessary to build a remote sensing image semantic segmentation model based on multi-granularity context modeling. The remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on a multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier. The ResNet-34 backbone feature extraction network is used to extract features of different scales. The long-distance dependency mining module based on the linear self-attention mechanism is used to perform spatial domain self-attention calculations and channel domain self-attention calculations, thereby realizing long-distance dependency mining. The local context perception module based on multi-scale asymmetric convolution is used to extract local context features and realize feature selection using the maximum pooling and average pooling strategies and the combined guided gating mechanism. The multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to achieve information fusion and obtain the final fusion feature. The convolution classifier is used to output the semantic segmentation result based on the final fusion feature.

[0058] In the specific implementation, after the server completes the construction of the remote sensing image semantic segmentation model, it needs to input the sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence (based on cross entropy loss training), and then tune the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model.

[0059] In an example, the specific structure of the remote sensing image semantic segmentation model is as follows Figure 2As shown in the figure, the ResNet-34 backbone feature extraction network uses ResNet-34 as the backbone network for feature extraction, which is mainly divided into four parts to extract feature maps of four different scales. When the server inputs the sample data in the training sample data set into the remote sensing image semantic segmentation model and iteratively trains the remote sensing image semantic segmentation model, it is necessary to input the sample data in the training sample data set into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network extracts features of different scales from the input sample data and obtains feature maps of sizes of 128×128×64, 64×64×128, 32×32×256 and 16×16×512, which are respectively recorded as e1, e3, e3 and e4.

[0060] In an example, the specific structure of the remote sensing image semantic segmentation model is as follows Figure 2 As shown in Figure 2, the specific structure of the long-distance dependency mining module is as follows: Figure 3 As shown in the figure. In the spatial domain, the long-distance dependency mining module uses the linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial domain self-attention calculations on feature maps of different scales to obtain spatial domain feature maps of different scales. In the channel domain, the long-distance dependency mining module uses the classic self-attention mechanism to perform channel domain self-attention calculations on feature maps of different scales to obtain channel domain feature maps of different scales. Finally, the spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which are denoted as a1, a2, a3 and a4 respectively. The sizes of a1, a2, a3 and a4 are 128×128×64, 64×64×128, 32×32×256 and 16×16×512 respectively.

[0061] In the spatial domain, given a remote sensing image Where H, W, and C represent the height, width, and number of channels of the remote sensing image, respectively. Each channel of the remote sensing image is flattened along the spatial dimension, and the reduced-order data is represented as Where N = H × W, which represents the number of pixels in each channel. In the traditional self-attention mechanism, the computational complexity of calculating the weighted sum of the query matrix Q, the key matrix K, and the value matrix V is O(N 2 ), the number of parameters and the amount of calculation are extremely high, which seriously reduces the calculation efficiency. To address this problem, the server designs a linear self-attention mechanism based on the generalization of Bernoulli inequality and applies it to the spatial domain self-attention calculation.

[0062] The generalized form of Bernoulli's inequality is as follows:

[0063] If x>-1, 0≤r≤1, then (1+x) r≤1+rx, according to the cosine theorem, the following inequality holds:

[0064]

[0065] Substituting the generalized form of Bernoulli's inequality, we get:

[0066]

[0067] By simplifying the right side of the inequality, we can get the similarity calculation mechanism between the query matrix Q and the key matrix K as follows:

[0068]

[0069] Based on the above formula, the linear self-attention mechanism based on the generalization of Bernoulli inequality can be obtained as:

[0070]

[0071] z = 1, 2, 3, 4;

[0072] Among them, N represents the number of pixels on each channel after the feature map is flattened along the spatial dimension, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and D spat,z (Q,K,V) i Represents the calculation result of each sample row vector in the spatial domain feature map of the z-th scale.

[0073] Based on this, the computational complexity of the server's self-attention in the spatial domain is only O(N), which is a significant improvement in computational efficiency compared to the traditional self-attention mechanism. For the self-attention calculation in the channel domain, since the number of feature channels is much smaller than the number of pixels, that is, C<<N, the computational complexity of the softmax regularization function in the channel domain is only O(C 2 ), and the softmax regularization function has the effect of amplifying differences, so it has a great advantage in capturing and extracting features of discrete data. Therefore, the server still uses the traditional calculation method (classical self-attention mechanism) for self-attention calculation in the channel domain.

[0074] The server uses the classic self-attention mechanism to perform channel-domain self-attention calculations on feature maps of different scales to obtain channel-domain feature maps of different scales, which are expressed by the formula:

[0075]

[0076] Among them, C represents the number of pixels in each space after the feature map is flattened along the channel dimension, and D chan,z (Q,K,V) i Represents the calculation result of each sample row vector in the channel domain feature map of the z-th scale.

[0077] Finally, the server fuses the spatial domain feature maps of different scales with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which are expressed by the formula:

[0078]

[0079] Among them, D spat,z (Q,K,V)(e z ) represents the result of the residual connection between the spatial domain feature map of the z-th scale and the feature map of the z-th scale, D chan,z (Q,K,V)(e z ) represents the result of the residual connection between the channel domain feature map of the z-th scale and the feature map of the z-th scale.

[0080] In an example, the specific structure of the remote sensing image semantic segmentation model is as follows Figure 2 As shown in Figure 2, the specific structure of the local context awareness module is as follows: Figure 4 The local context perception module is specifically composed of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a maximum pooling branch, an average pooling branch and a gated splicing unit.

[0081] The first extraction branch consists of a 1×1 convolution, the second extraction branch consists of a 1×3 convolution and a 3×1 convolution, the third extraction branch consists of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch consists of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features of long-distance fusion feature maps of different scales through four stages. The four extraction branches simultaneously extract local context features of long-distance fusion feature maps of the same scale at the same stage.

[0082] The maximum pooling branch consists of a maximum pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to perform maximum pooling on long-distance fusion feature maps of different scales. The maximum pooling result will participate in the local context feature extraction of the four extraction branches.

[0083] The average pooling branch consists of an average pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to average pool the long-distance fusion feature maps of different scales. The average pooling results will participate in the local context feature extraction of the four extraction branches.

[0084] The gated concatenation unit is used to perform feature selection on the output of each extraction branch, and finally obtains local context features of different scales, which are denoted as h1, h2, h3 and h4, with sizes of 128×128×128, 64×64×256, 32×32×512 and 16×16×1024 respectively.

[0085] In one example, the gated concatenation unit performs feature selection on the output of each extraction branch, and finally obtains local context features of different scales, which is implemented by the following formula:

[0086]

[0087] g r,z =ReLU{MLP[MaxPool(a z )]};

[0088] p r,z =ReLU{MLP[Avgpool(a z )]};

[0089] f 1,z =ReLU{BatchNorm[Conv1×1(a z )]};

[0090] f 2,z =ReLU{BatchNorm[Conv3×1 <Conx1×3(a z )>]};

[0091] f 3,z =ReLU{BatchNorm[Conv5×1 <Conx1×5(a z )>]};

[0092] f 4,z =ReLU{BatchNorm[Conv7×1 <Conx1×7(a z )>]};

[0093] Among them, Conv1×1(·) represents 1×1 convolution, BatchNorm(·) represents batch normalization, ReLU(·) represents ReLU pooling, Conv1×3(·) represents 1×3 convolution, Conv3×1(·) represents 3×1 convolution, Conv1×5(·) represents 1×5 convolution, Conv5×1(·) represents 5×1 convolution, Conv1×7(·) represents 1×7 convolution, Conv7×1(·) represents 7×1 convolution, Avgpool(·) represents average pooling, MLP(·) represents multi-layer perceptron, Maxpool(·) represents maximum pooling, h zIt represents the local context feature of the z-th scale, and the subscript r represents the r-th extraction branch.

[0094] In an example, the specific structure of the remote sensing image semantic segmentation model is as follows Figure 2 As shown in Figure 2, the specific structure of the multi-level feature cascade aggregation module is as follows: Figure 5 As shown in Figure 2, the multi-level feature cascade aggregation module performs progressive upsampling of local context features of different scales to achieve information fusion of local context features of different scales and obtain the final fused features.

[0095] The final fusion feature is expressed by the formula:

[0096] final = FC[Upsample(d1)];

[0097] d1=Upsample[dd2+Upsample(h1)];

[0098] d2=Upsample[d3+Upsample(h2)];

[0099] d3=Upsample[d4+Upsample(h3)];

[0100] d4 = Upsample(h4);

[0101] Among them, Upsample(·) means upsampling, FC(·) means convolution classification layer, and final means final fusion feature. The spatial scale of final is consistent with the input remote sensing image (sample image), the number of channels is consistent with the number of image categories, and the size is 512×512×6. This is because the Vaihingen dataset contains 6 types of objects, so the final output result contains 6 channels.

[0102] Step 104: input the test target sample data set into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0103] In the specific implementation, after obtaining the optimal remote sensing image semantic segmentation model, the server can deploy it in the required application scenario. When using the model, the test target sample data set is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0104] In this embodiment, a remote sensing image semantic segmentation model with multi-granularity context modeling is constructed, trained, and used to perform semantic segmentation of remote sensing images. A long-distance dependency mining module based on a linear self-attention mechanism is designed in the model, which solves the problem of low segmentation accuracy caused by uneven target distribution in remote sensing images, enhances the model's long-distance perception ability of remote sensing images, and improves the model's semantic segmentation accuracy. A local context perception module based on multi-scale asymmetric convolution is designed in the model, which solves the problem of local multi-scale representation caused by multi-scale targets in remote sensing images, enhances the model's local multi-scale representation ability of targets in remote sensing images, and further improves the model's segmentation accuracy. A multi-level feature cascade aggregation module is also designed in the model, which can solve the information gap problem caused by feature layer sampling. After feature extraction by the ResNet-34 backbone feature extraction network in the model, long-distance dependency mining by the long-distance dependency mining module, local context feature extraction by the local context perception module, and cascade aggregation by the multi-level feature cascade aggregation module, the convolution classifier outputs the semantic segmentation result, which reduces the computational complexity, realizes long-distance context perception of remote sensing images, realizes local multi-scale ground object representation of remote sensing images, effectively improves the perception ability of remote sensing image semantic segmentation model for remote sensing images, and thus greatly improves the segmentation accuracy of remote sensing image semantic segmentation model.

[0105] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0106] Another embodiment of the present application proposes a remote sensing image semantic segmentation system based on multi-granularity context modeling. The implementation details of the remote sensing image semantic segmentation system based on multi-granularity context modeling proposed in this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this example. Figure 6 It is a structural diagram of a remote sensing image semantic segmentation system based on multi-granularity context modeling proposed in this embodiment, including: a sample acquisition module 201, a model construction module 202, a model training module 203 and a model use module 204.

[0107] The sample acquisition module 201 is used to acquire sample data marked with real semantic labels, and divide all sample data into a training sample data set, a verification sample data set and a test sample data set according to a preset ratio.

[0108] The model construction module 202 is used to construct a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolutional layer classifier; the ResNet-34 backbone feature extraction network is used to extract features of different scales; the long-distance dependency mining module based on the linear self-attention mechanism is used to perform spatial domain self-attention calculations and channel domain self-attention calculations to achieve long-distance dependency mining; the local context perception module based on multi-scale asymmetric convolution is used to utilize the maximum pooling and average pooling strategies and the jointly guided gating mechanism to extract local context features and achieve feature selection; the multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to achieve information fusion and obtain the final fusion feature; the convolution classifier is used to output the semantic segmentation result based on the final fusion feature.

[0109] The model training module 203 is used to input the sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and tune the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model.

[0110] The model using module 204 is used to input the test sample data set into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

[0111] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0112] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0113] Another embodiment of the present application provides an electronic device, whose specific structure is as follows: Figure 7As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in the above-mentioned method embodiments.

[0114] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0115] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0116] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in the above method embodiments.

[0117] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0118] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A remote sensing image semantic segmentation method based on multi-granularity context modeling, characterized in that: include: Obtain sample data marked with real semantic labels, and divide all sample data into a training sample data set, a verification sample data set, and a test sample data set according to a preset ratio; A remote sensing image semantic segmentation model based on multi-granularity context modeling is constructed, wherein the remote sensing image semantic segmentation model consists of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on a multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to extract features of different scales; the long-distance dependency mining module based on a linear self-attention mechanism is used to perform spatial domain self-attention calculations and channel domain self-attention calculations to achieve long-distance dependency mining; the local context perception module based on a multi-scale asymmetric convolution is used to utilize the maximum pooling and average pooling strategies and the jointly guided gating mechanism to extract local context features and achieve feature selection; the multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to achieve information fusion and obtain the final fusion feature; the convolution classifier is used to output a semantic segmentation result based on the final fusion feature; Inputting sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively training the remote sensing image semantic segmentation model until convergence, and optimizing the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model; The test target sample data set is input into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

2. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 1, characterized in that: When all sample data are divided into training sample data set, verification sample data set and test sample data set according to a preset ratio, the sample data needs to be preprocessed by random rotation, random scale transformation and random scale cropping to obtain sample data of size 512×512×3, and then divided into training sample data set, verification sample data set and test sample data set according to a preset ratio, wherein the three sets of data sets do not have any intersection in the sample space; When the sample data in the training sample data set is input into the remote sensing image semantic segmentation model and the remote sensing image semantic segmentation model is iteratively trained, the sample data in the training sample data set needs to be input into the ResNet-34 backbone feature extraction network. The ResNet-34 backbone feature extraction network performs feature extraction of different scales on the input sample data to obtain feature maps with sizes of 128×128×64, 64×64×128, 32×32×256 and 16×16×512, which are respectively denoted as e1, e2, e3 and e4.

3. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 2, characterized in that: The long-distance dependency mining module performs spatial domain self-attention calculation and channel domain self-attention calculation to achieve long-distance dependency mining, including: In the spatial domain, the long-distance dependency mining module uses a linear self-attention mechanism based on the generalization of Bernoulli's inequality to perform spatial self-attention calculations on feature maps of different scales to obtain spatial feature maps of different scales. In the channel domain, the long-distance dependency mining module uses the classic self-attention mechanism to perform channel domain self-attention calculations on feature maps of different scales to obtain channel domain feature maps of different scales. The spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which are denoted as a1, a2, a3 and a4 respectively.

4. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 3 is characterized in that: Using the linear self-attention mechanism based on the generalization of Bernoulli inequality, spatial domain self-attention calculation is performed on feature maps of different scales to obtain spatial domain feature maps of different scales, which are expressed by the formula: Among them, N represents the number of pixels on each channel after the feature map is flattened along the spatial dimension, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and D spat,z (Q,K,V) i Represents the calculation result of each sample row vector in the spatial domain feature map of the z-th scale; Using the classic self-attention mechanism, channel-domain self-attention calculation is performed on feature maps of different scales to obtain channel-domain feature maps of different scales, which are expressed by the formula: Among them, C represents the number of pixels in each space after the feature map is flattened along the channel dimension, and D chan,z (Q,K,V) i Represents the calculation result of each sample row vector in the channel domain feature map of the z-th scale; The spatial domain feature maps of different scales are fused with the channel domain feature maps of different scales to obtain long-distance fused feature maps of different scales, which can be expressed as follows: Among them, D spat,z (Q,K,V)(e z ) represents the result of the residual connection between the spatial domain feature map of the z-th scale and the feature map of the z-th scale, D chan,z (Q,K,V)(V z ) represents the result of the residual connection between the channel domain feature map of the z-th scale and the feature map of the z-th scale.

5. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 4, characterized in that: The local context perception module consists of a first extraction branch, a second extraction branch, a third extraction branch, a fourth extraction branch, a maximum pooling branch, an average pooling branch and a gated splicing unit; The first extraction branch consists of a 1×1 convolution, the second extraction branch consists of a 1×3 convolution and a 3×1 convolution, the third extraction branch consists of a 1×5 convolution and a 5×1 convolution, and the fourth extraction branch consists of a 1×7 convolution and a 7×1 convolution. The four extraction branches complete the extraction of local context features of long-distance fusion feature maps of different scales through four stages. The four extraction branches extract local context features of long-distance fusion feature maps of the same scale at the same stage. The maximum pooling branch consists of a maximum pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to perform maximum pooling on long-distance fusion feature maps of different scales respectively; The average pooling branch consists of an average pooling layer, a fully connected layer, and a ReLU pooling layer, which are used to perform average pooling on long-distance fusion feature maps of different scales respectively; The gated concatenation unit is used to perform feature selection on the output of each extraction branch, and finally obtains local context features of different scales, which are denoted as h1, h2, h3 and h4, with sizes of 128×128×128, 64×64×256, 32×32×512 and 16×16×1024 respectively.

6. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 5, characterized in that: The gated concatenation unit performs feature selection on the output of each extraction branch, and finally obtains local context features of different scales, which is achieved by the following formula: g r,z =ReLU{MLP[Maxpool(a z )]}; pp r,z =ReLU{MLP[Avgpool(a z )]}; f 1,z =ReLU{BatchNorm[Conv1×1(a z )]}; f 2,z =ReLU{BatchNORm[Conv3×1 <Conx1×3(a z )>]}; f 3,z =ReLU{BatchNorm[Conv5×1 <Conx1×5(a z )>]}; f 4,z =ReLU{BatchNorm[Conv7×1 <Conx1×7(a z )>]}; Among them, Conv1×1(·) represents 1×1 convolution, BatchNorm(·) represents batch normalization, ReLU(·) represents ReLU pooling, Conv1×3(·) represents 1×3 convolution, Conv3×1(·) represents 3×1 convolution, Conv1×5(·) represents 1×5 convolution, Conv5×1(·) represents 5×1 convolution, Conv1×7(·) represents 1×7 convolution, Conv7×1(·) represents 7×1 convolution, Avgpool(·) represents average pooling, MLP(·) represents multi-layer perceptron, Maxpool(·) represents maximum pooling, h z It represents the local context feature of the z-th scale, and the subscript r represents the r-th extraction branch.

7. The remote sensing image semantic segmentation method based on multi-granularity context modeling according to claim 6, characterized in that: The multi-level feature cascade aggregation module performs progressive upsampling of local context features of different scales to achieve information fusion of local context features of different scales and obtain the final fused features. final The fusion feature is expressed by the formula: final = FC[Upsample(d1)]; d1=Upsample[d2+Upsample(h1)]; d2=Upsample[d3+Upsample(h2)]; d3=Upsample[d4+Upsample(h3)]; d4 = Upsample(h4); Among them, Upsample(·) represents upsampling, FC(·) represents the convolution classification layer, and final represents the final fusion feature.

8. A remote sensing image semantic segmentation system based on multi-granularity context modeling, characterized in that: include: The sample acquisition module is used to obtain sample data marked with real semantic labels, and divide all sample data into a training sample data set, a verification sample data set, and a test sample data set according to a preset ratio; A model building module is used to build a remote sensing image semantic segmentation model based on multi-granularity context modeling, wherein the remote sensing image semantic segmentation model is composed of a ResNet-34 backbone feature extraction network, a long-distance dependency mining module based on a linear self-attention mechanism, a local context perception module based on a multi-scale asymmetric convolution, a multi-level feature cascade aggregation module, and a convolution classifier; the ResNet-34 backbone feature extraction network is used to extract features of different scales; the long-distance dependency mining module based on a linear self-attention mechanism is used to perform spatial domain self-attention calculations and channel domain self-attention calculations to achieve long-distance dependency mining; the local context perception module based on multi-scale asymmetric convolution is used to utilize the maximum pooling and average pooling strategies and the jointly guided gating mechanism to extract local context features and achieve feature selection; the multi-level feature cascade aggregation module is used to cascade aggregate local context features of different scales to achieve information fusion and obtain the final fusion feature; the convolution classifier is used to output a semantic segmentation result based on the final fusion feature; A model training module, used to input sample data in the training sample data set into the remote sensing image semantic segmentation model, iteratively train the remote sensing image semantic segmentation model until convergence, and optimize the trained remote sensing image semantic segmentation model based on the verification sample data set to obtain the optimal remote sensing image semantic segmentation model; The model usage module is used to input the test sample data set into the optimal remote sensing image semantic segmentation model to obtain the semantic segmentation result output by the optimal remote sensing image semantic segmentation model.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a remote sensing image semantic segmentation method based on multi-granularity context modeling as described in any one of claims 1 to 7.

Images