Remote sensing image segmentation method, device, computer equipment and storage medium
Through the combination of cross-dimensional self-attention and channel self-attention modules, the problem of insufficient global contextual relationship capture in remote sensing image segmentation is solved, and the accuracy and speed of image segmentation are improved, especially in small-target and large-target segmentation. Excellent.
Patent Information
- Application Number
- CN202210763691.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-06-30
AI Technical Summary
Existing remote sensing image segmentation methods are difficult to effectively capture global contextual relationships, resulting in weak distinction ability of similar objects in different semantic categories, especially poor segmentation performance for small targets or inconsistent internal segmentation of large targets.
Using the cross-dimensional self-attention module and the channel self-attention module, by dividing and rearranging the original feature maps on the channel dimension, the cross-dimensional interaction information between the position dimension and the channel dimension is captured, and sparse channel self-attention processing is performed. , reduces the computational complexity to improve processing speed and accuracy.
It improves the accuracy and speed of image segmentation, can obtain more accurate image segmentation results more quickly, and enhances the segmentation performance of small targets and the consistency of internal segmentation of large targets.
Smart Images

Figure CN114998594B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image segmentation method, device, computer equipment and storage medium. Background Art
[0002] With the development of remote sensing technology, it is easier to obtain high-resolution remote sensing images, which contain a lot of usable information. Image segmentation is a computer vision task that predicts the semantic category of each pixel in an image. It has important application value in land resource management, navigation, surveying and mapping, etc.
[0003] In recent years, convolutional neural networks (CNNs) have been widely used in image segmentation. Some researchers have proposed fully convolutional networks (FCNs) to achieve end-to-end semantic segmentation, but many details are lost in the process of downsampling; U-Net uses a skip connection between the encoder and the decoder to fuse low-level features with details and high-level features with semantic information; PSPNet proposes a spatial pyramid pooling module to obtain rich multi-scale context; Deeplab series proposes conditional random fields (CRF) to obtain context and atrous spatial pyramid pooling modules (ASPP) to embed multi-scale context; HRNet proposes a parallel structure backbone network that maintains high-resolution characteristics throughout the process. However, none of the above methods can effectively capture global contextual relationships, resulting in the network's weak ability to distinguish similar objects of different semantic categories.
[0004] Other researchers have proposed using self-attention mechanisms to model non-local dependencies, such as using DANet to capture long-range dependencies in position and channel dimensions, using CCNet's cyclic cross-attention module to model spatial global contextual correlations, and OCNet with staggered sparse self-attention as the core. These methods perform self-attention operations on the position dimension and channel dimension respectively, and the feature map is actually three-dimensional. The above methods ignore the cross-dimensional interaction information between the position dimension and the channel dimension, resulting in poor segmentation performance for small targets or inconsistent internal segmentation of large targets. Summary of the invention
[0005] In view of this, an object of the present invention is to provide a remote sensing image segmentation method, apparatus, computer equipment and storage medium to improve the accuracy of image segmentation.
[0006] A remote sensing image segmentation method, the method comprising:
[0007] Obtaining the original feature map of the remote sensing image to be segmented;
[0008] Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image;
[0009] Divide the original feature map in the channel dimension to obtain a plurality of initial local feature groups, segment each of the initial local feature groups to obtain a plurality of sub-feature maps corresponding to each of the initial local feature groups, recombine and arrange the plurality of sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain a plurality of rearranged local feature groups, and input the plurality of rearranged local feature groups into a pre-constructed channel self-attention module to obtain a plurality of local self-attention feature groups;
[0010] Recombining the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups having the same relative channel order as the sub-feature maps in the initial local feature group, and inputting the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps;
[0011] A segmentation result of the remote sensing image is obtained according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
[0012] In one embodiment, the multiple rearranged local feature groups are respectively input into a pre-built channel self-attention module to obtain multiple local self-attention feature groups, including:
[0013] The multiple rearranged local feature groups M∈R c / n×H×W Input the pre-built channel self-attention module respectively, and perform the multiple rearranged local feature groups M∈R c / n×H×W Perform dimension adjustment respectively to obtain the first dimension adjusted local feature group M∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups;
[0014] Adjust the local feature group M∈R according to the first dimension c / n×HW Get the first channel-to-channel interaction metric score, and according to the first channel-to-channel interaction metric score and the corresponding rearranged local feature group M∈R c / n×H×W Multiple local self-attention feature groups are obtained.
[0015] In one embodiment, the restored local feature groups are respectively input into a pre-built channel self-attention module to obtain a sparse channel self-attention map, including:
[0016] The restored local feature group M′∈R c / n×H×WInput the pre-built channel self-attention module respectively, and restore the multiple local feature groups M′∈R c / n×H×W Perform dimension adjustment respectively to obtain the second dimension adjusted local feature group M′∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups;
[0017] Adjust the local feature group M′∈R according to the second dimension c / n×HW Get the second channel-to-channel interaction metric score, and according to the second channel-to-channel interaction metric score and the corresponding restored local feature group M′∈R c / n×H×W Obtain multiple local self-attention feature groups;
[0018] The multiple local self-attention feature groups are fused to obtain a sparse channel self-attention map.
[0019] In one embodiment, the location dimensions include height and width;
[0020] The step of inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image includes:
[0021] Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image;
[0022] Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image;
[0023] The first cross-dimensional interactive self-attention map and the second cross-dimensional interactive self-attention map are fused to obtain a final cross-dimensional interactive self-attention map.
[0024] In one embodiment, the original feature map is input into a pre-built cross-dimensional self-attention module to obtain a first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image, including:
[0025] The pre-built cross-dimensional self-attention module is used to convolve the original feature map to obtain the first query vector Q1∈R of the original feature map C×H×W , the first key vector K1∈R C×H×W And the first value vector V1∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map;
[0026] For the first query vector Q1∈R C×H×W 、The first key vector K1∈R C×H×WTake the value vector V1∈R C×H×W Adjust the dimension to get the second query vector Q2∈R HC×W , the second key vector K2∈R HC×W And the second value vector V2∈R HC×W ;
[0027] According to the second query vector Q2∈R HC×W and the second key vector K2∈R HC×W Get the cross-dimensional interaction metric score of the height and channel, and according to the cross-dimensional interaction metric score of the height and channel, the second value vector V2∈R HC×W The first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image is obtained by combining the original feature map.
[0028] In one embodiment, the original feature map is input into a pre-built cross-dimensional self-attention module to obtain a second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image, including:
[0029] The pre-built cross-dimensional self-attention module is used to convolve the original feature map to obtain a third query vector Q3∈R of the original feature map. C×H×W , the third key vector K3∈R C×H×W And the third value vector V3∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map;
[0030] For the third query vector Q3∈R C×H×W , the third key vector K3∈R C×H×W With the third value vector V3∈R C×H×W The fourth query vector Q4∈R is obtained by dimension adjustment WC×H , the fourth vector K4∈R WC×H And the fourth value vector V4∈R WC×H ;
[0031] According to the fourth query vector Q4∈R WC×H and the fourth key vector K4∈R WC×H Get the cross-dimensional interaction metric score of width and channel, and according to the cross-dimensional interaction metric score of width and channel, the fourth value vector V4∈R WC×H The second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image is obtained by combining the original feature map.
[0032] In one embodiment, the step of obtaining an original feature map of a remote sensing image to be segmented includes:
[0033] Acquire a remote sensing image to be segmented;
[0034] The remote sensing image is input into a pre-trained ResNet to obtain an original feature map of the remote sensing image to be segmented.
[0035] A remote sensing image segmentation device, the device comprising:
[0036] An acquisition module, used for acquiring an original feature map of a remote sensing image to be segmented;
[0037] A cross-dimensional interaction module, used for inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image;
[0038] A first channel self-attention processing module is used to divide the original feature map in the channel dimension to obtain multiple initial local feature groups, respectively segment each of the initial local feature groups to obtain multiple sub-feature maps corresponding to each of the initial local feature groups, recombin and arrange the multiple sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain multiple rearranged local feature groups, and respectively input the multiple rearranged local feature groups into a pre-constructed channel self-attention module to obtain multiple local self-attention feature groups;
[0039] A second channel self-attention processing module is used to recombine the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups with the same relative channel order as the sub-feature maps in the initial local feature group, and input the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps;
[0040] A segmentation result calculation module is used to obtain the segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
[0041] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0042] Obtaining the original feature map of the remote sensing image to be segmented;
[0043] Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image;
[0044] Divide the original feature map in the channel dimension to obtain a plurality of initial local feature groups, segment each of the initial local feature groups to obtain a plurality of sub-feature maps corresponding to each of the initial local feature groups, recombine and arrange the plurality of sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain a plurality of rearranged local feature groups, and input the plurality of rearranged local feature groups into a pre-constructed channel self-attention module to obtain a plurality of local self-attention feature groups;
[0045] Recombining the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups having the same relative channel order as the sub-feature maps in the initial local feature group, and inputting the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps;
[0046] A segmentation result of the remote sensing image is obtained according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0048] Obtaining the original feature map of the remote sensing image to be segmented;
[0049] Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image;
[0050] Divide the original feature map in the channel dimension to obtain a plurality of initial local feature groups, segment each of the initial local feature groups to obtain a plurality of sub-feature maps corresponding to each of the initial local feature groups, recombine and arrange the plurality of sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain a plurality of rearranged local feature groups, and input the plurality of rearranged local feature groups into a pre-constructed channel self-attention module to obtain a plurality of local self-attention feature groups;
[0051] Recombining the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups having the same relative channel order as the sub-feature maps in the initial local feature group, and inputting the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps;
[0052] A segmentation result of the remote sensing image is obtained according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
[0053] The above-mentioned remote sensing image segmentation method, device, computer equipment and storage medium divide the original feature map of the remote sensing image in the channel dimension to obtain multiple initial local feature groups, and then divide each initial local feature group to obtain a sub-feature map of each initial local feature group, and then recombine and arrange the sub-feature maps according to the relative channel order of the sub-feature maps of each initial local feature group to obtain multiple rearranged local feature groups, and then input the multiple rearranged local feature groups into a pre-constructed channel self-attention module respectively, and perform the first channel self-attention processing after the original feature map is divided and rearranged. The separate processing after division greatly reduces the computational complexity and thus improves the processing speed. Since the number of channels of the features input to the channel self-attention module is reduced after division, only the channel dependency within a small range can be captured. Therefore, the present invention also performs rearrangement after division, so as to achieve that each rearranged local feature group input into the pre-constructed channel self-attention module can capture the channel dependency approximately within the original channel range. Effect, in general, the result of dividing and rearranging and then performing channel self-attention processing is to capture the global and long-distance channel dependency while reducing the computational complexity; next, the multiple local self-attention feature groups obtained after the first channel self-attention processing are recombined to obtain multiple restored local feature groups with the same relative channel order as the sub-feature map in the initial local feature group, that is, to restore the original channel arrangement, and then input the restored local feature groups into the pre-constructed channel self-attention module respectively to obtain sparse channel self-attention maps, that is, to obtain local and short-distance channel dependencies; on the other hand, the present invention also considers the interactive information between the position dimension and the channel dimension, and inputs the original feature map into the pre-constructed cross-dimensional self-attention module to obtain the cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image, and finally obtains the segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map, that is, the feature representation ultimately used for pixel-level prediction. The present invention comprehensively considers the dependencies between channels and the interactive information between channels and positions, and at the same time captures the dependencies between channels to divide, rearrange and restore the order of the original feature map, thereby reducing the computational complexity, speeding up the processing speed, and generally obtaining more accurate image segmentation results more quickly. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A schematic diagram of a remote sensing image segmentation method in one embodiment;
[0055] Figure 2 A schematic diagram of a calculation process of a sparse channel self-attention (SCAM) graph in one embodiment;
[0056] Figure 3Schematic diagram of the process of cross-dimensional interactive self-attention module processing in one embodiment: (a) cross-dimensional interactive operation of height and channel, (b) cross-dimensional interactive operation of width and channel;
[0057] Figure 4 A schematic diagram of a process flow of a channel self-attention module processing in another embodiment;
[0058] Figure 5 A schematic diagram of a remote sensing image segmentation network architecture in one embodiment;
[0059] Figure 6 It is a feature visualization result of a remote sensing image segmentation method on the ISPRS Vaihingen dataset in one embodiment;
[0060] Figure 7 1 is a visualization result of a remote sensing image segmentation method in an embodiment on a Vaihingen and Potsdam dataset, wherein the first two rows show the visualization result on the Vaihingen dataset, and the last two rows show the visualization result on the Potsdam dataset;
[0061] Figure 8 is a structural block diagram of a remote sensing image segmentation device in one embodiment;
[0062] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0064] In one embodiment, Figure 1 As shown, a remote sensing image segmentation method is provided, comprising the following steps:
[0065] Step 102: Obtain an original feature map of the remote sensing image to be segmented.
[0066] The original feature map of the remote sensing image can be obtained by inputting the remote sensing image into a feature extraction network, such as the ResNet101 feature extraction backbone network with dilated convolution pre-trained on the ImageNet dataset.
[0067] Step 104, input the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image.
[0068] Since the feature map is actually three-dimensional, in order to improve the problem of lack of cross-dimensional interaction information between spatial positions and channels, the present invention designs a cross-dimensional self-attention module, which can explore the cross-dimensional interaction clues between height and channel, and width and channel.
[0069] Step 106, divide the original feature map in the channel dimension to obtain multiple initial local feature groups, segment each initial local feature group respectively to obtain multiple sub-feature maps corresponding to each initial local feature group, recombin and arrange the multiple sub-feature maps according to the relative channel order of the sub-feature maps in each initial local feature group to obtain multiple rearranged local feature groups, input the multiple rearranged local feature groups into the pre-constructed channel self-attention module respectively to obtain multiple local self-attention feature groups.
[0070] Here is an example to illustrate that the original feature map X∈R C×H×W It is divided into 4 initial local feature groups in the channel dimension, namely Four groups, each initial local feature group is segmented respectively to obtain multiple sub-feature maps corresponding to each initial local feature group, that is, the four groups A, B, C, and D are further subdivided into four subgroups {A1, A2, A3, A4}, {B1, B2, B3, B4}, {C1, C2, C3, C4}, and {D1, D2, D3, D4}, each subgroup includes 4 sub-feature maps, and the multiple sub-feature maps are recombined and arranged according to the relative channel order of the sub-feature maps in each initial local feature group. , multiple rearranged local feature groups are obtained, that is, for each initial local feature group, the sub-feature maps with the same relative position (such as {A1, B1, C1, D1}) are taken out and rearranged to generate new rearranged local feature groups {A1, B1, C1, D1}{A2, B2, C2, D2}, {A3, B3, C3, D3} and {A4, B4, C4, D4}, and then, these four rearranged local feature groups are respectively input into the pre-built channel self-attention module to obtain multiple local self-attention feature groups.
[0071] The original feature map is divided and rearranged before the first channel self-attention processing is performed. The separate processing after division greatly reduces the computational complexity, thereby improving the processing speed. Since the number of channels of the features input to the channel self-attention module is reduced after division, for example, from C to C / 4, only channel dependencies within a small range can be captured. Therefore, the present invention also performs rearrangement after division, so as to achieve the effect that each rearranged local feature group input pre-constructed channel self-attention module can capture channel dependencies approximately within the original channel range. In general, the result of dividing and rearranging and then performing channel self-attention processing is to capture global, long-distance channel dependencies while reducing computational complexity.
[0072] Step 108, recombining the sub-feature maps in the local self-attention feature group to obtain multiple restored local feature groups with the same relative channel order as the sub-feature maps in the initial local feature group, and inputting the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps.
[0073] Recombination means restoring the original channel arrangement, and then inputting the restored local feature groups into the pre-built channel self-attention modules respectively. The outputs corresponding to each restored local feature group are fused to obtain a sparse channel self-attention map, that is, to obtain local and close channel dependencies.
[0074] like Figure 2 As shown, a schematic diagram of the calculation flow of the sparse channel self-attention (SCAM) map is provided, wherein CAM is a channel self-attention module, the two Merges are respectively the recombining of the sub-feature maps in the local self-attention feature group after the first channel self-attention processing and the fusion of the output corresponding to each restored local feature group after the second channel self-attention processing to obtain the sparse channel self-attention map, the two Permutes respectively refer to the rearrangement of the sub-feature maps before the first channel self-attention processing and the reorganization of the sub-feature maps before the second channel self-attention processing to restore the original channel arrangement, the two Divides respectively represent that the inputs of the two channel self-attention processing are N (for example, 4) local feature groups, and each channel self-attention processing includes 4 parallel channel self-attention calculations.
[0075] Step 110, obtaining a segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map, and the cross-dimensional interactive self-attention map.
[0076] To achieve better image segmentation performance, we fuse the output features of the feature extraction backbone network, the cross-dimensional self-attention module, and SCAM to obtain the final feature representation for pixel-level prediction.
[0077] In the above-mentioned remote sensing image segmentation method, the original feature map of the remote sensing image is divided in the channel dimension to obtain multiple initial local feature groups, and then each initial local feature group is segmented to obtain a sub-feature map of each initial local feature group, and then the sub-feature maps are recombined and arranged according to the relative channel order of the sub-feature maps of each initial local feature group to obtain multiple rearranged local feature groups, and then the multiple rearranged local feature groups are respectively input into a pre-constructed channel self-attention module, and the original feature map is divided and rearranged to perform the first channel self-attention processing. The separate processing after division greatly reduces the computational complexity and thus improves the processing speed. Since the number of channels of the features input into the channel self-attention module after division is reduced, only the channel dependency within a small range can be captured. Therefore, the present invention also performs rearrangement after division, so as to achieve the effect that each rearranged local feature group input into the pre-constructed channel self-attention module can capture the channel dependency approximately within the original channel range. In general, The result of performing division and rearrangement followed by channel self-attention processing is to capture global, long-range channel dependencies while reducing computational complexity; next, the multiple local self-attention feature groups obtained after the first channel self-attention processing are recombined to obtain multiple restored local feature groups with the same relative channel order as the sub-feature maps in the initial local feature group, that is, to restore the original channel arrangement, and then the restored local feature groups are respectively input into the pre-constructed channel self-attention module to obtain sparse channel self-attention maps, that is, to obtain local, short-range channel dependencies; on the other hand, the present invention also considers the interactive information between the position dimension and the channel dimension, and inputs the original feature map into the pre-constructed cross-dimensional self-attention module to obtain the cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image, and finally obtains the segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map, that is, the feature representation ultimately used for pixel-level prediction. The present invention comprehensively considers the dependencies between channels and the interactive information between channels and positions, and at the same time captures the dependencies between channels to divide, rearrange and restore the order of the original feature map, thereby reducing the computational complexity, speeding up the processing speed, and generally obtaining more accurate image segmentation results more quickly.
[0078] In one embodiment, the position dimension includes height and width, and the original feature map is input into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image, including:
[0079] The original feature map is input into the pre-built cross-dimensional self-attention module to obtain the first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image: The original feature map is convolved with the pre-built cross-dimensional self-attention module to obtain the first query vector Q1∈R of the original feature map C×H×W, the first key vector K1∈R C×H×W And the first value vector V1∈R C ×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; for the first query vector Q1∈R C×H×W , the first key vector K1∈R C×H×W Take a value vector V1∈R C×H×W Adjust the dimension to get the second query vector Q2∈R HC×W , the second key vector K2∈R HC×W And the second value vector V2∈R HC×W ; According to the second query vector Q2∈R HC×W and the second bond vector K2∈R HC×W Get the cross-dimensional interaction metric score of height and channel, according to the cross-dimensional interaction metric score of height and channel, the second value vector V2∈R HC×W The first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image is obtained by combining the original feature map.
[0080] The original feature map is input into the pre-built cross-dimensional self-attention module to obtain the second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image: The original feature map is convolved with the pre-built cross-dimensional self-attention module to obtain the third query vector Q3∈R of the original feature map C×H×W , the third key vector K3∈R C×H×W And the third value vector V3∈R C ×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; for the third query vector Q3∈R C×H×W , the third key vector K3∈R C×H×W With the third value vector V3∈R C×H×W The fourth query vector Q4∈R is obtained by dimension adjustment WC×H , the fourth vector K4∈R WC×H And the fourth value vector V4∈R WC×H ; According to the fourth query vector Q4∈R WC×H and the fourth bond vector K4∈R WC×H Get the cross-dimensional interaction metric score of width and channel, according to the cross-dimensional interaction metric score of width and channel, the fourth value vector V4∈R WC×H The second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image is obtained by combining the original feature map.
[0081] Among them, the cross-dimensional self-attention module includes the cross-dimensionality between height and channel and the cross-dimensionality between width and channel. The parameters used for the two convolutions of the original feature map are different, and the obtained Q, K, V are also different, that is, Q1∈R C×H×W Different from Q3∈RC×H×W , K1∈R C×H×W Different from K3∈R C×H×W , V1∈R C×H×W Different from V3∈R C×H×W .
[0082] The first cross-dimensional interactive self-attention map and the second cross-dimensional interactive self-attention map are fused to obtain the final cross-dimensional interactive self-attention map.
[0083] To facilitate understanding, the specific implementation of cross-dimensional interaction is illustrated by example:
[0084] In order to improve the problem of lack of cross-dimensional interaction information between spatial position and channel, the present invention designs CDAM, namely cross-dimensional interaction self-attention module, which can model the cross-dimensional interaction clues between height and channel, width and cross-dimensional interaction. CDAM consists of two similar self-attention operations, such as Figure 3 shown.
[0085] We introduce CDAM by taking the cross-dimensional interaction of height and channel as an example. Given a feature map X∈R C×H×W , first convolve X to generate features Q1∈R C×H×W , K1∈R C×H×W , V1∈R C×H×W Then reshape Q1, K1 and V1 to R HC×W . Multiply the transpose of K1 by Q1 and use the softmax layer to calculate the attention map S1∈R HC×HC :
[0086]
[0087] In the formula, S1 measures the cross-dimensional interaction between height H and channel C. The closer the interaction between the two dimensions, the greater the correlation. Then V1 is multiplied by the transpose of S1, and the result is reshaped to R C×H×W Finally, the product is multiplied by the scale factor α and added to the original feature X to obtain the final feature map E1∈R C×H×W as follows
[0088]
[0089] The weight α is initialized to 0 and the weight is gradually learned.
[0090] The operation of cross-dimensional interaction between width and channels is similar. The difference is that Q2, K2 and V2 are reshaped to R WC×H , note that S2∈R WC×WC .
[0091] The final output feature map of CDAM is obtained by adding the outputs of these two parts.
[0092] In one embodiment, multiple rearranged local feature groups are respectively input into a pre-built channel self-attention module to obtain multiple local self-attention feature groups, including:
[0093] Multiple rearranged local feature groups M∈R c / n×H×W Input the pre-built channel self-attention module respectively, and perform multiple rearranged local feature groups M∈R c / n×H×W Perform dimension adjustment respectively to obtain the first dimension adjusted local feature group M∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups;
[0094] Adjust the local feature group M∈R according to the first dimension c / n×HW Get the first channel-to-channel interaction metric score, and according to the first channel-to-channel interaction metric score and the corresponding rearranged local feature group M∈R c / n×H×W Multiple local self-attention feature groups are obtained.
[0095] In one embodiment, the restored local feature groups are respectively input into a pre-built channel self-attention module to obtain a sparse channel self-attention map, including:
[0096] Restore the local feature group M′∈R c / n×H×W Input the pre-built channel self-attention module respectively, and restore multiple local feature groups M′∈R c / n×H×W Perform dimension adjustment respectively to obtain the second dimension adjusted local feature group M′∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups;
[0097] Adjust the local feature group M′∈R according to the second dimension c / n×HW Get the second channel-to-channel interaction metric score, and according to the second channel-to-channel interaction metric score and the corresponding restored local feature group M′∈R c / n×H×W Obtain multiple local self-attention feature groups;
[0098] Multiple local self-attention feature groups are fused to obtain a sparse channel self-attention map.
[0099] For ease of understanding, the specific implementation of sparse channel self-attention processing is illustrated by example:
[0100] In order to more effectively model the interdependence information between channels, this paper proposes sparse channel self-attention (SCAM) based on channel self-attention module (CAM). The structure of channel self-attention is shown in Figure 4 shown.
[0101] Input: local feature map X∈R C×H×W ,
[0102] Grouping and rearrangement: Divide feature X into 4 parts in the channel dimension and rearrange them into the shape of
[0103] Processing: Adjust the dimension by reshape
[0104] Get the attention feature map
[0105] Then we get the final feature map
[0106] The weight of β is initialized to 0, and the weight is gradually learned.
[0107] Feature Map Then concatenate them into Y∈R C×H×W , restore the channels of Y to their original combination, and then perform the subsequent second CAM operation.
[0108] Result: Feature map Z∈R C×H×W , that is, the sparse channel self-attention map.
[0109] The CAM module ( Figure 3 ) in the dimension R C×H×W is a general formal parameter. C, H, and W can vary according to the specific usage scenario. The dimensions of the usage scenario in the present invention are C / 4, H, and W, respectively. This is because the original feature map X∈R C×H×W The inputs of the two CAM operations are both local feature groups obtained after division, so the number of channels of each local feature group becomes the original C / n, and n can be designed and selected according to the specific situation.
[0110] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0111] Here, we first introduce the dataset and implementation details, then conduct ablation studies to verify the effectiveness of our framework, and compare our method with several other state-of-the-art methods on the ISPRS Vaihingen and Potsdam datasets.
[0112] (I) Dataset and implementation details:
[0113] 1. Datasets and evaluation indicators: ISPRS Vaihingen is a high-resolution remote sensing dataset containing 33 images with an average size of 2496×2046 pixels and a ground sampling distance (GSD) of 9 cm; ISPRS Potsdam consists of 38 high-resolution remote sensing images, each with a size of 6000×6000 pixels and a GSD of 5 cm. For ISPRS Vaihingen and Potsdam, there are six object categories: impervious surface, building, low vegetation, tree, car, and clutter / background. All images are cropped into image patches of 512×512 pixels, which are randomly divided into training, validation, and test sets in a ratio of 6:2:2.
[0114] In order to evaluate the performance of image segmentation, the present invention adopts three evaluation indicators, including mean intersection over union (mIoU), F1-score (F1) and overall pixel accuracy (OA).
[0115] 2. Implementation details: All experiments are implemented using PyTorch on an NVIDIA GeForce RTX 1080Ti GPU, with the optimizer set to standard stochastic gradient descent (SGD) and a learning rate of 0.0009. All methods use cross entropy loss as the loss function.
[0116] (II) Ablation experiment
[0117] In this section, an extensive ablation study is conducted on the ISPRS Vaihingen dataset to verify the effectiveness of each component of the proposed method. The experimental results are shown in Table 1.
[0118] Table 1: Comparison of different variants of the network on the ISPRS Vaihingen dataset
[0119]
[0120] 1. Backbone
[0121] A pre-trained ResNet with dilated convolution is used as the backbone network. The output of the last stage of ResNet is used for image segmentation. On the ISPRS Vaihingen dataset, the mIOU based on ResNet50 is 70.48%, and the mIOU based on ResNet101 is 71.24%.
[0122] 2. Cross-dimensional Self-Attention Module CDAM
[0123] Compared with the baseline, we introduce CDAM to capture cross-dimensional interaction information. For backbone ResNet50, CDAM improves mIOU by 2.42% to 72.9%. Meanwhile, for backbone ResNet101, CDAM brings 2.23% mIOU improvement and mIOU gets 73.47%. Experimental results demonstrate the effectiveness of the CDAM method, which models the interaction between spatial and channel dimensions.
[0124] 3. Sparse Channel Self-Attention Module SCAM
[0125] In order to effectively capture the interdependencies between channels, SCAM is introduced to enhance the output of the backbone network. Compared with the baseline ResNet50 and ResNet101, SCAM's mIOU is improved by 2.58% and 2.25%, respectively, and the mIOU reaches 73.06% and 73.49%, respectively. Obviously, SCAM is effective in modeling the information between channels and achieves better segmentation performance.
[0126] 4. Complete network structure
[0127] Integrate CDAM and SCAM into the baseline to generate a complete network architecture, such as Figure 5 As shown. Compared with the above methods, the network proposed in the present invention uses CDAM to model the cross-dimensional information between space and channels, and effectively captures the interdependence between channels. Compared with the baseline (ResNet50), our method improves mIOU by 3.32%, mIOU is 73.80%, F1 is 84.58%, and OA is 87.95%. In addition, compared with the baseline (ResNet101), the network has an mIOU of 74.2%, F1 of 84.86%, and OA of 88.19%, which is the best performance. A large number of experiments show that this method can effectively improve the semantic segmentation performance of images.
[0128] 5. Feature visualization analysis
[0129] In order to compare the segmentation performance of several network combinations, the visualization results of the feature map are shown. Each image corresponds to 6 channels, and each channel represents a semantic category. Figure 6 As shown in Figure 2, the channel features of some images are visualized. Note that the clutter / background category is not shown. Compared with the baseline, the network with CDAM, SCAM and the full architecture achieves better feature representation for each semantic category. Specifically, for Figure 5 For the impervious surface in the feature map, the confidence difference between the pixels representing the impervious surface class and the pixels representing other classes in the feature map is more significant. For example, the confidence of the pixels representing cars is lower, which is conducive to improving the classification accuracy. In addition, for Figure 6 For the building category in , the proposed network has clearer edge features than other methods. Meanwhile, for all categories, our CSANet obtains more competitive feature representations.
[0130] Table 2 Comparative experimental results on the ISPRS Vaihingen dataset
[0131]
[0132] Table 3 Comparative experimental results on the ISPRS Potsdam dataset
[0133]
[0134] Comparative experiment
[0135] To demonstrate the advantages of the proposed network, we compare it with several existing semantic segmentation methods, including DANet and CCNet based on self-attention, as well as other general semantic segmentation networks such as UNet, FCN, Deeplabv3+, PSPNet and HRNet. The experimental results on the ISPRS Vaihingen and Potsdam datasets are shown in Tables 2 and 3, respectively.
[0136] Compared with the typical segmentation network FCN, the mIOU / F1 / OA of the proposed network is improved by 1.81% / 1.24% / 0.9%, reaching 74.20% / 84.86% / 88.19% on the ISPRS Vaihingen dataset. In addition, the mIOU / F1 / OA of the proposed CSANet is 1.52% / 1.09% / 0.47% higher than that of the self-attention-based DANet. In summary, the proposed method achieves state-of-the-art semantic segmentation performance on the ISPRS Vaihingen and Potsdam datasets.
[0137] The visualization results are as follows Figure 7As shown in Figure 2, it can be observed that the overall visual effect of the proposed method is better than that of other methods. Specifically, our network can obtain accurate segmentation results for small objects (such as Figure 7 In addition, for larger objects, our method helps with intra-class consistency. Figure 7 In the second row, for large buildings, the proposed method maintains the consistency of the categories, while other methods incorrectly predict that some pixels representing buildings are impermeable surfaces. This shows that the proposed method can achieve better image semantic segmentation performance and visual effects.
[0138] In one embodiment, Figure 8 As shown, a remote sensing image segmentation device is provided, including: a processing module, a cross-dimensional interaction module, a first channel self-attention processing module, a second channel self-attention processing module, and a segmentation result calculation module, wherein:
[0139] An acquisition module, used for acquiring an original feature map of a remote sensing image to be segmented;
[0140] A cross-dimensional interaction module is used to input the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image;
[0141] A first channel self-attention processing module is used to divide the original feature map in the channel dimension to obtain multiple initial local feature groups, respectively segment each initial local feature group to obtain multiple sub-feature maps corresponding to each initial local feature group, recombin and arrange the multiple sub-feature maps according to the relative channel order of the sub-feature maps in each initial local feature group to obtain multiple rearranged local feature groups, and respectively input the multiple rearranged local feature groups into a pre-constructed channel self-attention module to obtain multiple local self-attention feature groups;
[0142] The second channel self-attention processing module is used to recombine the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups with the same relative channel order as the sub-feature maps in the initial local feature group, and input the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps;
[0143] The segmentation result calculation module is used to obtain the segmentation result of the remote sensing image based on the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
[0144] In one embodiment, the cross-dimensional interaction module is further used to input the original feature map into a pre-built cross-dimensional self-attention module to obtain a first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image: the original feature map is convolved using the pre-built cross-dimensional self-attention module to obtain a first query vector Q1∈R of the original feature mapC×H×W , the first key vector K1∈R C×H×W And the first value vector V1∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; for the first query vector Q1∈R C×H×W , the first key vector K1∈R C×H×W Take a value vector V1∈R C×H×W Adjust the dimension to get the second query vector Q2∈R HC×W , the second key vector K2∈R HC×W And the second value vector V2∈R HC×W ; According to the second query vector Q2∈R HC×W and the second bond vector K2∈R HC×W Get the cross-dimensional interaction metric score of height and channel, according to the cross-dimensional interaction metric score of height and channel, the second value vector V2∈R HC×W The original feature map is convolved with the pre-built cross-dimensional self-attention module to obtain the first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image; the original feature map is input into the pre-built cross-dimensional self-attention module to obtain the second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image: the original feature map is input into the pre-built cross-dimensional self-attention module to obtain the second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image: the pre-built cross-dimensional self-attention module is used to convolve the original feature map to obtain the third query vector Q3∈R of the original feature map C×H×W , the third key vector K3∈R C×H×W And the third value vector V3∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; for the third query vector Q3∈R C×H×W , the third key vector K3∈R C×H×W With the third value vector V3∈R C×H×W The fourth query vector Q4∈R is obtained by dimension adjustment WC×H , the fourth vector K4∈R WC×H And the fourth value vector V4∈R WC×H ; According to the fourth query vector Q4∈R WC×H and the fourth bond vector K4∈R WC×H Get the cross-dimensional interaction metric score of width and channel, according to the cross-dimensional interaction metric score of width and channel, the fourth value vector V4∈R WC×H The second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image is obtained by combining the original feature map; the final cross-dimensional interactive self-attention map is obtained by fusing the first cross-dimensional interactive self-attention map and the second cross-dimensional interactive self-attention map.
[0145] In one embodiment, the first channel self-attention processing module is further used to reorder the multiple local feature groups M∈R c / n×H×WInput the pre-built channel self-attention module respectively, and perform multiple rearranged local feature groups M∈R c / n×H×W Perform dimension adjustment respectively to obtain the first dimension adjusted local feature group M∈R c / n×HW , adjust the local feature group M∈R according to the first dimension c / n×HW Get the first channel-to-channel interaction metric score, and according to the first channel-to-channel interaction metric score and the corresponding rearranged local feature group M∈R c / n×H×W Multiple local self-attention feature groups are obtained, where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups.
[0146] In one embodiment, the second channel self-attention processing module is further used to restore the local feature group M′∈R c / n×H×W Input the pre-built channel self-attention module respectively, and restore multiple local feature groups M′∈R c / n×H×W Perform dimension adjustment respectively to obtain the second dimension adjusted local feature group M′∈R c / n×HW , adjust the local feature group M′∈R according to the second dimension c / n×HW Get the second channel-to-channel interaction metric score, and according to the second channel-to-channel interaction metric score and the corresponding restored local feature group M′∈R c / n×H×W Multiple local self-attention feature groups are obtained, and multiple local self-attention feature groups are fused to obtain a sparse channel self-attention map, where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups.
[0147] The specific definition of the remote sensing image segmentation device can be found in the definition of the remote sensing image segmentation method above, which will not be repeated here. Each module in the above remote sensing image segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0148] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a remote sensing image segmentation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0149] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0150] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synch link) DRAM (SLDRAM), memory bus (Ram bus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0154] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A remote sensing image segmentation method, characterized in that: The method comprises: Obtaining the original feature map of the remote sensing image to be segmented; Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and channel dimension of the remote sensing image; Divide the original feature map in the channel dimension to obtain a plurality of initial local feature groups, segment each of the initial local feature groups to obtain a plurality of sub-feature maps corresponding to each of the initial local feature groups, recombine and arrange the plurality of sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain a plurality of rearranged local feature groups, and input the plurality of rearranged local feature groups into a pre-constructed channel self-attention module to obtain a plurality of local self-attention feature groups; Recombining the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups having the same relative channel order as the sub-feature maps in the initial local feature group, and inputting the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps; Obtaining a segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map, and the cross-dimensional interactive self-attention map; The multiple rearranged local feature groups are respectively input into the pre-built channel self-attention module to obtain multiple local self-attention feature groups, including: The multiple rearranged local feature groups M∈R c / n×H×W Input the pre-built channel self-attention module respectively, and perform the multiple rearranged local feature groups M∈R c / n×H×W Perform dimension adjustment respectively to obtain the first dimension adjusted local feature group M∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups; Adjust the local feature group M∈R according to the first dimension c / n×HW Get the first channel-to-channel interaction metric score, and according to the first channel-to-channel interaction metric score and the corresponding rearranged local feature group M∈R c / n×H×W Obtain multiple local self-attention feature groups; The restored local feature groups are respectively input into the pre-built channel self-attention modules to obtain a sparse channel self-attention map, including: The restored local feature group M′∈R c / n×H×W Input the pre-built channel self-attention module respectively, and restore the multiple local feature groups M′∈R c / n×H×W Perform dimension adjustment respectively to obtain the second dimension adjusted local feature group M′∈R c / n×HW ; Where C is the number of channels of the original feature map, H is the height of the original feature map, W is the width of the original feature map, and n is the number of rearranged local feature groups; Adjust the local feature group M′∈R according to the second dimension c / n×HW Get the second channel-to-channel interaction metric score, and according to the second channel-to-channel interaction metric score and the corresponding restored local feature group M′∈R c / n×H×W Obtain multiple local self-attention feature groups; The multiple local self-attention feature groups are fused to obtain a sparse channel self-attention map.
2. The method according to claim 1, characterized in that The position dimensions include height and width; The step of inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image includes: Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image; Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image; The first cross-dimensional interactive self-attention map and the second cross-dimensional interactive self-attention map are fused to obtain a final cross-dimensional interactive self-attention map.
3. The method according to claim 2, characterized in that The original feature map is input into a pre-built cross-dimensional self-attention module to obtain a first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image, including: The pre-built cross-dimensional self-attention module is used to convolve the original feature map to obtain the first query vector Q1∈R of the original feature map C×H×W , the first key vector K1∈R C×H×W And the first value vector V1∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; For the first query vector Q1∈R C×H×W 、The first key vector K1∈R C×H×W Take the value vector V1∈R C×H×W Adjust the dimension to get the second query vector Q2∈R HC×W , the second key vector K2∈R HC×W And the second value vector V2∈R HC×W ; According to the second query vector Q2∈R HC×W and the second bond vector K2∈R HC×W Get the cross-dimensional interaction metric score of the height and channel, and according to the cross-dimensional interaction metric score of the height and channel, the second value vector V2∈R HC×W The first cross-dimensional interactive self-attention map of the height and channel of the remote sensing image is obtained by combining the original feature map.
4. The method according to claim 2, characterized in that: Inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image, including: The pre-built cross-dimensional self-attention module is used to convolve the original feature map to obtain a third query vector Q3∈R of the original feature map. C×H×W , the third key vector K3∈R C×H×W And the third value vector V3∈R C×H×W ; Where C is the number of channels, H is the height of the feature map, and W is the width of the feature map; For the third query vector Q3∈R C×H×W , the third key vector K3∈R C×H×W With the third value vector V3∈R C ×H×W The fourth query vector Q4∈R is obtained by dimension adjustment WC×H , the fourth key vector K4∈R WC×H And the fourth value vector V4∈R WC×H ; According to the fourth query vector Q4∈R WC×H and the fourth key vector K4∈R WC×H Get the cross-dimensional interaction metric score of width and channel, and according to the cross-dimensional interaction metric score of width and channel, the fourth value vector V4∈R WC×H The second cross-dimensional interactive self-attention map of the width and channel of the remote sensing image is obtained by combining the original feature map.
5. The method according to claim 1, characterized in that The step of obtaining the original feature map of the remote sensing image to be segmented comprises: Acquire a remote sensing image to be segmented; The remote sensing image is input into a pre-trained ResNet to obtain an original feature map of the remote sensing image to be segmented.
6. A remote sensing image segmentation device, characterized in that: Used to implement the remote sensing image segmentation method according to any one of claims 1 to 5, the device comprises: An acquisition module, used for acquiring an original feature map of a remote sensing image to be segmented; A cross-dimensional interaction module, used for inputting the original feature map into a pre-built cross-dimensional self-attention module to obtain a cross-dimensional interactive self-attention map of the position dimension and the channel dimension of the remote sensing image; A first channel self-attention processing module is used to divide the original feature map in the channel dimension to obtain multiple initial local feature groups, respectively segment each of the initial local feature groups to obtain multiple sub-feature maps corresponding to each of the initial local feature groups, recombin and arrange the multiple sub-feature maps according to the relative channel order of the sub-feature maps in each of the initial local feature groups to obtain multiple rearranged local feature groups, and respectively input the multiple rearranged local feature groups into a pre-constructed channel self-attention module to obtain multiple local self-attention feature groups; A second channel self-attention processing module is used to recombine the sub-feature maps in the local self-attention feature group to obtain a plurality of restored local feature groups with the same relative channel order as the sub-feature maps in the initial local feature group, and input the restored local feature groups into the pre-constructed channel self-attention modules respectively to obtain sparse channel self-attention maps; A segmentation result calculation module is used to obtain the segmentation result of the remote sensing image according to the original feature map, the sparse channel self-attention map and the cross-dimensional interactive self-attention map.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
