A High-Resolution Remote Sensing Extraction Method for Damaged Buildings Based on Weakly Supervised Semantic Segmentation

Abstract

This invention discloses a high-resolution remote sensing extraction method for damaged buildings based on weakly supervised semantic segmentation. To improve the quality of the Class Activation Map (CAM) and suppress noise within damaged buildings, this invention employs a cascaded dense connection method using dilated convolution to enhance the activation response of relevant regions and suppress the activation response of irrelevant regions. It also optimizes the global spatial correlation between pixels by comprehensively considering and integrating the feature dependencies of each pixel over a large scale. This solves problems such as poor CAM quality, severe noise within damaged buildings, and uneven stitching, thereby enhancing the network's ability to perceive details, alleviating blurred target boundaries, and achieving smooth stitching. This invention is suitable for the field of remote sensing image processing.

Description

Technical Field

[0001] This invention relates to a high-resolution remote sensing extraction method for damaged buildings based on weakly supervised semantic segmentation, specifically a method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation. Background Technology

[0002] Following disasters such as earthquakes, rapid and accurate assessment of damaged buildings is crucial for initially estimating the extent of the damage and providing information for rescue efforts. Unlike on-site surveys, high-resolution remote sensing can observe disaster areas with minimal contact with harsh and unsafe environments, enabling fine-grained identification of damaged buildings, even in remote areas.

[0003] Against this backdrop, researchers are focusing more on developing methods for automatically detecting damaged buildings from high-resolution post-disaster remote sensing imagery. Post-disaster damaged buildings in high-resolution remote sensing imagery exhibit characteristics such as chaotic appearance, irregular structure, blurred boundaries, and varying sizes. Debris falling from buildings onto surrounding roads during collapse can easily cause them to be confused with the surrounding landscape. Furthermore, completely collapsed buildings can easily be mistaken for bare ground or other objects. These factors present significant challenges to the automatic extraction of damaged buildings from remote sensing imagery.

[0004] The standard approach to overcome this challenge is to train a semantic segmentation network with finely annotated pixel-level samples. However, annotating finely annotated pixel-level samples is very difficult, especially for damaged buildings with the above characteristics, which can easily lead to bias in the annotation of human pixel-level samples.

[0005] Currently, due to the powerful feature extraction and representation capabilities of Convolutional Neural Networks (CNNs), CNN-based methods are showing increasing potential in the automatic detection of damaged buildings. To obtain semantic segmentation results, i.e., pixel-by-pixel recognition results, many researchers have directly introduced Fully Convolutional Neural Networks (FCNs) to extract damaged buildings. However, FCN-based methods require collecting a large amount of pixel-level annotations to train the model, which is both time-consuming and costly, especially for the annotation of damaged buildings.

[0006] Weakly supervised semantic segmentation (WSSS) has recently gained significant attention due to its ability to achieve pixel-level interpretation results using scene-level labeled samples. Generally, weakly supervised methods are implemented using two main strategies: sliding window classification and class activation maps (CAM). However, sliding window classification methods often struggle to infer precise target locations, frequently resulting in very coarse target segmentation. For this reason, most state-of-the-art weakly supervised semantic segmentation methods rely on CAMs generated by pre-trained CNN classification networks. Therefore, the quality of the CAM is crucial to the performance of weakly supervised semantic segmentation methods.

[0007] Due to the high local response characteristics of CAMs (Camera-like Objects), CAMs generated by image classification networks are often incompletely and incorrectly activated in the target region. This problem poses a significant challenge to weakly supervised semantic segmentation methods based on image-level annotation. To address this issue, a multi-dilated convolutional augmentation classification network has been proposed to improve the discrimination ability of low-response regions of the target in CAMs. However, this technique only considers optimizing the activation range of the target region and does not address the consistency and noise issues in target region extraction. Another approach proposes a self-supervised isovariant attention mechanism network, which employs consistency regularization and pixel correlation modules to generate more consistent and complete CAMs from natural images. While these techniques demonstrate the feasibility of using weakly supervised semantic segmentation methods to extract damaged buildings from high-resolution remote sensing imagery, they do not specifically consider the distribution characteristics of damaged buildings in post-disaster remote sensing imagery. Clearly, the blurred boundaries and messy textures of damaged buildings can negatively impact the acquisition of high-quality CAMs.

[0008] Therefore, a solution is needed that takes into account the special characteristics of damaged buildings, improves the quality of the Class Activation Graph (CAM), suppresses noise inside damaged buildings, enhances the network's ability to perceive details, and alleviates the problem of blurred target boundaries. Summary of the Invention

[0009] To solve or partially solve some or all of the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0010] A high-resolution remote sensing extraction method for damaged buildings based on weakly supervised semantic segmentation is disclosed. A first network receives input remote sensing images with a resolution of H×W, where H and W are both positive integers. The remote sensing images are input into a ResNet-1o1 network to obtain feature maps learned by the ResNet-1o1 network and the output of the second layer of the ResNet-1o1 network. The feature maps learned by the ResNet-1o1 network are input into a multi-scale dependency (MSD) module and a spatial correlation optimization (SCR) module, and then, after convolution and upsampling, a first processing result is obtained. The ResNet-1o1 network... The second layer output is convolved to obtain the second processing result; the first and second processing results are concatenated to obtain the fused feature map; then, it is sequentially processed through the Spatial Relationship Optimization (SCR) module and convolution to obtain the third processing result; by applying the Grad-CAM strategy to the third processing result, a CAM of the damaged building with H / 4×W / 4 resolution is obtained, and then after size deformation, a CAM of the damaged building with H×W resolution is obtained; finally, the denseCRFs strategy is applied to the CAM of the damaged building with H×W resolution to obtain the extraction result of the damaged building based on the CAM generated from the original image.

[0011] In one embodiment, the third processing result is sequentially passed through a GAP layer, a fully connected layer, and a softmax classifier to calculate the probability that the input image patch belongs to each category.

[0012] In one type of embodiment, a cascaded dense connection method using dilated convolution is employed to improve the activation response of target-related regions and suppress the activation response of target-irrelevant regions.

[0013] In one type of embodiment, the global spatial correlation between pixels is optimized by comprehensively considering and integrating the feature interdependencies of each pixel over a large scope.

[0014] In one embodiment, the CAM generated by the image blocks based on the overlapping segmentation of the original image is first stitched together to obtain the CAM corresponding to the original image, and then the damaged building extraction result of the original image is obtained by denseCRFs.

[0015] In one type of embodiment, the multi-scale dependency (MSD) module is specifically implemented as follows:

[0016] The input feature map F of the Multi-Scale Dependency (MSD) module is passed through a 1×1 convolutional layer to obtain feature map F′; feature map F′ is passed through a dilated convolution with a dilation rate of 3 to obtain feature map F1; feature map F′ and feature map F1 are concatenated to obtain feature map F′1; feature map F′1 is passed through a dilated convolution with a dilation rate of 6 to obtain feature map F2; feature map F′ and feature map F′1 are concatenated to obtain feature map F″1; feature map F″1 and feature map F2 are concatenated to obtain feature map F′2; feature map F′2 is passed through a dilated convolution with a dilation rate of 9 to obtain feature map F3. Feature map F′ and feature map F″1 are concatenated to obtain feature map F″′1, feature map F″′1 and feature map F′2 are concatenated to obtain feature map F″2, feature map F″2 and feature map F3 are concatenated to obtain feature map F′3; feature map F′3 is dilated by a dilation convolution with a dilation rate of 12 to obtain feature map F4; feature map F′ is obtained by global average pooling and upsampling operations from feature map F′; the output feature map F″ of the multi-scale dependency (MAD) module is obtained by concatenating feature maps F′, F1, F2, F3, F4 and F5.

[0017] In one embodiment, the Spatial Relationship Optimization (SCR) module is specifically implemented as follows: a feature map F of size C×H×W is used as the input to the SCR module, where C, H, and W represent the number of channels, height, and width of the feature map, respectively; feature map F i F j F k The feature map F is obtained by convolution of size 1×1; for feature map F i F k By applying matrix transformations and transpose operations, a new feature map F′ is generated. i and F′ k For feature map F j Apply matrix transformations to generate a new feature map F′ j ; By analyzing the feature map F′ i and F′ j Matrix multiplication and softmax operation yield a spatial relation graph S of size HW×HW; the spatial relation graph S and feature map F′ are then compared. k After matrix multiplication, transpose and matrix transformation, a 1×1 convolution operation is applied, and the result of the convolution operation is added to the feature map F to obtain the spatial cross-relation optimized feature map S′, which is the output of the spatial cross-relation optimization (SCR) module.

[0018] The various embodiments disclosed in this invention have one or more of the following beneficial technical effects, and more beneficial effects will be described in specific embodiments:

[0019] 1) A novel MSD module based on dilated convolution and dense connectivity is introduced to expand the target-related activation region and suppress the target-independent activation region. This facilitates complete and dense localization of damaged buildings in CAM.

[0020] 2) A novel SCR module is designed to further optimize the consistency relationship within the target, which is beneficial for suppressing noise inside damaged buildings in CAM.

[0021] 3) The designed unified MSD-SCR-Net network with encoder-decoder structure can fuse low-level feature maps to enhance the network's ability to perceive details and alleviate the problem of blurred target boundaries.

[0022] The above technical solutions / features are intended to summarize the technical solutions and features described in the Detailed Implementation section, and therefore the scope of the description may not be entirely the same. However, these new technical solutions and features disclosed in this section, together with the technical features disclosed in the subsequent Detailed Implementation section, disclose more technical solutions in a reasonable combination.

[0023] The technical solution formed by combining all the technical features disclosed at any position in this invention is used for summarizing the technical solution, modifying the patent document, and disclosing the technical solution. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the principle of a method for extracting damaged buildings under weak supervision;

[0025] Figure 2 This is a schematic diagram of a Multiscale Dependency (MSD) module;

[0026] Figure 3 This is a schematic diagram of the Spatial Relationship Optimization (SCR) module. Detailed Implementation

[0027] The method steps or modules described in the embodiments disclosed in this invention can be implemented in hardware, software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application or design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of protection claimed by this invention.

[0028] The serial numbers such as "first" and "second" in any position in this invention are merely descriptive distinguishing marks and do not imply an absolute order in time or space, nor do they imply that the terms prefixed with such serial numbers necessarily refer to different terms than the same terms prefixed with other modifiers.

[0029] Since it is impossible to exhaustively describe all alternative solutions, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Other technical solutions and details not disclosed in detail below generally belong to technical objectives or features that can be achieved by conventional means in the art, and due to space limitations, they will not be described in detail here.

[0030] This invention describes various key points used to combine into various specific embodiments, which will be incorporated into various methods and products. In this invention, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes that technical feature.

[0031] The description of the existence or inclusion of a step, module, or feature at any location in this invention does not imply that such existence is exclusive or unique. Those skilled in the art can obtain other embodiments by supplementing the technical solutions disclosed in this invention with other technical means. Based on the key points described in the specific embodiments of this invention, those skilled in the art can substitute, delete, add, combine, or change the order of certain technical features to obtain a technical solution that still follows the concept of this invention. These solutions that do not depart from the technical concept of this invention are also within the protection scope of this invention.

[0032] To overcome the challenge of pixel-level sample annotation in interpreting post-disaster damaged buildings, this invention designs a weakly supervised classification network that couples a multi-scale feature dependency mechanism with a global spatial relationship optimization mechanism to learn the features of damaged buildings in scene images, and interprets post-disaster damaged buildings at the pixel level.

[0033] This invention discloses a novel weakly supervised semantic segmentation network based on scene-level annotation for extracting damaged buildings from high-resolution post-earthquake remote sensing images. The proposed network, called MSD-SCR-Net, integrates a multi-scale dependency (MSD) module and a spatial correlation optimization (SCR) module into a unified framework, achieving high-precision extraction of damaged buildings by improving the quality of the generated CAM. Specifically, it is implemented as follows:

[0034] refer to Figure 1The network proposed in this invention uses ResNet-101 as its backbone. The input image is fed into the ResNet-101 network to obtain feature maps learned by the backbone, which are then sequentially fed into the MSD module and SCR module proposed in this invention. Furthermore, to enhance the network's ability to perceive details, this invention specifically adds a decoding process to fuse low-level feature maps. The output feature map from the SCR module is convolved and upsampled before being concatenated with the low-level feature maps. To maintain feature consistency, the fused feature map is processed again by the SCR module. The feature map then passes through a final convolutional layer, and the probability of the input image patch belonging to each category is calculated sequentially using a GAP layer, a fully connected layer, and a softmax classifier.

[0035] For CAM generation from the original image, it's important to note that due to GPU memory limitations, the original large-scale, high-resolution remote sensing image cannot be directly input into the network to generate CAMs. Therefore, this invention requires segmenting the original image into image blocks with a certain degree of overlap as input. The Grad-CAM strategy is then used to generate the CAM corresponding to each image block based on the last convolutional layer of MSD-SCR-Net. Finally, denseCRFs are used to obtain the extracted damaged buildings based on the generated CAMs from the original image.

[0036] refer to Figure 2 This invention discloses the workflow of the MSD module. Traditional CAM struggles to capture complete and dense target regions, especially for damaged buildings with blurred boundaries and messy textures. To address this issue, the invention introduces the MSD module, composed of stacked dilated convolutions. By expanding the receptive field of the convolution kernel and incorporating contextual information, it transfers the target discrimination response region from high-relevance areas to low-relevance areas. The dilated convolution operation is the same as ordinary convolution, using kernel movement to perform convolution calculations for corresponding regions. However, dilated convolution allows for increasing the kernel size and expanding the receptive field while keeping parameters constant, thereby capturing larger-scale features. Therefore, by stacking dilated convolutions with different dilation rates, features of different scales in the image can be captured.

[0037] like Figure 2As shown, the MSD module stacks dilated convolutions and global average pooling (GAP) layers with different dilation rates, where the dilation rates are set to 3, 6, 9, and 12, respectively. By using dilated convolutions with different dilation rates to sample feature maps in parallel, image feature information at multiple scales can be captured, achieving multi-scale feature representation. Furthermore, this invention finds that cascaded dense connections facilitate the tight connection of multi-scale feature maps obtained through multiple paths, enabling better adaptive fusion between features of different scales and maximizing the advantages of multi-scale features. Therefore, to more effectively utilize multi-scale features and prevent erroneous activation responses in irrelevant regions around the target, this invention employs a cascaded dense connection method of dilated convolutions to suppress activation responses in irrelevant regions of the target.

[0038] Specifically, the workflow of the MSD module is as follows: Assuming a given intermediate feature map... As input to the MSD module, C, H, and W represent the number of channels, height, and width of the feature map, respectively. First, the feature map F is passed through a 1×1 convolutional layer to obtain the final feature map. F′ is used to obtain the feature map through dilation convolution with a dilation rate of 3. Feature map F′ is obtained by concatenating feature map F′ and feature map F1. Feature map F′1 is then subjected to dilation convolution with a dilation rate of 6 to obtain feature map F′1. Feature map F′ and feature map F′1 are concatenated to obtain feature map F″1, and feature map F″1 and feature map F2 are concatenated to obtain feature map F″2. Feature map F″2 is then subjected to dilation convolution with a dilation rate of 9 to obtain feature map F″2. Feature map F′ and feature map F″1 are concatenated to obtain feature map F″′1, feature map F″′1 and feature map F′2 are concatenated to obtain feature map F″2, and feature map F″2 and feature map F3 are concatenated to obtain feature map F′3. Feature map F′3 is then subjected to dilation convolution with a dilation rate of 12 to obtain feature map F′3. It is obtained from the feature map F′ through global average pooling (GAP) and upsampling operations. Finally, the output feature map of this module is... It is obtained by cascading feature maps F′, F1, F2, F3, F4 and F5.

[0039] refer to Figure 3The invention discloses the workflow of the SCR module. The internal structures of most damaged buildings are irregular, exhibiting high intraclass heterogeneity in appearance within high-resolution remote sensing imagery. This characteristic significantly hinders the consistent identification of damaged buildings. To alleviate this problem, the invention designs an SCR module that further optimizes the consistency relationships within the target activation response region to suppress noise within damaged buildings in CAM. The SCR module aims to optimize the global spatial correlation between pixels by comprehensively considering and integrating the feature interdependencies of each pixel over a large scale.

[0040] The structure of the SCR module is as follows: Figure 3 As shown. Specifically, the SCR module's workflow is as follows: Assuming a given intermediate feature map... As input to the SCR module, C, H, and W represent the number of channels, height, and width of the feature map, respectively. Feature map It is obtained by convolving F with a 1×1 matrix, and then generating a new feature map through matrix transformation and transpose operations. and

[0041] Then, spatial relationship diagram It can be done through F′ i and F′ j Matrix multiplication and softmax operation are used to obtain the result. Normalized values ​​in the spatial relationship graph S can be used to measure the spatial relationships between pixels. Finally, the spatial relationship optimization feature map is used. Also the output of the SCR module, it can be obtained through the following calculation:

[0042] S′=F+Conv 1×1 [(S×F′ k ) Transpose&Reshape ].

[0043] For input image scene classification, after constructing the final feature map, the probability of the input image patch belonging to each category is calculated using a GAP layer, a fully connected layer, and a softmax classifier. Regarding the acquisition of the input image CAM, it's important to note that due to GPU memory limitations, the original large-scale, high-resolution remote sensing images cannot be directly input into the network to generate CAMs. Therefore, this invention requires segmenting the original image into image patches with a certain degree of overlap as input, generating a CAM for each patch separately. This invention uses the Grad-CAM strategy to generate the CAM for the corresponding input image patch based on the last convolutional layer in MSD-SCR-Net. Specifically, the CAM for any category c is denoted as H. c , can be represented as:

[0044]

[0045] in F represents the CAM corresponding to category c. k This represents the k-th channel of the feature map output from the last convolutional layer. The weight representing the k-th channel of the corresponding feature map can be calculated using the following formula:

[0046]

[0047] Here P c The probability that the input image block belongs to category c in scene classification is represented by (i,j), where (i,j) represents the pixel position.

[0048] Generally, for large-scale, high-resolution remote sensing images, this invention can process the CAMs obtained from each segmented image block using denseCRFs to obtain pixel-by-pixel classification results, and then perform an image stitching step to obtain the result of the entire image. However, this processing may result in uneven stitching positions in the image, with obvious seams. Therefore, this invention first stitches the CAMs generated from the overlapping segmented image blocks of the original image to obtain the CAMs corresponding to the original image, and then uses denseCRFs to obtain the damaged building extraction results from the original image. For CAM stitching, since each pixel may be covered by multiple CAMs, this invention adopts the following strategy: the CAM value of each pixel p is taken as the largest CAM value at that pixel position among all the CAMs covering it, i.e. CAM p This represents the final CAM value of pixel p. This represents the CAM value of pixel p in the nth covered CAM. The denseCRFs described are techniques from prior art 1, which are incorporated herein by reference in their entirety.

[0049] Prior Art 1: L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and ALYuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrousconvolution, and fully connected crfs," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.40, no.4, pp.834–848, 2017.

[0050] Although the invention has been described with reference to specific features and embodiments, various modifications, combinations, and substitutions can be made therein without departing from the invention. The scope of protection of this invention is not limited to the specific embodiments of processes, machines, manufactures, material compositions, apparatuses, methods, and steps described in the specification, and these methods and modules may also be implemented in one or more related, interdependent, cooperative, or upstream / downstream products or methods.

[0051] The technical features mentioned in the appended claims may have alternative technical features, or the order of certain technical processes or material organization may be rearranged. Those skilled in the art, upon learning of this invention, will readily conceive of these alternative means, or alter the order of the technical processes or material organization, and then employ substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, these modifications, alterations, and substitutions should all fall within the scope of protection of the claims based on the principle of equivalents.

Claims

1. A method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation, characterized in that: The first network receives the input remote sensing image, the resolution of which is H×W, where H and W are both positive integers; The remote sensing image is input into the ResNet-101 network to obtain the feature map learned by the ResNet-101 network, as well as the output of the second layer in the ResNet-101 network; The feature maps learned by the ResNet-101 network are input into the Multi-Scale Dependency (MSD) module and the Spatial Relationship Optimization (SCR) module, and then after convolution and upsampling, the first processing result is obtained. The output of the second layer in the ResNet-101 network is convolved to obtain the second processing result; After concatenating the first processing result and the second processing result, a fused feature map is obtained; Then, it passes through the Spatial Relationship Optimization (SCR) module and convolution in sequence to obtain the third processing result; By applying the Grad-CAM strategy to the third processing result, a CAM of the damaged building with H / 4×W / 4 resolution is obtained, and then a CAM of the damaged building with H×W resolution is obtained after dimensional deformation. Then, the denseCRFs strategy is applied to the CAM of the damaged building at H×W resolution to obtain the extraction result of the damaged building based on the CAM generated from the original image.

2. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1, characterized in that: The third processing result is then passed through a GAP layer, a fully connected layer, and a softmax classifier in sequence to calculate the probability that the input image patch belongs to each category.

3. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1, characterized in that: A cascaded dense connection method using dilated convolution is employed to improve the activation response in target-relevant regions and suppress the activation response in target-irrelevant regions.

4. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1, characterized in that: The global spatial correlation between pixels is optimized by comprehensively considering and integrating the feature interdependencies of each pixel over a large scale.

5. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1, characterized in that: First, the CAM generated from the image blocks based on the overlapping segmentation of the original image is stitched together to obtain the CAM corresponding to the original image. Then, the damaged buildings in the original image are extracted using denseCRFs.

6. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1 or 3, characterized in that: The multi-scale dependency (MSD) module is specifically implemented as follows: The input feature map F of the multi-scale dependency (MSD) module is passed through a 1×1 convolutional layer to obtain the feature map F′; Feature map F′ is obtained by dilation convolution with a dilation rate of 3; Feature map F′ is obtained by concatenating feature map F′ and feature map F1; feature map F′1 is obtained by dilating convolution with dilation rate of 6; Feature map F′ and feature map F′ i Feature map F″1 is obtained by concatenating features; feature map F″1 and feature map F2 are concatenated to obtain feature map F′2; feature map F′2 is then subjected to dilated convolution with a dilation rate of 9 to obtain feature map F3. Feature map F′ and feature map F″1 are concatenated to obtain feature map F″′1, feature map F″′1 and feature map F′2 are concatenated to obtain feature map F″2, feature map F″2 and feature map F3 are concatenated to obtain feature map F′3; feature map F′3 is then subjected to dilation convolution with a dilation rate of 12 to obtain feature map F4. Feature map F5 is obtained by performing global average pooling and upsampling operations on feature map F′; The output feature map F″ of the Multi-Scale Dependency (MAD) module is obtained by cascading feature maps F′, F1, F2, F3, F4 and F5.

7. The method for high-resolution remote sensing extraction of damaged buildings based on weakly supervised semantic segmentation according to claim 1 or 4, characterized in that: The Spatial Relationship Optimization (SCR) module is specifically implemented as follows: A feature map F with dimensions C×H×W is used as input to the Spatial Relationship Optimization (SCR) module, where C, H, and W represent the number of channels, height, and width of the feature map, respectively. Feature map F i F j F k The feature maps F are obtained by convolution of size 1×1. For feature map F i F k By applying matrix transformations and transpose operations, a new feature map F′ is generated. i and F′ k ; For feature map F j Apply matrix transformations to generate a new feature map F′ j ; By analyzing the feature map F′ i and F′ j Matrix multiplication and softmax operation yield a spatial relationship graph S with dimensions HW×HW; Spatial relationship diagram S and feature diagram F′ k After matrix multiplication, transpose and matrix transformation, a 1×1 convolution operation is applied, and the result of the convolution operation is added to the feature map F to obtain the spatial cross-relation optimized feature map S′, which is the output of the spatial cross-relation optimization (SCR) module.

Images