Remote sensing image spatio-temporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism

Abstract

The present application relates to a kind of remote sensing image spatio-temporal fusion method combined with multiscale and multidimensional mixed attention mechanism, belong to remote sensing image processing field.The method combines multiscale feature extraction and multidimensional mixed attention mechanism, and uses depth separable dilated convolution to carry out feature extraction, to maintain good receptive field, avoid detailed information loss.MMNet is injected into mechanism by homologous heterochronous and heterochronous simultaneous multiscale feature difference, and multidimensional mixed attention mechanism, optimizes information fusion, to improve the accuracy and reliability of spatio-temporal fusion.The method can effectively capture global information and local details in image, and make full use of relatively complete data in remote sensing image, provide a new solution for remote sensing image processing field.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing and relates to a remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanisms. Background Art

[0002] With the rapid advancement of remote sensing data processing technology, remote sensing data has been widely used in a variety of fields, including plateau land cover classification, forest health monitoring, analysis of the impact of urbanization on vegetation, and crop type identification. These applications typically require high-spatial-resolution, frequently updated remote sensing imagery to accurately capture dynamic changes in the Earth's surface. However, a single satellite cannot simultaneously provide remote sensing imagery with high spatial and temporal resolution. Consequently, over the past decade, a growing number of integrated satellite sensor data have been used to address this shortfall. Although the next generation of Landsat satellites is planned to reduce the revisit period to six days, current Landsat satellites are still limited to a 16-day revisit period. While their spatial resolution is high (30 meters), their temporal resolution is low. In contrast, MODIS imagery has a higher temporal resolution (daily) but a lower spatial resolution (250 meters / 500 meters). Image fusion techniques can combine this complementary spatiotemporal information to produce remote sensing imagery with high spatial and temporal resolution, further enhancing its effectiveness and value in practical applications.

[0003] Currently, mainstream remote sensing image fusion methods are typically based on learning techniques, aiming to model the complex nonlinear relationships between high spatial resolution images (HSLT) and high temporal resolution images (LSHT). For example, the SPSTFM and EBSCDL methods reconstruct high spatial resolution images from low-resolution images through sparse representation. Subsequently, convolutional neural networks (CNNs) have been applied to spatiotemporal image fusion (STF). STFDCNN utilizes CNNs to model the relationship between coarse- and fine-resolution images and uses a high-pass fusion model for prediction. However, these methods still have limitations in mapping complex nonlinear relationships between images. To overcome this issue, VDCNNSTF employs a deeper network structure that more effectively extracts deep features, thereby more accurately mapping these nonlinear relationships. The DCSTFN method further optimizes the fusion effect by fusing three parts of the image: an expanded high spatiotemporal resolution image, extracted high-frequency components, and fused features. EDCSTFN improves the accuracy and robustness of the model by reducing its reliance on a single reference image. The AMNet method directly incorporates MODIS residual imagery into network training and uses a multi-scale mechanism to extract spatiotemporal variations and spatial details at different scales, enhancing the detail information in feature maps. Furthermore, a hybrid attention mechanism is employed to enhance the saliency of feature information. Overall, these deep learning-based fusion methods have significantly improved the quality of remote sensing imagery and effectively addressed the challenge of fusion accuracy.

[0004] Although the STF of remote sensing imagery has garnered widespread attention and research, several challenges remain. First, atmospheric and radiometric corrections often produce significant errors in remote sensing images. Many methods fail to effectively account for these errors, leading to the direct input of images into the network. This leads to the accumulation of errors during training, particularly when dealing with regions with significant spatiotemporal variation. This accumulated error often compromises the final prediction accuracy. Second, in practical applications, cloud cover often results in the loss of local information in remote sensing images, while data in other areas is relatively complete. Utilizing the effective portions of these incomplete images remains an urgent challenge. Summary of the Invention

[0005] To address the errors caused by atmospheric and radiometric corrections, which can lead to blurred spatial details in predicted images, distorted temporal variations, and ineffectively utilize missing local information in remote sensing images, this paper proposes a remote sensing image spatiotemporal fusion method called MMNet. This network combines multi-scale feature extraction with a multi-dimensional hybrid attention mechanism. Specifically, MMNet employs a mechanism for injecting multi-scale features from the same source at different times and from different sources simultaneously, as well as a multi-dimensional hybrid attention mechanism, to optimize information fusion. To further improve network performance, MMNet replaces all conventional convolutions with depthwise separable dilated convolutions. This method aims to reduce the errors caused by atmospheric and radiometric corrections while capturing both global information and local details in the image, thereby improving the accuracy and reliability of the STF.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism includes the following steps:

[0008] Basic information extraction: extract high-frequency components from fine images;

[0009] Multi-scale feature difference information injection: extract the difference feature map of remote sensing images with the same source and different time; extract the difference feature map of remote sensing images with different sources and at the same time; stack the above two difference feature maps and add them to the feature extraction and basic information extraction of multi-scale homologous residual images;

[0010] Depthwise Separable Dilated Convolution: Use depthwise separable dilated convolution for feature extraction to maintain a good receptive field and avoid loss of detailed information;

[0011] Multi-dimensional fusion attention mechanism: Add the homologous residual feature map to the refined image frame map; input the added feature map into the multi-dimensional hybrid attention mechanism, including the coordinate attention mechanism and the spatial attention mechanism, to enhance the saliency of the feature information;

[0012] Image reconstruction: The final predicted image is obtained through upsampling and convolution operations.

[0013] Furthermore, in the multi-scale feature difference information injection, the step of extracting the difference feature map of the remote sensing images of the same source and different time includes:

[0014] Convolution is performed on the residual image obtained by subtracting the remote sensing images of the same source but different time;

[0015] Upsample the convolved feature map;

[0016] The upsampled feature map is input into the convolutional layer again to obtain the final difference feature map of the remote sensing images of the same source and different time.

[0017] Furthermore, in the multi-scale feature difference information injection, the step of extracting the heterogeneous simultaneous remote sensing image difference feature map includes:

[0018] Convolution is performed on the residual image obtained by subtracting the remote sensing images from different sources simultaneously;

[0019] Upsample the convolved feature map;

[0020] The upsampled feature map is input into the convolutional layer again to obtain the final heterogeneous simultaneous remote sensing image difference feature map.

[0021] Furthermore, in the depthwise separable dilated convolution, the depthwise separable dilated convolution includes:

[0022] Depth dilated convolution: Perform depth dilated convolution on the input feature map, and perform dilated convolution operation independently on each input channel of the feature map;

[0023] Point-by-point convolution: Use a 1×1 convolution kernel to convolve the output feature map after depth convolution.

[0024] Furthermore, in the image reconstruction, the upsampling operation includes:

[0025] Upsampling operation: Upsampling the feature map to increase the image resolution;

[0026] Transposed convolution operation: Perform convolution operation on the feature map to increase the image resolution.

[0027] Furthermore, in the image reconstruction, the convolution operation includes:

[0028] Depth-wise dilated convolution operation;

[0029] Point-wise convolution operation.

[0030] The beneficial effects of this invention are as follows: It aims to break through the limitations of existing fusion methods; MMNet uses the SSDT and DSST multi-scale feature difference information injection mechanism to add channels to the feature map to extract detailed information about the features, enhancing the model's feature learning and generalization capabilities. At the same time, it uses a multi-dimensional fusion attention mechanism to automatically learn and selectively focus on key input information, strengthening the representation of objects of interest. In addition, it uses depthwise separable dilated convolution to capture global information and local details, effectively preventing the loss of detailed information.

[0031] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0033] Figure 1 This is the overall structure of MMNet. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0036] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0037] The MMNet model consists of four core modules: basic information extraction, same-source different times (SSDT) ​​and different-source same times (DSST) multi-scale feature difference information injection mechanism, depthwise separable dilated convolution, and multi-dimensional hybrid attention mechanism. In this model, the basic information extraction module is responsible for extracting high-frequency details from high-resolution images. The multi-scale feature difference information injection mechanism extracts temporal and spatial difference information from SSDT and DSST remote sensing images of different scales, and integrates this information into the feature extraction and basic information extraction processes of the same-source residual image, playing a compensatory role. The depthwise separable dilated convolution module avoids the problem of detail information loss by expanding the receptive field. The multi-dimensional hybrid attention mechanism is used to enhance the expressive power of important feature areas to better highlight key information. The innovations of this fusion model are reflected in the following aspects:

[0038] 1. SSDT and DSST multi-scale feature difference information injection mechanism

[0039] To address the negative impact of error accumulation on prediction results, this proposal proposes a multi-scale feature difference information injection mechanism for SSDT and DSST. This mechanism extracts the feature differences between SSDT and DSST remote sensing images and uses these features as a complement to capture and extract temporal variation while preserving the spatial details of the images, thereby more comprehensively mining the spatiotemporal information in remote sensing images. By processing the difference feature maps of SSDT and DSST through convolutional layers, the number of channels in the feature maps can be increased, allowing for the extraction of more refined feature information. Because single-scale features are difficult to fully represent image content, and convolution and pooling operations often lead to the loss of spatial detail in large-scale remote sensing images, a multi-scale mechanism is introduced to enhance the comprehensive description of the image.

[0040] 2. Depthwise Separable Dilated Convolution

[0041] To effectively utilize the relatively complete data in remote sensing images, this solution introduces depthwise separable dilated convolution. Unlike traditional dilated convolution, depthwise separable dilated convolution simplifies the linear combination of multiple channels into addition and multiplication operations, significantly reducing the number of model parameters.

[0042] 3. Multi-dimensional Fusion Attention Mechanism

[0043] To more comprehensively capture features and enhance the representation of objects of interest, MMNet employs a multi-dimensional fusion attention mechanism. This model's attention mechanism includes coordinate attention and spatial attention. Introducing the attention mechanism during image fusion enables the neural network to automatically learn and selectively focus on key parts of the input image. This helps capture important information that might be overlooked in deep convolution processing, resulting in more accurate reconstruction of the target image.

[0044] Figure 1 The overall architecture of the MMNet spatiotemporal fusion method is presented. First, the algorithm performs multi-level feature extraction through upsampling, max pooling, and different convolutional layers to obtain basic image information as well as additional spatial details and temporal variation features, while maintaining the independence of temporal and spatial difference feature maps. Next, the SSDT and DSST multi-scale feature difference information injection mechanisms are used as compensation to further extract image information and integrate it into the feature extraction and basic information extraction process of the multi-scale homologous residual image, while maintaining the correlation of temporal and spatial difference features. Subsequently, the homologous residual feature map is added to the frame map obtained from the fine image and input into the multidimensional hybrid attention mechanism to further focus on important features along the channel and spatial axes. Throughout the process, the convolution used is depthwise separable dilated convolution. In the first stage, the spatial detail features of the MODIS and Landsat images are extracted by convolving their residual image ΔML. In the second stage, temporal image variations are extracted by convolving the residual image ΔM12 of the MODIS images at times t1 and t2 and combining it with ΔML to generate Fc. Simultaneously, the refined image is convolved and combined with ΔML to generate Fd, extracting the image's primary information and framework. In the third stage, Fp is generated by adding the feature maps Fc and Fd generated in the second stage to enhance spatial texture and temporal variation. Finally, coordinate and spatial attention mechanisms are used to enhance model performance. The Fp feature map is sequentially fed into the coordinate attention and spatial attention mechanisms to enhance the representation of the object of interest and more comprehensively capture feature information. Finally, the predicted image is obtained through upsampling and convolution operations.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanisms, characterized by: The following steps are involved: Basic information extraction: extract high-frequency components from the fine image, i.e., the Landsat image, and input the fine image into the convolution layer; Multi-scale feature difference information injection: extracting the difference feature map of remote sensing images of the same source and time; said extracting the difference feature map of remote sensing images of the same source and time includes: performing convolution on the residual image obtained by subtracting the remote sensing images of the same source and time, i.e., the MODIS images at time t1 and t2; upsampling the convolved feature map; inputting the upsampled feature map into the convolution layer again to obtain the final difference feature map of remote sensing images of the same source and time; extracting the difference feature map of remote sensing images of different sources and time; said extracting the difference feature map of remote sensing images of different sources and time includes: performing convolution on the residual image obtained by subtracting ... The residual image obtained by subtracting the simultaneous remote sensing images, i.e., the Landsat image and the MODIS image at the same time, is convolved; the convolved feature map is upsampled; the upsampled feature map is input into the convolution layer again to obtain the final heterogeneous simultaneous remote sensing image difference feature map; the final homogeneous and heterogeneous remote sensing image difference feature map is combined with the final heterogeneous simultaneous remote sensing image difference feature map to generate a feature map Fc; the final heterogeneous simultaneous remote sensing image difference feature map is combined with the refined image after the convolution layer to generate a feature map Fd; Multi-dimensional fusion attention mechanism: Add the feature map Fc and the feature map Fd; input the added feature map into the multi-dimensional hybrid attention mechanism, including the coordinate attention mechanism and the spatial attention mechanism, to enhance the saliency of the feature information; Image reconstruction: The final predicted image is obtained through upsampling and convolution operations; the above convolutions are all depth-wise separable dilated convolutions. Depth-wise separable dilated convolutions: Depth-wise separable dilated convolutions are used for feature extraction to maintain a good receptive field and avoid loss of detailed information.

2. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanisms according to claim 1, characterized in that: In the image reconstruction, the upsampling operation includes: Upsampling operation: Upsampling the feature map to increase the image resolution; Transposed convolution operation: Perform a transposed convolution operation on the feature map to increase the image resolution.

Images