Remote sensing image space-time fusion method combining multi-scale and multi-dimensional mixed attention mechanism

Abstract

The invention relates to a remote sensing image space-time fusion method combining a multi-scale and multi-dimensional mixed attention mechanism, and belongs to the field of remote sensing image processing. According to the method, multi-scale feature extraction and a multi-dimensional mixed attention mechanism are combined, and feature extraction is performed by using depth separable expansion convolution, so that a good receptive field is maintained, and detailed information loss is avoided. MMNet optimizes information fusion through a homologous different-time and different-source simultaneous multi-scale feature difference injection mechanism and a multi-dimensional mixed attention mechanism, so that the accuracy and reliability of space-time fusion are improved. According to the method, global information and local details in the image can be effectively captured, relatively complete data in the remote sensing image are fully utilized, and a new solution is provided for the field of remote sensing image processing.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing and relates to a remote sensing image spatiotemporal fusion method combining a multi-scale and multi-dimensional hybrid attention mechanism. Background Art

[0002] With the rapid development of remote sensing data processing technology, remote sensing data has been widely used in many fields, including plateau land cover classification, forest health monitoring, analysis of the impact of urbanization on vegetation, and crop type identification. These applications usually require high spatial resolution and frequently updated remote sensing images to accurately capture the dynamic changes of the surface. However, it is difficult for a single satellite to provide remote sensing image data with high temporal and spatial resolution at the same time. Therefore, in the past decade, more and more integrated satellite sensor data have been used to make up for this deficiency. Although the next generation of Landsat satellites plans to shorten the revisit cycle to 6 days, the current Landsat satellites are still limited to a 16-day revisit cycle. Although their spatial resolution is high (30 meters), their temporal resolution is low. In contrast, MODIS images have a higher temporal resolution (daily), but their spatial resolution is low (250 meters / 500 meters). Through image fusion technology, these complementary spatiotemporal information can be combined to generate remote sensing images with high spatiotemporal resolution, further improving their effectiveness and value in practical applications.

[0003] At present, mainstream remote sensing image fusion methods are usually based on learning techniques, aiming to model the complex nonlinear relationship between high spatial resolution images (HSLT) and high temporal resolution images (LSHT). For example, the SPSTFM and EBSCDL methods reconstruct low-resolution images into high spatial resolution images through sparse representation. Subsequently, convolutional neural networks (CNNs) were applied to spatiotemporal image fusion (STF), where STFDCNN uses CNN to model the relationship between coarse-resolution and fine-resolution images and uses a high-pass fusion model for prediction. However, these methods still have certain limitations in mapping the complex nonlinear relationships between images. To overcome this problem, VDCNNSTF adopts a deeper network structure that can more effectively extract deep features and thus more accurately map these nonlinear relationships. The DCSTFN method further optimizes the fusion effect by using three-part image fusion, including the extended high-temporal resolution image, the extracted high-frequency component, and the fused features. EDCSTFN improves the accuracy and robustness of the model by reducing the dependence on a single reference image. The AMNet method directly introduces MODIS residual images into network training, and extracts spatiotemporal changes and spatial details of different scales through a multi-scale mechanism, enhances the detail information of feature maps, and uses a hybrid attention mechanism to improve the significance of feature information. Overall, these deep learning-based fusion methods have improved the quality of remote sensing images to a certain extent and effectively solved the challenge of fusion accuracy.

[0004] Although the STF of remote sensing images has received extensive attention and research, there are still some challenges. First, remote sensing images usually produce large errors after atmospheric and radiation correction. Many methods fail to effectively consider these errors, resulting in images being directly input into the network. As a result, errors continue to accumulate during the training process, especially when dealing with areas with significant spatiotemporal changes. This error accumulation often affects the final prediction accuracy. Second, in practical applications, cloud cover often leads to the loss of local information in remote sensing images, while the data in other areas are relatively complete. How to make full use of the effective parts of these incomplete images remains a problem that needs to be solved. Summary of the invention

[0005] In order to solve the error problem caused by atmospheric and radiation correction, which leads to blurred spatial details of predicted images, distortion of temporal changes, and difficulty in effectively utilizing the missing local information in remote sensing images, the present invention proposes a remote sensing image spatiotemporal fusion method called MMNet. The network combines multi-scale feature extraction and multi-dimensional hybrid attention mechanism. Specifically, MMNet adopts the same-source and heterogeneous multi-scale feature difference injection mechanism, as well as the multi-dimensional hybrid attention mechanism, to optimize information fusion. In order to further improve the performance of the network, MMNet replaces all conventional convolutions with depth-separable dilated convolutions. This method aims to reduce the errors caused by atmospheric and radiation correction, and can capture global information and local details in the image, thereby improving the accuracy and reliability of STF.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism includes the following steps:

[0008] Basic information extraction: extract high-frequency components from fine images;

[0009] Multi-scale feature difference information injection: extract the difference feature map of remote sensing images of the same source and different time; extract the difference feature map of remote sensing images of different sources and at the same time; add the above two difference feature maps to the feature extraction and basic information extraction of multi-scale homologous residual images by stacking;

[0010] Depthwise Separable Dilated Convolution: Use depthwise separable dilated convolution for feature extraction to maintain a good receptive field and avoid loss of detailed information;

[0011] Multi-dimensional fusion attention mechanism: add the homologous residual feature map to the refined image frame map; input the added feature map into the multi-dimensional hybrid attention mechanism, including the coordinate attention mechanism and the spatial attention mechanism, to enhance the saliency of the feature information;

[0012] Image reconstruction: The final predicted image is obtained through upsampling and convolution operations.

[0013] Furthermore, in the multi-scale feature difference information injection, the step of extracting the difference feature map of the remote sensing images of the same source and different time includes:

[0014] Convolution is performed on the residual image obtained by subtracting the remote sensing images of the same source but different time;

[0015] Upsample the convolved feature map;

[0016] The upsampled feature map is input into the convolution layer again to obtain the final difference feature map of remote sensing images of the same source and different time.

[0017] Furthermore, in the multi-scale feature difference information injection, the step of extracting the heterogeneous simultaneous remote sensing image difference feature map comprises:

[0018] Convolution is performed on the residual image obtained after subtracting the remote sensing images from different sources simultaneously;

[0019] Upsample the convolved feature map;

[0020] The upsampled feature map is input into the convolution layer again to obtain the final heterogeneous simultaneous remote sensing image difference feature map.

[0021] Further, in the depthwise separable dilated convolution, the depthwise separable dilated convolution includes:

[0022] Deep dilated convolution: Perform deep dilated convolution on the input feature map, and perform dilated convolution operation independently on each input channel of the feature map;

[0023] Point-by-point convolution: Use a 1×1 convolution kernel to convolve the output feature map after depth convolution.

[0024] Furthermore, in the multi-dimensional fusion attention mechanism, the coordinate attention mechanism includes:

[0025] Average pooling: Average pooling is performed along the horizontal and vertical directions in space to obtain two one-dimensional vectors;

[0026] Convolution dimensionality reduction: concatenate in the spatial dimension and reduce the channel dimension through 1×1 convolution;

[0027] One-dimensional feature encoder: Through batch normalization and nonlinear activation function, the feature map is encoded in one dimension along two spatial directions to capture long-range dependencies and retain precise location information respectively;

[0028] Attention weighted feature map: The spatial dimension is split and restored into horizontal and vertical components, and the number of channels is restored through 1×1 convolution, and weights are generated through the gating mechanism.

[0029] Furthermore, in the multi-dimensional fusion attention mechanism step, the spatial attention mechanism includes:

[0030] Convolutional layer: concatenates and convolves the feature maps to obtain a single feature vector;

[0031] Attention weight calculation: weights are assigned through the Sigmoid activation function;

[0032] Attention-weighted feature map: Add the attention weights to the feature map to get the attention-weighted feature map.

[0033] Furthermore, in the image reconstruction, the upsampling operation includes:

[0034] Upsampling operation: Upsample the feature map to increase the image resolution;

[0035] Convolution operation: Perform convolution operation on the feature map to increase the image resolution.

[0036] Furthermore, in the image reconstruction, the convolution operation includes:

[0037] Depth-wise dilated convolution operation;

[0038] Point-wise convolution operation.

[0039] The beneficial effects of the present invention are as follows: The present invention aims to break through the limitations of existing fusion methods; MMNet uses the SSDT and DSST multi-scale feature difference information injection mechanism to add channels to the feature map to extract detailed information about the features, enhance the model's feature learning and generalization capabilities, and adopts a multi-dimensional fusion attention mechanism to automatically learn and selectively focus on the key information of the input, thereby strengthening the representation of the object of interest. In addition, the deep separable dilated convolution is used to capture global information and local details, effectively preventing the loss of detailed information.

[0040] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 This is the overall structure of MMNet. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0045] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0046] The MMNet model consists of four core modules: basic information extraction, same source different times (SSDT) ​​and different source same times (DSST) multi-scale feature difference information injection mechanism, deep separable dilated convolution and multi-dimensional hybrid attention mechanism. In this model, the basic information extraction module is responsible for extracting high-frequency details from high-resolution images. The multi-scale feature difference information injection mechanism extracts time difference and spatial difference information from SSDT and DSST remote sensing images of different scales, and integrates this information into the feature extraction and basic information extraction process of the homologous residual image, playing a compensatory role. The deep separable dilated convolution module avoids the problem of detail information loss by expanding the receptive field. The multi-dimensional hybrid attention mechanism is used to enhance the expression ability of important feature areas to better highlight key information. The innovation of this fusion model is reflected in the following aspects:

[0047] 1. SSDT and DSST multi-scale feature difference information injection mechanism

[0048] In order to solve the negative impact of error accumulation on prediction results, this scheme proposes a multi-scale feature difference information injection mechanism for SSDT and DSST. This mechanism extracts the feature differences of SSDT and DSST remote sensing images and uses the feature information of the two as a complement to capture and extract temporal variation features while retaining the spatial details of the image, thereby more comprehensively mining the spatiotemporal information in remote sensing images. By processing the difference feature maps of SSDT and DSST through convolutional layers, the number of channels of the feature maps can be increased and more detailed feature information can be extracted. Since it is difficult for features of a single scale to fully represent the image content, and convolution and pooling operations often lead to the loss of spatial details in large-scale remote sensing images, a multi-scale mechanism is introduced to enhance the comprehensive description of the image.

[0049] 2. Depthwise Separable Dilated Convolution

[0050] In order to solve the problem of how to effectively utilize the relatively complete data in remote sensing images, this scheme introduces depthwise separable dilated convolution. Different from traditional dilated convolution, depthwise separable dilated convolution simplifies the linear combination operation of multiple channels into addition and multiplication operations, thereby greatly reducing the number of model parameters.

[0051] 3. Multi-dimensional Fusion Attention Mechanism

[0052] In order to capture features more comprehensively and enhance the representation of objects of interest, MMNet adopts a multi-dimensional fusion attention mechanism. The attention mechanism of this model includes coordinate attention and spatial attention. Introducing the attention mechanism in the image fusion process enables the neural network to automatically learn and selectively focus on the key parts of the input image. This helps capture important information that may be ignored in deep convolution processing, thereby reconstructing the target image more accurately.

[0053] Figure 1The overall architecture of the MMNet spatiotemporal fusion method is presented. First, the algorithm performs multi-level feature extraction through upsampling, max pooling, and different convolutional layers to obtain basic image information as well as additional spatial details and temporal variation features, while maintaining the independence of temporal and spatial difference feature maps. Next, the SSDT and DSST multi-scale feature difference information injection mechanisms are used as compensation to further extract image information and integrate it into the feature extraction and basic information extraction process of the multi-scale homologous residual image, while maintaining the correlation of temporal and spatial difference features. Subsequently, the homologous residual feature map is added to the frame map obtained from the fine image and input into the multi-dimensional hybrid attention mechanism to further focus on the important features on the channel and spatial axis. Throughout the process, the convolutions used are all depthwise separable dilated convolutions. In the first stage, the spatial detail features of the MODIS and Landsat images are extracted by convolving the residual image ΔML of the two. In the second stage, the temporal changes of the image are extracted by convolving the residual image ΔM12 of the MODIS image at time t1 and t2 and combining it with ΔML to generate Fc; at the same time, the fine image is convolved and combined with ΔML to generate Fd to extract the main information and framework of the image. In the third stage, Fp is generated by adding the feature maps Fc and Fd generated in the second stage to enhance the spatial texture and temporal change features. Finally, the coordinate and spatial attention mechanisms are used to improve the model performance. The Fp feature map is sequentially input into the coordinate attention and spatial attention mechanisms to enhance the representation of the object of interest and capture the feature information more comprehensively. Finally, the predicted image is obtained through upsampling and convolution operations.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanisms, characterized by: The following steps are involved: Basic information extraction: extract high-frequency components from fine images; Multi-scale feature difference information injection: extract the difference feature map of remote sensing images of the same source and different time; extract the difference feature map of remote sensing images of different sources and at the same time; pass the above two difference feature maps through the convolution layer respectively, add the channel of the feature map and extract the fine information about the features; Depthwise Separable Dilated Convolution: Use depthwise separable dilated convolution for feature extraction to maintain a good receptive field and avoid loss of detailed information; Multi-dimensional fusion attention mechanism: add the homologous residual feature map to the refined image frame map; input the added feature map into the multi-dimensional hybrid attention mechanism, including the coordinate attention mechanism and the spatial attention mechanism, to enhance the saliency of the feature information; Image reconstruction: The final predicted image is obtained through upsampling and convolution operations.

2. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to claim 1, characterized in that: In the multi-scale feature difference information injection, the step of extracting the difference feature map of remote sensing images of the same source and different time includes: Upsampling of fine-grained images; Perform a maximum pooling operation on the upsampled fine image and the low temporal resolution image; The pooled image is input into the convolutional layer to obtain the difference feature map of remote sensing images of the same source and different time.

3. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to claim 1, characterized in that: In the multi-scale feature difference information injection, the step of extracting the heterogeneous simultaneous remote sensing image difference feature map comprises: Upsampling of fine-grained images; Perform a maximum pooling operation on the upsampled fine image and the low spatial resolution image; The pooled image is input into the convolutional layer to obtain the difference feature map of heterogeneous simultaneous remote sensing images.

4. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to claim 1, characterized in that: In the depthwise separable dilated convolution, the depthwise separable dilated convolution comprises: Separation convolution layer: perform deep separation on the input feature map and perform convolution operations on each channel separately; Dilated convolution layer: Perform a dilated convolution operation on the feature map output by the separation convolution layer to expand the receptive field.

5. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to claim 1, characterized in that: In the multi-dimensional fusion attention mechanism, the coordinate attention mechanism includes: One-dimensional feature encoder: One-dimensional feature encoding is performed on the feature map along two spatial directions to capture long-range dependencies and retain precise location information respectively; Attention weight calculation: Calculate the attention weight based on the output of the one-dimensional feature encoder; Attention-weighted feature map: Multiply the attention weight by the feature map to obtain the attention-weighted feature map.

6. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to claim 1, characterized in that: In the multi-dimensional fusion attention mechanism step, the spatial attention mechanism includes: Global pooling layer: performs global pooling operation on the feature map to obtain a single feature vector; Attention weight calculation: Calculate the attention weight based on the output of the global pooling layer; Attention-weighted feature map: Multiply the attention weight by the feature map to obtain the attention-weighted feature map.

7. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to any one of claims 1 to 6, characterized in that: In the image reconstruction, the upsampling operation includes: Deconvolution operation: Deconvolution operation is performed on the feature map to increase the image resolution; Transposed convolution operation: Perform a transposed convolution operation on the feature map to increase the image resolution.

8. The remote sensing image spatiotemporal fusion method combining multi-scale and multi-dimensional hybrid attention mechanism according to any one of claims 1 to 7, characterized in that: In the image reconstruction, the convolution operation includes: Standard convolution operation; Depthwise separable convolution operation; Depthwise separable dilated convolution operation.

Images