Hyperspectral and multispectral image fusion method based on dilated convolution and cross attention

Abstract

The application discloses a hyperspectral and multispectral image fusion method based on dilated convolution and cross attention, and is specifically implemented according to the following steps: step 1, the original data is preprocessed, and a hyperspectral and multispectral image dataset is generated by simulation, and then the generated hyperspectral and multispectral image dataset is divided into a training set and a test set; step 2, a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention is constructed; step 3, the hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention is trained; and step 4, the trained hyperspectral and multispectral image fusion network model is used to test the test set. The application solves the problems of the prior art, such as difficulty in considering spatial and spectral information in feature extraction, difficulty in balancing calculation efficiency and accuracy, and insufficient feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image fusion technology in image processing, specifically involving a hyperspectral and multispectral image fusion method based on dilated convolution and cross attention. Background Technology

[0002] In the field of remote sensing, hyperspectral images (HSI) and multispectral images (MSI), with their unique data characteristics, have become key data sources for acquiring information about the Earth's surface. Hyperspectral images refer to image data obtained through hyperspectral imaging technology, which images the same target or scene across multiple consecutive narrow bands. This high-resolution spectral information can be used to identify the composition of ground features, accurately distinguishing various ground features such as vegetation and minerals, providing crucial evidence for research in ecology, geology, and the environment. For example, in geological exploration, potential mineral resources can be detected by analyzing the spectral characteristics of different minerals; in agricultural monitoring, the health status of crops can be accurately assessed, and early signs of pests and diseases can be identified. However, hyperspectral images have relatively low spatial resolution, lack sufficient image detail, and struggle to clearly present the spatial distribution and shape characteristics of ground features. Multispectral images, on the other hand, typically consist of several to dozens of wider bands, focusing on acquiring differences in reflectance or radiation of different ground features in specific bands. They have relatively high spatial resolution, clearly showing the spatial location and shape of ground features, and are widely used in urban planning, land use monitoring, and other fields. For example, in urban planning, multispectral images can clearly distinguish different functional areas, such as residential areas, commercial areas, and industrial areas. However, due to the limited number of bands, multispectral images are insufficient for accurate material identification and struggle to perform fine classification of complex land features.

[0003] To fully leverage the high spectral resolution of hyperspectral images and the high spatial resolution of multispectral images, hyperspectral and multispectral image fusion techniques have emerged. Early fusion methods were mostly based on simple data-level fusion, such as weighted averaging, which directly weighted and combined the pixel values ​​of the two images. While simple to operate, the fusion effect was limited, failing to effectively preserve the spectral and spatial information of the images. With technological advancements, transform-domain-based fusion methods have gradually emerged, such as wavelet transform. These methods decompose the image into different frequency sub-bands, fuse the sub-bands of the hyperspectral and multispectral images separately, and then reconstruct the fused image, improving the fusion quality to some extent. In recent years, with the rapid development of deep learning technology, neural network-based fusion methods have made significant progress. Utilizing the powerful feature extraction and learning capabilities of deep neural networks, they can more effectively mine the complementary information of the two images, achieving more accurate fusion.

[0004] Deep learning-based hyperspectral and multispectral image fusion methods mainly fall into three categories: convolutional neural networks (CNNs), generative adversarial networks (GANs), and autoencoders (AEs). CNNs automatically extract features using convolutional layers, fusing features from hyperspectral and multispectral images through multi-layer convolution and pooling operations. Their advantages include automatic feature learning and effective detail extraction, but they are computationally intensive and prone to overfitting. GANs use a generator to fuse images, and a discriminator to determine authenticity. Adversarial training optimizes the generator, resulting in more realistic images; however, training is unstable, and pattern collapse and artifacts are common. AEs use an encoder to map images to a low-dimensional feature space, and then a decoder reconstructs the fused image. They offer good feature extraction, compression, and denoising capabilities, but may lose details and are less adaptable to complex scenes. Summary of the Invention

[0005] The purpose of this invention is to provide a hyperspectral and multispectral image fusion method based on dilated convolution and cross attention, which solves the problems in the prior art, such as the difficulty in balancing spatial and spectral information in feature extraction, the difficulty in balancing computational efficiency and accuracy, and the insufficient feature fusion.

[0006] The technical solution adopted in this invention is a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, specifically implemented according to the following steps:

[0007] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0008] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross-attention;

[0009] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0010] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0011] The invention is further characterized in that,

[0012] Step 1 is implemented in the following steps:

[0013] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0014] Step 1.2: Divide the dataset. Set the 128×128 region in the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 as the test set. Use the rest of the image outside the center region as the training region. Before training, perform zero-padding on the test region to ensure that the training can cover the entire image area. Finally, randomly crop a 128×128 region from the training region as the training set for training.

[0015] The hyperspectral and multispectral image fusion network model in step 2 consists of three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module.

[0016] In step 2, the feature extraction module is implemented according to the following steps:

[0017] To address the different characteristics of HR-MSI and LR-HSI data, corresponding processing methods are implemented, divided into two branches. For HR-MSI, the upper branch first uses a downsampling module and a band reshaping module. The band reshaping module is a combination of four 3×3 convolutions and a ReLU activation function. By downsampling HR-MSI to reduce the image size and reshaping the bands, it achieves a match with LR-HSI. Then, cascaded convolutional layers are used to extract the spatial information of the multispectral image to obtain the HR feature. Specifically, each layer uses a 3×3 convolution and a ReLU activation function. For LR-HSI, the lower branch first uses an upsampling module to match LR-HSI with HR-MSI in spatial scale. Then, multi-scale dilated convolutional blocks are used to extract the spectral information of the hyperspectral image to obtain the LR feature. Specifically, each layer uses a 3×3 dilated convolution and a ReLU activation function. The dilation coefficients of the 3×3 dilated convolutions are 2, 4, and 2.

[0018] In step 2, the feature fusion module is implemented according to the following steps:

[0019] The HR-MSI and LR-HSI features obtained from the feature extraction part are input into the cross-attention fusion module. First, the HR-MSI and LR-HSI features are processed through linear layers to generate the query matrix Q, key matrix K, and value matrix V required by the attention mechanism. Then, the correlation between Q and K is established in the feature mapping, which is achieved through the calculation of the feature relationship matrix, as shown below:

[0020]

[0021] d k =(dim / heads) (2)

[0022] Among them, A i Let Q1, K1, and V1 represent the feature relation matrices between the two input queries and keys, respectively. Q1, K1, and V1 are the query matrix, key matrix, and value matrix generated by HR-MSI, while Q2, K2, and V2 are the query matrix, keyword matrix, and value matrix generated by LR-HSI. T represents the transpose operation, dim is the feature dimension of the input, heads is the number of heads followed by multiple heads, and softmax represents the normalization operation.

[0023] The calculated feature relation matrix is ​​used as weights and multiplied with the value matrix of the other branch input to obtain feature soft attention. This is then added to the original features of the current branch input to perform feature fusion across the spatial and spectral domains, as shown below:

[0024]

[0025] Where LR and HR represent the hyperspectral and multispectral features of the input cross-attention fusion module, respectively, and M and N are the fused features obtained after cross-attention calculation. Finally, a linear layer is used to shape M and N, and the shaped results of M and N are added together to obtain the final fused feature F, as shown below:

[0026] F=Linear(M)+Linear(N) (4).

[0027] In step 2, the feature reconstruction module is implemented according to the following steps:

[0028] First, the output feature F of the cross-attention fusion module in the feature fusion part is concatenated with the two input features HR and LR of the cross-attention fusion module. Then, it enters the mixed-order convolutional feature reconstruction module. The features processed by the band reshaping module are input into the cascaded convolutional layer and the multi-scale dilated convolutional block for feature reconstruction. Then, they are combined by addition. The addition result is then input into the band reshaping module and the upsampling module in turn to perform nonlinear transformation on the features. Finally, the features processed in the above way are first concatenated with the HR-MSI of the original input of the overall fusion network. Then, the concatenated result is concatenated with the LR-HSI↑↑ obtained by the LR-MSI through two upsampling modules and band reshaping is performed to reconstruct a high spatial resolution hyperspectral image HR-HSI with high quality in both spatial and spectral dimensions.

[0029] Step 3 is implemented in the following steps:

[0030] Step 3.1: Use the root mean square error (RMSE) as the loss function, as shown below:

[0031]

[0032] Among them, L fus Represents the loss function. This is the HR-HSI for reference, and The HR-HSI obtained by reconstruction is represented by H and W, which represent the height and width of the image, respectively, and C represents the number of spectral channels.

[0033] Step 3.2: Use Adam as the optimizer to adjust the learning rate. Set the initial learning rate to 0.0001. Use the training set generated in Step 1 as the input to the hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention in Step 2 for supervised training. Stop training when the loss value converges or the set maximum number of iterations is reached. Finally, a set of weight parameters corresponding to the image fusion network model based on dilated convolution and cross attention is obtained.

[0034] Step 4 is implemented in the following steps:

[0035] The hyperspectral and multispectral images of the test set are input into the image fusion network model based on dilated convolution and cross attention trained in step 3, and the trained weight parameters are substituted to finally output the fused image.

[0036] The beneficial effects of this invention are as follows: The hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention firstly employs ordinary convolution and dilated convolution for feature extraction according to the characteristics of different images. Dilated convolution can expand the receptive field without increasing the number of parameters or computational cost, allowing the network to capture image features at different scales, thereby extracting richer contextual information and contributing to a more comprehensive understanding of image content, greatly improving the quality of the fused image. Secondly, a cross-attention mechanism is introduced in the fusion stage. By calculating the attention weights between different inputs, the model can focus on important feature regions, explore the correlation between hyperspectral and multispectral image features, and better integrate the advantages of both, improving the accuracy and effectiveness of the fusion result. Finally, skip connections are used in the reconstruction stage, continuously concatenating the reconstructed image feature processing with previous inputs. This operation can fully preserve multi-source information, enhance feature expression, and avoid losing key information during model processing. The new feature map contains both basic details and high-level semantics, helping the model better understand image content and improving the fusion effect. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the implementation of the hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention in this invention.

[0038] Figure 2 This is a diagram of the network model framework used in the hyperspectral and multispectral image fusion method based on dilated convolution and cross attention in this invention.

[0039] Figure 3 This is a framework diagram of the multi-scale dilated convolution block in the hyperspectral and multispectral image fusion method based on dilated convolution and cross attention in this invention;

[0040] Figure 4 This is a framework diagram of the cross-attention fusion module in the hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention of the present invention;

[0041] Figure 5 This is the mixed-order convolution feature reconstruction module in the hyperspectral and multispectral image fusion method based on dilated convolution and cross attention in this invention;

[0042] Figure 6 This is the fusion result of a comparative experiment on the Urban dataset of the network model in the hyperspectral and multispectral image fusion method based on dilated convolution and cross attention in this invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0044] In the feature extraction stage, this invention employs ordinary convolution and dilated convolution respectively, based on the different characteristics of multispectral and hyperspectral images, to achieve targeted and efficient feature extraction for both types of images. In the fusion stage, a cross-attention mechanism is introduced, enabling deep interaction between the spatial features of the multispectral image and the spectral features of the hyperspectral image. Attention weights are dynamically allocated according to feature importance, fully exploring the intrinsic connections between the features of the two types of images and achieving feature complementarity. This method provides a superior solution for the fusion of hyperspectral and multispectral images, and is expected to improve the quality of the fused image and the effectiveness of subsequent applications.

[0045] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0046] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0047] Step 1 is implemented in the following steps:

[0048] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0049] Step 1.2: Divide the dataset. The 128×128 region at the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 is set as the test set. The remaining image outside this central region is used as the training region. Before training, zero-padding is applied to the test region to ensure that the training covers the entire image area. It is important to note that the test set and training set have no overlap. Finally, a 128×128 region is randomly cropped from the training region to serve as the training set.

[0050] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0051] The hyperspectral and multispectral image fusion network model in step 2 is as follows: Figure 2As shown, it comprises three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module. The feature extraction module uses a dual-branch structure to process HR-MSI and LR-HSI respectively. The upper branch for extracting HR-MSI features consists of a downsampling module, a band reshaping module, and cascaded convolutional layers. The lower branch for extracting LR-MSI features consists of an upsampling module and a multi-scale dilated convolutional block. The structure of the multi-scale dilated convolutional block is shown below. Figure 3 As shown, it consists of three consecutive dilated convolutions and a ReLU activation function, with dilation coefficients of 2, 4, and 2 respectively. The feature fusion module inputs the HR-MSI and LR-HSI features obtained from the upper and lower branches of the feature extraction part into the cross-attention fusion module to obtain the fused features. The structure of the cross-attention fusion module is as follows: Figure 4 As shown, a dual-branch structure is used to process the input HR-MSI and LR-HSI features respectively. The upper and lower branches have the same structure. Both first generate a query matrix Q, a key matrix K, and a value matrix V through a linear layer. Then, the feature relation matrix A is calculated by performing feature relation matrix calculation on the query matrix Q and the key matrix K. Subsequently, the feature relation matrix A is multiplied by the value matrix V of the other branch. Then, the result of the matrix multiplication is added to the original input features of this branch. Finally, the calculation results of the two branches are output through a linear layer and added together to obtain the fused feature. The feature reconstruction part consists of a mixed-order convolutional feature reconstruction module, a skip connection, and a band reshaping module. The structure of the mixed-order convolutional feature reconstruction module is as follows: Figure 5 As shown, features of size (H / 2, W / 2, 3C) are used as input. First, the band dimensions are adjusted by the band reshaping module. Then, the features are split into two paths: one path enters a cascaded convolutional layer, and the other path enters a multi-scale dilated convolutional block. After different feature reconstructions are completed, the results of the two paths are added and fused. The fused result is then adjusted by the band reshaping module, and finally, the spatial resolution is improved by the upsampling module, outputting features of size (H, W, C), thus realizing feature reconstruction.

[0052] Combination Figure 2 and Figure 3 In step 2, the feature extraction module is implemented according to the following steps:

[0053] To address the different characteristics of HR-MSI and LR-HSI data, corresponding processing methods are implemented, divided into two branches. For HR-MSI, the upper branch first uses a downsampling module and a band reshaping module. The band reshaping module is a combination of four 3×3 convolutions and a ReLU activation function. By downsampling HR-MSI to reduce image size and reshaping bands, it aims to match LR-HSI, reduce computational cost, and expand the receptive field. Then, cascaded convolutional layers are used to extract spatial information from the multispectral image to obtain the HR feature. Specifically, each layer uses a 3×3 convolution and a ReLU activation function. This design achieves stronger spatial detail capture and has certain advantages in handling spatial correlations. For LR-HSI, the lower branch first uses an upsampling module to match LR-HSI spatially with HR-MSI. Then, multi-scale dilated convolutional blocks are used to extract spectral information from the hyperspectral image to obtain the LR feature. Figure 3 As shown, each layer specifically uses a 3×3 dilated convolution and a ReLU activation function, with dilation coefficients of 2, 4, and 2 for the 3×3 dilated convolution. This design can significantly expand the receptive field and better capture global spectral features without increasing the number of parameters.

[0054] Combination Figure 2 and Figure 4 In step 2, the feature fusion module is specifically implemented according to the following steps:

[0055] The HR-MSI and LR-HSI features obtained from the feature extraction part are input into the cross-attention fusion module, which is as follows: Figure 4 As shown in the diagram, firstly, the input HR-MSI and LR-HSI features are processed through linear layers to generate the query matrix Q, key matrix K, and value matrix V required by the attention mechanism. Then, the correlation between Q and K is established in the feature mapping, calculated through the feature relationship matrix, as shown below:

[0056]

[0057] d k =(dim / heads) (2)

[0058] Among them, A i Let Q1, K1, and V1 represent the feature relation matrices between the two input queries and keys, respectively. Q1, K1, and V1 are the query matrix, key matrix, and value matrix generated by HR-MSI, while Q2, K2, and V2 are the query matrix, keyword matrix, and value matrix generated by LR-HSI. T represents the transpose operation, dim is the feature dimension of the input, heads is the number of heads followed by multiple heads, and softmax represents the normalization operation.

[0059] The calculated feature relation matrix is ​​used as weights and multiplied with the value matrix of the other branch input to obtain feature soft attention. This is then added to the original features of the current branch input to perform feature fusion across the spatial and spectral domains, as shown below:

[0060]

[0061] Where LR and HR represent the hyperspectral and multispectral features of the input cross-attention fusion module, respectively, and M and N are the fused features obtained after cross-attention calculation. Finally, a linear layer is used to shape M and N, and the shaped results of M and N are added together to obtain the final fused feature F, as shown below:

[0062] F=Linear(M)+Linear(N) (4).

[0063] This approach enables more comprehensive information exchange and fusion between hyperspectral and multispectral images in both spectral and spatial dimensions, reducing information loss and improving fusion performance.

[0064] Combination Figure 2 and Figure 5 In step 2, the feature reconstruction module is specifically implemented according to the following steps:

[0065] First, the output feature F of the cross-attention fusion module in the feature fusion part is concatenated with the two input features HR and LR of the cross-attention fusion module. This operation preserves the basic information and details in the original input features and prevents information loss during fusion. Then, it enters the mixed-order convolutional feature reconstruction module, as follows... Figure 5 As shown, the features processed by the band reshaping module are input into a cascaded convolutional layer and a multi-scale dilated convolutional block for feature reconstruction in two paths. These features are then combined through addition. During this process, information from different scales and characteristics complements each other, refining the feature representation. The addition result is then sequentially input into the band reshaping module and the upsampling module to further perform nonlinear transformations on the features, enhancing their expressive power. Finally, the features processed above are first concatenated with the original HR-MSI input from the overall fusion network. Then, the concatenated result is concatenated with the LR-HSI↑↑ obtained from the LR-MSI after two upsampling modules and band reshaping is performed. This fully utilizes the advantages of features at different stages, enriching the diversity and completeness of information, and reconstructing a high-spatial-resolution hyperspectral image HR-HSI with high quality in both spatial and spectral dimensions.

[0066] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0067] Step 3 is implemented in the following steps:

[0068] Step 3.1: Use the root mean square error (RMSE) as the loss function, as shown below:

[0069]

[0070] Among them, L fus Represents the loss function. This is the HR-HSI for reference, and The HR-HSI obtained by reconstruction is represented by H and W, which represent the height and width of the image, respectively, and C represents the number of spectral channels.

[0071] Step 3.2: Use Adam as the optimizer to adjust the learning rate. Set the initial learning rate to 0.0001. Use the training set generated in Step 1 as the input to the hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention in Step 2 for supervised training. Stop training when the loss value converges or the set maximum number of iterations is reached. Finally, a set of weight parameters corresponding to the image fusion network model based on dilated convolution and cross attention is obtained.

[0072] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0073] Step 4 is implemented in the following steps:

[0074] The hyperspectral and multispectral images of the test set are input into the image fusion network model based on dilated convolution and cross attention trained in step 3, and the trained weight parameters are substituted to finally output the fused image.

[0075] For the fused image, the fused image obtained above is compared with the original HR-HSI image to perform quantitative evaluation. Four representative evaluation indicators are selected: Root Mean Square Error (RMSE), Peak Signal to Noise Ratio (PSNR), Relative Average Spectral Error (RASE), and Spectral Angle Mapper (SAM). Simultaneously, qualitative analysis is performed based on the fused image.

[0076] Example 1

[0077] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0078] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0079] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0080] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0081] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0082] Example 2

[0083] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0084] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0085] Step 1 is implemented in the following steps:

[0086] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0087] Step 1.2: Divide the dataset. The 128×128 region at the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 is set as the test set. The remaining image outside this central region is used as the training region. Before training, zero-padding is applied to the test region to ensure that the training covers the entire image area. It is important to note that the test set and training set have no overlap. Finally, a 128×128 region is randomly cropped from the training region to serve as the training set.

[0088] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0089] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0090] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0091] Example 3

[0092] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0093] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0094] Step 1 is implemented in the following steps:

[0095] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0096] Step 1.2: Divide the dataset. The 128×128 region at the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 is set as the test set. The remaining image outside this central region is used as the training region. Before training, zero-padding is applied to the test region to ensure that the training covers the entire image area. It is important to note that the test set and training set have no overlap. Finally, a 128×128 region is randomly cropped from the training region to serve as the training set.

[0097] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0098] The hyperspectral and multispectral image fusion network model in step 2 is as follows: Figure 2 As shown, it consists of three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module.

[0099] Combination Figure 2 and Figure 3 In step 2, the feature extraction module is implemented according to the following steps:

[0100] To address the different characteristics of HR-MSI and LR-HSI data, corresponding processing methods are implemented, divided into two branches. For HR-MSI, the upper branch first uses a downsampling module and a band reshaping module. The band reshaping module is a combination of four 3×3 convolutions and a ReLU activation function. By downsampling HR-MSI to reduce image size and reshaping bands, it aims to match LR-HSI, reduce computational cost, and expand the receptive field. Then, cascaded convolutional layers are used to extract spatial information from the multispectral image to obtain the HR feature. Specifically, each layer uses a 3×3 convolution and a ReLU activation function. This design achieves stronger spatial detail capture and has certain advantages in handling spatial correlations. For LR-HSI, the lower branch first uses an upsampling module to match LR-HSI spatially with HR-MSI. Then, multi-scale dilated convolutional blocks are used to extract spectral information from the hyperspectral image to obtain the LR feature. Figure 3 As shown, each layer specifically uses a 3×3 dilated convolution and a ReLU activation function, with dilation coefficients of 2, 4, and 2 for the 3×3 dilated convolution. This design can significantly expand the receptive field and better capture global spectral features without increasing the number of parameters.

[0101] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0102] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0103] Example 4

[0104] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0105] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0106] Step 1 is implemented in the following steps:

[0107] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0108] Step 1.2: Divide the dataset. The 128×128 region at the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 is set as the test set. The remaining image outside this central region is used as the training region. Before training, zero-padding is applied to the test region to ensure that the training covers the entire image area. It is important to note that the test set and training set have no overlap. Finally, a 128×128 region is randomly cropped from the training region to serve as the training set.

[0109] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0110] The hyperspectral and multispectral image fusion network model in step 2 is as follows: Figure 2 As shown, it consists of three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module.

[0111] Combination Figure 2 and Figure 3 In step 2, the feature extraction module is implemented according to the following steps:

[0112] To address the different characteristics of HR-MSI and LR-HSI data, corresponding processing methods are implemented, divided into two branches. For HR-MSI, the upper branch first uses a downsampling module and a band reshaping module. The band reshaping module is a combination of four 3×3 convolutions and a ReLU activation function. By downsampling HR-MSI to reduce image size and reshaping bands, it aims to match LR-HSI, reduce computational cost, and expand the receptive field. Then, cascaded convolutional layers are used to extract spatial information from the multispectral image to obtain the HR feature. Specifically, each layer uses a 3×3 convolution and a ReLU activation function. This design achieves stronger spatial detail capture and has certain advantages in handling spatial correlations. For LR-HSI, the lower branch first uses an upsampling module to match LR-HSI spatially with HR-MSI. Then, multi-scale dilated convolutional blocks are used to extract spectral information from the hyperspectral image to obtain the LR feature. Figure 3 As shown, each layer specifically uses a 3×3 dilated convolution and a ReLU activation function, with dilation coefficients of 2, 4, and 2 for the 3×3 dilated convolution. This design can significantly expand the receptive field and better capture global spectral features without increasing the number of parameters.

[0113] Combination Figure 2 and Figure 4 In step 2, the feature fusion module is specifically implemented according to the following steps:

[0114] The HR-MSI and LR-HSI features obtained from the feature extraction part are input into the cross-attention fusion module, which is as follows: Figure 4 As shown in the diagram, firstly, the input HR-MSI and LR-HSI features are processed through linear layers to generate the query matrix Q, key matrix K, and value matrix V required by the attention mechanism. Then, the correlation between Q and K is established in the feature mapping, calculated through the feature relationship matrix, as shown below:

[0115]

[0116] d k =(dim / heads) (2)

[0117] Among them, A i Let Q1, K1, and V1 represent the feature relation matrices between the two input queries and keys, respectively. Q1, K1, and V1 are the query matrix, key matrix, and value matrix generated by HR-MSI, while Q2, K2, and V2 are the query matrix, keyword matrix, and value matrix generated by LR-HSI. T represents the transpose operation, dim is the feature dimension of the input, heads is the number of heads followed by multiple heads, and softmax represents the normalization operation.

[0118] The calculated feature relation matrix is ​​used as weights and multiplied with the value matrix of the other branch input to obtain feature soft attention. This is then added to the original features of the current branch input to perform feature fusion across the spatial and spectral domains, as shown below:

[0119]

[0120] Where LR and HR represent the hyperspectral and multispectral features of the input cross-attention fusion module, respectively, and M and N are the fused features obtained after cross-attention calculation. Finally, a linear layer is used to shape M and N, and the shaped results of M and N are added together to obtain the final fused feature F, as shown below:

[0121] F=Linear(M)+Linear(N) (4).

[0122] This approach enables more comprehensive information exchange and fusion between hyperspectral and multispectral images in both spectral and spatial dimensions, reducing information loss and improving fusion performance.

[0123] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0124] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0125] Example 5

[0126] This invention relates to a hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, with the following process: Figure 1 As shown, please follow these steps:

[0127] Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set.

[0128] Step 1 is implemented in the following steps:

[0129] Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI.

[0130] Step 1.2: Divide the dataset. The 128×128 region at the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 is set as the test set. The remaining image outside this central region is used as the training region. Before training, zero-padding is applied to the test region to ensure that the training covers the entire image area. It is important to note that the test set and training set have no overlap. Finally, a 128×128 region is randomly cropped from the training region to serve as the training set.

[0131] Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention, and determine the structure of each module;

[0132] The hyperspectral and multispectral image fusion network model in step 2 is as follows: Figure 2 As shown, it consists of three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module.

[0133] Combination Figure 2 and Figure 5 In step 2, the feature reconstruction module is specifically implemented according to the following steps:

[0134] First, the output feature F of the cross-attention fusion module in the feature fusion part is concatenated with the two input features HR and LR of the cross-attention fusion module. This operation preserves the basic information and details in the original input features and prevents information loss during fusion. Then, it enters the mixed-order convolutional feature reconstruction module, as follows... Figure 5As shown, the features processed by the band reshaping module are input into a cascaded convolutional layer and a multi-scale dilated convolutional block for feature reconstruction in two paths. These features are then combined through addition. During this process, information from different scales and characteristics complements each other, refining the feature representation. The addition result is then sequentially input into the band reshaping module and the upsampling module to further perform nonlinear transformations on the features, enhancing their expressive power. Finally, the features processed above are first concatenated with the original HR-MSI input from the overall fusion network. Then, the concatenated result is concatenated with the LR-HSI↑↑ obtained from the LR-MSI after two upsampling modules and band reshaping is performed. This fully utilizes the advantages of features at different stages, enriching the diversity and completeness of information, and reconstructing a high-spatial-resolution hyperspectral image HR-HSI with high quality in both spatial and spectral dimensions.

[0135] Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention;

[0136] Step 3 is implemented in the following steps:

[0137] Step 3.1: Use the root mean square error (RMSE) as the loss function, as shown below:

[0138]

[0139] Among them, L fus Represents the loss function. This is the HR-HSI for reference, and The HR-HSI obtained by reconstruction is represented by H and W, which represent the height and width of the image, respectively, and C represents the number of spectral channels.

[0140] Step 3.2: Use Adam as the optimizer to adjust the learning rate. Set the initial learning rate to 0.0001. Use the training set generated in Step 1 as the input to the hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention in Step 2 for supervised training. Stop training when the loss value converges or the set maximum number of iterations is reached. Finally, a set of weight parameters corresponding to the image fusion network model based on dilated convolution and cross attention is obtained.

[0141] Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

[0142] Example 6

[0143] The effects of the present invention will be further explained below with reference to experiments:

[0144] 1. Experimental conditions:

[0145] The experiments of this invention used an NVIDIA GeForce RTX 4090D (24G), the deep learning framework used was PyTorch 2.1.0, and the programming language was Python 3.10.

[0146] 2. Introduction to the dataset

[0147] The Urban dataset used in the experiments of this invention was obtained from images of Copper Tree Bay, Texas, USA, taken by the Hydice sensor in 1995. The dataset contains 307×307 images, and the raw data covers 210 bands. After preprocessing operations such as noise reduction and absorbent strip removal, typically 162 bands are retained for subsequent processing and analysis.

[0148] 3. Parameter settings:

[0149] This invention selects the Adam algorithm as the optimizer, and the initial learning rate is set to 0.0001.

[0150] 4. Experimental Comparison

[0151] To verify the effectiveness of the proposed method, it was compared with seven existing hyperspectral and multispectral image fusion methods. The fusion results of the proposed method were quantitatively evaluated and qualitatively analyzed. Table 1 shows the evaluation metrics of different fusion methods on the Urban dataset, and the fusion results of different fusion methods on the Urban dataset are shown below. Figure 6 As shown.

[0152] Table 1. Fusion performance of different methods on the Urban dataset.

[0153] method RMSE↓ PSNR↑ ERGAS↓ SAM↓ MSDCNN 3.1317 35.9676 1.7652 3.1199 TFNet 3.0405 36.2243 1.7146 2.8524 ResTFNet 2.8916 36.6604 1.598 2.7355 SSFCNN 8.5801 27.2133 4.2196 8.7402 ConSSFCNN 4.9788 34.1876 1.8615 3.1634 MCT 2.5517 37.7465 1.34 2.3299 Proposed 2.4556 38.0799 1.2776 2.2704

[0154] Table 1 shows that the proposed method achieved strong competitive results across all four metrics. Even with the large number of bands in the Urban dataset, the method still achieved satisfactory fusion results, fully demonstrating its effectiveness in preserving both spectral and spatial information. Figure 6 The fusion result image more intuitively shows the fusion effect. Comparison revealed that the method proposed in this invention best preserves the spatial structure and detailed features of the image; that is, the outlines and textures of ground features are clearest and essentially consistent with the original image. Therefore, the method proposed in this invention has excellent performance in hyperspectral and multispectral image fusion.

Claims

1. A hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention, characterized in that, The specific steps are as follows: Step 1: Preprocess the raw data to simulate and generate a hyperspectral and multispectral image dataset. Then, divide the generated hyperspectral and multispectral image dataset into a training set and a test set. Step 2: Construct a hyperspectral and multispectral image fusion network model based on dilated convolution and cross-attention; The hyperspectral and multispectral image fusion network model in step 2 consists of three parts: a feature extraction module, a feature fusion module, and a feature reconstruction module. In step 2, the feature extraction module is implemented according to the following steps: The algorithm addresses the different characteristics of HR-MSI and LR-HSI data by processing them in two branches. For HR-MSI, the upper branch first uses a downsampling module and a band reshaping module. The band reshaping module is a combination of four 3×3 convolutions and a ReLU activation function. By downsampling HR-MSI to reduce the image size and reshaping the bands, it matches LR-HSI. Then, cascaded convolutional layers are used to extract the spatial information of the multispectral image to obtain the HR feature. Specifically, each layer uses a 3×3 convolution and a ReLU activation function. For LR-HSI, the lower branch first uses an upsampling module to match LR-HSI with HR-MSI in terms of spatial scale. Then, multi-scale dilated convolutional blocks are used to extract the spectral information of the hyperspectral image to obtain the LR feature. Specifically, each layer uses a 3×3 dilated convolution and a ReLU activation function. The dilation coefficients of the 3×3 dilated convolutions are 2, 4, and 2. The HR-MSI and LR-HSI features obtained from the feature extraction part are input into the cross-attention fusion module; The calculated feature relation matrix is ​​used as a weight and multiplied with the value matrix of the other branch input to obtain feature soft attention. Then, it is added to the original features of the current branch input to perform feature fusion across spatial and spectral domains. Finally, through a linear layer The fused features obtained after cross-attention calculation are shaped, and the shaped results are added together to obtain the final fused features; In the feature reconstruction module, the specific steps are as follows: First, the output feature F of the cross-attention fusion module in the feature fusion part is concatenated with the two input features HR and LR of the cross-attention fusion module. Then, it enters the mixed-order convolutional feature reconstruction module. The features processed by the band reshaping module are input into the cascaded convolutional layer and the multi-scale dilated convolutional block for feature reconstruction. Then, they are combined by addition. The addition result is then input into the band reshaping module and the upsampling module in turn to perform nonlinear transformation on the features. Finally, the features processed above are first concatenated with the HR-MSI of the original input of the overall fusion network. Then, the concatenated result is concatenated with the LR-HSI↑↑ obtained by the LR-MSI through two upsampling modules and band reshaping is performed to reconstruct a high spatial resolution hyperspectral image HR-HIS with high quality in both spatial and spectral dimensions. Step 3: Train a hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention; Step 4: Test the test set using the trained hyperspectral and multispectral image fusion network model.

2. The hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention according to claim 1, characterized in that, Step 1 is implemented in the following steps: Step 1.1: Preprocess the dataset by performing a 5×5 Gaussian filter on the high spatial resolution hyperspectral image HR-HSI in the dataset to blur the HR-HSI. Then, downsample the blurred high spatial resolution hyperspectral image HR-HSI four times to obtain the low spatial resolution hyperspectral image LR-HSI. Finally, extract five bands at equal intervals from HR-HSI to obtain the high spatial resolution multispectral image HR-MSI. Step 1.2: Divide the dataset. Set the 128×128 region in the center of the low spatial resolution hyperspectral image LR-HSI and the high spatial resolution multispectral image HR-MSI generated in Step 1.1 as the test set. Use the rest of the image outside the center region as the training region. Before training, perform zero-padding on the test region to ensure that the training can cover the entire image area. Finally, randomly crop a 128×128 region from the training region as the training set for training.

3. The hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention according to claim 2, characterized in that, In step 2, the feature fusion module is implemented according to the following steps: The HR-MSI and LR-HSI features obtained from the feature extraction part are input into the cross-attention fusion module. First, the HR-MSI and LR-HSI features are processed through linear layers to generate the query matrix Q, key matrix K, and value matrix V required by the attention mechanism. Then, the correlation between Q and K is established in the feature mapping, which is achieved through the calculation of the feature relationship matrix, as shown below: (1) (2) in, These represent the feature relationship matrices between the query and the key, respectively. , and It consists of a query matrix, a key matrix, and a value matrix generated by HR-MSI. , and It consists of a query matrix, a keyword matrix, and a value matrix generated by LR-HSI. This indicates the transpose operation. It is the input feature dimension. It's the number of heads that the bulls are focusing on. This indicates a normalization operation; The calculated feature relation matrix is ​​used as weights and multiplied with the value matrix of the other branch input to obtain feature soft attention. This is then added to the original features of the current branch input to perform feature fusion across the spatial and spectral domains, as shown below: (3) in, and These represent the hyperspectral and multispectral features of the input cross-attention fusion module, respectively. and The fused features are obtained after cross-attention calculation and finally passed through a linear layer. right and Perform plastic surgery and and The results of the plastic surgery are added together to obtain the final fusion feature. , means as follows: (4)。 4. The hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention according to claim 3, characterized in that, Step 3 is implemented in the following steps: Step 3.1: Use the root mean square error (RMSE) as the loss function, as shown below: (5) in, Represents the loss function. This is the HR-HSI for reference, and This represents the HR-HSI obtained from the reconstruction. and These represent the height and width of the image, respectively. Indicates the number of spectral channels; Step 3.2: Use Adam as the optimizer to adjust the learning rate. Set the initial learning rate to 0.0001. Use the training set generated in Step 1 as the input to the hyperspectral and multispectral image fusion network model based on dilated convolution and cross attention in Step 2 for supervised training. Stop training when the loss value converges or the set maximum number of iterations is reached. Finally, a set of weight parameters corresponding to the image fusion network model based on dilated convolution and cross attention is obtained.

5. The hyperspectral and multispectral image fusion method based on dilated convolution and cross-attention according to claim 4, characterized in that, Step 4 is implemented in the following steps: The hyperspectral and multispectral images of the test set are input into the image fusion network model based on dilated convolution and cross attention trained in step 3, and the trained weight parameters are substituted to finally output the fused image.

Images