Hyperspectral image and multispectral image fusion method based on multi-scale visual attention

Through the multi-scale visual attention network method, combined with multi-scale large-core attention and joint attention module, the problem of spectral and spatial distortion in the fusion of hyperspectral images and multi-spectral images is solved, achieving high-quality image fusion and improving Feature extraction and characterization capabilities are achieved.

CN118918016APending Publication Date: 2024-11-08TIANJIN UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202410567207.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art cannot efficiently mine spatial and spectral information in the fusion of hyperspectral images and multispectral images, resulting in spectral and spatial distortion problems in the fusion image. The method based on convolutional neural networks has problems with small perception domains and limited generalization capabilities.

Method used

Using a multi-scale visual attention network method, through a multi-scale large-core attention mechanism and joint attention module, combined with shallow feature coding and deep feature extraction, the loss function is designed to optimize image reconstruction and improve the space and spectrum of the image. quality.

Benefits of technology

It effectively improves the spatial and spectral details quality of the fusion image, balances the learning of non-local and local information, reduces spectral distortion, and improves the model's inference speed and feature extraction ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918016A_ABST
    Figure CN118918016A_ABST
Patent Text Reader

Abstract

The invention relates to a hyperspectral image and multispectral image fusion method based on a multiscale visual attention network. Belongs to the field of computer vision and remote sensing image processing. The fusion of the multispectral image with high spatial resolution and the hyperspectral image with low spatial resolution aims to improve the spatial resolution of the hyperspectral image and can be better applied to downstream tasks. In order to solve the problem of image space and spectrum distortion caused by limitation of feature extraction capability in a current fusion method, an efficient fusion model composed of a multi-scale visual attention network is designed, the model combines a convolutional neural network and a self-attention mechanism, global and local features can be captured at the same time, and the fusion accuracy is improved. And spectral correlation in the image is maintained, and the fused image space and spectral quality are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for fusing hyperspectral images and multispectral images based on a multi-scale visual attention network, and belongs to the field of computer vision and remote sensing image processing. Background Art

[0002] As images widely used in the field of remote sensing, hyperspectral images and multispectral images play an important role in applications such as environmental monitoring, agricultural planning, and mineral exploration. However, due to the limitations of the imaging platform, acquisition equipment usually needs to make a compromise between spectral resolution and spatial resolution, and a single optical camera cannot capture hyperspectral images with high spatial resolution and high spectral resolution at the same time. Therefore, fusing hyperspectral images with multispectral images to enhance the spatial resolution of hyperspectral images has become a commonly used method.

[0003] Hyperspectral image and multispectral image fusion algorithms can be mainly divided into two categories, namely traditional machine learning methods and deep learning methods. Traditional machine learning methods use image prior information such as low rank, spectral correlation and other features, use matrix, tensor decomposition or Bayesian representation, and combine optimization algorithms to obtain the final fusion results. However, these traditional methods are limited by high computational complexity or suboptimal feature representation, which seriously affects their actual performance and applicability. Deep learning methods have excellent performance in the fusion of hyperspectral images and multispectral images. Compared with traditional machine learning methods, they have greatly improved accuracy and robustness. However, the fusion method based on convolutional neural networks has the disadvantages of small perception domain and limited generalization ability, which makes it difficult to construct long-range dependencies between features, seriously affecting the spatial quality of the fused image. The fusion method based on Transformer improves the quality of the fused image with its high-powered and spatial long-range correlation capture ability. However, the neglect of channel adaptability affects the spectral quality of the fused image, and spectral distortion is prone to occur. The high computational complexity brought by high-resolution image input also seriously affects the inference speed of the model.

[0004] In the task of fusion of hyperspectral images and multispectral images, the attention to spatial local and global information and the mining of spectral correlation information both affect the spatial and spectral quality of the fused image. The current method cannot efficiently mine the spatial and spectral information of the two types of images, resulting in a certain degree of spectral and spatial distortion in the fused image. In view of the above problems, the present invention proposes a multi-scale-based visual attention network for realizing high-quality fusion of hyperspectral images and multispectral images. The introduction of visual attention can efficiently mine the spatial and channel information of the image. Combined with the multi-scale mechanism, this method can make full use of the spatial features at different scales, effectively mine the global and local features, and realize the fusion of high-quality hyperspectral images and multispectral images. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned deficiencies of the prior art and provide a method for fusing hyperspectral images and multispectral images based on a multi-scale visual attention network. To this end, the present invention adopts the following technical solution.

[0006] The method for fusing a hyperspectral image with a multispectral image includes the following steps:

[0007] 1. Data processing and synthesis. Since there is no high-spatial-resolution hyperspectral image that can be directly used as a reference image, the present invention uses the original hyperspectral image data set as a reference image, and the width and height of the image are W and H respectively. On the basis of the above, Gaussian blur and n-fold downsampling methods are used successively to obtain a low-spatial-resolution hyperspectral image, whose width and height are W / n and H / n respectively. The multispectral image is composed of five bands taken from the original image at equal intervals, and its size is consistent with the size of the original image;

[0008] 2. Crop the reference image, hyperspectral image, and multispectral image into small-sized images in equal proportions. Ensure the correspondence between the three images, and randomly select the processed data as the training set and validation set in a certain proportion;

[0009] 3. Based on the visual attention network, combined with the multi-scale learning mechanism, design the network structure, determine the number of input node layers, output node layers, convolution kernel size, step size and other parameters of each layer of the neural network, give the learning rate, set the learning rate attenuation method, select the optimization function, and design a suitable loss function;

[0010] 4. Image upsampling: The low spatial resolution hyperspectral image is upsampled n times in the spatial dimension by interpolation to make its spatial resolution consistent with that of the multispectral image:

[0011] 5. Send the multispectral image and the upsampled hyperspectral image to the shallow feature encoding module at the same time;

[0012] 6. The feature maps of the multispectral image after shallow feature encoding and the upsampled hyperspectral image are passed to the deep feature extraction part composed of several joint attention modules;

[0013] 7. Perform feature-level connection on the obtained feature maps, and then pass the connected feature maps into the feature aggregation and image reconstruction module, add the original image, and generate the final output result;

[0014] 8. Calculate the loss of the output result and the reference image, and back-propagate to update the network parameters;

[0015] 9. Repeat steps 4 to 8 to train the specified number of epochs. Finally, the fusion model with the highest average peak signal-to-noise ratio (PSNR) of the test image is selected as the optimal image fusion model and saved for use.

[0016] 10. Input any low spatial resolution hyperspectral image and high spatial resolution multispectral image into the trained neural network model. The output of the model is the fused high spatial resolution hyperspectral image.

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 :Flowchart of hyperspectral and multispectral image fusion method based on multi-scale visual attention network

[0019] Figure 2 : Structure diagram of the multi-scale large-core attention module

[0020] Figure 3 : Structure diagram of the local spatial attention module

[0021] Figure 4 :Structural diagram of feature aggregation and image reconstruction module

[0022] Figure 5 : Fusion results of some test images

[0023] The beneficial effects of the present invention are:

[0024] 1. Multi-scale large-core attention mechanism: This paper integrates the multi-scale mechanism into the large-core attention and proposes a new multi-scale fusion architecture MLKA-Net, which can capture global and local features at the same time, thereby improving the feature extraction and characterization capabilities, flexibly absorbing the advantages of convolutional neural networks and white attention mechanisms, and effectively balancing the learning of non-local and local information.

[0025] 2. High fusion quality: The present invention can not only effectively improve the fine spatial and spectral details of the fused image, but also inject a spectral angle mapping loss term into the fusion process to reduce the image spectral distortion, so as to better maintain the original spectral internal correlation of the hyperspectral image. DETAILED DESCRIPTION

[0026] The following is a detailed description of the specific implementation steps of the method for fusion of hyperspectral images and multispectral images based on multi-scale visual attention provided by the present invention in conjunction with the accompanying drawings. The fusion process is performed in the following steps:

[0027] Step 1: Data processing and synthesis, using the original hyperspectral image dataset as the reference image, the width and height of the image are W and H respectively. Based on the above, the Gaussian blur method is first used to denoise and smooth the image, while blurring the image details. Then, n-fold spatial downsampling is performed to obtain a synthesized low spatial resolution hyperspectral image, whose width and height are W / n and H / n respectively. The multispectral image is composed of five bands taken from the original image at equal intervals, and its size is consistent with the size of the original image;

[0028] The Gaussian blur formula is as follows:

[0029]

[0030] Where (x, y) represents the pixel position in the image and σ represents the standard deviation of the Gaussian function.

[0031] Step 2: Crop the reference image, hyperspectral image, and multispectral image into small-sized images in equal proportions to ensure the correspondence between the three, and randomly sample the processed data in a certain proportion as the training set and test set;

[0032] Step 3: Based on the visual attention network, combined with the multi-scale learning mechanism, design the network structure, set the learning rate, set the learning rate decay method, select the optimization function, and design a suitable loss function. Figure 1 As shown: The network is mainly composed of three parts, namely shallow feature extraction part, deep feature extraction part, feature integration and image reconstruction part. The following is a detailed introduction

[0033] a. Shallow feature extraction is completed by combining convolutional layer, batch normalization layer and LeakyRelu activation function.

[0034] b. Deep feature extraction is performed by stacking multiple joint attention modules, which are composed of multi-scale large kernel attention modules and local spatial attention modules. Large Kernel Attention (LKA)

[0035] It integrates the advantages of self-attention mechanism and convolution, and can adaptively capture long-range dependencies and local texture information. To reduce the computational complexity and ensure dynamic feature extraction, the size of {Ki, D i The large convolution kernel of} is decomposed into three components: the kernel size is 2D i -1 depth convolution (Depth-Wise Convolution, DW-Conv), kernel size is K i / D iDepth-Wise-Dilation Convolution (DWD-Conv) and Point-WiseConvolution (PW-Conv). K i represents the convolution kernel size, D i In order to mine valuable spatial features, a convolution kernel size of K is introduced. i / D i Therefore, the calculation process of LKA is as follows:

[0036]

[0037] Among them, F represents the input feature, Conv dw 、Conv dwd 、Conv pw They represent depth convolution, depth expansion convolution and point convolution respectively. is the inner product operation.

[0038] Considering that features of different scales can represent rich information, we set different kernel sizes and expansion coefficients for large kernel convolution, integrate the multi-scale mechanism into LKA, and realize the learning and complementarity of multi-scale features. Figure 2 As shown in Figure 1, we first divide the input feature G into three groups of equal size according to the channel domain, represented by G1, G2, and G3. Then, we input them into their corresponding LKAs, using {K i , D i}, including {7, 2}, {21, 3} and {35, 4} respectively. The multi-scale mechanism enables our model to extract more diverse and informative features at different receptive scales, thereby effectively improving feature representation. The calculation process of the multi-scale large-core attention module is as follows:

[0039]

[0040] Among them, F is the input feature map, LN(·) is the linear normalization operation, Conv pw represents point convolution, LKA1, LKA2, and LKA3 represent large-core attention modules of three different scales. Concat(·) is a channel-level connection operation. is the inner product operation, λ is a learnable scale coefficient, and point convolution is used to maintain the number of channels of the feature map.

[0041] c. Design a local spatial attention module to extract spatial texture information and perform effective redundant filtering on the feature maps captured by the multi-scale large-core attention module.

[0042] like Figure 3As shown in the figure, based on the lightweight design, only a single deep convolution is used to generate the feature map. The specific process is as follows:

[0043]

[0044] Where F refers to the input feature, LN(·) is the linear normalization operation, is the inner product operation, Conv pw Represents point convolution, Conv dw It represents the depth convolution, which performs the inner product operation on the result of the depth convolution of F1 and F2, and finally uses the point convolution to adjust the number of channels.

[0045] d. Design feature aggregation and image reconstruction modules to filter out redundant information, aggregate effective information, and achieve high-quality image reconstruction. Figure 4 As shown in Figure 2, this module is also built on the large kernel attention mechanism, which can effectively integrate spectral and spatial information and improve the quality of reconstructed images. The calculation process is as follows:

[0046]

[0047] Among them, Conv dw 、Conv dwd 、Conv pw . represent depthwise convolution, depthwise dilated convolution, and pointwise convolution, respectively. GELU(·) represents Gaussian error linear unit as the activation function.

[0048] e. The loss function consists of two terms, including L1 and L sam , L1 is used to calculate the spatial loss between the reconstructed image and the reference image, L sum Used to calculate the spectral loss between the reconstructed image and the reference image. The combination of the two loss terms improves the image spatial quality while maintaining the image spectral quality. The loss function is as follows:

[0049] L(θ)=L1+λL sam (6)

[0050]

[0051]

[0052] Where X is the generated high spatial resolution hyperspectral image, X ref is the reference high spatial resolution hyperspectral image. λ is a non-negative weight coefficient, and its value is 0.03. ref(i, j) and X(i, j) represent the reference high spatial resolution hyperspectral image and the generated high spatial resolution hyperspectral image pixel vector at the specified position (i, j), respectively. In addition, <·, ·> represents the inner product function, ||·||, ||·||2 are the L1 norm and L2 norm, respectively, arccos(·) represents the inverse cosine trigonometric function, and H and W are the height and width of the image.

[0053] Step 4: Image upsampling: The low spatial resolution hyperspectral image is upsampled n times in the spatial dimension by interpolation to make its spatial resolution consistent with that of the multispectral image.

[0054] Step 5: Send the multispectral image and the upsampled hyperspectral image to the shallow feature encoding module to perform shallow feature extraction.

[0055] Step 6: The feature maps of the multispectral image after shallow feature extraction and the upsampled hyperspectral image are passed to the deep feature extraction part composed of several joint attention modules.

[0056] Step 7: Perform feature-level concatenation on the deep feature maps extracted from both. Then, the concatenated feature maps are passed to the feature aggregation and image reconstruction module, and added with the original image to generate the final output result.

[0057] Step 8: Calculate the loss of the output result and the reference image, and back-propagate to update the network parameters.

[0058] Step 9: Repeat steps 4 to 8 to train for the specified number of epochs. Finally, select the image fusion model with the highest PSNR value of the test image as the optimal model and save it for use.

[0059] In the field of remote sensing image fusion, some performance metrics are needed to judge the quality of a model. The most commonly used ones are PSNR, Root Mean Square Error (RMSE), and Spectral Angle Mapper (SAM). The formula is as follows

[0060]

[0061]

[0062]

[0063] Among them, X and X represent the generated image and the reference image respectively, max(·) represents the maximum function, arccos(·) represents the inverse cosine trigonometric function, and L, H, and W represent the number of image bands, height, and width respectively. The higher the PSNR value, the better the quality of the fused image, and the smaller the RMSE and SAM values, the higher the quality of the fused image.

[0064] Step 10: Input any low spatial resolution hyperspectral image and a corresponding high spatial resolution multispectral image into the trained neural network model. The output of the model is the fused high spatial resolution hyperspectral image.

[0065] This paper proposes a hyperspectral and multispectral image fusion method based on a multi-scale visual attention network. This method efficiently characterizes the spatial and spectral features of hyperspectral images and multispectral images through multi-scale large kernel attention. By learning long-range correlations and local granular information at different scales, the spatial and spectral quality of the fused hyperspectral image is improved.

[0066] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The present invention relates to a method for fusing hyperspectral images and multispectral images based on a multi-scale visual attention network. The method comprises the following steps: Step 1: Data processing and synthesis. Since there is no high-spatial-resolution hyperspectral image that can be directly used as a reference image, the present invention uses the original hyperspectral image data set as a reference image, and the width and height of the image are W and H respectively. On the basis of the above, Gaussian blur and n-fold downsampling methods are used successively to obtain a low-spatial-resolution hyperspectral image, and its width and height are W / n and H / n respectively. The multispectral image is composed of five bands taken from the original image at equal intervals, and its size is consistent with the original image. Step 2: Crop the reference image, hyperspectral image, and multispectral image into small-sized images in equal proportions to ensure the correspondence between the three images, and randomly sample the processed data in a certain proportion as the training set and validation set. Step 3: Based on the visual attention network and combined with the multi-scale learning mechanism, design the network structure, determine the number of input node layers, output node layers, convolution kernel size, step size and other parameters of each layer of the neural network, give the learning rate, set the learning rate decay method, select the optimization function, and design a suitable loss function. Step 4: Image upsampling: The low spatial resolution hyperspectral image is upsampled n times in the spatial dimension by interpolation to make its spatial resolution consistent with that of the multispectral image. Step 5: Send the multispectral image and the upsampled hyperspectral image to the shallow feature encoding module simultaneously. Step 6: The feature maps of the multispectral image after shallow feature encoding and the upsampled hyperspectral image are passed to the deep feature extraction part composed of several joint attention modules. Step 7: Perform feature-level connection on the obtained feature maps, and then pass the connected feature maps into the feature aggregation and image reconstruction module, add the original image, and generate the final output result. Step 8: Calculate the loss of the output result and the reference image, and back-propagate to update the network parameters. Step 9: Repeat steps 4 to 8 to train for a specified number of epochs. The update of the fusion model is mainly based on the Peak Signal-to-Noise Ratio (PSNR), and finally the optimal image fusion model is obtained. Step 10: Input any low spatial resolution hyperspectral image and high spatial resolution multispectral image into the trained neural network model. The output of the model is the fused high spatial resolution hyperspectral image.

2. The method for fusion of hyperspectral image and multispectral image based on multi-scale visual attention network according to claim 1, characterized in that: Data processing and synthesis, using the original hyperspectral image dataset as the reference image, the width and height of the image are W and H respectively. On the basis of the above, the Gaussian blur method is first used to denoise and smooth the image, while blurring the image details. Then, n-fold spatial downsampling is performed to obtain a synthesized low spatial resolution hyperspectral image, whose width and height are W / n and H / n respectively. The multispectral image is composed of five bands taken from the original image at equal intervals, and its size is consistent with the size of the original image; The Gaussian blur formula is as follows: Where (x, y) represents the pixel position in the image and σ represents the standard deviation of the Gaussian function.

3. The method for fusion of hyperspectral image and multispectral image based on multi-scale visual attention network according to claim 1, characterized in that: The deep feature extraction part consists of stacking multiple joint attention modules, which are composed of a multi-scale large core attention module and a local space attention module. The multi-scale large core attention module efficiently integrates the multi-scale mechanism with the large core attention module, enabling the model to extract richer and more diverse spatial features at different spatial scales. The process is as follows: Among them, F is the input feature map, Conv pw represents point convolution. LN(·) is a linear normalization operation. LKA1, LKA2, and LKA3 represent large-core attention modules of three different scales. Concat(·) is a channel-level connection operation. is the inner product operation, λ is a learnable scale coefficient, and point convolution is used to maintain the number of channels of the feature map. The local spatial attention module is designed to extract spatial texture information and effectively filter the redundant feature maps captured by the multi-scale large-core attention module. The process is as follows: Among them, F refers to the input feature, Conv dw 、Conv pw denote depth convolution and point convolution respectively, LN(·) is the linear normalization operation, It is an inner product operation. The result of the depthwise convolution of F1 is inner-producted with F2, and finally the number of channels is adjusted using point convolution.

4. The method for fusion of hyperspectral image and multispectral image based on multi-scale visual attention network according to claim 1, characterized in that: The feature aggregation and image reconstruction module is designed to filter out redundant information, aggregate valuable information, and achieve high-quality image reconstruction. The specific process is as follows: Among them, F represents the input feature, Conv dw 、Conv dwd 、Conv pw They represent depthwise convolution, depthwise dilated convolution, and pointwise convolution, respectively. GELU(·) represents Gaussian error linear unit as activation function. is the inner product operation.

5. The method for fusion of hyperspectral image and multispectral image based on multi-scale visual attention network according to claim 1, characterized in that: The loss function used in the model training process is composed of L1 and L sam L1 is used to calculate the overall spatial loss between the reconstructed image and the reference image, and L sam It is used to calculate the spectral loss between the reconstructed image and the reference image. This loss function can effectively constrain the training of the model, improve the spatial quality of the fused image, and maintain the spectral quality of the fused image. The specific details are as follows: L(θ)=L1+λL sam (5) Where X is the generated high spatial resolution hyperspectral image, X ref is the reference high spatial resolution hyperspectral image. λ is a non-negative weight coefficient, and its value is 0.

03. ref (i, j) and X(i, j) represent the reference high spatial resolution hyperspectral image and the generated high spatial resolution hyperspectral image pixel vector at the specified position (i, j), respectively. In addition, <·, ·> represents the inner product function, ||·||, ||·||2 are the L1 norm and L2 norm, respectively, arccos(·) represents the inverse cosine trigonometric function, and H and W are the height and width of the image.

6. The method for fusion of hyperspectral image and multispectral image based on multi-scale visual attention network according to claim 1, characterized in that: When training the model, the average PSNR value of the test image is used as an indicator. The model with the highest PSNR value of the test image is selected as the optimal model and saved for use.

Citation Information

Patent Citations

  • Hyper-spectral and panchromatic image fusion method for extracting multi-scale spatial-spectral characteristics based on AE

    CN112634137A

  • Remote sensing image fusion method of multi-scale attention deep convolutional network based on 3D convolution

    CN112819737A

  • Multi-scale and global feature hyperspectral and multispectral remote sensing fusion method

    CN115861083A

  • Hyperspectral and multispectral image fusion method based on attention mechanism

    CN117474781A

Cited By

  • Hyperspectral and multispectral image fusion method based on adaptive multi-scale features

    CN120047326A

  • Hyperspectral and Multispectral Image Fusion Method Based on Adaptive Multi-Scale Features

    CN120047326B

  • Hyperspectral and multispectral image fusion method based on heterogeneous double-branch collaboration

    CN120526278A

  • Hyperspectral and multispectral image fusion method based on heterogeneous dual-branch collaboration

    CN120526278B