Hyperspectral image fusion method based on dual-resolution residual cross-guided network

Abstract

This invention provides a hyperspectral image fusion method based on a dual-resolution residual mutual guidance network, comprising: Step 1, establishing a training dataset of hyperspectral images and visible light images; downsampling all visible light images in the training dataset and forming low-resolution data pairs with the hyperspectral images in the training dataset; upsampling all hyperspectral images in the training dataset and forming high-resolution data pairs with the visible light images in the training dataset; Step 2, obtaining the fusion result of the hyperspectral images and visible light images in the high-resolution and low-resolution data pairs datasets through a dual-resolution residual mutual guidance network model; Step 3, training the dual-resolution residual mutual guidance network model using the training dataset using a stochastic gradient descent algorithm to obtain a fusion model. Finally, testing on data in a test dataset shows that this invention effectively improves the final image fusion result.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of information technology, relates to image processing technology, and particularly relates to a hyperspectral image fusion method based on a dual-resolution residual mutual guidance network. BACKGROUND

[0002] With the development of optical imaging technology, a hyperspectral image (HS) with tens or even hundreds of spectral bands can be obtained. Although the hyperspectral image contains rich spectral information, compared with a visible light image (RGB), the spatial resolution of the hyperspectral image is much lower because the photon collection in the hyperspectral sensor needs to be performed in a larger spatial area to ensure a high enough signal-to-noise ratio. The low spatial resolution in the hyperspectral image leads to the mixing of different materials in the scene, which greatly affects the performance of scene analysis and understanding. Therefore, how to combine the high spectral resolution of the hyperspectral image with the high spatial resolution of the visible light image to obtain a hyperspectral image with high spatial and spectral resolution is a problem that needs to be solved at present. In the past few years, a convolutional neural network (CNN) with high nonlinearity has been widely applied to the field of image processing. In the aspect of hyperspectral image fusion, a dual-branch structure based on the convolutional neural network has achieved remarkable fusion effect. Such a dual-branch structure is mostly designed to extract spatial and spectral features respectively, but such a structure mostly performs up-sampling processing on the original hyperspectral image before extracting the spatial and spectral features respectively, which may lose part of the spectral features of the original hyperspectral image. In addition, the existing dual-branch fusion structure inevitably affects the spectral features and spatial features in the process of fusing two features. SUMMARY

[0003] In view of this, the application provides a hyperspectral image fusion method based on a dual-resolution residual mutual guidance network to solve the problem that the existing dual-branch structure may cause the loss of spectral information of the original hyperspectral image in the feature extraction network. The specific scheme is as follows:

[0004] A hyperspectral image fusion method based on a dual-resolution residual mutual guidance network, comprising the following steps:

[0005] Step one, first, a data set of hyperspectral images and visible light images is established; then all the visible light images in the data set are down-sampled and combined with the hyperspectral images in the training data set to form a low-resolution data pair; then all the hyperspectral images in the data set are up-sampled and combined with the visible light images in the training data set to form a high-resolution data pair;

[0006] Step two, a dual-resolution residual mutual guidance network model DRRMGN is used to obtain the fusion result of the hyperspectral images and the visible light images in the data set of the high-resolution data pair and the low-resolution data pair.

[0007] Step three, using the random gradient descent algorithm, using the training data set to train the dual-resolution residual mutual guidance network model to obtain a fusion model.

[0008] Preferably, the hyperspectral images and visible light images in the training data set are multiple pairs of hyperspectral images and visible light images of multiple scenes;

[0009] The hyperspectral image in the low-resolution data pair is the same size as the hyperspectral image in the training data set.

[0010] Preferably, the method of upsampling and downsampling processing comprises the following steps:

[0011] Step 3.1, using an interpolation method to downsample the visible light image R or in the training data set by an interval of 4, and the downsampled visible light image is represented as R down ∈R h×w×3 ; wherein h and w represent the length and width of the spatial dimension, respectively, and 3 represents the number of spectral bands; the downsampled visible light image R down and the hyperspectral image H or ∈R h×w×c in the training data set form a low-resolution image pair; the downsampled visible light image is represented as formula (1)

[0012] R down = R or ↓, R down ∈R h×w×3 formula (1)

[0013] wherein ↓ represents a downsampling operation;

[0014] Step 3.2, using bilinear interpolation to upsample the hyperspectral image H or in the training data set by a scaling factor p = 4 to obtain a high-resolution hyperspectral image H up ∈R H×W×c , wherein H and W represent the length and width of the spatial dimension, respectively, and c represents the number of spectral bands; the hyperspectral image H up and the visible light image R or in the training set form a high-resolution image pair, and the upsampled hyperspectral image is represented as formula (2):

[0015] H up = H or ↑, H up ∈R H×W×c formula (2)

[0016] wherein ↑ represents an upsample operation.

[0017] Preferably, the dual-resolution residual mutual guidance network model comprises feature extraction, feature fusion and image reconstruction.

[0018] Preferably, the specific process for obtaining the fusion result of the hyperspectral image and the visible light image is as follows:

[0019] Step 5.1, spatial and spectral features of the high-resolution image pair and the low-resolution image pair are extracted by the feature extraction network Φ H and Φ L respectively, and after feature extraction, a first group of feature maps Φ HH and Φ HR and a second group of feature maps Φ LH and Φ LR are obtained respectively.

[0020] Φ HH = Φ HH (H up )

[0021] Φ HR = Φ HR (R or )

[0022] Φ LH = Φ LH (H or )

[0023] Φ LR = Φ LR (R down )

[0024] wherein Φ HH and Φ HR and Φ LH and Φ LR represent two sub-networks of Φ H and Φ L , (.) represents a convolution operation, H up represents an image obtained by four times up-sampling of the hyperspectral image in the training set, R or represents a visible light image in the training set, H or represents a hyperspectral image in the training set, R down represents an image obtained by four times down-sampling of the visible light image in the training set.

[0025] Step 5.2, the first group of feature maps Φ HH and Φ HR and the second group of feature maps Φ LH and Φ LR ​The feature maps are connected together and then preliminarily fused using a first-level fusion network (FFN) to form compact features containing both spatial and spectral information and Specifically, as shown in formula (3):

[0026]

[0027]

[0028] wherein, represents the compact features of the branch of the high-resolution data in the FFN network, represents the compact features of the branch of the low-resolution data in the FFN network, represents the feature fusion function of the FFN sub-network of the branch of the high-resolution data, represents the feature fusion function of the FFN sub-network of the branch of the low-resolution data, and "+" represents a series operation;

[0029] Step 5.3, the compact features and are respectively sent into a second-level fusion network SFN, the relationship between the compact features and is established through residual mutual guidance blocks, and two new feature maps and are generated by learning a more complex combination between spatial resolution and spectral resolution, which is represented by formula (4):

[0030]

[0031]

[0032] wherein, represents the feature map of the branch of the high-resolution data in the SFN network, represents the compact features of the branch of the low-resolution data in the FFN network, represents the feature fusion function of the SFN sub-network of the branch of the high-resolution data, represents the feature fusion function of the SFN sub-network of the branch of the low-resolution data;

[0033] Step 5.4, in the preliminary fusion process of step 5.3, the required high-resolution hyperspectral image is recovered, and a transpose convolution is used to reconstruct the fusion result of the hyperspectral image and the visible light image , original hyperspectral image and visible light image and image information after four times up-sampling of the original hyperspectral image are introduced into the reconstruction network in a hierarchical manner through a skip connection, which is represented by formula (5):

[0034]

[0035] Where f1(.), f2(.), and f3(.) represent convolution operations, and "+" represents concatenation operations. The feature map representing the branch formed by high-resolution data in the SFN network. This represents the compact characteristics of low-resolution data pairs forming branches in an FFN network.

[0036] Step 5.5, replace the ordinary convolutional neural network units with residual convolutional neural network units, wherein the general form of each residual unit is expressed as Equation (6):

[0037] x l =h(y l )+R(y l w l )

[0038] y l+1 =f(x) l Equation (6)

[0039] Where y l and y l+1 Represents the input and output of the l-th residual unit, R(.) represents the residual function, and w l f(x) represents the network weight of the l-th residual unit. l ) is the activation function and h(y) l ) is the feature mapping function.

[0040] Preferably, the feature extraction network Φ H and Φ L Each contains the first set of subnetworks Φ HH or Φ LH and the second set of subnetworks Φ HR or Φ LR ;

[0041] The first group of sub-networks Φ HH or Φ LH With the second group of sub-networks Φ HR or Φ LR With the same structure but different weights, the first group of subnetworks Φ HH or Φ LH With the second group of sub-networks Φ HR or Φ LR One of them takes a multi-band hyperspectral image as input, and the other takes a three-band visible light image as input.

[0042] Preferably, the hybrid loss function in the fusion model includes:

[0043] Step 7.1, in the function shown in the main network, the L1-norm loss function and SAM are selected to be jointly constrained, and the main loss function of the main network is shown in formula (7):

[0044]

[0045] Wherein l1 represents the loss function, SAM represents the constraint function, represents the spectral vector of the reference image in the spectral dimension, represents the spectral vector of the result image in the spectral dimension.

[0046] Step 7.2, in the resolution loss function, the L1-norm loss function and MSE are selected to be jointly constrained, and the resolution loss function is formula (8):

[0047]

[0048]

[0049] Wherein l1 represents the loss function, MSE represents the constraint function.

[0050] Finally, according to steps 7.1 and 7.2, the loss function formula (9) and formula (10) shown in the fusion model are constructed:

[0051]

[0052]

[0053] Wherein, L H represents the comprehensive loss function of the branch for high-resolution data, L L represents the comprehensive loss function of the branch for low-resolution data; y represents the reference image, represents the fusion result of the high light image and the visible light image; y1, y2 respectively represent the output feature maps of two groups of different resolution data in the second level fusion network, λ1, λ2 represent the weight of the resolution mutual guidance loss function, L r is the resolution mutual learning loss function, L M is the network main loss function, represents the spectral vector of the reference image in the spectral dimension, represents the spectral vector of the result image in the spectral dimension.

[0054] Step 7.3, in the process of training the network under the stochastic gradient descent algorithm, the loss function is continuously optimized loss until convergence.

[0055] Preferably, in step 7.3, the process of optimizing loss specifically includes:

[0056] In the training data set, p samples are selected to form a small batch of samples, and then in each iteration:

[0057] The random gradient descent for the branch of high-resolution data pairs is represented as formula (11):

[0058]

[0059] wherein, is the loss function L constructed in steps 7.1 and 7.2 H , where m represents p samples, and then the model is updated by gradient descent:

[0060]

[0061] Here, is the partial derivative of the loss function with respect to the parameter θ i , and α is the learning rate set for the model. After updating the model by gradient descent, formula (11) can be simplified as formula (12):

[0062]

[0063] In formula (12), represents the gradient of the loss function, and in each model update, a sample is randomly selected to update the parameters;

[0064] At the same time, the random gradient descent for the branch of low-resolution data pairs is as formula (13):

[0065]

[0066] wherein, is the loss function L constructed in steps 7.1 and 7.2 L , and then the network is updated by gradient descent similar to the branch of high-resolution data pairs.

[0067] Preferably, a hybrid loss function is designed in the process of training the network by random gradient descent algorithm, and the loss function is continuously optimized in the training until convergence, wherein the loss function is as formula (14),

[0068]

[0069]

[0070] wherein, L H represents the comprehensive loss function of the branch of high-resolution data pairs, L L represents the comprehensive loss function of the branch of low-resolution data pairs; y represents the reference image, represent the fusion result of hyperspectral image and visible light image; y1, y2 respectively represent the output features of two groups of different resolution data pairs in the second level fusion network, λ1, λ2 represent the weights of the resolution mutual guidance loss function, L r is the resolution mutual guidance loss function, L M is the network main loss function.

[0071] Compared with the prior art, the application has the beneficial effects that:

[0072] In the process of obtaining the fusion result of the hyperspectral image and the visible light image , the feature extraction networks Φ H and Φ L are used to extract the spatial and spectral features of the high-resolution image pair and the low-resolution image pair respectively, so as to retain the spectral information of the original hyperspectral image and the spatial information of the visible light image as much as possible, and make up for the problem that the existing double-branch structure may lose part of the spectral features of the original low-resolution hyperspectral image.

[0073] 2. In the process of sending the compact features and into the second level fusion network SFN for fusion respectively, a new residual mutual guidance block RMGB is explored, which establishes a connection between the fusion features of two different resolutions, so as to learn more complex combinations between the spatial resolution and the spectral resolution, and generate two new feature maps and The two branches of different resolutions can learn the features of each other.

[0074] 3. The application introduces the reconstructed network by hierarchical jump connection of the original hyperspectral image and the visible light image, and the image information after four times up-sampling of the original hyperspectral image, that is, multi details are injected, and the original information of the hyperspectral and visible light images of different resolutions is introduced into the image reconstruction network, so that the quality of the reconstructed image is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0075] The accompanying drawings are used to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application.

[0076] In the drawings:

[0077] Figure 1 is the framework diagram of the hyperspectral image fusion method DRRMGN based on the double-resolution residual mutual guidance network of the application;

[0078] Figure 2A structural diagram of a residual mutual guidance block RMGB in a dual-resolution residual mutual guidance network-based hyperspectral image fusion method of the present application;

[0079] Figure 3 Fusion results of different fusion methods on a "balloon" image in CAVE data in an embodiment simulation experiment of the dual-resolution residual mutual guidance network-based hyperspectral image fusion method of the present application;

[0080] Figure 4 Fusion results of different fusion methods on a "real and fake pepper comparison" image in CAVE data in an embodiment simulation experiment of the dual-resolution residual mutual guidance network-based hyperspectral image fusion method of the present application;

[0081] Wherein, 1-convolution layer, 2-PReLU, 3-transposed convolution layer. DETAILED DESCRIPTION

[0082] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0083] A dual-resolution residual mutual guidance network-based hyperspectral image fusion method, comprising the following steps:

[0084] Step one, first, a data set of hyperspectral images and visible light images is established; then all visible light images in the data set are down-sampled and combined with hyperspectral images in the training data set to form low-resolution data pairs; then all hyperspectral images in the data set are up-sampled and combined with visible light images in the training data set to form high-resolution data pairs;

[0085] Step two, a dual-resolution residual mutual guidance network model DRRMGN is used to obtain fusion results of hyperspectral images and visible light images in the data set of the high-resolution data pairs and the low-resolution data pairs

[0086] Step three, a random gradient descent algorithm is used to train the dual-resolution residual mutual guidance network model using the training data set to obtain a fusion model.

[0087] It should be noted that in step one, the data set includes multiple pairs of hyperspectral images and visible light images of multiple scenes, wherein one sample in the data set is a pair of hyperspectral images and visible light images of the same scene and same angle;

[0088] The dual-resolution residual mutual guidance network model of step two is trained using the training data set until convergence is achieved to obtain a fusion model, and a hybrid loss function is designed in the process of training the network under the stochastic gradient descent algorithm, and the loss function is continuously optimized in the training until convergence is achieved.

[0089] Further, the hyperspectral image and the visible light image in the training data set are a plurality of pairs of hyperspectral images and visible light images of multiple scenes;

[0090] The hyperspectral image in the low-resolution data pair has the same size as the hyperspectral image in the training data set.

[0091] Further, the up-sampling and down-sampling processing method comprises the following steps:

[0092] Step 3.1, using an interpolation method to scale the visible light image R or in the training data set by a factor of 4 to obtain a down-sampled visible light image R down ∈R h×w×3 ; wherein h and w represent the length and width of the spatial dimension, respectively, and 3 represents the number of spectral bands; the down-sampled visible light image R down and the hyperspectral image H or ∈R h×w×c in the training data set form a low-resolution image pair; the down-sampled visible light image is represented by formula (1)

[0093] R down = R or ↓, R down ∈R h×w×3 formula (1);

[0094] wherein ↓ represents a down-sampling operation.

[0095] Step 3.2, using bilinear interpolation to scale the hyperspectral image H or in the training data set by a factor of p = 4 to obtain a high-resolution hyperspectral image H up ∈R H×W×c , wherein H and W represent the length and width of the spatial dimension, respectively, and c represents the number of spectral bands; the hyperspectral image H up and the visible light image R or in the training set form a high-resolution image pair, and the up-sampled hyperspectral image is represented by formula (2):

[0096] H up = H or ↑, H up ∈R H×W×c formula (2)

[0097] wherein ↑ represents an up-sampling operation.

[0098] Further, the dual-resolution residual mutual guidance network model comprises feature extraction, feature fusion and image reconstruction.

[0099] Further, the specific process for obtaining the fusion result of the hyperspectral image and the visible light image is as follows:

[0100] Step 5.1, spatial and spectral features of the high-resolution image pair and the low-resolution image pair are extracted by the feature extraction network Φ H and Φ L respectively, and after feature extraction, a first group of feature maps Φ HH and Φ HR and a second group of feature maps Φ LH and Φ LR are obtained respectively.

[0101] Φ HH = Φ HH (H up )

[0102] Φ HR = Φ HR (R or )

[0103] Φ LH = Φ LH (H or )

[0104] Φ LR = Φ LR (R down )

[0105] wherein Φ HH and Φ HR and Φ LH and Φ LR represent two sub-networks of Φ H and Φ L , (.) represents convolution operation, H up represents an image obtained by four times up-sampling of the hyperspectral image in the training set, R or represents the visible light image in the training set, H or represents the hyperspectral image in the training set, R down represents an image obtained by four times down-sampling of the visible light image in the training set.

[0106] Step 5.2, the first group of feature maps Φ HH and Φ HR and the second group of feature maps Φ LH and Φ LR ​The feature maps are concatenated together and then preliminary fused using a first fusion network FFN to form compact features containing both spatial and spectral information and Specifically, as shown in formula (3):

[0107]

[0108]

[0109] wherein, represents the compact features formed by the high-resolution data branch in the FFN network, represents the compact features formed by the low-resolution data branch in the FFN network, represents the feature fusion function of the FFN sub-network of the high-resolution data branch, represents the feature fusion function of the FFN sub-network of the low-resolution data branch, and "+" represents a series operation;

[0110] Step 5.3, the compact features and are sent into a second fusion network SFN, and a new residual mutual guidance block is explored in this module. The residual mutual guidance block establishes the connection between the compact features and of two different resolutions, and generates two new feature maps and by learning more complex combinations between spatial resolution and spectral resolution, which is expressed as formula (4):

[0111]

[0112]

[0113] wherein, represents the feature maps formed by the high-resolution data branch in the SFN network, represents the compact features formed by the low-resolution data branch in the FFN network, represents the feature fusion function of the SFN sub-network of the high-resolution data branch, represents the feature fusion function of the SFN sub-network of the low-resolution data branch;

[0114] Step 5.4, the high-resolution hyperspectral image required in the preliminary fusion process of step 5.3 is recovered, and a transpose convolution is used to reconstruct the fusion result of the hyperspectral image and the visible light image Meanwhile, in order to restore fine and realistic details, the original hyperspectral image, the visible light image, and the image information after four times upsampling of the original hyperspectral image are introduced into the reconstruction network in a hierarchical manner through skip connections, which is expressed as Equation (5):

[0115]

[0116] Where f1(.), f2(.), and f3(.) represent convolution operations, and "+" represents concatenation operations. The feature map representing the branch formed by high-resolution data in the SFN network. This represents the compact characteristics of low-resolution data pairs forming branches in an FFN network.

[0117] Step 5.5, replace the ordinary convolutional neural network units with residual convolutional neural network units, wherein the general form of each residual unit is expressed as Equation (6):

[0118] x l =h(y l )+R(y l w l )

[0119] y l+1 =f(x) l Equation (6)

[0120] Where y l and y l+1 Represents the input and output of the l-th residual unit, R(.) represents the residual function, and w l f(x) represents the network weight of the l-th residual unit. l ) is the activation function and h(y) l ) is the feature mapping function.

[0121] It should be noted that, because high-resolution hyperspectral images must have both spatial and spectral high resolution, their features must capture both spectral and spatial information simultaneously. Therefore, the two feature maps obtained from each feature extraction network in step 5.1 are concatenated together; then, a first-level fusion network (FFN) is used for preliminary fusion to form a compact feature map that simultaneously contains both spatial and spectral information. and

[0122] Furthermore, the feature extraction network Φ H and Φ L All contain the first set of subnetworks Φ HH or Φ LH and the second set of subnetworks Φ HR or Φ LR ;

[0123] The first group of sub-networks Φ HH or Φ LH The second group of sub-networks Φ HR or Φ LR The structure is consistent, the weight is different, the first group of sub-networks Φ HH or Φ LH The second group of sub-networks Φ HR or Φ LR One of them takes multi-band hyperspectral image as input, and the other takes three-band visible light image as input.

[0124] Further, the hybrid loss function in the fusion model includes:

[0125] Step 7.1, in the function of the main network, L1-norm loss function and SAM are selected to be jointly constrained, and the main loss function of the main network is shown in formula (7):

[0126]

[0127] Wherein l1 represents the loss function, SAM represents the constraint function, represents the spectral vector of the reference image in the spectral dimension, represents the spectral vector of the result image in the spectral dimension;

[0128] Step 7.2, in the resolution loss function, L1-norm loss function and MSE are selected to be jointly constrained, and the resolution loss function is formula (8):

[0129]

[0130]

[0131] Wherein l1 represents the loss function, and MSE represents the constraint function.

[0132] Finally, according to steps 7.1 and 7.2, the loss function specific to the fusion model is constructed as shown in formula (9) and formula (10):

[0133]

[0134]

[0135]

[0136] Wherein, L H represents the comprehensive loss function of the branch for high-resolution data, L L represents the comprehensive loss function of the branch for low-resolution data; y represents the reference image, The fusion result of the hyperspectral image and the visible light image is represented; y1 and y2 represent the output feature maps of the second-level fusion network for two groups of different resolution data, respectively, and λ1 and λ2 represent the weights of the resolution mutual guidance loss function, and L r is the resolution mutual learning loss function, and LM is the network main loss function, represents the spectral vector of the reference image in the spectral dimension, represents the spectral vector of the result image in the spectral dimension.

[0137] Step 7.3, the loss function constructed according to steps 7.1 and 7.2 is continuously optimized in the process of training the network under the random gradient descent algorithm until convergence.

[0138] Further, in step 7.3, the process of optimizing the loss specifically includes: selecting p samples in the training data set to form a small batch of samples, here p = 32 is selected, and then in each iteration:

[0139] The random gradient descent for the high-resolution data pair branch is represented as formula (11):

[0140]

[0141] wherein, is the loss function L constructed in steps 7.1 and 7.2 H , and m represents the p samples, and then the gradient descent method is used to update the model:

[0142]

[0143] Here, is the partial derivative of the loss function with respect to the parameter θ i , and α is the learning rate set by the model, and after updating the model by the gradient descent method, formula (11) can be simplified as formula (12):

[0144]

[0145] In formula (12), represents the gradient of the loss function, and in each model update, a sample is randomly taken to update the parameters,

[0146] At the same time, the random gradient descent for the low-resolution data pair branch is as formula (13):

[0147]

[0148] wherein, is the loss function L constructed in steps 7.1 and 7.2 LAfter that, the gradient descent method is used to update the network similar to the branch of high resolution data.

[0149] In the fusion feature of step 7.3, the required high resolution hyperspectral image is recovered, and the transpose convolution is used to reconstruct the fusion result of the hyperspectral image and the visible light image At the same time, in order to restore fine and real details, the original hyperspectral image and visible light image, and the image information after four times upsampling of the original hyperspectral image are introduced into the reconstruction network through hierarchical jump connection.

[0150] Further, in the process of training the network by the stochastic gradient descent algorithm, a hybrid loss function is designed, and the loss function is continuously optimized in the training until convergence, wherein the loss function is shown in formula (14),

[0151]

[0152]

[0153] Wherein, L H represents the comprehensive loss function of the high resolution data branch, L L represents the comprehensive loss function of the low resolution data branch; y represents the reference image, represents the fusion result of the hyperspectral image and the visible light image; y1 and y2 respectively represent the output features of two groups of different resolution data pairs in the second level fusion network, λ1 and λ2 represent the weights of the resolution mutual guidance loss function, L r is the resolution mutual guidance loss function, and L M is the network main loss function.

[0154] In the present application, the hyperspectral resolution in the hyperspectral image and the high spatial resolution in the visible light image are fused to obtain a hyperspectral image with high spatial resolution and spectral resolution. First, two feature extraction networks with double branch structure are used to extract spectral and spatial information of the hyperspectral image and the visible light image of two different resolution pairs respectively. Then, the spatial and spectral information extracted by each feature extraction network is preliminarily fused to form compact features containing spatial and spectral information. Then a new residual mutual guidance block is explored, which establishes a connection between the fusion features of two different resolutions to learn more complex combinations between spatial resolution and spectral resolution. Finally, the desired high spatial resolution hyperspectral image is recovered from the fusion features through the image reconstruction network. At the same time, in order to restore fine and real details, the original information of the hyperspectral image and the visible light image of different resolutions is introduced into the image reconstruction network, which effectively improves the fusion result of the final image.

[0155] To verify the fusion result of the present application, the effectiveness of the DRRMGN image fusion method proposed in the present application is verified by simulation as follows:

[0156] First, in one embodiment of the present application, a set of balloon contrast images and a set of real and fake pepper contrast images are fused by the DRRMGN image fusion method proposed in the present application, and the specific process is as follows:

[0157] The hyperspectral data-CAVE data set is used to verify the effectiveness of the fusion algorithm. The hyperspectral image in the CAVE data set contains balloons or real and fake peppers and objects in an indoor environment with controlled lighting. Each balloon or real and fake pepper image contains 31 bands, with a wavelength range of 400-700 nm, a resolution of 512x512 pixels, and a resolution of 10 nm per band. The resolution of the balloon or real and fake pepper hyperspectral image and the RGB image of the balloon or real and fake pepper is reduced by 4 times, and the balloon or real and fake pepper hyperspectral image is used as the standard fusion result for reference. In the training, 60% of the total data is selected as the training set, and the remaining data is used as the test set, and finally the balloon or real and fake pepper fusion result image is obtained. In this embodiment, the Pytorch software package is used to train the network, about 15000 iterations are performed, and the batch size is set to 32. For the stochastic gradient descent algorithm, the weight decay is set to 10 -3 , and the momentum is set to 0.9.

[0158] Finally, the fusion result of the simulation experiment is analyzed:

[0159] Figure Figure 3 is the fusion result of the "balloon" image in the CAVE data using different fusion methods in the simulation experiment; in figure Figure 3 (a)-(f) are the fusion result images of the six comparison methods, where (a) is CNMF, (b) is Bayesian naive, (c) is GFPCA, (d) is MSDCNN, (e) is SSFCNN, (f) is ResTFNET, (g) is the fusion image of the DRRMGN method proposed in the present application, and (h) is the reference image; by comparing Figure 3The intuitive feeling of the fusion image and the difference value graph of the reference image can be compared, and it can be seen that all the methods from (a) to (f) can improve the spatial resolution of the original hyperspectral image, but it is obvious that the color deviation of the image (a) is visually obvious, and the edge of the image of the methods (b) and (c) is too sharp; for (d) to (g), the difference is not large in vision, but it can be seen from the two illustrations that the fusion result in (g) is closer to the reference image, so it can be concluded that the method of the present application can improve the spatial resolution of the hyperspectral image while better preserving the spectral information of the source image, and the obtained fusion image is better and more natural.

[0160] Figure 1 is a schematic diagram of a fusion method of the present application. Figure 4 Figure 1 is a schematic diagram of a fusion method of the present application. Figure 4 Figure 1 is a schematic diagram of a fusion method of the present application.

[0161] The visual evaluation comparison can have a more intuitive understanding of the fusion result, but it is difficult to give the most accurate judgment of the fusion result by relying on subjective evaluation alone, so it is necessary to combine objective indexes to evaluate the fusion result. In this embodiment, five objective evaluation indexes, PSNR, SAM, CC, RMSE and ERGAS, are used to comprehensively evaluate the image. Among them, PSNR represents the peak signal-to-noise ratio, which is an objective standard for evaluating images; SAM represents global spectral distortion measurement, which reflects the color difference between two images, and the optimal value is 0; CC represents the correlation coefficient, which evaluates the spectral and spatial information similarity between the images of each band of two images from the perspective of statistical correlation; RMSE is the root mean square error; ERGAS is a weighted sum based on RMSE, which represents the fidelity of the image, and the optimal value is 0.

[0162] Tables 1 and 2 are objective indexes of the "balloon" image and "pepper comparison" image in the CAVE data set for different fusion methods, respectively. From Tables 1 and 2, it can be seen that most of the objective indexes of the present embodiment are better than those of other methods, among which the PSNR and SAM values of the present embodiment are obviously better than those of other methods, indicating that the method given in the present embodiment can well improve the spatial resolution of the hyperspectral image while maintaining the spectral characteristics of the hyperspectral image.

[0163] Based on the comprehensive visual evaluation and the objective index evaluation, the hyperspectral image fusion method based on the dual-resolution residual mutual guidance network given in this embodiment can well obtain a fused image with high spatial and high spectral resolution.

[0164] Table 1: Objective index of fusion results of "balloon" image experiment

[0165]

[0166]

[0167] Table 2: Objective index of fusion results of "pepper comparison" image experiment

[0168]

[0169] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and the equivalent technologies thereof, the present application also intends to include these modifications and variations.

Claims

1. A dual-resolution residual guided network based method for hyperspectral image fusion, characterized in that, The method comprises the following steps: Step 1, establishing a training data set of hyperspectral images and visible light images; All visible light images in the training data set are down-sampled and combined with the hyperspectral images in the training data set to form low-resolution data pairs; all hyperspectral images in the training data set are up-sampled and combined with the visible light images in the training data set to form high-resolution data pairs; Step two, obtaining the fusion result of the hyperspectral image and the visible light image in the data set of the high-resolution data pair and the low-resolution data pair through a dual-resolution residual mutual guidance network model DRRMGN ; Specifically, first, two dual-branch structure feature extraction networks are used to extract spectral and spatial information of hyperspectral images and visible light images of two different resolution pairs respectively, then the spatial and spectral information extracted by each feature extraction network is preliminarily fused to form compact features containing spatial and spectral information, then a residual mutual guidance block establishes a connection between the fusion features of two different resolutions to learn more complex combinations between spatial resolution and spectral resolution; finally, the desired hyperspectral image with high spatial resolution is recovered from the fusion features through an image reconstruction network, and the original information of the hyperspectral image and the visible light image with different resolutions is introduced into the image reconstruction network to obtain the fusion result of the final image ; Step 3, using a random gradient descent algorithm to train the dual-resolution residual mutual guidance network model using the training data set to obtain a fusion model.

2. The dual-resolution-based residual mutual guidance network-based hyperspectral image fusion method according to claim 1, characterized in that, The hyperspectral images and visible light images in the training data set are multiple pairs of hyperspectral images and visible light images of multiple scenes. The hyperspectral images in the low-resolution data pairs have the same size as the hyperspectral images in the training data set.

3. The dual-resolution-based residual guided network for hyperspectral image fusion method of claim 1, wherein, The method of up-sampling and down-sampling comprises the following steps: Step 3.1, Interpolation method is used to the visible light images in the training dataset Down-sampling with interval 4, the down-sampled visible light image is denoted as ; wherein, h and w denote the length and width of the spatial dimension, respectively, and 3 represents the number of spectral bands; the down-sampled visible light image and the hyperspectral image in the training dataset form a low resolution image pair; the down-sampled visible light image is denoted as equation (1): Formula (1); wherein represents a down-sampling operation; Step 3.2, upsampling the hyperspectral image in the training dataset using bilinear interpolation with a scaling factor to obtain a high-resolution hyperspectral image H up wherein, H and W denote the length and width of the spatial dimension, respectively, c represents the number of spectral bands, and the hyperspectral image H up and the visible light image in the training dataset form a high-resolution image pair, and the upsampled hyperspectral image is denoted as equation (2): Formula (2); wherein represents an up-sampling operation.

4. The dual-resolution-based residual guided network for hyperspectral image fusion method of claim 1, wherein, The dual-resolution residual mutual guidance network model comprises feature extraction, feature fusion and image reconstruction.

5. The dual-resolution-based residual guided network for hyperspectral image fusion method of claim 1, wherein, Obtaining the hyperspectral image and visible light image fusion result The specific process is as follows: Step 5.1, spatial and spectral features of high-resolution image pairs and low-resolution image pairs are extracted respectively by a feature extraction network and After feature extraction, the first set of feature maps and and the second set of feature maps and are obtained respectively ； ； ； ； wherein, with and with representing respectively and two subnetworks, (.) representing a convolution operation, representing an image obtained by four times up-sampling the hyperspectral image in the training dataset, representing a visible light image in the training dataset, representing a hyperspectral image in the training dataset, representing an image obtained by four times down-sampling the visible light image in the training dataset; Step 5.2, mapping the first set of features With feature maps and mapping the second set of features With feature maps are respectively connected together, and then preliminary fusion is performed using a first-level fusion network FFN to form compact features containing both spatial and spectral information And Specifically as shown in formula (3): ； Equation (3); wherein, represents the compact features formed by the branch on the FFN network for the high resolution data, represents the compact features formed by the branch on the FFN network for the low resolution data, (.) represents the feature fusion function of the branch FFN subnetwork for the high resolution data, represents the feature fusion function of the branch FFN subnetwork for the low resolution data, "+" represents the concatenation operation; Step 5.3, the compact features and are fed into the second stage fusion network SFN, which establishes the link between the compact features and through residual cross-guided blocks, and produces two new feature maps and by learning more complex combinations between spatial resolution and spectral resolution, which are denoted as equation (4): ； Formula (4); wherein, represents a feature map formed by the branch on the SFN network for high resolution data, represents a compact feature formed by the branch on the FFN network for low resolution data, (.) represents a feature fusion function of the SFN sub-network for the branch on the high resolution data, (.) represents a feature fusion function of the SFN sub-network for the branch on the low resolution data. Step 5.4, restore the high-resolution hyperspectral image required in the preliminary fusion process of step 5.3, use transpose convolution to reconstruct the fusion result of hyperspectral image and visible light image The original hyperspectral image and visible light image and the image information after four times up-sampling of the original hyperspectral image are introduced into the reconstruction network in a hierarchical manner through a skip connection, which is represented as formula (5): Formula (5); wherein , , represents a convolution operation, "+" represents a concatenation operation, ; representing high resolution data to branch forming feature maps in SFN network, representing low resolution data to branch forming compact features in FFN network; Step 5.5, using a residual convolutional neural network unit to replace a general convolutional neural network unit, wherein the general form of each residual unit is represented by formula (6): The hybrid loss function in the fusion model comprises: ； Formula (6); wherein, and represent the input and output of the th residual unit, represents the residual function, represents the network weight of the th residual unit, is an activation function and is a feature mapping function.

6. The dual-resolution-based residual guided network method for hyperspectral image fusion according to claim 5, wherein, The feature extraction network And Both comprise a first group of subnetworks Or And a second group of subnetworks Or ; The first group of sub-networks Or The second group of sub-networks Or The first group of sub-networks Or The second group of sub-networks Or One of them takes multi-band hyperspectral images as input, and the other takes three-band visible light images as input.

7. The dual-resolution-based residual guided network for hyperspectral image fusion method of claim 1, wherein, Finally, according to steps 7.1 and 7.2, the loss function specific to the fusion model is constructed as shown in formula (9) and formula (10): Step 7.1, in the function shown in the main network, select -norm loss function and constraints, the main network main loss function is shown in equation (7): Equation (7); wherein represents a loss function, represents a constraint function, represents a spectral vector of the reference image in the spectral dimension, represents a spectral vector of the result image in the spectral dimension; Step 7.2, in the resolution loss function, select -norm loss function and come together to constrain, the resolution loss function is formula (8): Formula (8); wherein represents a loss function, represents a constraint function; Step 7.3, in the process of training the network by the random gradient descent algorithm, the loss function is continuously optimized until convergence. Equation (9); Equation (10); wherein, a comprehensive loss function of the branch for high-resolution data, a comprehensive loss function of the branch for low-resolution data; a reference image, a fusion result of the hyperspectral image and the visible light image; 、 respectively represent output feature maps of two groups of different resolution data in the second-level fusion network, 、 denotes a weight of the resolution mutual guidance loss function, is a resolution mutual learning loss function, is a network main loss function, denotes a spectral vector of the reference image in the spectral dimension, denotes a spectral vector of the result image in the spectral dimension; In step 7.3, the process of optimizing the loss comprises:

8. The dual-resolution-based residual guided network method for hyperspectral image fusion according to claim 7, wherein, The random gradient descent for the high-resolution data pair branch is represented by formula (11): A small batch of samples is composed of p samples in the training dataset, and then the following is performed in each iteration: At the same time, the random gradient descent for the low-resolution data pair branch is represented by formula (13): Equation (11); where, Loss function constructed for steps 7.1 and 7.2 , here m represents p samples, after which the model is updated using gradient descent: ； Here, is the partial derivative of the loss function with respect to the parameter , is the learning rate set for the model, and after updating the model by gradient descent, equation (11) can be simplified to equation (12): Equation (12); In formula (12), denotes the gradient of the loss function obtained, at each model update, by randomly taking one sample to update the parameters; In the process of training the network by the random gradient descent algorithm, a hybrid loss function is designed, and the loss function is continuously optimized in the training until convergence, wherein the loss function is represented by formula (14), Equation (13); where, Loss function constructed for steps 7.1 and 7.2 After that, the branch is updated with gradient descent similar to the high-resolution data.

9. The dual-resolution-based residual guided network for hyperspectral image fusion method of claim 1, wherein, ​ Formula (14); wherein, a comprehensive loss function representing the branch for high-resolution data, a comprehensive loss function representing the branch for low-resolution data; a reference image, a fusion result of the hyperspectral image and the visible light image; 、 respectively represent the output features of two groups of different resolution data pairs in the second-level fusion network, 、 denotes the weight of the resolution mutual guidance loss function, is the resolution mutual guidance loss function, is the network main loss function.

Images