Non-reference quality evaluation method for hyperspectral fusion image based on deep learning

Abstract

The invention discloses a non-reference quality evaluation method for a hyperspectral fusion image based on deep learning, and the method comprises the steps: 1, building a spatial quality evaluation network, and obtaining a spatial quality evaluation result; the spatial quality evaluation network comprises a spatial feature difference extractor and a spatial feature regression device; 2, establishing a spectrum quality evaluation network, and obtaining a spectrum quality evaluation result; the spectrum quality evaluation network comprises a spectrum characteristic difference extractor and a spectrum characteristic regression device; 3, establishing an overall quality evaluation network, and obtaining an overall quality evaluation result; the overall quality evaluation network comprises a spatial feature difference extractor, a spectral feature difference extractor and an overall feature regression device; and step 4, training and testing the space quality evaluation network, the spectrum quality evaluation network and the overall quality evaluation network, and finally performing non-reference quality evaluation on the hyperspectral fusion image after the training is completed. The method solves the problems that an existing image fusion method is poor in accuracy and not comprehensive in evaluation.

Description

Technical field

[0001] The present invention belongs to the technical field of hyperspectral fusion image quality assessment, and specifically relates to a reference-free quality assessment method for hyperspectral fusion images based on deep learning. [Background Technology]

[0002] Existing hyperspectral HS image fusion quality assessment methods basically use reference evaluation indicators such as ERGAS, Q 2n , SAM, etc. Reference quality assessment requires spatial degradation of the input image, which includes the original hyperspectral image and the original multispectral image, and then uses the original hyperspectral image and the original multispectral image as a reference. Although reference assessment methods can effectively evaluate the quality of hyperspectral fusion images, they rely on many assumptions. In practical applications, there are no high-spatial-resolution hyperspectral images as a reference for the fusion results. Therefore, the practical application value of reference quality assessment methods is relatively weak.

[0003] Existing no-reference assessment methods for pan-sharpened image quality are less than ideal when applied to hyperspectral fusion image quality assessment, and some methods are not directly applicable. Commonly used pan-sharpened image quality assessment methods such as QNR, FQNR, and HQNR are designed based on the characteristics of the pan-sharpening process and fused images. However, their accuracy and timeliness are poor when applied to HS-fused images. QNR-based methods require the calculation of Q values ​​between two bands. Hyperspectral images have dozens or even hundreds of bands, so using QNR methods to evaluate hyperspectral fusion images requires a large amount of computation and is very time-consuming. Furthermore, the pan-sharpening process and hyperspectral image fusion process differ significantly, using different modulation transfer filters. Pan-sharpened fusion images are typically sourced from a single remote sensing platform, while existing hyperspectral fusion images are mostly sourced from multiple different satellite platforms. Therefore, these pan-sharpened image quality assessment methods exhibit poor accuracy when applied to hyperspectral fusion quality assessment.

[0004] The few existing no-reference quality assessment methods for hyperspectral fusion images focus solely on the overall quality of the fusion result, without independently assessing spatial and spectral quality. The quality of hyperspectral fusion images is often divided into three categories: spatial quality, spectral quality, and overall quality. However, the two existing methods for evaluating hyperspectral fusion images focus solely on the overall quality of the fused image, failing to assess spatial or spectral quality separately. Consequently, the evaluation of the fused image is incomplete. [Summary of the invention]

[0005] The purpose of this invention is to provide a reference-free quality assessment method for hyperspectral fusion images based on deep learning, so as to solve the problems of poor accuracy and incomplete evaluation of existing fusion image methods.

[0006] The first technical solution adopted by the present invention is a reference-free quality assessment method for hyperspectral fusion images based on deep learning, which includes the following contents:

[0007] Step 1: Input the reduced-dimensional hyperspectral fusion image and the multispectral image into the spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain a spatial quality score, that is, to obtain the spatial quality assessment result; the spatial feature difference extractor and the spatial feature regressor together constitute the spatial quality assessment network;

[0008] Step 2: Input the hyperspectral fusion image and the spectral repair image into the spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain the spectral quality score, that is, the spectral quality assessment result; the spectral feature difference extractor and the spectral feature regressor together constitute the spectral quality assessment network;

[0009] Step 3: Concatenate the spatial feature difference matrix and the spectral feature difference matrix and input them into the overall feature regressor to obtain the overall quality score, that is, to obtain the overall quality assessment result; the spatial feature difference extractor, the spectral feature difference extractor and the overall feature regressor together constitute the overall quality assessment network;

[0010] Step 4: Train and test the spatial quality assessment network, spectral quality assessment network, and overall quality assessment network to finally complete the reference-free quality assessment of the hyperspectral fusion image.

[0011] Furthermore, in step 1, the hyperspectral image with low spatial resolution is fused with the multispectral image with high spatial resolution to obtain a hyperspectral fusion image; and the hyperspectral fusion image is subjected to band dimensionality reduction to obtain a hyperspectral fusion image after dimensionality reduction.

[0012] Furthermore, the spatial feature difference extractor consists of four convolutional layers and two maximum pooling layers, and the spatial feature regressor consists of one convolutional layer and two linear layers.

[0013] Furthermore, in step 2, the hyperspectral image is subjected to bicubic interpolation to obtain an interpolated image, and the multispectral image is subjected to a convolution operation and then added to the interpolated image to obtain a spectral restoration image.

[0014] Furthermore, the spectral feature difference extractor consists of five convolutional layers and two maximum pooling layers, and the spectral feature regressor consists of one convolutional layer and two linear layers.

[0015] Furthermore, in step 4, the dataset of the hyperspectral fusion image is divided into a training set, a test set, and a validation set without overlapping;

[0016] The quality of the hyperspectral fusion images in the training set was evaluated using reference quality assessment indicators, and the corresponding reference spatial quality scores, reference spectral quality scores and reference overall quality scores were obtained. The reference spatial quality scores, reference spectral quality scores and reference overall quality scores were selected as the true values ​​for network training.

[0017] Furthermore, the specific process of network training in step 4 is:

[0018] The spatial quality assessment network in step 1 is trained using the reference spatial quality score to obtain a trained spatial feature difference extractor, and all network parameters of the spatial feature difference extractor are frozen;

[0019] The spectral quality assessment network in step 2 is trained using the reference spectral quality scores to obtain a trained spectral feature difference extractor, and all network parameters of the spectral feature difference extractor are frozen;

[0020] The hyperspectral fusion image and the multispectral image are both input into the frozen spatial feature difference extractor to obtain the spatial feature difference matrix. The spectral repair image and the hyperspectral fusion image are both input into the frozen spectral feature difference extractor to obtain the spectral feature difference matrix. The spatial feature difference matrix and the spectral feature difference matrix are both input into the overall feature regressor in step 3, and the overall quality score is used to train the network parameters of the overall feature regressor.

[0021] Furthermore, the overall quality regressor consists of two convolutional layers and two linear layers.

[0022] The second technical solution adopted by the present invention is a reference-free quality assessment device for hyperspectral fusion images based on deep learning, comprising:

[0023] The spatial quality assessment module is used to input the hyperspectral fusion image and the multispectral image after dimensionality reduction into the spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain the spatial quality assessment result;

[0024] The spectral quality assessment module is used to input the hyperspectral fusion image and the spectral repair image into the spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain the spectral quality assessment result;

[0025] An overall quality assessment module is used to concatenate the spatial feature difference matrix and the spectral feature difference matrix and input them into the overall feature regressor to obtain an overall quality assessment result;

[0026] The no-reference quality assessment module is used to train and test the spatial quality assessment network, spectral quality assessment network and overall quality assessment network, and finally complete the no-reference quality assessment of the hyperspectral fusion image.

[0027] The third technical solution adopted by the present invention is a reference-free quality assessment device for hyperspectral fusion images based on deep learning, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, a reference-free quality assessment method for hyperspectral fusion images based on deep learning is implemented.

[0028] The beneficial effects of the present invention are as follows: it is designed for quality assessment of hyperspectral fusion images and includes three assessment networks: spatial quality, spectral quality, and overall quality. This is the first application of deep learning networks to the quality assessment of hyperspectral fusion images. Based on the distortion of hyperspectral image fusion, the spatial quality assessment network, the spectral quality assessment network, and the overall quality assessment network are designed, respectively. These networks can independently assess the spatial quality, spectral quality, and overall quality of the fused image.

[0029] The proposed method for assessing the quality of hyperspectral fusion images without the need for high-spatial-resolution hyperspectral images as a reference enables a reference-free assessment of the quality of hyperspectral fusion images, demonstrating its high practical value. The proposed method provides a more comprehensive assessment of the quality of fused images from three perspectives: spatial quality, spectral quality, and overall quality. Compared to traditional reference-free assessment methods, this method is more effective and accurate.

Brief Description of the Drawings

[0030] Figure 1 Schematic diagram of the spatial quality assessment network structure of the present invention;

[0031] Figure 2 Schematic diagram of the spectral quality assessment network structure of the present invention;

[0032] Figure 3 Schematic diagram of the overall quality assessment network structure of the present invention. [Specific implementation method]

[0033] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] The present invention provides a reference-free quality assessment method for hyperspectral fusion images based on deep learning, which includes the following contents:

[0035] Step 1. Input the reduced-dimensional hyperspectral fusion image and the multispectral image into the spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain a spatial quality score, that is, to obtain a spatial quality assessment result; the spatial feature difference extractor and the spatial feature regressor together constitute a spatial quality assessment network; the network structure of the spatial feature difference extractor can be diverse, and the spatial feature difference extractor can use other structures such as residual networks or deepened network depth for feature extraction.

[0036] The spatial feature difference matrix is:

[0037] SpaFeaDif=SpaNet(F D , M) (1),

[0038] Among them, SpaFeaDif is the spatial feature difference matrix, and SpaNet(A, B) is the spatial feature difference extractor of A and B;

[0039] The results of the space quality assessment are:

[0040] Q spa = SpaReg(SpaFeaDif) (2),

[0041] Among them, Q spa is the spatial quality assessment result, and SpaReg(·) is the spatial feature regressor.

[0042] Step 2: Input the hyperspectral fusion image and the spectral repair image into the spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain the spectral quality score, that is, the spectral quality assessment result; the spectral feature difference extractor and the spectral feature regressor together constitute a spectral quality assessment network; the network structure of the spectral feature extractor can be diverse, and the spectral feature extractor can use other structures such as residual network or deepen the network depth for feature extraction.

[0043] Input the spectral repair image and the hyperspectral fusion image into the spectral feature difference extractor and obtain the spectral feature difference matrix:

[0044] SpeFeaDif=SpeNet(H R , F) (3),

[0045] Among them, F is the hyperspectral fusion image, SpeFeaDif is the spectral feature difference matrix, and SpeNet(A, B) is the spectral feature difference extractor of A and B.

[0046] SpeFeaDif is input into the spectral feature regressor to obtain the spectral quality assessment result:

[0047] Q spe =SpeReg(SpeFeaDif) (4),

[0048] Among them, Q spe is the spectral quality assessment result, and SpeReg(·) is the spectral feature regressor.

[0049] Step 3: Figure 3 As shown, the spatial feature difference matrix and the spectral feature difference matrix are cascaded and input into the overall feature regressor to obtain the overall quality score, that is, the overall quality assessment result; the spatial feature difference extractor, the spectral feature difference extractor and the overall feature regressor together constitute the overall quality assessment network.

[0050] Overall quality assessment results:

[0051] Q global =GlobalReg(SpaFeaDif,SpeFeaDif) (5),

[0052] Among them, Q global is the overall quality assessment result, GlobalReg(A, B) is the overall feature regressor of A and B.

[0053] Step 4: Train and test the spatial quality assessment network, spectral quality assessment network, and overall quality assessment network to finally complete the reference-free quality assessment of the hyperspectral fusion image.

[0054] In some embodiments, in step 1, a hyperspectral image with low spatial resolution is fused with a multispectral image with high spatial resolution to obtain a hyperspectral fusion image; and the hyperspectral fusion image is subjected to band dimensionality reduction to obtain a hyperspectral fusion image after dimensionality reduction.

[0055] The formula for the dimensionality reduction process is as follows:

[0056] F D =SRF·F (6),

[0057] Among them, F is the hyperspectral fusion image, F D is the hyperspectral fusion image after dimensionality reduction. SRF is the spectral response function of the multispectral sensor. A·B is the product of matrix A and matrix B.

[0058] In some embodiments, the spatial feature difference extractor consists of four convolutional layers and two maximum pooling layers, where a Tanh activation function is used after each convolutional layer; the spatial feature regressor consists of one convolutional layer and two linear layers, where a Tanh activation function is used after the convolutional layer.

[0059] In some embodiments, in step 2, the hyperspectral image is subjected to bicubic interpolation to obtain an interpolated image, and the multispectral image is subjected to a convolution operation and then added to the interpolated image to obtain a spectral restoration image.

[0060] The design of the spectral information restoration network can be diverse. For example, the interpolation method can be bicubic interpolation or other interpolation methods.

[0061] The calculation is as follows:

[0062] H R =SpeRepairNet(H,M) (7),

[0063] Among them, H R is the spectral repair image, H is the hyperspectral image, M is the multispectral image, and SpeRepairNet(A,B) is the spectral information repair network of A and B.

[0064] In some embodiments, the spectral feature difference extractor consists of five convolutional layers and two maximum pooling layers, wherein each convolutional layer is followed by a Tanh activation function. The spectral feature regressor consists of one convolutional layer and two linear layers, wherein each convolutional layer is followed by a Tanh activation function.

[0065] In some embodiments, in step 4, the hyperspectral fusion image dataset is divided into a training set, a test set, and a validation set without overlapping;

[0066] The hyperspectral fusion images in the training set are evaluated using a reference quality assessment metric, resulting in a reference spatial quality score, a reference spectral quality score, and a reference overall quality score. These scores are then selected as ground truth for network training. Various ground truths can be used, such as PSNR, SAM, ERGAS, and other evaluation methods.

[0067] In some embodiments, the specific process of network training in step 4 is:

[0068] The spatial quality assessment network in step 1 is trained using the reference spatial quality score to obtain a trained spatial feature difference extractor, and all network parameters of the spatial feature difference extractor are frozen;

[0069] The spectral quality assessment network in step 2 is trained using the reference spectral quality scores to obtain a trained spectral feature difference extractor, and all network parameters of the spectral feature difference extractor are frozen;

[0070] The hyperspectral fusion image and the multispectral image are both input into the frozen spatial feature difference extractor to obtain the spatial feature difference matrix. The spectral repair image and the hyperspectral fusion image are both input into the frozen spectral feature difference extractor to obtain the spectral feature difference matrix. The spatial feature difference matrix and the spectral feature difference matrix are both input into the overall feature regressor in step 3, and the overall quality score is used to train the network parameters of the overall feature regressor.

[0071] In some embodiments, the overall quality regressor consists of two convolutional layers and two linear layers, and a Tanh activation function is used after each convolutional layer.

[0072] The present invention also provides a reference-free quality assessment device for hyperspectral fusion images based on deep learning, comprising:

[0073] The spatial quality assessment module is used to input the hyperspectral fusion image and the multispectral image after dimensionality reduction into the spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain the spatial quality assessment result;

[0074] The spectral quality assessment module is used to input the hyperspectral fusion image and the spectral repair image into the spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain the spectral quality assessment result;

[0075] An overall quality assessment module is used to concatenate the spatial feature difference matrix and the spectral feature difference matrix and input them into the overall feature regressor to obtain an overall quality assessment result;

[0076] The no-reference quality assessment module is used to train and test the spatial quality assessment network, spectral quality assessment network and overall quality assessment network, and finally complete the no-reference quality assessment of the hyperspectral fusion image.

[0077] The present invention also provides a reference-free quality assessment device for hyperspectral fusion images based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a reference-free quality assessment method for hyperspectral fusion images based on deep learning is implemented.

[0078] Example: We conducted hyperspectral image fusion and quality assessment experiments using a hyperspectral image dataset collected by a next-generation airborne visible-near-infrared spectral imager. The dataset covers 375 continuous bands in the 380-2510 nm wavelength range and contains hyperspectral images of 30 different scenes, all with a size of 256×256 pixels.

[0079] The input images for the hyperspectral image fusion experiment were generated by simulation. The input multispectral image was generated by spectrally filtering the original hyperspectral image using the spectral response function of the WorldView3 satellite. The input hyperspectral image was generated by spatially filtering the original hyperspectral image using a Gaussian low-pass filter to simulate a modulated transfer filter. The Gaussian filter kernel size was 7×7 and the standard deviation was 3. The fusion algorithms used in the hyperspectral image fusion experiment included CNN-FUS, LTMR, UTV, PCA, GSA, HySure, SFIM, GLP, and MAP-SMM fusion methods.

[0080] The quality evaluation experiment of the hyperspectral fusion image obtained by the hyperspectral image fusion experiment is carried out. The hyperspectral fusion image quality evaluation experiment is divided into reference quality evaluation and non-reference quality evaluation. The reference evaluation experiment uses the original hyperspectral image as a reference for evaluation, and selects four reference indicators: PSNR, SAM, ERGAS and Q 2n The PSNR, SAM, and ERGAS evaluation results were used as the spatial quality truth value, spectral quality truth value, and overall quality truth value, respectively, for network training of the proposed method. A no-reference evaluation experiment was conducted on a validation set after the network training of the proposed method to verify the effectiveness of the proposed method.

[0081] The experimental code running platform is Python 3.8, the computer operating system is Windows 11 Professional Edition, the computer processor is Intel(R) Core(TM) i9-10900K CPU, and the system graphics card is NVDIA GeForce RTX 3060.

[0082] Step 1: Establish a spatial quality assessment network:

[0083] The size of the hyperspectral fusion image F used in the hyperspectral fusion image quality assessment experiment is 256×256×375 pixels, which means that the hyperspectral fusion image F has 375 bands, and each band has 256 rows and 256 columns of pixels. The spectral response function (SRF) of the WorldView3 satellite is 375×16. Substituting the hyperspectral fusion image F and the spectral response function (SRF) into the dimensionality reduction formula and performing matrix multiplication, the reduced dimensionality hyperspectral fusion image F is obtained. DThe size is 256×256×16 pixels, representing the hyperspectral fusion image F after dimensionality reduction D There are 16 bands, each with 256 rows × 256 columns of pixels.

[0084] As attached Figure 1 As shown in the figure, a spatial feature difference extractor is created. The spatial feature difference extractor consists of a 4-layer convolutional neural network. The hyperspectral fusion image F after dimensionality reduction is D Substitute the multispectral image M into formula (1) and obtain the spatial feature difference matrix SpaFeaDif. Substitute the spatial feature difference matrix into formula (2) to obtain the spatial quality assessment result. The spatial feature regressor consists of 1 convolutional layer and 2 linear layers. The spatial quality assessment result is Q spa .

[0085] Step 2: Establish a spectral quality assessment network:

[0086] As attached Figure 2 As shown, a spectral information restoration network and a spectral feature difference extractor are created. The spectral restoration network includes image interpolation operations, two image convolution operations, and image summation operations. Substitute the low spatial resolution hyperspectral image H and the high spatial resolution multispectral image M into formula (7) to obtain the spectral restoration image H R .

[0087] Substitute the spectral repair image and the hyperspectral fusion image F into formula (3) and obtain the spectral feature difference matrix SpeFeaDif. Create a spectral feature regressor as shown in the following figure. Figure 2 As shown in Figure 2, the spectral feature regressor consists of 1 convolutional layer and 2 linear layers. Substituting the spectral feature difference matrix SpeFeaDif into formula (4) yields the spectral quality evaluation result Q spe .

[0088] Step 3: Establish an overall quality assessment network:

[0089] As attached Figure 3 As shown in Figure 2, an overall feature regressor is created, which contains two convolutional layers and two linear layers. The spatial feature difference matrix SpaFeaDif output by the spatial feature difference extractor and the spectral feature difference matrix SpaFeaDif output by the spectral feature difference extractor are substituted into formula (5) to obtain the overall quality evaluation result.

[0090] Step 4: Train and test the spatial quality assessment network, spectral quality assessment network, and overall quality assessment network:

[0091] The evaluation results of the reference evaluation indicators in the hyperspectral fusion image quality assessment experiment are used as the true values ​​for network training.

[0092] First, the evaluation results of the reference evaluation indicator PSNR are used as the true value of spatial quality to train and test the spatial quality evaluation network, and the parameters of the spatial feature difference extractor are frozen.

[0093] Secondly, the evaluation results of the reference evaluation index SAM are used as the true value of the spectral quality to train and test the spectral quality assessment network, and the parameters of the spectral feature difference extractor are frozen.

[0094] Then, the evaluation results of the reference evaluation index ERGAS are used as the true value of the overall quality, and the SpaFeaDif and speFeaDif output by the frozen spatial feature difference extractor and spectral feature difference extractor are substituted into the overall quality regressor for training and testing.

[0095] Finally, the effectiveness of each network is verified on the validation set.

[0096] Table 1 Quality evaluation results of various spatial indicators of hyperspectral fusion images on the validation set

[0097]

[0098] Table 2 Quality evaluation results of various spectral indicators of hyperspectral fusion images on the validation set

[0099]

[0100] Table 3 Quality evaluation results of various overall indicators of hyperspectral fusion images on the validation set

[0101]

[0102]

[0103] The fusion methods in Tables 1 to 3 are, from top to bottom, unidirectional total variation with Tucker decomposition (UTV), convolutional neural network-based fusion (CNN-Fus), low tensor multi-rank regularization (LTMR) based on subspace, Gram-Schmidt adaptive (GSA), hyperspectral super resolution (HySure) based on subspace regularization, generalized Laplacian pyramid (GLP), smoothing filter-based intensity modulation (SFIM), principal component analysis (PCA), and maximum a posterior estimation with a stochastic mixing model. Model, MAP-SMM), bicubic interpolation upsampling method (Expand, EXP). The spatial quality evaluation indicators in Table 1 include: Peak Signal-to-Noise Ratio, spatial quality evaluation network, the spatial index part of the Quality with No Reference and the spatial index part of the no-reference evaluation method based on multivariate Gaussian distribution (Multivariate Gaussian-based Quality with NoReference). The spectral quality evaluation indicators in Table 2 include: Spectral Angle Mapper (SAM), spectral quality evaluation network, the spectral index part of the no-reference quality evaluation method and the spectral index part of the no-reference evaluation method based on multivariate Gaussian distribution. The overall quality evaluation indicators in Table 3 include: Relative Global Comprehensive Error (ERGAS), Q 2n , overall quality assessment network, no-reference quality assessment method and no-reference assessment method based on multivariate Gaussian distribution, where Q 2nThis is a commonly used overall quality assessment metric in remote sensing image fusion quality assessment. Each column in Tables 1 through 3 represents the quality assessment score for the corresponding evaluation metric (e.g., peak signal-to-noise ratio, spatial quality assessment network, etc.). The quality assessment scores are sorted from highest to lowest.

[0104] From the data in Tables 1 to 3, it can be seen that the spatial quality assessment network in the no-reference quality assessment method of hyperspectral fusion images based on deep learning of the present invention has the same evaluation trend as the reference spatial index (peak signal-to-noise ratio), the spectral quality assessment network has the same evaluation trend as the reference spectral index (spectral angle mapping), and the overall quality assessment network has the same evaluation trend as the reference overall index (relative global comprehensive error and Q 2n ) evaluation trends are consistent, indicating that the deep learning-based no-reference quality assessment method for hyperspectral fusion images can effectively evaluate the spatial, spectral, and overall quality of hyperspectral fusion images without the need for a reference image. Furthermore, compared with two existing no-reference assessment methods for remote sensing image fusion, the spatial, spectral, and overall quality assessment networks can more accurately distinguish the quality of fusion results, and the evaluation scores are closer to those with reference evaluation indicators, demonstrating the accuracy of the proposed method.

Claims

1. A no-reference quality assessment method for hyperspectral fusion images based on deep learning, characterized by: Includes the following: Step 1: Input the reduced-dimensional hyperspectral fusion image and the multispectral image into a spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain a spatial quality score, that is, to obtain a spatial quality assessment result; the spatial feature difference extractor and the spatial feature regressor together constitute a spatial quality assessment network; Step 2: Input the hyperspectral fusion image and the spectral restoration image into a spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain a spectral quality score, that is, to obtain a spectral quality assessment result; the spectral feature difference extractor and the spectral feature regressor together constitute a spectral quality assessment network; Step 3: Concatenate the spatial feature difference matrix and the spectral feature difference matrix and input them into the overall feature regressor to obtain an overall quality score, that is, to obtain an overall quality assessment result; the spatial feature difference extractor, the spectral feature difference extractor and the overall feature regressor together constitute an overall quality assessment network; Step 4: training and testing the spatial quality assessment network, the spectral quality assessment network, and the overall quality assessment network, and finally completing the reference-free quality assessment of the hyperspectral fusion image.

2. The method for no-reference quality assessment of hyperspectral fusion images based on deep learning according to claim 1, wherein: In the step 1, a hyperspectral image with low spatial resolution is fused with a multispectral image with high spatial resolution to obtain a hyperspectral fusion image; and the hyperspectral fusion image is subjected to band dimensionality reduction to obtain a hyperspectral fusion image after dimensionality reduction.

3. The no-reference quality assessment method for hyperspectral fusion images based on deep learning according to claim 2, characterized in that: The spatial feature difference extractor consists of four convolutional layers and two maximum pooling layers, and the spatial feature regressor consists of one convolutional layer and two linear layers.

4. The method for no-reference quality assessment of hyperspectral fusion images based on deep learning according to claim 2 or 3, characterized in that: In step 2, the hyperspectral image is subjected to bicubic interpolation to obtain an interpolated image, and the multispectral image is subjected to a convolution operation and then added to the interpolated image to obtain a spectral restoration image.

5. The no-reference quality assessment method for hyperspectral fusion images based on deep learning according to claim 4, characterized in that: The spectral feature difference extractor consists of five convolutional layers and two maximum pooling layers, and the spectral feature regressor consists of one convolutional layer and two linear layers.

6. The no-reference quality assessment method for hyperspectral fusion images based on deep learning according to claim 4, characterized in that: In step 4, the data set of the hyperspectral fusion image is divided into a training set, a test set and a validation set without overlapping; Using a reference quality assessment index to evaluate the quality of the hyperspectral fusion image in the training set, and correspondingly obtaining a reference spatial quality score, a reference spectral quality score, and a reference overall quality score; The referenced spatial quality score, the referenced spectral quality score and the referenced overall quality score are selected as true values ​​for network training.

7. The no-reference quality assessment method for hyperspectral fusion images based on deep learning according to claim 6, characterized in that: The specific process of network training in step 4 is as follows: Using the reference spatial quality score to train the spatial quality assessment network in step 1 to obtain the trained spatial feature difference extractor, and freezing all network parameters of the spatial feature difference extractor; Using the reference spectral quality score to train the spectral quality assessment network in step 2 to obtain the trained spectral feature difference extractor, and freezing all network parameters of the spectral feature difference extractor; The hyperspectral fusion image and the multispectral image are both input into the frozen spatial feature difference extractor to obtain a spatial feature difference matrix, the spectral repair image and the hyperspectral fusion image are both input into the frozen spectral feature difference extractor to obtain a spectral feature difference matrix, the spatial feature difference matrix and the spectral feature difference matrix are both input into the overall feature regressor in step 3, and the network parameters of the overall feature regressor are trained using the overall quality score.

8. The no-reference quality assessment method for hyperspectral fusion images based on deep learning according to claim 7, characterized in that: The overall quality regressor consists of two convolutional layers and two linear layers.

9. A reference-free quality assessment device for hyperspectral fusion images based on deep learning, characterized in that: include: A spatial quality assessment module is configured to input the dimensionality-reduced hyperspectral fusion image and the multispectral image into a spatial feature difference extractor to obtain two spatial feature maps, subtract the two spatial feature maps to obtain a spatial feature difference matrix, and input the spatial feature difference matrix into the spatial feature regressor to obtain a spatial quality assessment result; A spectral quality assessment module is configured to input the hyperspectral fusion image and the spectral restoration image into a spectral feature difference extractor to obtain two spectral feature maps, subtract the two spectral feature maps to obtain a spectral feature difference matrix, and input the spectral feature difference matrix into the spectral feature regressor to obtain a spectral quality assessment result; An overall quality assessment module, configured to concatenate the spatial feature difference matrix and the spectral feature difference matrix and input the concatenated matrix into an overall feature regressor to obtain an overall quality assessment result; The no-reference quality assessment module is used to train and test the spatial quality assessment network, the spectral quality assessment network and the overall quality assessment network, and finally complete the no-reference quality assessment of the hyperspectral fusion image.

10. A device for non-reference quality assessment of hyperspectral fusion images based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images