Data structure for hyperspectral image fusion algorithm

Abstract

The invention discloses a data structure for a hyperspectral image fusion algorithm based on data-oriented integration. The data structure comprises the following main body data layers: a low-resolution hyperspectral and multispectral input data layer; a spectrum dictionary data layer; a spatial coefficient matrix data layer; a solution space data layer representing fusion result parameters; all the data layers interact in sequence and have global compatibility; the data structure is used for a datamation fusion algorithm for improving the spatial resolution of a hyperspectral image, and the algorithm combines external additional data to optimize a spatial coefficient based on data attributes of hyperspectral and multispectral images, and learns a spectral dictionary by using a non-negative matrix dictionary learning method. And carrying out algorithm fusion on the hyperspectral image and the multispectral image to obtain a hyperspectral image with target high spatial resolution.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a data structure for a hyperspectral image fusion algorithm based on data-oriented integration. Background Art

[0002] Hyperspectral imaging technology can simultaneously obtain scene information of different spectral wavelengths corresponding to hundreds of spectral bands. This kind of hyperspectral image has a high spectral resolution and can accurately identify substances. It is widely used in remote sensing, medical diagnosis, anomaly detection, face recognition and other fields. However, due to the limitations of imaging sensors, there is a mutually restrictive relationship between spectral resolution, spatial resolution and signal-to-noise ratio. Therefore, the enhancement of spatial resolution has become a basic problem of hyperspectral imaging, which will affect the potential application of hyperspectral imaging. Due to the limitations of hardware conditions, it is costly and challenging to improve spatial resolution through hardware means. Compared with hyperspectral images, multispectral images have lower spectral resolution, higher spatial resolution and signal-to-noise ratio, and can provide more spatial detail information of objects. Therefore, fusing hyperspectral images and multispectral images of the same scene has become a popular method to obtain high spatial resolution hyperspectral images.

[0003] At present, the fusion methods of hyperspectral and multispectral images can be divided into methods based on sparse representation, tensor decomposition, matrix subspace representation and deep learning. Researchers have also proposed many spatial-spectral fusion algorithms based on these methods and achieved good fusion results. However, after investigation and research, it was found that the existing hyperspectral and multispectral image fusion algorithms still face some challenges. Among them, the effective transmission of information between spatial patterns and spectral patterns is still a difficult problem. In addition, the algorithm has problems such as insufficient introduction of prior information, limited model expression ability, high time complexity, insufficient applicability of model parameter optimization algorithm and poor generalization performance. These problems need further research and improvement. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a data structure and its application in a hyperspectral image fusion algorithm.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0006] A data structure for a hyperspectral image fusion algorithm is used for a data-based fusion algorithm that improves the spatial resolution of hyperspectral images. The algorithm optimizes spatial coefficients based on data attributes of hyperspectral and multispectral images in combination with external additional data, learns spectral dictionaries using a non-negative matrix dictionary learning method, and algorithmically fuses hyperspectral and multispectral images to obtain hyperspectral images with a target high spatial resolution. The data structure includes the following main data layers: a low-resolution hyperspectral and multispectral input data layer; a spectral dictionary data layer; a spatial coefficient matrix data layer; and a solution space data layer that characterizes the parameters of the fusion result. Each data layer interacts in sequence and has global compatibility.

[0007] As a preferred technical solution of the present invention, the data-guided integration at least includes: the data attributes of the hyperspectral and multispectral images include the following data guidance: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following data guidance: additional prior information learned from external data; the spatial coefficients adopt second-order and above tensor data configurations and obey matrix multiplication operation rules; the algorithm adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combined norms, and multi-operator interactions for threshold-guided multiple iterations.

[0008] As a preferred technical solution of the present invention, in the spectral dictionary data layer, a non-negative structured sparse dictionary learning method is used to learn the spectral dictionary D from the hyperspectral image X; the hyperspectral image X is used as input data to characterize the solution of the spectral dictionary D as follows: Where U represents the spectral coefficient, ‖·‖ F represents the F norm; and the spectral dictionary D is learned using the existing non-negative structured sparse dictionary learning method.

[0009] As a preferred technical solution of the present invention, in the spatial coefficient matrix data layer, the number of initialization iterations is firstly set to 1, and the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, G5 are all initialized to matrices with all elements being 0; the iterative spatial coefficient matrix A is updated according to the following formula (1):

[0010]

[0011] Where B represents the spatial fuzzy matrix.

[0012] As a preferred technical solution of the present invention, in the solution space data layer representing the fusion result parameters, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of the above formula (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the same data process is used for iterative update of the split variables V1, V2, V3, V4, and V5; when the number of iterations reaches a threshold, the loop is exited and the spatial coefficient A is output.

[0013] The present invention also includes the following contents:

[0014] A data-based fusion algorithm for improving the spatial resolution of hyperspectral images is proposed. The algorithm optimizes spatial coefficients based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns the spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on hyperspectral and multispectral images to obtain a hyperspectral image with a target high spatial resolution.

[0015] As a preferred technical solution of the present invention, the algorithm includes the following main data structures in sequence according to the data derivation process: low-resolution hyperspectral and multi-spectral input data layer; spectral dictionary data layer; spatial coefficient matrix data layer; solution space data layer characterizing the fusion result parameters; each data layer interacts in sequence and has global compatibility.

[0016] As a preferred technical solution of the present invention, the data attributes of the hyperspectral and multispectral images include the following optional data guidance: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following optional data guidance: additional prior information learned from external data; the spatial coefficients adopt second-order and above tensor data configurations and obey matrix multiplication operation rules; the algorithm fusion adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combined norms, and multi-operator interactions for threshold-guided multiple iterations.

[0017] As a preferred technical solution of the present invention, the algorithm includes the following optional data processes:

[0018] 1) Simulation of low-resolution input data and construction of observation models;

[0019] 2) Using non-negative structured sparse dictionary learning method to learn spectral dictionary D from hyperspectral image X;

[0020] 3) Using hyperspectral image X, multispectral image Y and spectral dictionary D, a spatial coefficient optimization model is constructed to estimate the spatial coefficient A;

[0021] 4) Use the learned spectral dictionary D and spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fusion image Z.

[0022] As a preferred technical solution of the present invention, the data process of data process 2) includes:

[0023] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0024]

[0025] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, and U represents the spectral coefficient;

[0026] 2.2) Use the existing non-negative structured sparse dictionary learning method to learn the spectral dictionary D.

[0027] As a preferred technical solution of the present invention, the data process of data process 3) includes:

[0028] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 to matrices with all elements equal to 0, and initialize the penalty factor μ;

[0029] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0030]

[0031]

[0032] μ=γμ (8)

[0033] In the above formula, X represents the hyperspectral image, Y represents the multispectral image, B represents the spatial blur block circulant matrix, S represents the spatial downsampling matrix, R represents the spectral response function, ‖·‖1 represents the 1 norm, ‖·‖ * represents the nuclear norm, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0034] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). When optimizing and solving, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of equation (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is optimized by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is optimized by taking the partial derivative of equation (3) and setting the partial derivative to zero to derive the optimization equation and then update the variable V1. And solve the updated variable V2; optimize the solution model for the sub-problem of variable V3, and use the soft threshold operator to solve the updated variable V3; optimize the solution model for the sub-problem of variable V4, and use the singular value threshold algorithm to solve the nuclear norm optimization problem and then solve the updated variable V4; optimize the solution model for the sub-problem of variable V5, regard it as the denoising problem of the image (A-G5) with Gaussian noise with variance η3 / μ, and use the convolutional neural network denoising method to solve the updated variable V5, whose input data is the image (A-G5) and the noise variance η3 / μ, and the output data is the updated variable V5; when the number of iterations reaches the threshold, the loop is jumped out and the spatial coefficient A is output.

[0035] As a preferred technical solution of the present invention, in data process 3.2, the data process of establishing the sub-problem solving model shown in equations (1)-(8) includes:

[0036] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y and the target high spatial resolution hyperspectral fusion image Z:

[0037] X=ZBS+N H Y=RZ + N M (10)

[0038] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents zero-mean Gaussian noise in the observation model;

[0039] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. Then the relationship between the hyperspectral image X to be fused, the multispectral image Y and the target fusion image Z can be represented as follows:

[0040] X=DABS+N H Y=RDA+N M (11)

[0041] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents zero-mean Gaussian noise in the observation model;

[0042] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0043]

[0044] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0045] 3.2.4) The prior information of the estimated spatial coefficient A is introduced into the basic optimization model to obtain the following optimization model:

[0046]

[0047] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the local structure low-rank prior, For additional priors;

[0048] 3.2.5) Introduce splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, A represents the spatial coefficient to be estimated, B represents the spatial fuzzy matrix, and construct the following augmented Lagrangian function:

[0049]

[0050] In the above formula, L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0051] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8).

[0052] As a preferred technical solution of the present invention, in the data process 3.2.4), the local structure sparse low-rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing the convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization, which can be learned from any external grayscale image.

[0053] As a preferred technical solution of the present invention, the algorithm includes the following optional data processes:

[0054] 1) Simulation of low-resolution input data and construction of observation models;

[0055] 2) Using a non-negative structured sparse dictionary learning method to learn a spectral dictionary D from a hyperspectral image X; including:

[0056] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0057]

[0058] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, and U represents the spectral coefficient;

[0059] 2.2) Use the non-negative structured sparse dictionary learning method proposed by Dong Weisheng to learn the spectral dictionary D;

[0060] 3) Using hyperspectral image X, multispectral image Y and spectral dictionary D, a spatial coefficient optimization model is constructed to estimate the spatial coefficient A;

[0061] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 to matrices with all elements equal to 0, and initialize the penalty factor μ;

[0062] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0063]

[0064] μ=γμ (8)

[0065] In the above formula, X represents hyperspectral image, Y represents multispectral image, B represents spatial blur block circulant matrix, S represents spatial downsampling matrix, R represents spectral response function, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0066] The data process of establishing the model for solving the sub-problems of equations (1) to (8) includes:

[0067] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y and the target high spatial resolution hyperspectral fusion image Z:

[0068] X=ZBS+N H Y=RZ + N M (10)

[0069] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents zero-mean Gaussian noise in the observation model;

[0070] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. Then the relationship between the hyperspectral image X to be fused, the multispectral image Y and the target fusion image Z can be represented as follows:

[0071] X=DABS+N H Y=RDA+N M (11)

[0072] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents zero-mean Gaussian noise in the observation model;

[0073] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0074]

[0075] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0076] 3.2.4) The prior information of the estimated spatial coefficient A is introduced into the basic optimization model to obtain the following optimization model:

[0077]

[0078] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the local structure low-rank prior, is an additional prior; in this data process, the local structure sparse low-rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing the convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization, which can be learned from any external grayscale image;

[0079] 3.2.5) Introduce splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, A represents the spatial coefficient to be estimated, B represents the spatial fuzzy matrix, and construct the following augmented Lagrangian function:

[0080]

[0081] In the above formula, L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0082] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8);

[0083] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). When optimizing and solving, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of equation (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is optimized by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is optimized by taking the partial derivative of equation (3) and setting the partial derivative to zero to derive the optimization equation and then update the variable V1. The updated variable V2 is solved; the sub-problem solving model of variable V3 is optimized, and the soft threshold operator is used to solve the updated variable V3; the sub-problem solving model of variable V4 is optimized, and the singular value threshold algorithm is used to solve the nuclear norm optimization problem and then solve the updated variable V4; the sub-problem solving model of variable V5 is optimized, and it is regarded as the denoising problem of the image (A-G5) with Gaussian noise with variance η3 / μ, and the convolutional neural network denoising method is used to solve the updated variable V5, whose input data is the image (A-G5) and the noise variance η3 / μ, and the output data is the updated variable V5; when the number of iterations reaches the threshold, the loop is jumped out, and the spatial coefficient A is output;

[0084] 4) Use the learned spectral dictionary D and spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fusion image Z.

[0085] The present invention also includes a data model, a database group, a data platform and applications thereof constructed by the above algorithm.

[0086] The beneficial effects of adopting the above technical scheme are: in general, the present invention makes full use of the sparse priors, low-rank structural priors of hyperspectral and multispectral images and the additional prior information learned from external data to optimize the spatial coefficients, and uses the non-negative matrix dictionary learning method to learn the spectral dictionary, and then fuses the target high spatial resolution hyperspectral image, which can ensure the fusion efficiency on the basis of effectively improving the fusion quality, the data structure is clear, and the model has good generalization ability and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is the overall framework diagram of the present invention.

[0088] Figure 2 This is a basic flow chart for constructing an image fusion model for the present invention. The figure contains all the contents of data process 1) to data process 4). Figure 2 It can be seen that the hyperspectral image fusion problem is transformed into the estimation problem of the spectral dictionary D and the spatial coefficient A.

[0089] Figure 3 Schematic diagram of the effect of superpixel segmentation before and after segmentation in the present invention. Figure 3 It can be seen that the multispectral image Y after superpixel segmentation has local low-rank structural characteristics.

[0090] Figure 4 The fusion effect diagram of five fusion algorithms on the Hyperspec-VNIR Chikusei dataset. Figure 4 It can be seen that the high-resolution hyperspectral image fused by the fusion algorithm proposed in the present invention has the best spatial and spectral quality. DETAILED DESCRIPTION

[0091] The following examples describe the present invention in detail. Various raw materials and equipment used in the present invention are conventional commercial products, and can be directly obtained by market purchase. In the description of the following examples, for the purpose of illustration rather than limitation, specific details such as specific system structures, technology, etc. are proposed, so as to thoroughly understand the present application embodiments. However, it should be clear to those skilled in the art that the present application can also be realized in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.

[0092] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, data processes, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, data processes, operations, elements, components and / or their collections. It should also be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the items listed in association and all possible combinations, and includes these combinations. As used in the present specification and the appended claims, the term "if" can be interpreted as "when..." or "once" or "in response to determination" or "in response to detection" according to the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "once it is determined" or "in response to determination" or "once [described condition or event] is detected" or "in response to detection [described condition or event]" according to the context. In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0093] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0094] Example 1

[0095] See attached Figure 1-2, this embodiment provides a data fusion algorithm for improving the spatial resolution of hyperspectral images, based on the data attributes of hyperspectral and multispectral images combined with external additional data to optimize the spatial coefficients, using the non-negative matrix dictionary learning method to learn the spectral dictionary, and performing algorithm fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with a target high spatial resolution. The data attributes of the hyperspectral and multispectral images include the following optional data orientations: sparse priors and low-rank structural priors of the hyperspectral and multispectral images; the external additional data includes the following optional data orientations: additional prior information learned from external data; the spatial coefficients are configured as second-order and higher-order tensor data and obey the matrix multiplication operation rules; the algorithm fusion is constructed based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combination norms, and multiple operator interactions for threshold-guided multiple iterations for composite construction.

[0096] The main data integration of the algorithm of this embodiment includes: 1) simulation of low-resolution input data and construction of observation model; 2) using non-negative structured sparse dictionary learning method to learn spectral dictionary D from hyperspectral image X; 3) using hyperspectral image X, multispectral image Y and spectral dictionary D to build a spatial coefficient optimization model and estimate the spatial coefficient A; 4) using the learned spectral dictionary D and spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fusion image Z.

[0097] Example 2

[0098] This embodiment specifically discusses the data process 2 in embodiment 1). The detailed data process includes:

[0099] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0100]

[0101] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, ‖·‖ F represents the F norm, and U represents the spectral coefficient;

[0102] 2.2) The spectral dictionary D is learned using the non-negative structured sparse dictionary learning method proposed by Dong Weisheng.

[0103] Example 3

[0104] This embodiment specifically discusses the data process 3 in embodiment 1). The detailed data process includes:

[0105] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, G5 to matrices with all elements equal to 0, and initialize the penalty factor μ.

[0106] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0107]

[0108] μ=γμ (8)

[0109] In the above formula, X represents the hyperspectral image, Y represents the multispectral image, B represents the spatial blur block circulant matrix, S represents the spatial downsampling matrix, R represents the spectral response function, ‖·‖ F represents the F norm, ‖·‖1 represents the 1 norm, ‖·‖ * represents the nuclear norm, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0110] The data process of establishing the sub-problem solving model shown in equations (1)-(8) includes:

[0111] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y and the target high spatial resolution hyperspectral fusion image Z:

[0112] X=ZBS+N H Y=RZ + N M (10)

[0113] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents zero-mean Gaussian noise in the observation model;

[0114] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. Then the relationship between the hyperspectral image X to be fused, the multispectral image Y and the target fusion image Z can be represented as follows:

[0115] X=DABS+N HY=RDA+N M (11)

[0116] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents zero-mean Gaussian noise in the observation model;

[0117] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0118]

[0119] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0120] 3.2.4) The prior information of the estimated spatial coefficient A is introduced into the basic optimization model to obtain the following optimization model:

[0121]

[0122] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the local structure low-rank prior, For additional priors;

[0123] Among them, the local structure sparse low-rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing the convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization, which can be learned from any external grayscale image;

[0124] 3.2.5) Introduce splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, A represents the spatial coefficient to be estimated, B represents the spatial fuzzy matrix, and construct the following augmented Lagrangian function:

[0125]

[0126] In the above formula, L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0127] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8);

[0128] It can be seen that the main feature of the constructed fusion model is to estimate the spatial coefficient matrix A with more detailed spatial detail information by adding more prior constraints in the fusion process. Therefore, in order to make full use of the prior information of the image to be fused and the external image, this embodiment adopts a superpixel segmentation strategy to over-segment the multispectral image Y to learn a sparse low-rank prior, and uses a convolutional neural network to learn additional priors from the external grayscale image, aiming to improve the quality of the spatial coefficient matrix.

[0129] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). When optimizing and solving, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of equation (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is optimized by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is optimized by taking the partial derivative of equation (3) and setting the partial derivative to zero to derive the optimization equation and then update the variable V1. The updated variable V2 is solved; the sub-problem solving model of variable V3 is optimized, and the updated variable V3 is solved by the soft threshold operator; the sub-problem solving model of variable V4 is optimized, and the singular value threshold algorithm is used to solve the nuclear norm optimization problem and then solve the updated variable V4; the sub-problem solving model of variable V5 is optimized, and it is regarded as the denoising problem of the image (A-G5) with Gaussian noise with variance η3 / μ, and the convolutional neural network denoising method is used to solve the updated variable V5, whose input data is the image (A-G5) and the noise variance η3 / μ, and the output data is the updated variable V5; when the number of iterations reaches the threshold, the loop is jumped out and the spatial coefficient A is output. In this way, we have realized the direct insertion of an image denoising method trained on an external arbitrary grayscale image with a known noise level into the fusion process, without the need to train on a specific image, and with strong generalization performance. The experiment in this embodiment is compatible with the training results of K. Zhang et al. on grayscale images. The first layer is a convolution operation and a linear correction unit, the second layer is a convolution operation, a linear correction unit and batch processing, and the third layer is a convolution operation.

[0130] Example 4

[0131] In order to verify the hyperspectral and multispectral image fusion method based on local sparse low-rank and convolutional neural network in this embodiment, the hyperspectral image in the Hyperspec-VNIR Chikusei data set was selected as the experimental object to carry out the fusion experiment. In the simulation experiment, the Chikusei image with a spatial resolution of 128×128 and a spectral resolution of 93 was intercepted as the reference image, and the low-resolution hyperspectral image simulation under 4 times downsampling and 5×5 Gaussian blur (variance of 2.5) was performed, and the IKONOS_Like spectral response function was used to simulate and generate multispectral images. In order to compare the fusion effect of the fusion method of this embodiment, five typical hyperspectral and multispectral image fusion methods (GLP, LTMR, CNN-Fus, LRSE, OTD) were selected for comparative analysis. At the same time, in order to quantitatively compare the fusion effects under different algorithms, the method of this embodiment respectively selected six commonly used indicators of peak signal-to-noise ratio (PSNR), root mean square error (RMSE), spectral angle mapping (SAM), structural similarity index (SSIM), distortion (DD) and correlation coefficient (CC) as quantitative evaluation indicators. Among them, the larger the PSNR, SSIM, and CC values ​​are, the higher the fusion quality is, and the smaller the RMSE, SAM, and DD values ​​are, the better the fusion effect is. Table 1 shows the quantitative evaluation indicators of the fusion results of the fusion method proposed in this embodiment and five typical fusion methods on Chikusei data, and the best effect is shown in bold in the table.

[0132] Table 1: Objective evaluation indicators of the method in this embodiment and five typical hyperspectral and multispectral fusion methods

[0133]

[0134]

[0135] It can be seen from Table 1 that all quantitative evaluation indicators of the fusion algorithm proposed in this embodiment are better than other methods. The main reason is that the fusion algorithm (SLR_CNN) proposed in this embodiment makes full use of the image sparse prior, local structure low-rank prior and additional prior information learned from external grayscale images, effectively constrains the fusion model, and thus effectively improves the fusion quality.

[0136] In order to more intuitively demonstrate the fusion quality of different algorithms, Figure 4 The fusion effect diagrams of the fusion algorithm of this embodiment and five typical fusion algorithms are given. Figure 4 It can be seen that the high-resolution hyperspectral image fused by the fusion algorithm proposed in this embodiment has the best spatial and spectral quality.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] From the above embodiments, it can be seen that this study uses the non-negative structure sparse dictionary learning method to learn a more detailed hyperspectral dictionary of spectral features, and makes full use of the sparse prior of the image, the local structure low-rank prior and the additional prior information from external data to establish a spatial coefficient matrix optimization model. With the help of a norm to increase the sparse constraint, the superpixel segmentation strategy is introduced to impose the local structure low-rank prior, and the convolutional neural network is used to learn the external grayscale image as an additional prior, which well constrains the optimization process of the spatial coefficient matrix model and obtains a spatial coefficient matrix containing more spatial detail information; the convolutional neural network additional prior used in this study does not need to change the network structure and parameters, and has good generalization ability; the fusion method of this study is suitable for remote sensing images of different sensor types, and has the advantages of good fusion effect, low time complexity and strong noise robustness. The hyperspectral and multispectral image fusion method proposed in the present invention makes full use of the sparse prior of the image, the local structure low-rank prior and the additional prior from external data, and establishes a hyperspectral image fusion model under the regularization framework. First, the non-negative structure sparse dictionary learning method is used to learn the hyperspectral image to obtain the spectral dictionary; then, the superpixel segmentation strategy is used to over-segment the multispectral image, and the spatial coefficient optimization model is constructed by adding sparse constraints, local low-rank constraints and additional constraints, and the split augmented Lagrangian shrinkage algorithm (SALSA) is used to optimize and solve the model to obtain the spatial coefficient matrix; finally, the learned spectral dictionary and the spatial coefficient matrix are fused to obtain a high spatial resolution hyperspectral target image. Since the method of this embodiment makes full use of the image prior information, the learned spatial coefficient matrix contains more detailed spatial structure information, and the spectral dictionary based on the non-negative structure sparse dictionary learning contains richer spectral features of the ground objects. The combination of the two effectively improves the fusion effect. In addition, the network model using the convolutional neural network to provide external additional priors in this embodiment is only trained on grayscale images, and does not contain spectral information and can be directly introduced into the fusion model. While effectively adding external priors to improve the fusion quality, it also expands the application of deep learning in image fusion and has good generalization performance. By comparing the effects with the other five typical fusion algorithms, the high spatial resolution hyperspectral image fused in this embodiment has better quality and lower time complexity. The convolutional neural network denoising model used does not need to change the network structure and parameters, and has strong universality and robustness. In summary, the present invention can effectively realize the spatial-spectral fusion of low spatial resolution hyperspectral images and high spatial resolution multispectral images, and has the advantages of good fusion performance, low time complexity and strong noise robustness.

[0139] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0140] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data structure for a hyperspectral image fusion algorithm based on data-oriented integration, which is used for a data-based fusion algorithm to improve the spatial resolution of hyperspectral images. The algorithm optimizes the spatial coefficient based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns the spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with a target high spatial resolution; the data structure includes the following main data layers: a low-resolution hyperspectral and multispectral input data layer; a spectral dictionary data layer; a spatial coefficient matrix data layer; a solution space data layer that characterizes the parameters of the fusion result; each data layer interacts in sequence and has global compatibility; The data-oriented integration includes at least: The data attributes of the hyperspectral and multispectral images include the following data orientations: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following data orientations: additional prior information learned from external data; the spatial coefficients adopt second-order and higher-order tensor data configurations and obey matrix multiplication operation rules; the algorithm adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combined norms, and multi-operator interactions for threshold-guided multiple iterations.

2. The data structure according to claim 1, characterized in that: In the spectral dictionary data layer, a non-negative structured sparse dictionary learning method is used to learn the spectral dictionary D from the hyperspectral image X. The hyperspectral image X is used as input data, and the solution of the spectral dictionary D is characterized as follows: Where U represents the spectral coefficient, ‖·‖ F represents the F norm; and the spectral dictionary D is learned using the existing non-negative structured sparse dictionary learning method.

3. The data structure according to claim 2, characterized in that: In the spatial coefficient matrix data layer, the number of iterations is first initialized to 1, and the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, G5 are all initialized to matrices with all elements being 0; the iterative spatial coefficient matrix A is updated according to the following formula (1): Where B represents the spatial fuzzy matrix.

4. The data structure according to claim 3, characterized in that: In the solution space data layer representing the fusion result parameters, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of the above formula (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the same data process is used for iterative update of the split variables V1, V2, V3, V4, and V5; when the number of iterations reaches a threshold, the loop is exited and the spatial coefficient A is output.

Images