Data structure used in hyperspectral image fusion algorithm

Abstract

The present invention discloses a data structure for a hyperspectral image fusion algorithm based on data-oriented integration, which includes the following main data layers: a low-resolution hyperspectral and multispectral input data layer; a spectral dictionary data layer; a spatial coefficient matrix data layer; and a solution space data layer representing fusion result parameters. Each data layer interacts in sequence and has global compatibility. This data structure is used in a data-based fusion algorithm for improving the spatial resolution of hyperspectral images. The algorithm optimizes the spatial coefficients based on the data attributes of the hyperspectral and multispectral images in combination with external additional data, learns the spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with a target high spatial resolution.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a data structure for a hyperspectral image fusion algorithm based on data-oriented integration. Background Art

[0002] Hyperspectral imaging technology can simultaneously acquire scene information at different spectral wavelengths corresponding to hundreds of spectral bands. This type of hyperspectral image has high spectral resolution and can accurately identify substances. It is widely used in remote sensing, medical diagnosis, anomaly detection, face recognition and other fields. However, due to the limitations of imaging sensors, there is a mutually restrictive relationship between spectral resolution, spatial resolution and signal-to-noise ratio. Therefore, enhancing spatial resolution has become a fundamental issue in hyperspectral imaging, which affects the potential applications of hyperspectral imaging. Due to hardware limitations, improving spatial resolution through hardware means is costly and challenging. Compared with hyperspectral images, multispectral images have lower spectral resolution, higher spatial resolution and signal-to-noise ratio, and can provide more spatial details of objects. Therefore, fusing hyperspectral and multispectral images of the same scene has become a popular method for obtaining high-spatial-resolution hyperspectral images.

[0003] Currently, hyperspectral and multispectral image fusion methods can be categorized as those based on sparse representation, tensor decomposition, matrix subspace representation, and deep learning. Researchers have also proposed numerous spatial-spectral fusion algorithms based on these approaches, achieving promising results. However, investigations and studies have revealed that existing hyperspectral and multispectral image fusion algorithms still face several challenges. Among these challenges, effectively transferring information between spatial and spectral patterns remains a challenge. Furthermore, the algorithms suffer from insufficient prior information, limited model expressiveness, high time complexity, insufficient applicability of model parameter optimization algorithms, and poor generalization performance. These issues require further research and improvement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a data structure and its application in a hyperspectral image fusion algorithm.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0006] A data structure for a hyperspectral image fusion algorithm is disclosed. This data structure is used for a data-based fusion algorithm that improves the spatial resolution of hyperspectral images. The algorithm optimizes spatial coefficients based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns a spectral dictionary using a non-negative matrix dictionary learning method, and algorithmically fuses the hyperspectral and multispectral images to obtain a hyperspectral image with a target high spatial resolution. The data structure includes the following main data layers: a low-resolution hyperspectral and multispectral input data layer; a spectral dictionary data layer; a spatial coefficient matrix data layer; and a solution space data layer that represents the parameters of the fusion result. Each data layer interacts in sequence and has global compatibility.

[0007] As a preferred technical solution of the present invention, the data-guided integration at least includes: the data attributes of the hyperspectral and multispectral images include the following data guidance: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following data guidance: additional prior information learned from external data; the spatial coefficients adopt second-order and above tensor data configurations and obey matrix multiplication operation rules; the algorithm adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combination norms, and multi-operator interactions for threshold-guided multiple iterations.

[0008] As a preferred technical solution of the present invention, in the spectral dictionary data layer, a non-negative structured sparse dictionary learning method is used to learn the spectral dictionary D from the hyperspectral image X; the hyperspectral image X is used as input data to characterize the solution of the spectral dictionary D as follows: Where U represents the spectral coefficient, ‖·‖ F represents the F norm; and the spectral dictionary D is learned using the existing non-negative structured sparse dictionary learning method.

[0009] As a preferred technical solution of the present invention, in the spatial coefficient matrix data layer, the number of initialization iterations is first set to 1, and the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 are all initialized to matrices with all elements being 0; the iterative spatial coefficient matrix A is updated according to the following formula (1):

[0010]

[0011] Where B represents the spatial fuzzy matrix.

[0012] As a preferred technical solution of the present invention, in the solution space data layer representing the parameters of the fusion result, a sub-problem optimization solution model for the spatial coefficient matrix A is obtained by taking the partial derivative of the above formula (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the same data process is used for the split variables V1, V2, V3, V4, and V5 for iterative update; when the number of iterations reaches a threshold, the loop is exited and the spatial coefficient A is output.

[0013] The present invention also includes the following contents:

[0014] A data-based fusion algorithm for improving the spatial resolution of hyperspectral images is proposed. The algorithm optimizes the spatial coefficients based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns the spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with the target high spatial resolution.

[0015] As a preferred technical solution of the present invention, the algorithm includes the following main data structures in sequence according to the data derivation process: low-resolution hyperspectral and multispectral input data layer; spectral dictionary data layer; spatial coefficient matrix data layer; solution space data layer that characterizes the fusion result parameters; each data layer interacts in sequence and has global compatibility.

[0016] As a preferred technical solution of the present invention, the data attributes of the hyperspectral and multispectral images include the following optional data guidance: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following optional data guidance: additional prior information learned from external data; the spatial coefficients adopt second-order and above tensor data configurations and obey matrix multiplication operation rules; the algorithm fusion adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combination norms, and multi-operator interactions for threshold-guided multiple iterations.

[0017] As a preferred technical solution of the present invention, the algorithm includes the following optional data processes:

[0018] 1) Simulation of low-resolution input data and construction of observation models;

[0019] 2) Using non-negative structured sparse dictionary learning method to learn spectral dictionary D from hyperspectral image X;

[0020] 3) Using hyperspectral image X, multispectral image Y and spectral dictionary D, a spatial coefficient optimization model is constructed to estimate the spatial coefficient A;

[0021] 4) Use the learned spectral dictionary D and spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fusion image Z.

[0022] As a preferred technical solution of the present invention, the data process of data process 2) includes:

[0023] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0024]

[0025] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, and U represents the spectral coefficient;

[0026] 2.2) Use the existing non-negative structured sparse dictionary learning method to learn the spectral dictionary D.

[0027] As a preferred technical solution of the present invention, the data process of data process 3) includes:

[0028] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 to matrices with all elements set to 0, and initialize the penalty factor μ;

[0029] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, and V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, and G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0030]

[0031]

[0032] μ=γμ (8)

[0033] In the above formula, X represents hyperspectral image, Y represents multispectral image, B represents spatial blur block circulant matrix, S represents spatial downsampling matrix, R represents spectral response function, ‖·‖1 represents 1 norm, ‖·‖ * represents the nuclear norm, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0034] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). During the optimization, the sub-problem optimization solution model of the spatial coefficient matrix A is obtained by taking the partial derivative of equation (1) and setting it to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is obtained by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is obtained by taking the partial derivative of equation (3) and setting it to zero to derive the optimization equation and then update the coefficient matrix A. And solve the updated variable V2; for the sub-problem optimization solution model of variable V3, the soft threshold operator is used to solve the updated variable V3; for the sub-problem optimization solution model of variable V4, the singular value threshold algorithm is used to solve the nuclear norm optimization problem and then solve the updated variable V4; for the sub-problem optimization solution model of variable V5, it is regarded as the denoising problem of the image (A-G5) with Gaussian noise with variance η3 / μ, and the convolutional neural network denoising method is used to solve the updated variable V5, whose input data is the image (A-G5) and the noise variance η3 / μ, and the output data is the updated variable V5; when the number of iterations reaches the threshold, the loop is exited and the spatial coefficient A is output.

[0035] As a preferred technical solution of the present invention, in data process 3.2, the data process of establishing the sub-problem solving model shown in formulas (1)-(8) includes:

[0036] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y, and the target high spatial resolution hyperspectral fusion image Z:

[0037] X=ZBS + N H Y=RZ + N M (10)

[0038] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents the zero-mean Gaussian noise in the observation model;

[0039] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. The relationship between the hyperspectral image to be fused X, the multispectral image Y, and the target fusion image Z can be represented as follows:

[0040] X=DABS+N H Y=RDA + N M (11)

[0041] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents the zero-mean Gaussian noise in the observation model;

[0042] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0043]

[0044] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial blur downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0045] 3.2.4) The following optimization model is obtained by introducing the prior information of the spatial coefficient A to be estimated into the basic optimization model:

[0046]

[0047] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the low-rank prior for local structure, is an additional prior;

[0048] 3.2.5) Introduce the splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, where A represents the spatial coefficient to be estimated and B represents the spatial fuzzy matrix. Construct the following augmented Lagrangian function:

[0049]

[0050] In the above formula, L(A,V1,V2,V3,V4,V5,G1,G2,G3,G4,G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0051] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in Equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8).

[0052] As a preferred technical solution of the present invention, in the data process 3.2.4), the local structure sparse low rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of the local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing a convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization, which can be learned from any external grayscale image.

[0053] As a preferred technical solution of the present invention, the algorithm includes the following optional data processes:

[0054] 1) Simulation of low-resolution input data and construction of observation models;

[0055] 2) Using a non-negative structured sparse dictionary learning method to learn a spectral dictionary D from the hyperspectral image X; including:

[0056] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0057]

[0058] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, and U represents the spectral coefficient;

[0059] 2.2) Use the non-negative structured sparse dictionary learning method proposed by Dong Weisheng to learn the spectral dictionary D;

[0060] 3) Using hyperspectral image X, multispectral image Y and spectral dictionary D, a spatial coefficient optimization model is constructed to estimate the spatial coefficient A;

[0061] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 to matrices with all elements set to 0, and initialize the penalty factor μ;

[0062] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, and V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, and G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0063]

[0064] μ=γμ (8)

[0065] In the above formula, X represents hyperspectral image, Y represents multispectral image, B represents spatial blur block circulant matrix, S represents spatial downsampling matrix, R represents spectral response function, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0066] The data process of establishing the model for solving the sub-problems of equations (1)-(8) includes:

[0067] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y, and the target high spatial resolution hyperspectral fusion image Z:

[0068] X=ZBS + N H Y=RZ + N M (10)

[0069] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents the zero-mean Gaussian noise in the observation model;

[0070] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. The relationship between the hyperspectral image to be fused X, the multispectral image Y, and the target fusion image Z can be represented as follows:

[0071] X=DABS+N H Y=RDA + N M (11)

[0072] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents the zero-mean Gaussian noise in the observation model;

[0073] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0074]

[0075] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial blur downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0076] 3.2.4) The following optimization model is obtained by introducing the prior information of the spatial coefficient A to be estimated into the basic optimization model:

[0077]

[0078] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the low-rank prior for local structure, is an additional prior; in this data process, the local structure sparse low-rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of the local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing the convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization. It can be learned from any external grayscale image.

[0079] 3.2.5) Introduce the splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, where A represents the spatial coefficient to be estimated and B represents the spatial fuzzy matrix. Construct the following augmented Lagrangian function:

[0080]

[0081] In the above formula, L(A,V1,V2,V3,V4,V5,G1,G2,G3,G4,G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0082] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in Equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8);

[0083] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). During the optimization, the sub-problem optimization solution model of the spatial coefficient matrix A is obtained by taking the partial derivative of equation (1) and setting it to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is obtained by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is obtained by taking the partial derivative of equation (3) and setting it to zero to derive the optimization equation and then update the coefficient matrix A. And solve the updated variable V2; for the sub-problem optimization solution model of variable V3, the soft threshold operator is used to solve the updated variable V3; for the sub-problem optimization solution model of variable V4, the singular value threshold algorithm is used to solve the nuclear norm optimization problem and then solve the updated variable V4; for the sub-problem optimization solution model of variable V5, it is regarded as a denoising problem of an image (A-G5) with Gaussian noise of variance η3 / μ, and the convolutional neural network denoising method is used to solve the updated variable V5, whose input data is the image (A-G5) and the noise variance η3 / μ, and the output data is the updated variable V5; when the number of iterations reaches the threshold, the loop is jumped out and the spatial coefficient A is output;

[0084] 4) Use the learned spectral dictionary D and spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fusion image Z.

[0085] The present invention also includes a data model, a database group, a data platform and applications thereof constructed by the above algorithm.

[0086] The beneficial effects of adopting the above technical solution are: Overall, the present invention makes full use of the sparse priors, low-rank structural priors of hyperspectral and multispectral images and the additional prior information learned from external data to optimize the spatial coefficients, and uses the non-negative matrix dictionary learning method to learn the spectral dictionary, and then fuses the target high-spatial-resolution hyperspectral image, which can ensure the fusion efficiency on the basis of effectively improving the fusion quality, has a clear data structure, and the model has good generalization ability and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is the overall framework diagram of the present invention.

[0088] Figure 2 This is a basic flow chart for constructing an image fusion model for the present invention. The figure contains all the contents of data process 1) to data process 4). Figure 2 It can be seen that the hyperspectral image fusion problem is transformed into the estimation problem of the spectral dictionary D and the spatial coefficient A.

[0089] Figure 3 Schematic diagram of the segmentation effect before and after super pixel segmentation of the present invention. Figure 3 It can be seen that the multispectral image Y after superpixel segmentation has local low-rank structural characteristics.

[0090] Figure 4 The fusion effect diagram of five fusion algorithms on the Hyperspec-VNIR Chikusei dataset. Figure 4 It can be seen that the high-resolution hyperspectral image obtained by the fusion algorithm proposed in the present invention has the best spatial and spectral quality. DETAILED DESCRIPTION

[0091] The following examples illustrate the present invention in detail. The various raw materials and equipment used in the present invention are conventional commercial products and can be directly obtained by purchasing from the market. In the description of the following examples, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to provide a thorough understanding of the present application embodiments. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0092] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, data processes, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, data processes, operations, elements, components and / or their collections. It should also be understood that the term "and / or" used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. As used in this specification and the appended claims, the term "if" can be interpreted as "when..." or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context. In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0093] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0094] Example 1

[0095] See attached Figure 1-2This embodiment provides a data-based fusion algorithm for improving the spatial resolution of hyperspectral images. The algorithm optimizes spatial coefficients based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns the spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with the target high spatial resolution. The data attributes of the hyperspectral and multispectral images include the following optional data guidance: sparse priors and low-rank structural priors of the hyperspectral and multispectral images; the external additional data includes the following optional data guidance: additional prior information learned from external data; the spatial coefficients are configured as second-order and higher-order tensor data and obey the matrix multiplication operation rules; the algorithm fusion is constructed based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combination norms, and multi-operator interactions for threshold-guided multiple iterations for composite construction.

[0096] The main data integration of the algorithm in this embodiment includes: 1) simulation of low-resolution input data and construction of an observation model; 2) using a non-negative structured sparse dictionary learning method to learn a spectral dictionary D from a hyperspectral image X; 3) using the hyperspectral image X, the multispectral image Y, and the spectral dictionary D to construct a spatial coefficient optimization model and estimate the spatial coefficient A; 4) using the learned spectral dictionary D and the spatial coefficient A to perform matrix multiplication to obtain the target high-resolution hyperspectral fused image Z.

[0097] Example 2

[0098] This embodiment specifically discusses the data process 2 in embodiment 1. The detailed data process includes:

[0099] 2.1) Taking the hyperspectral image X as input data, the problem of solving the spectral dictionary D can be expressed as the following optimization problem:

[0100]

[0101] In the above formula, D represents the spectral dictionary to be optimized, X is the hyperspectral image, ‖·‖ F represents the F norm, U represents the spectral coefficient;

[0102] 2.2) The spectral dictionary D is learned using the non-negative structured sparse dictionary learning method proposed by Dong Weisheng.

[0103] Example 3

[0104] This embodiment specifically discusses the data process 3 in embodiment 1. The detailed data process includes:

[0105] 3.1) First, initialize the number of iterations to 1, initialize the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 to matrices with all elements set to 0, and initialize the penalty factor μ.

[0106] 3.2) Perform k iterations and iteratively update the model parameters using equations (1) to (8): update the iteration space coefficient matrix A according to equation (1), update the split variables V1, V2, V3, V4, and V5 according to equations (2) to (6), update the Lagrange multipliers G1, G2, G3, G4, and G5 according to equation (7), and update the penalty factor μ according to equation (8);

[0107]

[0108] μ=γμ (8)

[0109] In the above formula, X represents hyperspectral image, Y represents multispectral image, B represents spatial blur block circulant matrix, S represents spatial downsampling matrix, R represents spectral response function, ‖·‖ F represents the F norm, ‖·‖1 represents the 1 norm, ‖·‖ * represents the nuclear norm, η1, η2, η3, λ1 represent regularization parameters, represents the additional prior introduced, γ>1 is the update coefficient;

[0110] The data process of establishing the sub-problem solving model shown in formulas (1)-(8) includes:

[0111] 3.2.1) Establish the association representation model between the low spatial resolution hyperspectral image X to be fused, the high spatial resolution multispectral image Y, and the target high spatial resolution hyperspectral fusion image Z:

[0112] X=ZBS + N H Y=RZ + N M (10)

[0113] In the above formula, Z represents the target fusion image, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral response function. H and N M represents the zero-mean Gaussian noise in the observation model;

[0114] 3.2.2) The target fusion image Z can be further described as a linear representation of the spectral dictionary D and the spatial coefficient A. The relationship between the hyperspectral image to be fused X, the multispectral image Y, and the target fusion image Z can be represented as follows:

[0115] X=DABS+N HY=RDA + N M (11)

[0116] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial fuzzy downsampling matrix, S is the downsampling matrix, and N is the spectral dictionary. H and N M represents the zero-mean Gaussian noise in the observation model;

[0117] 3.2.3) According to equations (10) and (11), the basic optimization model of the spatial coefficient A is constructed as follows:

[0118]

[0119] In the above formula, X is the hyperspectral image, Y is the multispectral image, D is the spectral dictionary, A is the spatial coefficient to be optimized, R is the spectral response function, B is the spatial blur downsampling matrix, S is the downsampling matrix, and λ1 is the regularization parameter;

[0120] 3.2.4) The following optimization model is obtained by introducing the prior information of the spatial coefficient A to be estimated into the basic optimization model:

[0121]

[0122] In the above formula, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, ||A||1 is the sparse prior, is the low-rank prior for local structure, is an additional prior;

[0123] Among them, the local structure sparse low-rank prior constraint By implementing the superpixel segmentation strategy on the high spatial resolution multispectral image Y, we can obtain K superpixels of Y. Then use the nuclear norm ||A i || * Approximately express the low-rank prior of each superpixel, impose the low-rank prior of the local structure on the sparse coefficients of each spectral pixel in the same superpixel; additional prior imposed on the optimization model of the spatial coefficient A By introducing the convolutional neural network denoising algorithm for prior constraints, the convolutional neural network does not involve spectral information and can be directly inserted into the fusion process with strong generalization. It can be learned from any external grayscale image.

[0124] 3.2.5) Introduce the splitting variables V1 = AB, V2 = A, V3 = A, V4 = A, V5 = A, where A represents the spatial coefficient to be estimated and B represents the spatial fuzzy matrix. Construct the following augmented Lagrangian function:

[0125]

[0126] In the above formula, L(A,V1,V2,V3,V4,V5,G1,G2,G3,G4,G5) represents the Lagrangian function to be optimized and augmented, A is the spatial coefficient matrix to be estimated, X is the hyperspectral image, Y is the multispectral image, R is the spectral response function, B is the spatial fuzzy matrix, S is the downsampling matrix, D is the spectral dictionary, η1, η2, η3, λ1 represent regularization parameters, V1, V2, V3, V4, V5 represent splitting variables, G1, G2, G3, G4, G5 represent Lagrangian multipliers, and μ represents the penalty factor (μ>0);

[0127] 3.2.6) The Lagrangian function L(A, V1, V2, V3, V4, V5, G1, G2, G3, G4, G5) to be optimized and augmented as shown in Equation (14) is transformed and decomposed into multiple sub-problem solution models as shown in (1)-(8);

[0128] As can be seen, the main feature of the constructed fusion model is that it estimates the spatial coefficient matrix A with more detailed spatial detail information by adding more prior constraints during the fusion process. Therefore, to fully utilize the prior information of the image to be fused and the external image, this embodiment adopts a superpixel segmentation strategy to over-segment the multispectral image Y to learn a sparse low-rank prior, and uses a convolutional neural network to learn additional priors from the external grayscale image to improve the quality of the spatial coefficient matrix.

[0129] 3.3) The augmented Lagrangian shrinkage algorithm SALSA is used to iteratively optimize and solve the sub-problems in equations (1) to (8). During the optimization, the sub-problem optimization solution model of the spatial coefficient matrix A is obtained by taking the partial derivative of equation (1) and setting it to zero to derive the optimization equation and then update the coefficient matrix A; the sub-problem optimization solution model of the variable V1 is obtained by constructing a downsampling matrix composed of selected and unselected down-sampled pixels as a mask matrix, and taking the partial derivative of equation (2) to obtain the optimization equation to solve the updated variable V1; the sub-problem optimization solution model of the variable V2 is obtained by taking the partial derivative of equation (3) and setting it to zero to derive the optimization equation and then update the coefficient matrix A. The updated variable V2 is solved. A soft threshold operator is used to solve the updated variable V3. A singular value threshold algorithm is used to solve the nuclear norm optimization problem and then solve the updated variable V4. A convolutional neural network denoising method is used to solve the updated variable V5. The input data for the subproblem of variable V5 is the image (A-G5) with Gaussian noise of variance η3 / μ, and the output data is the updated variable V5. When the number of iterations reaches the threshold, the loop is exited and the spatial coefficient A is output. In this way, we have directly inserted an image denoising method trained on an arbitrary external grayscale image with a known noise level into the fusion process, eliminating the need for training on specific images and achieving strong generalization performance. The experiment in this embodiment is compatible with the training results of K. Zhang et al. on grayscale images. The first layer is a convolution operation and a linear correction unit, the second layer is a convolution operation, a linear correction unit and batch processing, and the third layer is a convolution operation.

[0130] Example 4

[0131] To validate the hyperspectral and multispectral image fusion method based on local sparse low-rank and convolutional neural networks in this embodiment, fusion experiments were conducted using hyperspectral images from the Hyperspec-VNIR Chikusei dataset. In the simulation, a Chikusei image with a spatial resolution of 128×128 and a spectral resolution of 93 was used as a reference image. A low-resolution hyperspectral image simulation was performed using 4x downsampling and a 5×5 Gaussian blur (with a variance of 2.5). The IKONOS_Like spectral response function was then used to simulate and generate a multispectral image. To compare the fusion performance of this embodiment's fusion method, five typical hyperspectral and multispectral image fusion methods (GLP, LTMR, CNN-Fus, LRSE, and OTD) were selected for comparative analysis. Furthermore, to quantitatively compare the fusion performance of different algorithms, six commonly used metrics were used as quantitative evaluation indicators: peak signal-to-noise ratio (PSNR), root mean square error (RMSE), spectral angle mapping (SAM), structural similarity index (SSIM), distortion factor (DD), and correlation coefficient (CC). Among them, larger PSNR, SSIM, and CC values ​​indicate higher fusion quality, while smaller RMSE, SAM, and DD values ​​indicate better fusion effect. Table 1 shows the quantitative evaluation indicators of the fusion results of the fusion method proposed in this example and five typical fusion methods on Chikusei data. The best results are shown in bold in the table.

[0132] Table 1: Objective evaluation indicators of the method in this embodiment and five typical hyperspectral and multispectral fusion methods

[0133]

[0134]

[0135] As can be seen from Table 1, all quantitative evaluation indicators of the fusion algorithm proposed in this embodiment are superior to those of other methods. The main reason is that the fusion algorithm (SLR_CNN) proposed in this embodiment makes full use of the image sparse prior, the local structure low-rank prior, and the additional prior information learned from the external grayscale image, effectively constraining the fusion model and thus effectively improving the fusion quality.

[0136] In order to more intuitively demonstrate the fusion quality of different algorithms, Figure 4 The fusion effect diagrams of this embodiment's fusion algorithm and five typical fusion algorithms are given. Figure 4 It can be seen that the high-resolution hyperspectral image obtained by the fusion algorithm proposed in this embodiment has the best spatial and spectral quality.

[0137] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0138] From the above examples, it can be seen that this study uses a non-negative structured sparse dictionary learning method to learn a hyperspectral dictionary with more detailed spectral features, and fully utilizes the sparse prior of the image, the low-rank prior of the local structure, and the additional prior information from external data to establish a spatial coefficient matrix optimization model. With the help of a norm to increase the sparse constraint, the superpixel segmentation strategy is introduced to impose the low-rank prior of the local structure, and the convolutional neural network is used to learn the external grayscale image as an additional prior, which well constrains the optimization process of the spatial coefficient matrix model and obtains a spatial coefficient matrix containing more spatial detail information; the additional prior of the convolutional neural network used in this study does not require changing the network structure and parameters and has good generalization ability; the fusion method of this study is applicable to remote sensing images of different sensor types and has the advantages of good fusion effect, low time complexity, and strong noise robustness. The hyperspectral and multispectral image fusion method proposed in this invention fully utilizes the sparse prior of the image, the low-rank prior of the local structure, and the additional prior from external data to establish a hyperspectral image fusion model under the regularization framework. First, the non-negative structured sparse dictionary learning method is used to learn the hyperspectral image to obtain a spectral dictionary; then, the superpixel segmentation strategy is used to over-segment the multispectral image, and a spatial coefficient optimization model is constructed by adding sparse constraints, local low-rank constraints and additional constraints. The model is optimized and solved using the split augmented Lagrangian shrinkage algorithm (SALSA) to obtain a spatial coefficient matrix; finally, the learned spectral dictionary and the spatial coefficient matrix are fused to obtain a high spatial resolution hyperspectral target image. Since the method of this embodiment makes full use of the image prior information, the learned spatial coefficient matrix contains more detailed spatial structure information, and the spectral dictionary learned based on the non-negative structured sparse dictionary contains richer spectral features of the ground objects. The combination of the two effectively improves the fusion effect. In addition, the network model using the convolutional neural network to provide external additional priors in this embodiment is only trained on grayscale images and does not contain spectral information, which can be directly introduced into the fusion model. While effectively adding external priors to improve the fusion quality, it also expands the application of deep learning in image fusion and has good generalization performance. Comparing the results of five other typical fusion algorithms, the high-spatial-resolution hyperspectral image fused in this embodiment has better quality and lower temporal complexity. The convolutional neural network denoising model used does not require changes to the network structure and parameters, and has strong universality and robustness. In summary, this invention can effectively achieve spatial-spectral fusion of low-spatial-resolution hyperspectral images and high-spatial-resolution multispectral images, with advantages such as good fusion performance, low temporal complexity, and strong noise robustness.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0140] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A hyperspectral image fusion algorithm based on data-oriented integration, characterized by: The algorithm optimizes spatial coefficients based on the data attributes of hyperspectral and multispectral images in combination with external additional data, learns a spectral dictionary using a non-negative matrix dictionary learning method, and performs algorithmic fusion on the hyperspectral and multispectral images to obtain a hyperspectral image with the target high spatial resolution. The data structure embedded in the algorithm includes the following main data layers: a low-resolution hyperspectral and multispectral input data layer; a spectral dictionary data layer; a spatial coefficient matrix data layer; and a solution space data layer representing the parameters of the fusion result. Each data layer interacts sequentially and has global compatibility. The data-oriented integration includes at least: the data attributes of the hyperspectral and multispectral images include the following data orientations: sparse priors and low-rank structural priors of hyperspectral and multispectral images; the external additional data includes the following data orientations: additional prior information learned from external data; the spatial coefficients adopt second-order and above tensor data configurations and obey matrix multiplication operation rules; the algorithm adopts a composite construction based on local sparse low-rank and convolutional neural networks and includes and is compatible with multi-dimensional parameters, combination norms, and multi-operator interactions for threshold-guided multiple iterations.

2. The algorithm according to claim 1, characterized in that: In the spectral dictionary data layer, a non-negative structured sparse dictionary learning method is used to learn the spectral dictionary D from the hyperspectral image X. With the hyperspectral image X as input data, the solution of the spectral dictionary D is represented as: Where U represents the spectral coefficient, ‖·‖ F represents the F norm; and the existing non-negative structured sparse dictionary learning method is used to learn the spectral dictionary D.

3. The algorithm according to claim 2, characterized in that: In the spatial coefficient matrix data layer, the number of iterations is first initialized to 1, and the spatial coefficient matrix A to be estimated, the introduced splitting variables V1, V2, V3, V4, V5, and the Lagrange multipliers G1, G2, G3, G4, and G5 are all initialized to matrices with all elements set to 0; the iterative spatial coefficient matrix A is updated according to the following formula (1): Where B represents the spatial fuzzy matrix.

4. The algorithm according to claim 3, characterized in that: In the solution space data layer representing the parameters of the fusion result, the sub-problem optimization solution model of the spatial coefficient matrix A is optimized by taking the partial derivative of the above formula (1) and setting the partial derivative to zero to derive the optimization equation and then update the coefficient matrix A; the same data process is used for the split variables V1, V2, V3, V4, and V5 to perform iterative updates; when the number of iterations reaches the threshold, the loop is exited and the spatial coefficient A is output.

Images