Panchromatic sharpening method and device based on deep subspace embedding
Through deep subspace embedding technology, combined with neural networks and mathematical optimization models, the problems of spectral distortion and ringing effects in full color sharpening are solved, and the accuracy and generalization performance of high-resolution multispectral images of remote sensing images are improved. Achieve more efficient full-color sharpening effect.
Patent Information
- Application Number
- CN202210227879.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-08
AI Technical Summary
The existing full-color sharpening methods have problems such as spectral distortion, ringing effect and data dependence, and the generalization performance is low, making it difficult to effectively solve the accuracy and efficiency of high-resolution multispectral images of remote sensing images.
Using a full-color sharpening method based on deep subspace embedding, the subspace characterization and mathematical optimization model are combined, and iteratively solve using neural networks and alternative direction multiplier methods to generate high-resolution multispectral images, combining variational optimization and deep learning technology to improve spectral fidelity and generalization performance.
It achieves better edge texture structure and spectral information accuracy for high-resolution multispectral images, improves the accuracy, generalization and efficiency of the algorithm, and reduces spectral distortion and ringing effects.
Smart Images

Figure CN114677293B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a full-color sharpening method and a device based on deep subspace embedding, and belongs to the field of full-color sharpening. Background Art
[0002] Panchromatic sharpening technology refers to panchromatic sharpening of panchromatic images and multispectral images collected by the same remote sensing satellite at the same time, obtaining rich spectral information of multispectral images while ensuring high spatial resolution, and finally obtaining high-resolution multispectral images. The huge advantage of panchromatic sharpening technology enables remote sensing images to be used for greater purposes, broadening the use scenarios and effects of remote sensing images. Therefore, the study of panchromatic sharpening is of great significance to scientific research, environmental protection, economic development, etc. in the country and even the world.
[0003] At present, there are three main types of traditional pan-sharpening methods: component replacement-based methods, multi-resolution analysis-based methods, and variational optimization-based methods. The component replacement method uses a panchromatic image to replace a part of the spatial information in the multispectral image after the projection transformation, and then generates a multispectral image with high resolution through inverse transformation. This type of method is simple to implement and has a fast operation speed, but it has a large dependence on the correlation between the panchromatic image and the replaced component. Therefore, due to the mismatch between the panchromatic image and the multispectral image, the multispectral image after pan-sharpening has serious spectral distortion. The pan-sharpening method based on multi-resolution analysis uses multi-scale decomposition technology, which applies a spatial filter to the panchromatic image to generate spatial information injected into the multispectral image. The multi-resolution analysis algorithm can ensure good spectral characteristics and effectively solve the problem of spectral distortion, but it is complex to implement and has a large computational burden and there will be degradation of spatial information, such as ringing effect. The panchromatic sharpening algorithm based on the variational optimization framework makes assumptions about the correlation between the spatial structure and spectral information of the multispectral image, the panchromatic image and the predicted image, and constructs a regularization term to solve the minimum value of the function to obtain a high-resolution multispectral image. This type of method has high spectral fidelity but has many problems, such as more parameters, complex model solving process, insufficient generalization ability, and unsatisfactory time efficiency. Therefore, this type of method still has many areas that need to be improved.
[0004] The above-mentioned classic panchromatic sharpening methods are prone to serious spectral distortion, ringing effect, lack of priors and other problems, and have certain limitations. With the successful application of deep learning in the field of image processing, panchromatic sharpening methods have made new developments. Masi et al. first proposed a panchromatic sharpening method based on convolutional neural networks, which uses a three-layer convolutional super-resolution reconstruction network. It includes nonlinear mapping of the input image and various radiation indices, which improves the performance of the method without increasing the complexity. Methods based on deep learning can provide better panchromatic sharpening results than other types of methods, but this requires more computing time and a large amount of training data sets. This type of method is too data-dependent and has low generalization performance. Summary of the invention
[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method and device for panchromatic sharpening based on deep subspace embedding, which can solve spectral distortion and improve data generalization performance.
[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:
[0007] In a first aspect, the present invention provides a full color sharpening method based on deep subspace embedding, the method comprising the following steps:
[0008] Step 1: Obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images PAN and multispectral images MS with exactly the same acquisition time and acquisition area;
[0009] Step 2: Based on the multispectral image MS, perform subspace representation on the MS data to obtain the subspace basis E;
[0010] Step 3: Input the panchromatic image PAN and the multispectral image MS into the neural network PanNet for panchromatic sharpening to obtain a high-resolution multispectral image X net ;
[0011] Step 4: According to PAN graph, MS graph, X net , build a mathematical model and obtain a mathematical optimization model;
[0012] Step 5: Solve the mathematical optimization model by using the alternative direction multiplier method, and iterate the X, U, and Z variables in turn until the termination condition is met; finally, the final result of the variable X is obtained, which is the target high-resolution multispectral image of the full-color sharpening task.
[0013] Furthermore, in step 2, the method for performing subspace characterization on MS data specifically includes:
[0014] Set the subspace dimension k of the subspace basis E to 5;
[0015] Perform singular value decomposition on MS data;
[0016] The subspace basis E is obtained from the first five left singular vectors of the MS data.
[0017] Furthermore, the mathematical optimization model is:
[0018]
[0019] stU=XB,EZ=X
[0020] Where, P represents the PAN image, Y represents the MS image, X represents the target result of the full color sharpening task, initialized to 0; U and Z are auxiliary variables initialized to 0 for auxiliary iteration X; λ, α, τ, β are weights, initialized to 0.001; S represents a sampling matrix composed of sparse components; B represents the Gaussian blur kernel; represents the upsampled version of the PAN graph; represent is a low-pass version of ; φ(·) represents the image prior.
[0021] Furthermore, in step 5, the iterative solution method for the mathematical optimization model specifically includes the following steps:
[0022] First, multiple auxiliary variables are introduced as constraints of the fusion model, and finally it is divided into multiple sub-problems and then solved iteratively, and the mathematical optimization model is Lagrangian expanded:
[0023]
[0024] Among them, ∧1, ∧2, η1, η2 are introduced auxiliary variables, initialized to 0;
[0025] According to the idea of fixing other variables while updating the current variable, the above formula is solved by alternating iterations in sequence:
[0026]
[0027]
[0028]
[0029] Step4: update ∧1, ∧2;
[0030] Iterate the X, U, and Z variables in sequence until |X is satisfied i+1 -X i |<10 -5The final result of variable X is obtained, which is the target high-resolution multispectral image of the panchromatic sharpening task.
[0031] Further, iterative solution: The formula methods include:
[0032]
[0033] The above formula can be rewritten as:
[0034]
[0035] in:
[0036]
[0037] g(x)=||X||1
[0038]
[0039] Introduce the proximal gradient descent algorithm and let X k represents the result after the kth update of X. For X→X k When , f(X) is expressed as:
[0040]
[0041]
[0042] Where L>=1, is the step length, so we get the iteration X k+1 The formula is:
[0043]
[0044]
[0045]
[0046] Finally, we get: and X k+1 The output is X i+1 ;
[0047] Further, iterative solution The methods include:
[0048]
[0049] make: in
[0050] , The symbol represents the Kronecker product, vec(·) represents the vectorization operator, According to the above formula, we can deduce:
[0051]
[0052] According to the above formula, we can directly calculate U i+1 .
[0053] Further, iterative solution The methods include:
[0054]
[0055]
[0056] Since E is a semi-orthogonal matrix, this means that for all Has isometric properties:
[0057] ||Ex||2=||x||2
[0058] So we can get:
[0059]
[0060] make:
[0061] get: Directly introduce the "three-dimensional block matching algorithm" to solve this problem, and the output is Z i+1 .
[0062] In a second aspect, the present invention provides a full color sharpening device based on deep subspace embedding, comprising:
[0063] Image acquisition module: used to obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images PAN and multispectral images MS with exactly the same acquisition time and acquisition area;
[0064] Subspace module: used to perform subspace representation on MS data according to the multispectral image MS and obtain the subspace basis E;
[0065] Neural network module: used to input the panchromatic image PAN and the multispectral image MS into the neural network PanNet for panchromatic sharpening to obtain a high-resolution multispectral image X net ;
[0066] Mathematical modeling module: used to calculate the net , build a mathematical model and obtain a mathematical optimization model;
[0067] Sharpening output module: used to solve the mathematical optimization model by using the alternative direction multiplier method, and iterate the X, U, and Z variables in turn until the termination condition is met; finally, the final result of the variable X is obtained, which is the target high-resolution multispectral image of the full-color sharpening task.
[0068] In a third aspect, an embodiment of the present invention further provides a full color sharpening device based on deep subspace embedding, including a processor and a storage medium;
[0069] The storage medium is used to store instructions;
[0070] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.
[0071] Compared with the prior art, the present invention has the following beneficial effects:
[0072] The present invention proposes a panchromatic sharpening method based on deep subspace embedding. The method proposes a universal panchromatic sharpening framework, which combines traditional variational optimization and the proposed deep subspace technology for panchromatic sharpening, aiming to obtain advantages from both variational optimization and deep learning methods, and utilize the good modeling interpretation and data generalization performance of variational optimization methods and the accuracy of deep learning technology. In particular, the introduction of deep subspace can better process multi-spectral high-dimensional and huge data, so that the panchromatic sharpening algorithm has better accuracy, generalization and efficiency.
[0073] The present invention proposes a panchromatic sharpening method based on deep subspace embedding. The high-resolution multispectral image obtained by this method has better edge texture structure and more accurate spectral information and spatial information, and the generalization performance of this method is also excellent. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 It is the overall framework diagram of the model of the present invention.
[0075] Figure 2 , Figure 3 The target enhancement effect of the method of the present invention is demonstrated.
[0076] Figure 4 , Figure 5 It is an intuitive visual comparison display of different comparison methods. DETAILED DESCRIPTION
[0077] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0078] Embodiment 1:
[0079] A method for full color sharpening based on deep subspace embedding is provided, the method comprising the following steps:
[0080] Panchromatic image and low-resolution multispectral image to be pan-sharpened;
[0081] The panchromatic image to be panchromatically sharpened and the low-resolution multispectral image are input into a MATLAB program written according to the panchromatic sharpening method based on deep subspace embedding to obtain a high-resolution multispectral image.
[0082] Specifically, the method for obtaining the full color sharpening result includes:
[0083] Step 1: Obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images (PAN) and multispectral images (MS) with exactly the same acquisition time and acquisition area;
[0084] Step 2: Based on the MS graph data obtained in the previous step, perform subspace representation on the MS data to obtain the subspace basis (represented by E);
[0085] Step 3: Based on the MS and PAN images obtained in step 1, input them into the PanNet neural network designed by Dr. Yang Junfeng for panchromatic sharpening to obtain a high-resolution multispectral image output by the network (using X net express);
[0086] Step 4: According to the above steps, the PAN graph (indicated by P), MS graph (indicated by Y), and X net , the mathematical model of the method of the present invention is constructed to obtain a mathematical optimization model (where X represents the target result of the full color sharpening task, initialized to 0; U and Z are auxiliary variables initialized to 0 for auxiliary iteration X; λ, α, τ, β are weights, initialized to 0.001; S represents a sampling matrix composed of sparse components; B represents a Gaussian blur kernel; represents the upsampled version of the PAN graph; represent The low-pass version of φ(·) represents the image prior):
[0087]
[0088] stU=XB,EZ=X
[0089] Step 5: Solve the mathematical optimization model in step 4 by using the alternative direction multiplier method, iterating the X, U, and Z variables in turn until |X is satisfied. i+1 -X i |<10 -5The final result of variable X is the target high-resolution multispectral image of the panchromatic sharpening task.
[0090] Specific analysis of the steps:
[0091] In step 2, the method for performing subspace characterization on MS data specifically includes: first setting the subspace dimension k of the subspace basis E to 5, then performing singular value decomposition on the MS data, and finally obtaining the subspace basis E from the first 5 left singular vectors of the MS data.
[0092] In step 5, the iterative solution method of the mathematical optimization model specifically includes: firstly, introducing multiple auxiliary variables as constraints of the fusion model, and finally dividing it into multiple sub-problems and then iteratively solving them respectively, and expanding the mathematical optimization model in step 4 by Lagrangian (where ∧1, ∧2, η1, η2 are the introduced auxiliary variables, which are initialized to 0):
[0093]
[0094] According to the idea of fixing other variables while updating the current variable, the above formula is solved by alternating iterations in sequence:
[0095]
[0096]
[0097]
[0098] Step 4: update ∧1, ∧2
[0099] ① Iterative solution: formula:
[0100]
[0101] The above formula can be rewritten as:
[0102]
[0103] in:
[0104]
[0105] g(x)=||X||1
[0106]
[0107] Introduce the proximal gradient descent algorithm and let X k represents the result after the kth update of X. For X→X k When , f(X) can be approximately expressed as:
[0108]
[0109]
[0110] Where L>=1, is the step length, so we can get the iteration X k+1 The formula is:
[0111]
[0112]
[0113]
[0114] Finally, we get: And X k+1 The output is X i+1 .
[0115] ② Iterative solution
[0116]
[0117] make: in ,
[0118] The symbol represents the Kronecker product, vec(·) represents the vectorization operator, According to the above formula, we can deduce:
[0119]
[0120] According to the above formula, we can directly calculate U i+1
[0121] ③Iterative solution
[0122]
[0123]
[0124] Since E is a semi-orthogonal matrix, this means that for all Has isometric properties:
[0125] ||Ex||2=||x||2
[0126] So we can get:
[0127]
[0128] make:
[0129] get: Directly introduce the "three-dimensional block matching algorithm" to solve this problem, and the output is Z i+1 .
[0130] Iterate the X, U, and Z variables in sequence until |X is satisfied i+1 -X i |<10 -5 The final result of variable X is the target high-resolution multispectral image of the panchromatic sharpening task.
[0131] Specifically, this embodiment is as follows Figure 1 The method shown achieves obtaining a high-resolution multispectral image by sharpening a high-resolution monochrome image and a low-resolution panchromatic image, and conducts comparative experiments on the panchromatic sharpening method of the present invention and several other traditional infrared target detection algorithms on the same real data set. The present invention uses MATLAB as the implementation software platform.
[0132] The specific steps of the present invention are as follows:
[0133] The full color sharpening method based on deep subspace embedding proposed in the present invention consists of five stages.
[0134] First, build a "variational optimization + deep subspace" panchromatic sharpening framework and prepare high spatial resolution panchromatic images and low resolution multispectral images to be panchromatic sharpened;
[0135] Then, by referring to the downsampled version of the high-resolution multispectral image, a blurring operation is designed based on the point spread function of the remote sensing satellite sensor for spectral fidelity;
[0136] Third, the multi-resolution analysis principle of multi-spectral fidelity and the sparsity of high-resolution multi-spectrum are fully utilized to introduce the spatial structure enhancement term based on L1 regularization term constraint and detail injection;
[0137] Fourth, the prior information of CNN is introduced into a robust subspace method to achieve deep subspace embedding;
[0138] Finally, the unified optimization framework is solved by the alternative direction multiplier method to calculate the final panchromatic sharpening result and achieve panchromatic sharpening.
[0139] The specific steps include:
[0140] Step 1: Build a "variational optimization + deep subspace" panchromatic sharpening model, prepare panchromatic images and low-resolution multispectral images, and collect WorldView-3 and WorldView-2 remote sensing datasets.
[0141] In order to solve the spectral distortion and data dependency problems of pan-sharpening, a general pan-sharpening framework is proposed, which combines the traditional variational optimization and the proposed deep subspace technology for pan-sharpening. This paper proposes a "variational optimization + deep subspace" pan-sharpening model:
[0142]
[0143] where f DS (X,X net ) represents the deep subspace embedding term, which plays the role of a priori term: f spec (X,Y) represents the spectral fidelity term; f spat (X,P) represents the spatial fidelity term; stands for High Resolution Multispectral Imagery; represents low-resolution multispectral imagery; represents a high-resolution monochrome image; EZ = X is a subspace representation, represents the basis for the subspace representation of X, represents the coefficient of X relative to E, and EZ refers to the matrix multiplication of E and Z after expansion in the third dimension (i.e. ), λ, α are the designed parameter weights.
[0144] Then download the public WorldView-3 and WorldView-2 remote sensing datasets from Maxar’s official website, which contain panchromatic images and low-resolution multispectral images.
[0145] Step 2: Spectral fidelity item construction
[0146] Many existing spatial-spectral pan-sharpening methods upsample the low-resolution multispectral image to the panchromatic image scale. However, using this (often biased) interpolation process may carry imprecise information that can affect performance. This method references a downsampled version of the underlying high-resolution multispectral image. This operation is performed based on the point spread function of the remote sensing satellite sensor, aiming to design the blur operation. Therefore, modeling:
[0147] Y=XBS+ξ1
[0148] in: is the matrix of the fuzzy operator, represents the sampling matrix composed of sparse components, and ξ1 is Gaussian noise. So we get the following formula:
[0149]
[0150] in It is the Frobenius norm, or F-norm for short, which is a matrix norm.
[0151] Step 3: Build a spatial structure enhancement item based on L1 regularization constraint
[0152] Based on the multi-resolution analysis principle of multispectral fidelity and the sparsity of high-resolution multispectral images, this method builds a new spatial structure enhancement scheme. The outstanding feature of this spatial structure enhancement scheme is the introduction of L1 regularization constraints and detail injection ideas to deal with the spatial deviation problem and panchromatic sharpening noise problem in the restoration process of high-resolution multispectral images. The general multi-resolution analysis framework can be expressed as
[0153]
[0154] For i = 1, 2, ..., S, where Respectively expressed in and an upsampled version of Y(i) on the scale of the injection coefficients, symbol represents element-wise multiplication, yes The low-pass version of i Filtering, that is Where * indicates the convolution operation. Considered as Image details of the Qualcomm version.
[0155] Replace Y with XB H , to avoid introducing low-resolution multispectral images, thereby independently modeling the relationship from unknown high-resolution multispectral images to panchromatic images. Therefore, considering the matrix form, there is
[0156]
[0157] Taking into account the image sparsity of multispectral images, the L1 regularization constraint is applied to the spatial fidelity task of panchromatic sharpening, and the following spatial fidelity model based on multi-resolution analysis detail injection and L1 regularization constraint can be obtained:
[0158]
[0159] Among them: Symbol represents element-wise multiplication; yes In this paper, a high-pass modulation scheme is adopted to handle detail extraction. That is, define (Use XB instead of Y directly H ), so:
[0160]
[0161] in is defined using a filter based on the modulation transfer function, so the final spatial fidelity that relies on high-pass modulation and L1 regularization constraints can be formalized as follows:
[0162]
[0163] Step 4: Create WorldView-3 and WorldView-2 training sets, input the full-color image and low-resolution multispectral image into the CNN network to obtain prior information, and then introduce a subspace representation method to achieve deep subspace embedding
[0164] The obtained WorldView-3 dataset is divided into training set, validation set, and test set according to the ratio of 80%, 10%, and 10%. The training set has a total of 12,000 image arrays of {panchromatic images, low-resolution multispectral images, and true high-resolution multispectral images}.
[0165] Taking into account the huge amount of high-resolution multispectral image data and the high correlation between channels, this method specifically introduces the deep subspace embedding term into a robust subspace method. This method converts the huge data of high-resolution multispectral images into the product of the subspace basis E and its representation coefficient Z, and at the same time the subspace basis E can be learned from observations. Since the size of Z is much smaller than X, this means that a better-conditioned problem is being solved. Compared with other sparse representations, the (orthogonal) subspace representation is a sparse table, which means that the signal can be well reconstructed using fewer transform domain coefficients. The DS deep subspace embedding term is defined as follows:
[0166]
[0167] where f DS (X,X net ) is the deep subspace embedding term, which plays the role of a priori term, and β is a positive parameter that balances variational optimization and deep subspace terms; represents the output of the general convolutional neural network method based on CNN; EZ = X is the subspace representation, represents the basis for the subspace representation of X, represents the coefficient of X relative to E, and EZ refers to the matrix multiplication of E and Z after expansion in the third dimension (i.e. ), φ(·) is a regularization term that represents the prior information customized to the image.
[0168] Step 5: Solve the unified optimization framework by alternative direction multiplier method to calculate the final pan-sharpening result
[0169] By combining the above spectral fidelity term, spatial fidelity term, and deep subspace term, we get the optimization solution model:
[0170]
[0171] stU=XB,EZ=X
[0172] The formulated mixing problem is solved by the alternating direction multiplier method, multiple auxiliary variables are introduced as constraints of the pan-sharpening model, and finally it is divided into multiple sub-problems and then solved iteratively:
[0173]
[0174] According to the idea of fixing other variables while updating the current variable, the above formula is solved by alternating iterations in sequence:
[0175]
[0176]
[0177]
[0178] Step 4: update ∧1, ∧2
[0179] (I) Iterative solution: formula:
[0180]
[0181] The above formula can be rewritten as:
[0182]
[0183] in:
[0184]
[0185] g(x)=||X||1
[0186]
[0187] Introduce the proximal gradient descent algorithm and let X k represents the result after the kth update of X. For X→X k When , f(X) can be approximately expressed as:
[0188]
[0189]
[0190] Where L>=1, is the step length, so we can get the iteration X k+1 The formula is:
[0191]
[0192]
[0193]
[0194] Finally, we get:
[0195] (II) Iterative solution
[0196]
[0197] make: in ,
[0198] The symbol represents the Kronecker product, vec(·) represents the vectorization operator, According to the above formula, we can deduce:
[0199]
[0200] (III) Iterative solution
[0201]
[0202]
[0203] Since E is a semi-orthogonal matrix, this means that for all Has isometric properties:
[0204] ||Ex||2=||x||2
[0205] So we can get:
[0206]
[0207] make:
[0208] get: Directly introduce BM3D to solve this problem
[0209] Finally, the solution to the panchromatic sharpening target X is to solve each of the above sub-problems separately until the termination condition is reached, and finally a high-resolution multispectral image is obtained.
[0210] The overall algorithm flow is as follows:
[0211] ① Input: panchromatic image P, low-resolution multispectral image Y.
[0212] ② Initialization: Build the model, initialize the corresponding weights and initial values of iterative variables, and preprocess Y to obtain E
[0213] ③ Iterative solution of X i+1 :
[0214]
[0215] ④Calculation
[0216] ⑤ Update U i+1 Variable, according to the formula
[0217] ⑥Update Z i+1 Variables, according to the formula:
[0218]
[0219] ⑧Update
[0220] ∧1 i+1 :=∧1 i +η1(X i+1 BU i+1 )
[0221] ∧2 i+1 : = ∧2 i +η2(X i+1 -EZ i+1 )
[0222] ⑨Iteratively optimize each variable until the termination condition is met.
[0223] ⑩ Output high-resolution multispectral images
[0224] Two real remote sensing image datasets (Worldview-3 and Worldview-2) are used to illustrate the performance of the invented method. The CNN is only trained on the Worldview-3 dataset, while Worldview-2 is used to test the generalization performance of the method. The effects of pan-sharpening performance and parameters are discussed through actual experiments. Finally, different basic methods are used to experiment on infrared images to confirm the superiority of the invented method.
[0225] First, the panchromatic sharpening effect of the invented method on the Worldview-3 dataset is shown in the attached figure. Figure 2 As shown. (LRMS stands for low-resolution multispectral image, PAN stands for monochrome image, HRMS stands for the panchromatic sharpening result of this method, and GT stands for the real image).
[0226] Secondly, the panchromatic sharpening effect of the invented method on the Worldview-2 dataset (to test the generalization performance) is shown in the attached figure. Figure 3 shown.
[0227] Then, the intuitive visual comparison of four different comparison methods (traditional method and deep learning method) is compared. The comparison algorithms used are: BDSD-PC is a robust panchromatic sharpening method with correlated spatial details (Vivone G.Robust Band-Dependent Spatial-Detail Approaches for Panchromatic Sharpening[J].IEEE transactions on geoscience and remote sensing:a publication of the IEEE Geoscience and Remote Sensing Society,2019.); MTF-GLP is a panchromatic sharpening method based on full-scale regression injection coefficients (Gemine,Vivone,Rocco,et al.Full Scale Regression-Based Injection Coefficients for Panchromatic Sharpening[J].IEEE Transactions on Image Processing,2018.); SR-D is a panchromatic sharpening method based on sparse representation of injected details (Vicinanza MR,Restaino R,Vivone G,et al.A Pansharpening Method Based on the Sparse Representation of Injected Details[J].IEEE Geoscience & Remote Sensing Letters, 2014, 12 (1): 180-184.); PanNet is a deep network architecture for pan-sharpening (Yang J, FuX, HuY, et al. PanNet: A Deep Network Architecture for Pan-Sharpening [C] / / 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017.). The results on the Worldview-3 dataset are shown in the attached figure. Figure 4 shown.
[0228] For the deep learning methods compared, none of them were trained on the Worldview-2 dataset. Instead, the generalization performance was tested using the models trained on the Worldview-3 dataset. The visual comparison of the four different methods on the Worldview-2 dataset is shown in the attached figure. Figure 5 shown.
[0229] In summary, the full-color sharpening method based on deep subspace embedding proposed in the present invention obtains a high-resolution multispectral image with better edge texture structure and more accurate spectral information and spatial information, and the generalization performance of this method is also excellent, with good accuracy, generalization and efficiency.
[0230] Embodiment 2:
[0231] A full color sharpening device based on deep subspace embedding is provided, comprising:
[0232] Image acquisition module: used to obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images PAN and multispectral images MS with exactly the same acquisition time and acquisition area;
[0233] Subspace module: used to perform subspace representation on MS data according to the multispectral image MS and obtain the subspace basis E;
[0234] Neural network module: used to input the panchromatic image PAN and the multispectral image MS into the neural network PanNet for panchromatic sharpening to obtain a high-resolution multispectral image X net ;
[0235] Mathematical modeling module: used to calculate the net , build a mathematical model and obtain a mathematical optimization model;
[0236] Sharpening output module: used to solve the mathematical optimization model by using the alternative direction multiplier method, and iterate the X, U, and Z variables in turn until the termination condition is met; finally, the final result of the variable X is obtained, which is the target high-resolution multispectral image of the full-color sharpening task.
[0237] The device of this embodiment can be used to implement the steps of the method described in the first embodiment.
[0238] Embodiment three:
[0239] This embodiment also provides a full color sharpening device based on deep subspace embedding, including a processor and a storage medium;
[0240] The storage medium is used to store instructions;
[0241] The processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1.
[0242] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0243] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0244] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0245] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0246] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A full color sharpening method based on deep subspace embedding, characterized in that: The method comprises the following steps: Step 1: Obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images PAN and multispectral images MS with exactly the same acquisition time and acquisition area; Step 2: Based on the multispectral image MS, perform subspace representation on the MS data to obtain the subspace basis E; Step 3: Input the panchromatic image PAN and the multispectral image MS into the neural network PanNet for panchromatic sharpening to obtain a high-resolution multispectral image X net ; Step 4: According to PAN graph, MS graph, X net , build a mathematical model and obtain a mathematical optimization model; Step 5: Solve the mathematical optimization model by using the alternative direction multiplier method, and iterate the X, U, and Z variables in turn until the termination condition is met; finally, the final result of the variable X is obtained, which is the target high-resolution multispectral image of the panchromatic sharpening task; The mathematical optimization model is: Where, P represents the PAN image, Y represents the MS image, X represents the target result of the full color sharpening task, initialized to 0; U and Z are auxiliary variables initialized to 0 for auxiliary iteration X; λ, α, τ, β are weights, initialized to 0.001; S represents a sampling matrix composed of sparse components; B represents the Gaussian blur kernel; represents the upsampled version of the PAN graph; represent The low-pass version of ; φ(·) represents the image prior; In step 5, the iterative solution method for the mathematical optimization model specifically includes the following steps: First, multiple auxiliary variables are introduced as constraints of the fusion model, and finally it is divided into multiple sub-problems and then solved iteratively, and the mathematical optimization model is Lagrangian expanded: Among them, ∧1, ∧2, η1, η2 are introduced auxiliary variables, initialized to 0; According to the idea of fixing other variables while updating the current variable, the above formula is solved by alternating iterations in sequence: Step1: Step2: Step3: Step4: update ∧1,∧2; Iterate the X, U, and Z variables in sequence until |X is satisfied i+1 -X i |<10 -5 The final result of variable X is obtained, which is the target high-resolution multispectral image of the panchromatic sharpening task.
2. The method for full color sharpening based on deep subspace embedding according to claim 1, characterized in that: In step 2, the method for performing subspace characterization on MS data specifically includes: Set the subspace dimension k of the subspace basis E to 5; Perform singular value decomposition on MS data; The subspace basis E is obtained from the first five left singular vectors of the MS data.
3. The method for full color sharpening based on deep subspace embedding according to claim 1, characterized in that: Iterative solution: The formula methods include: The above formula can be rewritten as: in. Introduce the proximal gradient descent algorithm and let X k represents the result after the kth update of X. For X→X k When , f(X) is expressed as: in is the step length, so we get the iteration X k+1 The formula is: Finally, we get: and X k+1 The output is X i+1 ; 4. The method for full color sharpening based on deep subspace embedding according to claim 1, characterized in that: Iterative solution The methods include: make: in The symbol represents the Kronecker product, vec(·) represents the vectorization operator, According to the above formula, we can deduce: According to the above formula, we can directly calculate U i+1 .
5. The method for full color sharpening based on deep subspace embedding according to claim 1, characterized in that: Iterative solution The methods include: Since E is a semi-orthogonal matrix, this means that for all Has isometric properties: ||Ex||2=||x||2 So we can get: make: get: Directly introduce the "three-dimensional block matching algorithm" to solve this problem, and the output is Z i+1 .
6. A full color sharpening device based on deep subspace embedding, characterized in that: include: Image acquisition module: used to obtain panchromatic images and multispectral images from remote sensing satellite data sets, perform pixel matching, ensure that the acquisition areas of the two images are consistent, and obtain panchromatic images PAN and multispectral images MS with exactly the same acquisition time and acquisition area; Subspace module: used to perform subspace representation on MS data according to the multispectral image MS and obtain the subspace basis E; Neural network module: used to input the panchromatic image PAN and the multispectral image MS into the neural network PanNet for panchromatic sharpening to obtain a high-resolution multispectral image X net ; Mathematical modeling module: used to calculate the net , build a mathematical model and obtain a mathematical optimization model; Sharpening output module: used to solve the mathematical optimization model by using the alternative direction multiplier method, and iterate the X, U, and Z variables in turn until the termination condition is met; finally, the final result of the variable X is obtained, which is the target high-resolution multispectral image of the panchromatic sharpening task; The mathematical optimization model is: Where, P represents the PAN image, Y represents the MS image, X represents the target result of the full color sharpening task, initialized to 0; U and Z are auxiliary variables initialized to 0 for auxiliary iteration X; λ, α, τ, β are weights, initialized to 0.001; S represents a sampling matrix composed of sparse components; B represents the Gaussian blur kernel; represents the upsampled version of the PAN graph; represent The low-pass version of ; φ(·) represents the image prior; In step 5, the iterative solution method for the mathematical optimization model specifically includes the following steps: First, multiple auxiliary variables are introduced as constraints of the fusion model, and finally it is divided into multiple sub-problems and then solved iteratively, and the mathematical optimization model is Lagrangian expanded: Among them, ∧1, ∧2, η1, η2 are introduced auxiliary variables, initialized to 0; According to the idea of fixing other variables while updating the current variable, the above formula is solved by alternating iterations in sequence: Step1: Step2: Step3: Step4: update ∧1,∧2; Iterate the X, U, and Z variables in sequence until |X is satisfied i+1 -X i |<10 -5 The final result of variable X is obtained, which is the target high-resolution multispectral image of the panchromatic sharpening task.
7. A full color sharpening device based on deep subspace embedding, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to claims 1-5.
