Remote sensing image panchromatic sharpening method based on spatial details and low-rank constraints

Abstract

The invention provides a remote sensing image panchromatic sharpening method based on spatial details and low-rank constraints, which relates to the technical field of image processing and application, and comprises the following steps: pre-constructing a panchromatic sharpening model comprising data items, spatial detail constraint items and low-rank constraint items; acquiring an MS image and a PAN image to be fused; inputting the MS image and the PAN image into a panchromatic sharpening model for loop iterative calculation until a relative error between an image quality index of an output image of a current iteration round and an image quality index of an output image of a previous iteration round is smaller than a preset error threshold value, and ending the loop iterative calculation; outputting a fusion result of the MS image and the PAN image; wherein after each round of loop iteration calculation is finished, if the condition for finishing the loop iteration calculation is not met, the optimization variable of the panchromatic sharpening model is updated. According to the scheme, the description precision of the image detail information can be improved, the global information of the fused image is fully analyzed, and the quality of the fused image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing and application technology, and in particular to a remote sensing image full color sharpening method based on spatial details and low rank constraints. Background Art

[0002] With the continuous updating of sensors, remote sensing images are widely used in fields such as information transmission, geological exploration, and ecological environment monitoring. Its advantage is that it is not restricted by geographical location and can observe and image quickly and accurately in a large scale. At present, there are various image information collected by earth observation satellites, among which multispectral (MS) images and panchromatic (PAN) images are common. These images can describe the information of ground objects, but there are certain limitations. For example, PAN images have good spatial information but poor spectral information, and MS images have good spectral information but poor spatial information. Therefore, it is very important to fuse and classify different types of remote sensing images for practical applications. The remote sensing image pan-sharpening technology uses advanced algorithms to fuse the MS image and the corresponding PAN image of the same scene. After fine processing, a fused image that can accurately depict the scene information is obtained. The fused image not only contains rich spectral information and spatial information, but also contains more detailed surface information, which can realize the rapid and accurate investigation of surface resources and provide comprehensive data support for urban planning.

[0003] As a special type of image fusion problem, panchromatic sharpening of remote sensing images mainly integrates the spatial information of high spatial resolution images and the spectral information of low spatial resolution images to obtain a fused image with high spatial resolution and high spectral resolution, so as to achieve a fine expression of ground object information. Existing panchromatic sharpening methods have problems such as spatial distortion and spectral distortion in the fusion process. For example, in traditional methods, there may be obvious color differences between the fused image and the MS image, resulting in spectral distortion. Spatial distortion may occur due to insufficient injection of detail information. Due to different sensors, the acquired MS image and PAN image have differences in spectral range, spatial resolution, etc., which affects the quality of the fused image. Summary of the invention

[0004] In view of this, in order to address the above shortcomings, it is necessary to propose a remote sensing image panchromatic sharpening method based on spatial details and low-rank constraints to improve the accuracy of image detail information, fully analyze the global information of the fused image, and thus improve the quality of the fused image.

[0005] In a first aspect, the present invention provides a remote sensing image pan-sharpening method based on spatial details and low-rank constraints, comprising:

[0006] Pre-constructing a pan-sharpening model including data terms, spatial detail constraint terms and low-rank constraint terms;

[0007] Acquire the MS image and PAN image to be fused;

[0008] Inputting the MS image and the PAN image into the full color sharpening model for loop iterative calculation until the relative error between the image quality index of the output image of the current iteration round and the image quality index of the output image of the previous iteration round is less than a preset error threshold, and the loop iterative calculation is terminated;

[0009] Output the fusion result of MS image and PAN image;

[0010] Wherein, after each round of loop iterative calculation ends, if the condition for ending the loop iterative calculation is not met, the optimization variables of the panchromatic sharpening model are updated.

[0011] Preferably, the pan-sharpening model includes:

[0012]

[0013] Let X = M H , XB=M U , the panchromatic sharpening model is simplified to obtain the following simplified panchromatic sharpening model:

[0014]

[0015] Among them, the MS image to be fused is PAN image is P, To fuse the images, B and S represent the downsampling matrix and blur matrix respectively, μ1, μ2 and α are all positive penalty parameters, is the MS image upsampled to the same size as the PAN image, φ represents the injection coefficient that controls the degree of detail injection, i is the i-th band of the image, P~ is the low-pass approximation of the PAN image, ||·|| TNN represents the tensor nuclear norm; X represents the fused image, Y represents the MS image with low spatial resolution; ||·|| F is the F-norm, Used to represent data items, Used to represent spatial detail constraints, ||X|| TNN Used to characterize the low-rank constraint term, defined as: but Where n = min(n1,n2), σ i (X (j) ) indicates X (j) The i-th singular value of .

[0016] Preferably, the updating of the optimization variables of the pan-sharpening model comprises:

[0017] Introducing auxiliary vectors U and V, where U = XB, V = X, the augmented Lagrangian function form of the simplified pan-sharpening model is expressed as:

[0018]

[0019] Among them, L(X,U,V,λ1,λ2) is the augmented Lagrangian function, η1,η2 are positive penalty parameters, λ1,λ2 are Lagrangian multipliers;

[0020] The alternating multiplier direction method is used to optimize and update the variables X, U, and V, and to update the Lagrange multipliers.

[0021] Preferably, the optimizing and updating of variables X, U, and V includes:

[0022] Use the following calculation formula to update X, U, and V respectively:

[0023]

[0024] Among them, argmin is used to find the value of the variable when the objective function reaches the minimum value.

[0025] Preferably, the updating of the Lagrange multiplier comprises:

[0026] The Lagrange multiplier is updated using the following formula:

[0027]

[0028] Here, k is used to represent the kth iteration calculation.

[0029] Preferably, P~ is calculated by the following formula:

[0030]

[0031] Among them, ε i,1 and ε i,2 represents the parameters, I is the intensity component of the upsampled MS image, L G P is the result of low-pass filtering of the PAN image.

[0032] Preferably, the image quality indicator includes at least one of the following:

[0033] Universal image quality index UIQI, spectral angle mapping SAM, global relative index ERGAS, spatial correlation coefficient SCC, peak signal-to-noise ratio PSNR and structural similarity SSIM.

[0034] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any of the methods described in the first aspect.

[0035] In a third aspect, the present invention provides a computing device, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, any method described in the first aspect is implemented.

[0036] It can be seen from the above technical scheme that in the remote sensing image pan-sharpening method based on spatial details and low-rank constraints provided by the embodiment of the present invention, a pan-sharpening model including data items, spatial detail constraint items and low-rank constraint items is first pre-constructed, and then the MS image and PAN image to be fused are obtained, and the MS image and PAN image are input into the pan-sharpening model for cyclic iterative calculation until the relative error between the image quality index of the output image of the current iteration round and the image quality index of the output image of the previous iteration round is less than the preset error threshold, then the cyclic iterative calculation is terminated, and the fusion result of the MS image and the PAN image is output. It can be seen that this scheme constructs detail constraint items by analyzing data correlation, improves the accuracy of image detail information description, and introduces low-rank regularization items to characterize the low-rank of the fused image, which not only helps to maintain the low-rank of the image, can capture a large amount of low-rank redundant information in the image, and can effectively improve the quality of the fused image. Therefore, this scheme can solve the problems of insufficiently accurate description of image detail information and insufficient analysis of global information of the fused image in the existing pan-sharpening method, and can greatly improve the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a remote sensing image pan-sharpening method based on spatial details and low-rank constraints is provided in an embodiment of the present invention.

[0038] Figure 2 This is the decomposition result of this scheme and other comparison methods under simulated image data.

[0039] Figure 3 The decomposition results and quantitative indicators of this scheme and other comparison methods under real image data. DETAILED DESCRIPTION

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] like Figure 1As shown, the present invention provides a remote sensing image panchromatic sharpening method based on spatial details and low-rank constraints, comprising:

[0042] Step 101: pre-constructing a pan-sharpening model including a data item, a spatial detail constraint item, and a low-rank constraint item;

[0043] Step 102: Acquire the MS image and PAN image to be fused;

[0044] Step 103: Input the MS image and the PAN image into the full-color sharpening model for loop iterative calculation until the relative error between the image quality index of the output image of the current iteration round and the image quality index of the output image of the previous iteration round is less than a preset error threshold, and the loop iterative calculation is terminated; wherein, after each round of loop iterative calculation is completed, if the condition for terminating the loop iterative calculation is not met, the optimization variables of the full-color sharpening model are updated;

[0045] Step 104: Output the fusion result of the MS image and the PAN image.

[0046] In this embodiment, by analyzing the data correlation to construct detail constraint items, the accuracy of image detail information is improved, and at the same time, a low-rank regularization item is introduced to characterize the low-rank property of the fused image, which not only helps to maintain the low-rank property of the image, but also can capture a large amount of low-rank redundant information in the image, and can effectively improve the quality of the fused image. Therefore, this solution can solve the problems of inaccurate description of image detail information and insufficient analysis of global information of the fused image in the existing pan-sharpening method, and can greatly improve the quality of the fused image.

[0047] For step 101, a pan-sharpening model including a data item, a spatial detail constraint item, and a low-rank constraint item is pre-built;

[0048] In this step, the input MS image is analyzed The constructed data items can be represented as By analyzing the input MS image and PAN image P, the constructed spatial detail constraint term can be expressed as in ε i,1 and ε i,2 represents the parameters, I is the intensity component of the upsampled MS image, L G P is the result of low-pass filtering of the PAN image. Furthermore, TNN is used to regularize the fused image, which not only helps to maintain the low rank of the image, but also captures a large amount of low-rank redundant information in the image, and can effectively improve the quality of the fused image. The low-rank constraint term can be expressed as ||M H || TNN , which can be defined as: but Where n = min(n1,n2), σ i (X (j) ) indicates X (j) Thus, by combining the above data terms, spatial detail terms and low-rank constraint terms, the panchromatic sharpening model can be obtained: Furthermore, in order to simplify the model, let X = M H , XB=M U , simplify the panchromatic sharpening model to obtain the simplified panchromatic sharpening model

[0049] Among them, the MS image to be fused is PAN image is P, To fuse the images, B and S represent the downsampling matrix and blur matrix respectively, μ1, μ2 and α are all positive penalty parameters, is the MS image upsampled to the same size as the PAN image, φ represents the injection coefficient that controls the degree of detail injection, i is the i-th band of the image, is a low-pass approximation of the PAN image, ||·|| TNN represents the tensor nuclear norm; X represents the fused image, Y represents the MS image with low spatial resolution; ||·|| F is the F-norm, Used to represent data items, Used to represent spatial detail constraints, ||X|| TNN Used to characterize low-rank constraints.

[0050] For step 102: obtaining the MS image and the PAN image to be fused;

[0051] In this step, it is considered to obtain image information collected by earth observation satellites, and the collected image information includes multispectral MS images and full-color PAN images, which are used as MS images and PAN images to be fused.

[0052] For step 103, the MS image and the PAN image are input into the full color sharpening model for loop iterative calculation until the relative error between the image quality index of the output image of the current iteration round and the image quality index of the output image of the previous iteration round is less than a preset error threshold, and the loop iterative calculation is terminated;

[0053] In this step, the MS image and the PAN image are input into the pan-sharpening model for cyclic iterative calculation. Specifically, the pan-sharpening model is solved using the alternating direction method of multipliers ADMM. Auxiliary vectors U and V are introduced, where U = XB, V = X, and the augmented Lagrangian function form of the simplified pan-sharpening model is expressed as:

[0054]

[0055] Among them, L(X,U,V,λ1,λ2) is the augmented Lagrangian function, η1,η2 are positive penalty parameters, λ1,λ2 are Lagrangian multipliers;

[0056] The alternating multiplier direction method is used to optimize and update the variables X, U, and V, and to update the Lagrange multipliers.

[0057] Specifically, X, U, and V can be updated using the following formulas:

[0058]

[0059] Among them, argmin is used to find the value of the variable when the objective function reaches the minimum value.

[0060] In this embodiment, for the subproblem of updating X, other variables are fixed to obtain X. The subproblem X is a least squares problem and has a closed solution under periodic boundary conditions. Its least squares solution is obtained by applying the fast Fourier transform F(·) Among them, F -1 (·) is the inverse fast Fourier transform.

[0061] For the subproblem of updating U, fix other variables to obtain U. The symbol represents the Kronecker product, K represents a sparse matrix, and vec(·) represents a vector operation. Based on this, the solution to the U subproblem can be expressed as

[0062] For the subproblem of updating V, fix other variables to obtain V. Let the truncated singular value decomposition T-SVD of X+λ2 / η2 be Q*S*R T , then the closed-form solution of the V subproblem can be accurately calculated by the singular value threshold as V = Q*S*R T , where S is a diagonal tensor whose forward slice in the Fourier domain is Both Q and R are orthogonal tensors.

[0063] Specifically, when updating the Lagrange multiplier, it can be achieved through the gradient descent method, which can be achieved through the following calculation formula:

[0064]

[0065] Image quality indicators may include the universal image quality index UIQI, spectral angle mapping SAM, global relative index ERGAS, spatial correlation coefficient SCC, peak signal-to-noise ratio PSNR and structural similarity SSIM, etc.

[0066] In addition, the initial value of the iteration is The current iteration number is 0.

[0067] For step 104, the fusion result of the MS image and the PAN image is output.

[0068] The iterative calculation is performed through step 103. When the loop iterative calculation is completed, the output result of the full color sharpening model is the fusion result of the MS image and the PAN image.

[0069] The effect of the method proposed in the present invention is further illustrated below in conjunction with specific simulation experiments.

[0070] The simulated images and real images are used as input data to evaluate the fusion results of the algorithm, and the Universal Image Quality Index (UIQI), Spectral Angle Mapper (SAM), Erreur Relative Globale Adimensionnellede Synthèse (ERGAS), Spatial Correlation Coefficient (SCC), Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used to evaluate the simulated image fusion results. The spectral information distortion index D is used λ , spatial information distortion index D s And Quality with No Reference (QNR) is used to evaluate the real data results.

[0071] like Figure 2 The decomposition results of the proposed method and other comparison methods under simulated image data are shown; Figure 2 (a) is the ground truth (GT) image, Figure 2 (c)-(k) are some comparison methods. Figure 2 (l) is the fusion result of the present invention. As can be seen from the figure, the traditional Figure 2 Methods (c)-(e) lack the ability to preserve spatial information, resulting in obvious edge blurring. When observing the magnified area, we can see Figure 2Methods (h) and (j) have slight blurring and missing spatial information on road boundaries; Figure 2 The sharpening effects of methods (f)-(g) and (k) are relatively good. Further observation and analysis show that the method proposed in this application Figure 2 (l) It is superior to the contrast method in terms of spatial detail preservation, and the color and detail information of the image are closer to the GT image.

[0072] In addition, Table 1 gives Figure 2 Quantitative comparison of the corresponding objective experimental results, the optimal value is indicated in bold. As can be seen from Table 1, compared with the comparison methods, the proposed model achieves the best results in the six indicators of UIQI, SAM, ERGAS, SCC, PSNR and SSIM, especially in the PSNR and SSIM indicators, which have been greatly improved. In the PSNR indicator, the proposed model has a maximum improvement of about 14.5% and a minimum improvement of 9.7% over the other comparison methods; in the SSIM indicator, the proposed model has a maximum improvement of about 17% and a minimum improvement of about 3.6%. Therefore, in the objective experimental results of highway terrain in GaoFen-2 data, the qualitative results of the proposed model are consistent with the quantitative performance, and both have achieved good results.

[0073] Table 1 Figure 2 Corresponding simulation experiment objective evaluation results

[0074]

[0075] Figure 3 The decomposition results and quantitative indicators of the proposed method and other comparison methods on real image data are shown. Figure 3 (a) is the input MS image, Figure 3 (b) is the input PAN image, Figure 3 (c)-(k) are some comparison methods. Figure 3 (l) is the fusion result of the proposed method. As can be seen from the figure, Figure 3 (c) and (h) have vague descriptions of the outlines of urban green areas, and the details of the plants are not clear enough; Figure 3 The color of (j) is seriously distorted. From the enlarged image given in the figure, it can be seen that the color of the green roof in the figure becomes lighter. The proposed method fully retains the spectral information of the image, and the spatial information is also very close to the PAN image.

[0076] Table 2 gives Figure 3Quantitative comparison of the corresponding objective experimental results. As can be seen from Table 2, the three indicators all achieved the optimal values. For the indicators, the proposed model improved by at least about 49% and at most about 90% compared with the comparison method; for the indicators, the proposed model improved by at least about 62% and at most about 82% compared with the comparison method; for the QNR indicator, the proposed model improved by at least about 9% and at most about 35% compared with the comparison method.

[0077] Table 2 Figure 3 Corresponding real experimental objective evaluation results

[0078]

[0079] Depend on Figure 2 and Figure 3 The experimental results show that for both simulated and real images, the present invention not only fully considers the spectral information of MS images and the spatial information of PAN images, but also performs low-rank analysis on the fused images. The present invention can achieve better fusion results in terms of both subjective visual effects and objective indicators. It fully proves that the proposed spatial detail and low-rank constraint model is effective, and at the same time greatly improves the fusion effect.

[0080] In summary, in order to solve the problems of inaccurate description of image detail information and insufficient analysis of global information of fused images in the pan-sharpening method, the present invention proposes a pan-sharpening method for remote sensing images based on spatial details and low-rank constraints. First, the data item is constructed by correlation analysis of the fused image and the MS image, which avoids the redundancy of the upsampled MS image. Secondly, from the perspective of fully retaining the image detail information, by analyzing the correlation between the downsampled PAN image and the MS image blurred data after filtering and the original MS image, a detail injection regularization term is proposed to analyze the spectral and spatial information of the image. Thirdly, the fused image is analyzed by regularization constraint using the tensor nuclear norm (TNN), which better interprets the low rank of the image. Finally, the alternate direction method of multipliers (ADMM) is used to solve the problem, which greatly improves the fusion effect.

[0081] The present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute a method in any one of the embodiments in the specification.

[0082] The present specification also provides a computing device, including a memory and a processor, wherein executable codes are stored in the memory, and when the processor executes the executable codes, a method in any embodiment of the present specification is implemented.

[0083] Since the device embodiments provided by the present invention are based on the same inventive concept as the method embodiments of this specification, the specific contents can be found in the description of the method embodiments of this specification and will not be repeated here.

[0084] The modules or units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. The above disclosure is only the preferred embodiment of the present invention, and of course it cannot be used to limit the scope of the rights of the present invention. Those skilled in the art can understand that all or part of the processes of the above embodiment are implemented, and the equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A remote sensing image pan-sharpening method based on spatial details and low-rank constraints, characterized in that: include: Pre-constructing a pan-sharpening model including data terms, spatial detail constraint terms and low-rank constraint terms; Acquire the MS image and PAN image to be fused; Inputting the MS image and the PAN image into the full color sharpening model for loop iterative calculation until the relative error between the image quality index of the output image of the current iteration round and the image quality index of the output image of the previous iteration round is less than a preset error threshold, and the loop iterative calculation is terminated; Output the fusion result of MS image and PAN image; Wherein, after each round of loop iterative calculation ends, if the condition for ending the loop iterative calculation is not met, the optimization variables of the panchromatic sharpening model are updated.

2. The remote sensing image pan-sharpening method based on spatial detail and low-rank constraints according to claim 1, characterized in that: The pan-sharpening model includes: Let X = M H , XB=M U , the panchromatic sharpening model is simplified to obtain the following simplified panchromatic sharpening model: Among them, the MS image to be fused is PAN image is P, To fuse the images, B and S represent the downsampling matrix and blur matrix respectively, μ1, μ2 and α are all positive penalty parameters, is the MS image upsampled to the same size as the PAN image, φ represents the injection coefficient that controls the degree of detail injection, i is the i-th band of the image, is a low-pass approximation of the PAN image, ||·|| TNN represents the tensor nuclear norm; X represents the fused image, Y represents the MS image with low spatial resolution; ||·|| F is the F-norm, Used to represent data items, Used to represent spatial detail constraints, ||X|| TNN Used to characterize the low-rank constraint term, defined as: but Where n = min(n1,n2), σ i (X (j) ) indicates X (j) The i-th singular value of .

3. The remote sensing image pan-sharpening method based on spatial detail and low-rank constraints according to claim 2, characterized in that: The updating of the optimization variables of the pan-sharpening model includes: Introducing auxiliary vectors U and V, where U = XB, V = X, the augmented Lagrangian function form of the simplified pan-sharpening model is expressed as: Among them, L(X,U,V,λ1,λ2) is the augmented Lagrangian function, η1,η2 are positive penalty parameters, λ1,λ2 are Lagrangian multipliers; The alternating multiplier direction method is used to optimize and update the variables X, U, and V, and to update the Lagrange multipliers.

4. The method for pan-sharpening remote sensing images based on spatial details and low-rank constraints according to claim 3, characterized in that: The optimizing and updating of variables X, U, and V includes: Use the following calculation formula to update X, U, and V respectively: Among them, argmin is used to find the value of the variable when the objective function reaches the minimum value.

5. The method for pan-sharpening of remote sensing images based on spatial details and low-rank constraints according to claim 4, characterized in that: The updated Lagrange multiplier includes: The Lagrange multiplier is updated using the following formula: Here, k is used to represent the kth iteration calculation.

6. The method for pan-sharpening of remote sensing images based on spatial details and low-rank constraints according to claim 2, characterized in that: Calculated by the following formula: Among them, ε i,1 and ε i,2 represents the parameters, I is the intensity component of the upsampled MS image, L G P is the result of low-pass filtering of the PAN image.

7. The remote sensing image pan-sharpening method based on spatial detail and low-rank constraints according to claim 1, characterized in that: The image quality indicator includes at least one of the following: Universal image quality index UIQI, spectral angle mapping SAM, global relative index ERGAS, spatial correlation coefficient SCC, peak signal-to-noise ratio PSNR and structural similarity SSIM.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

9. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images