Remote sensing image restoration method, terminal device and storage medium

Abstract

The application discloses a remote sensing image recovery method, a terminal device and a storage medium. According to the relationship between n-rank and Tucker factor matrix rank, a core norm regularization is introduced to a Tucker decomposition factor matrix, so that the difficulty of finding an optimal tensor rank is avoided, and the calculation amount of a singular value decomposition optimization step is reduced. A nearest neighbor graph based on tensor slicing is constructed to obtain geometric neighbor information of a remote sensing image, and the manifold information is fused into Tucker decomposition through a graph Laplacian operator. Non-negative constraints are added to the remote sensing image data to be recovered, so that the model has more practical application value.

Description

Technical Field

[0001] This invention relates to tensor filling technology, and in particular to a remote sensing image restoration method, terminal device, and storage medium. Background Technology

[0002] A key idea behind tensor filling is that many real-world datasets are highly structured, and their corresponding tensors can be approximated by low-rank decomposition. Unlike matrix decomposition, tensor decomposition has no unique definition; the two most well-known are CP decomposition (CANDECOMP / PARAFAC, abbreviated as CP) and Tucker decomposition. CP decomposition is a highly compact representation that decomposes a tensor into the sum of components of rank-one factor tensors. Compared to CP decomposition, Tucker decomposition is more flexible. The number of components in each modulus of Tucker decomposition can vary, and these components are connected through a core tensor, allowing us to model more complex hidden data structures. Existing tensor filling models based on Tucker decomposition mainly suffer from the following shortcomings:

[0003] (1) The choice of Tucker rank is very sensitive. Tucker rank, as a parameter, needs to be given in advance, and inaccurate rank estimation can lead to overfitting or underfitting problems. At the same time, due to the large-scale Singular Value Decomposition (SVD) computation, its time cost is expensive.

[0004] (2) The inherent geometric structure of remote sensing images is not fully utilized. Remote sensing images contain many repetitive local structures, which indicate that their geometric proximity correlation exists not only in adjacent pixels but also in non-adjacent pixels. The commonly used total variation (TV) prior regularization can effectively preserve the piecewise smooth structure of remote sensing images, but it does not adequately mine this geometric proximity information, thus making it difficult to obtain satisfactory restoration results.

[0005] (3) The nonnegativity of the image was ignored. Each element of the image is nonnegative, therefore, it is necessary to impose a nonnegativity constraint on the tensor of the remote sensing image to be recovered. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a remote sensing image restoration method, terminal device and storage medium that fully utilize the inherent correlation of high-dimensional images to improve the restoration accuracy of remote sensing images, in order to address the shortcomings of the existing technology.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a remote sensing image restoration method, comprising the following steps:

[0008] S1. Obtain the tensor of the observed remote sensing image. And its corresponding index tensor Ω, where the elements of Ω are 0 or 1, where 0 indicates that the element at that position is missing, and 1 indicates that the element at that position is not missing;

[0009] S2, along the tensor Constructing a slice-based nearest neighbor graph G in 3 directions n That is, to obtain the tensor of the image to be recovered. Information about the manifold in all directions; where n = 1, 2, 3;

[0010] S3. The manifold information is combined with graph Laplacian decomposition and low-rank Tucker decomposition to establish a non-negative tensor filling model based on factor prior and manifold regularization; the expression of the non-negative tensor filling model is:

[0011]

[0012]

[0013]

[0014]

[0015] in and Let these represent the tensor of the image to be recovered and the tensor of the observed image, respectively. express Projection onto Ω; α n λ1 and λ2 are regularization parameters, ∑ represents the summation operation, and ||·|| * Let represent the nuclear norm of the matrix, and tr represent the trace operation of the matrix. Let I represent the space of positive real numbers. n Let X be the dimension of the tensor in each direction. (n) Let L be a modulo-n expansion matrix. n Represents the graph Laplace matrix; express Tucker decomposition, and U n These are the kernel tensor and the factor matrix, respectively. n Represents the product of a tensor and a matrix modulo - n;

[0016] S4. Calculate the non-negative tensor filling model to obtain the restored image.

[0017] The nearest neighbor graph constructed in this invention can intuitively describe the nonlinear relationships of remote sensing images, thus making better use of their inherent geometric structure; the low-rank Tucker decomposition, by combining low-rank factor priors, can further explore the low-rank nature hidden in the factor matrix, thus better characterizing the global correlation of remote sensing images; the non-negative constraint makes the model more interpretable, thereby improving the recovery accuracy of remote sensing images.

[0018] The specific implementation process of step S2 includes:

[0019] Each forward slice X in the observed remote sensing image tensor k stretch into column vector x k Arrange them in lexicographical order to obtain a modulo-3 expanded matrix; then expand each column vector x... k Treating each vertex as a distinct vertex, we construct a nearest neighbor graph G3 using these vertices. The weight matrix of the nearest neighbor graph is denoted as W. n Where any two vertices i and j are column vectors x i and x j The weights W between n (i,j) is defined as:

[0020]

[0021] N(x i ) represents the column vector x i The top k nearest neighbors, N(x) j ) represents the column vector x j The topk nearest neighbors;

[0022] manifold information Where L n =D n -W n It is the graph Laplace matrix, D n It is a diagonal matrix, diagonal elements

[0023] The specific implementation process of step S4 includes:

[0024] Introduce auxiliary variables to the nonnegative tensor filling model The following equivalent model is obtained:

[0025]

[0026] stR n =X (n) ,

[0027] V n =U n ,

[0028]

[0029]

[0030] in express The indicator function, i.e.

[0031]

[0032] The equivalent model is calculated using the augmented Lagrangian function to obtain the restored image.

[0033] As an inventive concept, the present invention also provides a terminal device, comprising:

[0034] One or more processors;

[0035] A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the steps of the method described above.

[0036] As an inventive concept, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. This invention applies a low-rank prior to the factor matrix of Tucker decomposition, avoiding the burden of finding the optimal Tucker rank and reducing the computational cost of handling singular value decomposition problems.

[0039] 2. This invention constructs a nearest neighbor graph based on tensor slices to explore neighborhood information in remote sensing images. This information can be well combined with Tucker decomposition through graph Laplacian. Therefore, the low-rank tensor filling model proposed in this invention can more faithfully preserve the inherent geometric structure of remote sensing image data.

[0040] 3. This invention applies non-negative constraints to the recovered tensor image, enabling the model to have a reasonable physical interpretation, thereby improving the recovery accuracy.

[0041] 4. This invention employs the alternating direction method of multipliers (ADMM) to solve the proposed model. Numerical results verify the good performance of the proposed method. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the slice-based manifold regularization construction according to an embodiment of the present invention;

[0044] Figure 3 Comparison of recovery performance for each band in embodiments of the present invention: (a) PSNR, (b) SSIM;

[0045] Figure 4 Visual restoration results of different methods on WDC with a missing rate of 95%. (a) Original band 50, (b) Simulated noise band, (c) HaLRTC, (d) STDC, (e) KBR-TC, (f) TNN, (g) LRTV, (h) TRLRF, (i) LRTC-TV-II, (j) Method of the present invention;

[0046] Figure 5 Visual restoration results of different methods on Toys with a missing rate of 90%. (a) Original band 25, (b) Observed image, (c) HaLRTC, (d) STDC, (e) KBR-TC, (f) TNN, (g) LRTV, (h) TRLRF, (i) LRTC-TV-II, (j) Method of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] In this document, the terms "first," "second," and other similar words are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. The terms "one," "a," and other similar words are not intended to indicate the existence of only one of the stated things, but rather that the description refers only to one of the stated things, which may have one or more. The terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0049] Example 1

[0050] This invention provides a remote sensing image restoration method based on manifold learning and factor priors, comprising the following steps:

[0051] (1) Obtain the tensor of the observed remote sensing image by random sampling. And its corresponding index tensor Ω, where the elements of Ω are 0 or 1: 0 indicates that the element at that position is missing, and 1 indicates that the element at that position is not missing. Simultaneously, the remote sensing image tensor to be recovered... Initialize to

[0052] (2) Along the tensor in step (1) Constructing a slice-based nearest neighbor graph G in 3 directions n (n = 1, 2, 3), that is, the image to be restored is obtained. Manifold information in all directions;

[0053] (3) The manifold information obtained in step (2) is combined with graph Laplacian and low-rank Tucker decomposition to establish a non-negative tensor filling model based on factor prior and manifold regularization for remote sensing image restoration:

[0054]

[0055]

[0056]

[0057]

[0058] in Ω represents the recovered image tensor, the observed image tensor, and the index tensor, respectively. Let I represent the space of positive real numbers. n Let X be the dimension of the tensor in each direction, tr denote the trace operation of the matrix, and X be the dimension of the tensor in each direction. t Let X represent the transpose of matrix X. (n) express The modulus-n expansion matrix, α n λ1, λ2 are regularization parameters. express Tucker decomposition, and U n These are the kernel tensor and factor matrix, respectively. n This represents the product of a tensor and a matrix modulo -n. This represents the graph Laplace matrix.

[0059] (4) Design the ADMM algorithm to solve the tensor filling model in step (3) to obtain the restored image. Note: In each step of the ADMM algorithm iteration, the nearest neighbor graph G... n According to the latest information Dynamic adjustments will be made; the specific steps are in (2-4).

[0060] Figure 2 (a) is a remote sensing image. Step (2) above specifically includes the following sub-steps:

[0061] (2-1) will Figure 2 (a) Remote sensing image Modeled as a 3D tensor, we obtain in Representing remote sensing images Each band, i.e. Figure 1 Each forward slice in (b3). I1×I2 is... The spatial dimension, I3 is the total number of bands;

[0062] (2-2) For each forward slice X in step (2-1) k stretch into column vector x j Arrange them in dictionary order to obtain Figure 2 The modulus-3 expansion matrix in (c3). To preserve the hyperspectral image. In the geometric structure of the spectral dimension, it is naturally desirable that: Figure 2 The two slices X in (b3) i and X j Very close, then Figure 2 The corresponding column vector x in (c3) i and x j Also maintain closeness;

[0063] (2-3) For each column vector x obtained in step (2-2) k View as Figure 2 The graph (d3) is composed of different vertices, and a nearest neighbor graph G3 is constructed for these vertices. The weight matrix of each edge in G3 is defined as follows:

[0064]

[0065] Where N(x) i ) represents x i The top k nearest neighbors. In this embodiment of the invention, the Euclidean distance between any two points is calculated, and then the k-nearest neighbor algorithm (KNN) is used to find the set of the top k nearest neighbors of each node. The algorithm finds the top k nearest neighbors of each node if and only if x... i ∈N(x j), and x j ∈N(x i If both conditions are met, then there exists an edge connecting x. i and x j .

[0066] (2-4) Based on the neighbor graph G3 and weight matrix W3 obtained in step (2-3), minimize the following function to preserve the hyperspectral image. Geometric information in the spectral dimension:

[0067]

[0068] Where L3 = D3 - W3 is the Graph Laplacian matrix, D3 is a diagonal matrix, and the diagonal elements are... Without loss of generality, the manifold regularization formula can be generalized to: Where the Thulaplatz matrix L n The same derivation can be made similarly for (n = 1, 2, 3).

[0069] Step (4) above specifically includes the following sub-steps:

[0070] (4-1) Introduce auxiliary variables for the tensor filling model in step (3). The following equivalent model is obtained:

[0071]

[0072] stR n =X (n) ,

[0073] V n =U n ,

[0074]

[0075]

[0076] Where δ C (x) represents the indicator function of C, i.e.

[0077] (4-2) The augmented Lagrangian function of the model in step (4-1) is as follows:

[0078]

[0079] in X is a Lagrange multiplier, ρ is the penalty parameter, and X is a Lagrange multiplier. (n) Tensor The modulo-n expansion matrix is ​​then obtained. Furthermore, the update expression for each variable in the (t+1)th iteration is specifically solved.

[0080] (4-3) Update variable R in step (4-2) n :

[0081]

[0082] (4-4) Update variable U in step (4-2) n :

[0083]

[0084] Where D α (·) denotes the singular value threshold operator.

[0085] (4-5) Update variable V in step (4-2) n :

[0086]

[0087] in It is the Kronecker product.

[0088] (4-6) Update the variables in step (4-2).

[0089] make but

[0090]

[0091] Where Ω c It is the complement of Ω. Represents the projection operator, i.e.

[0092]

[0093] (4-7) Update the variables in step (4-2).

[0094]

[0095] Where vec(·) represents turning a tensor into a vector, and I is the identity matrix.

[0096] (4-8) Update the multipliers and penalty parameters in step (4-2).

[0097]

[0098]

[0099]

[0100] ρ t+1=min(μρ) t ,ρ max ), where μ is a hyperparameter, ρ max This is the upper bound of the penalty parameter.

[0101] To verify the effectiveness of the tensor filling method SMF-NLRTC proposed in this embodiment, this embodiment was compared with seven other tensor filling methods on a remote sensing image dataset, including HaLRTC, KBR-TC, STDC, TNN, LRTV, TRLRF, and LRTC-TV-II.

[0102] This embodiment verifies the good performance of the proposed model using one hyperspectral image (WDC) and four multispectral images (Toys, Feathers, Balloons, and Flowers). For quantitative evaluation, four image quality metrics—MPSNR, MSSIM, ERGAS, and MSA—were used to measure the restoration results. Tables 1 and 2 show the results of these four metrics for the WDC and CAVE datasets at different sampling ratios (SR). The best results for each quality metric are marked in bold.

[0103] It is easy to see that, for different sample rates, the method proposed in this embodiment of the invention exhibits the best recovery performance, and its advantage increases with the increase of the missing rate. Taking the WDC dataset with SR=5% as an example, the method proposed in this embodiment of the invention outperforms the best comparison method KBR-TC by 4dB, 0.0456, 26.66, and 0.0330 in MPSNR, MSSIM, ERGAS, and MSA, respectively. Furthermore, to verify the recovery performance of the proposed method in each band of hyperspectral (multispectral) images, this embodiment of the invention... Figure 3 The paper presents a comparison of PSNR and SSIM recovery for each band under different methods when WDC has SR = 5%. Clearly, the embodiments of the present invention achieve the highest PSNR and SSIM values ​​in almost every band, demonstrating that the combination of low-rank factor prior and manifold regularization helps the tensor filling model better utilize the underlying hidden structure of high-dimensional images.

[0104] Table 1. Quantitative evaluation of the recovery results of the WDC dataset at different sample rates.

[0105]

[0106] Table 2 Quantitative evaluation of the recovery results of the CAVE dataset at different sample rates

[0107]

[0108] The following embodiments of the present invention will illustrate the effectiveness of the proposed model and algorithm from a visual perspective. Figures 4-5 The images show a visual comparison of a representative band from WDC (SR=95%) and Toys (SR=90%). To better observe detail recovery, the same sub-region in each image was first marked and then magnified within the green box. By comparing the magnified areas, it is evident that HaLRTC and TRLRF perform poorly in terms of recovery because they ignore the local smooth structure of remote sensing images. Secondly, LRTV, as an enhancement model of HaLRTC, achieves clearer and smoother edges due to the inclusion of isotropic total variational regularization. However, for the WDC dataset, LRTV's recovery ability still has limitations, indicating that isotropic TV has limited ability to preserve detail information. Although STDC uses manifold learning to characterize the potential structural relationships between decomposition factors, it still fails to achieve the expected recovery performance. There may be two reasons for this: (1) The graph-Laplacian regularization in the model is based on the factor matrix and cannot make full use of the high correlation of the spectral dimension; (2) The construction of the graph depends on the filling results of HaLRTC, but the filling accuracy of HaLRTC is not high, so the constructed neighbor graph is inaccurate and cannot utilize accurate geometric information. Figures 3-4 Although (e) and (f) in the text have good visual effects, Figure 3 Some local details are still lost in (e) to (f), such as the letters in the magnified area.

[0109] In contrast, thanks to slice-based manifold regularization and non-negative constraints, the embodiments of the present invention exhibit the best visual restoration results.

[0110] Example 2

[0111] Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device can be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0112] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0113] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0114] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0115] Example 3

[0116] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, they implement the steps of the method of Embodiment 1 above.

[0117] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0122] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for remote sensing image restoration, characterized in that, Includes the following steps: S1. Obtain the tensor of the observed remote sensing image. and its corresponding index tensor ,in The element is 0 or 1, where 0 indicates that the element at the corresponding position is missing, and 1 indicates that the element at the corresponding position is not missing; S2, along the tensor Constructing a slice-based nearest neighbor graph in three directions That is, to obtain the tensor of the image to be recovered. Manifold information in all directions; where, ; S3. The manifold information is combined with graph Laplacian decomposition and low-rank Tucker decomposition to establish a non-negative tensor filling model based on factor prior and manifold regularization; the expression of the non-negative tensor filling model is: + ； s.t. = ； in and Let them represent the tensor of the image to be recovered and the tensor of the observed image, respectively. express exist Projection on; , , It is a regularization parameter. Summation operation, Let represent the nuclear norm of the matrix, and tr represent the trace operation of the matrix. Represents the space of positive real numbers. Let be the dimension of the tensor in each direction. for Modulo-n expansion matrix, Represents the graph Laplace matrix; express Tucker decomposition, and These are the kernel tensor and the factor matrix, respectively. Represents the product of a tensor and a matrix modulo - n; S4. Calculate the non-negative tensor filling model to obtain the restored image.

2. The remote sensing image restoration method according to claim 1, wherein the specific implementation process of step S2 includes: Each forward slice in the observed remote sensing image tensor Pull into column vector Arrange them in lexicographical order to obtain a modulo-3 expanded matrix; then expand each column vector... Treating them as distinct vertices, we construct a nearest neighbor graph using these vertices. The weight matrix of the nearest neighbor graph Where any two vertices i and j are column vectors and Weights between Defined as: ； Represents column vectors The topk nearest neighbors, Represents column vectors The topk nearest neighbors; manifold information , It is a graph Laplace matrix. It is a diagonal matrix, diagonal elements .

3. The remote sensing image restoration method according to claim 1, characterized in that, The specific implementation process of step S4 includes: Introduce auxiliary variables to the nonnegative tensor filling model The following equivalent model is obtained: + + ； s.t. , ， ， = ， in express The indicator function, i.e. ； The equivalent model is calculated using the augmented Lagrangian function to obtain the restored image.

4. A terminal device, characterized in that, include: One or more processors; A memory having stored one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to perform the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 3.

Images