Image processing method, device, storage medium and electronic device

Abstract

The present invention discloses an image processing method, device, storage medium and electronic device, which relate to the field of image processing technology, including: inputting an original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, and the regularization term constraints are constructed based on linear constraints and nonlinear constraints on a low-resolution hyperspectral image. The present invention inputs an original image with low spatial resolution into an image processing model, and outputs a target image with high spatial resolution, and since the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, the regularization term constraints take into account linear constraints and nonlinear constraints combined in spatial or spectral dimensions, and both characterizes the low-dimensional space and characterizes the nonlinear structure in the spatial and spectral dimensions, and combines linear and nonlinear constraints to improve the spatial resolution of the target image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, device, storage medium and electronic equipment. Background Art

[0002] Hyperspectral images have rich spatial and spectral information, which makes them widely used in land object classification, special operations, computer vision tasks, etc. However, due to the limitations of instruments and imaging optics, hyperspectral sensors often cannot obtain data with high spatial resolution, which greatly reduces the detection and recognition performance required in the application field. Summary of the invention

[0003] The main purpose of the present invention is to provide an image processing method, device, storage medium and electronic device, aiming to solve the problem of low spatial resolution of hyperspectral images in the prior art.

[0004] To achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, comprising the following steps:

[0006] Get the original image;

[0007] The original image is input into the image processing model to obtain the target image; wherein the image processing model is constructed based on the spatial resolution degradation model and regularization term constraints of the hyperspectral image, and the regularization term constraints are constructed based on the linear constraints and nonlinear constraints on the low-resolution hyperspectral image.

[0008] In a possible implementation manner of the first aspect, before inputting the original image into the image processing model to obtain the target image, the image processing method further includes:

[0009] Based on the linear and nonlinear constraints on the low-resolution hyperspectral image, the regularization term constraint is constructed;

[0010] Based on the spatial resolution degradation model and regularization term constraints of hyperspectral images, an image processing model is constructed.

[0011] In a possible implementation of the first aspect, a regularization term constraint is constructed based on a linear constraint and a nonlinear constraint on a low-resolution hyperspectral image, including:

[0012] Based on the linear constraint on the low-resolution hyperspectral image, the first regularization term constraint is obtained;

[0013] Based on the nonlinear constraint on the low-resolution hyperspectral image, a second regularization term constraint is obtained;

[0014] A regular term constraint is constructed according to the first regular term constraint and the second regular term constraint.

[0015] In a possible implementation manner of the first aspect, obtaining a first regularization term constraint based on a linear constraint on a low-resolution hyperspectral image includes:

[0016] Segmentation is performed based on low-resolution hyperspectral images to obtain three-dimensional full-band image blocks;

[0017] Search for similar cubes of the target center cube on the full-band image block according to the k-nearest neighbor algorithm to obtain several similar cubes;

[0018] Stack similar cubes separately to obtain several fourth-order tensors;

[0019] Based on several fourth-order tensors, a low-rank regularization term is constructed to obtain the first regularization term constraint.

[0020] In a possible implementation manner of the first aspect, based on a nonlinear constraint on a low-resolution hyperspectral image, a second regularization term constraint is obtained, including:

[0021] Hypergraph representation of nonlinear structures based on low-resolution hyperspectral images;

[0022] According to the hypergraph form expression, a hypergraph matrix is ​​obtained;

[0023] The hypergraph matrix is ​​used to characterize the spatial and spectral manifold structures in low-resolution hyperspectral images, and the second regularization term constraint is obtained.

[0024] In a possible implementation of the first aspect, based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint, before constructing the image processing model, the image processing method further includes:

[0025] The degradation model of two-dimensional images is extended to obtain the spatial resolution degradation model of hyperspectral images.

[0026] In a possible implementation of the first aspect, after constructing an image processing model based on a spatial resolution degradation model of a hyperspectral image and a regularization term constraint, the image processing method further includes:

[0027] Solve the image processing model based on the alternating direction multiplier method to obtain the target image processing model;

[0028] Input the original image into the image processing model to obtain the target image, including:

[0029] Input the original image into the target image processing model to obtain the target image.

[0030] In a second aspect, an embodiment of the present invention provides an image processing device, including:

[0031] An acquisition module, which is used to obtain an original image;

[0032] The target module is used to input the original image into the image processing model to obtain the target image; wherein the image processing model is constructed based on the spatial resolution degradation model and regularization term constraints of the hyperspectral image, and the regularization term constraints are constructed based on the linear constraints and nonlinear constraints on the low-resolution hyperspectral image.

[0033] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, the image processing method provided in any one of the first aspects above is implemented.

[0034] In a fourth aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to load and execute a computer program so that the electronic device executes the image processing method provided in any one of the first aspects above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] An image processing method, device, storage medium and electronic device proposed in an embodiment of the present invention include: obtaining an original image; inputting the original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, and the regularization term constraints are constructed based on linear constraints and nonlinear constraints on a low-resolution hyperspectral image. The present invention inputs an original image with low spatial resolution into an image processing model, and outputs a target image with high spatial resolution. Since the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, it is able to mine the prior knowledge expression of the latent space and spectrum, wherein the regularization term constraints take into account the linear constraints and nonlinear constraints combined in the spatial or spectral dimensions, characterizing the low-dimensional space, and characterizing the nonlinear structure in the spatial and spectral dimensions, and combining linear and nonlinear constraints to improve the spatial resolution of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present invention;

[0041] Figure 3A schematic diagram of an original image in the image processing method provided by an embodiment of the present invention;

[0042] Figure 4 Schematic diagram for comparing the effects of different super-resolution algorithms in the image processing method provided by an embodiment of the present invention, wherein (b) is an effect diagram of NN algorithm processing, (c) is an effect diagram of NARM algorithm processing, (d) is an effect diagram of SSGS algorithm processing, (e) is an effect diagram of LRTV algorithm processing, (f) is an effect diagram of the algorithm of the present invention processing, and (g) is a high-resolution reference image of the original image;

[0043] Figure 5 A schematic diagram of a module of an image processing device provided by an embodiment of the present invention;

[0044] Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] The main solution of the embodiment of the present invention is: obtaining an original image; inputting the original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model of a hyperspectral image and regularization term constraints, and the regularization term constraints are constructed based on linear constraints and nonlinear constraints on a low-resolution hyperspectral image.

[0047] Hyperspectral images have rich spatial and spectral information, which makes them widely used in land object classification, special operations, computer vision tasks, etc. However, due to the limitations of instruments and imaging optics, hyperspectral sensors often cannot obtain data with high spatial resolution, which greatly reduces the detection and recognition performance required in civil and military fields. Therefore, enhancing the spatial resolution of hyperspectral data has become an important research topic. In practical applications, improving imaging optics or sensor hardware is costly, so researchers have studied the use of post-processing techniques to improve resolution as a better alternative.

[0048] In order to overcome the inherent resolution limitation of imaging sensors, various hyperspectral image super-resolution reconstruction techniques have been proposed in the prior art. In recent years, hyperspectral image super-resolution algorithms based on prior information constraints have attracted more and more attention. For example, based on the guidance of non-local similarity in space, a regularization method with non-local constraints has been proposed. However, this type of method ignores the correlation between different bands. Although the hyperspectral super-resolution method based on global spatial-spectral correlation and local spatial-spectral smoothing characteristics can be used to overcome this defect, this method introduces new problems: on the one hand, the low-dimensional space representation method based on the nuclear norm leads to modeling bias in low-rank structure estimation; on the other hand, the nonlinear structure of hyperspectral images in spatial and spectral dimensions is ignored, resulting in a decrease in super-resolution performance.

[0049] In recent years, convolutional neural networks have attracted increasing attention in hyperspectral super-resolution due to their high efficiency and advanced performance. Most of these methods use a dual-channel convolutional neural network to extract spatial and spectral features from the input hyperspectral image, and then feed the feature cascade to the fully connected layer to generate a hyperspectral image. However, the large-scale parameters in the hyperspectral super-resolution methods based on convolutional neural networks lead to reduced stability of the network and affect the accuracy of fusion. In addition, these methods require additional data for training, but often lack hyperspectral data for pre-training. More importantly, the generalization ability of deep convolutional neural network models is weak. For example, based on data-driven network models, network models trained on one type of hyperspectral data are difficult to obtain good super-resolution performance on other types of images.

[0050] To this end, the present invention provides a solution, by inputting the original image with low spatial resolution into the image processing model, and outputting the target image with high spatial resolution. Since the image processing model is constructed based on the spatial resolution degradation model and regularization term constraints of the hyperspectral image, it is able to mine the prior knowledge expression of the latent space and spectrum. The regularization term constraints take into account the linear constraints and nonlinear constraints combined in the spatial or spectral dimensions, which not only characterizes the low-dimensional space, but also represents the nonlinear structure in the spatial and spectral dimensions, and combines the linear and nonlinear constraints to improve the spatial resolution of the target image.

[0051] See attached Figure 1 , attached Figure 1This is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present invention. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or it may be a stable non-volatile memory (NVM), such as at least one disk storage. The processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or it may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0052] Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] As attached Figure 1 As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module and an image processing device.

[0054] In the attached Figure 1 In the electronic device shown, the network interface 103 is mainly used for data communication with a network server; the user interface 104 is mainly used for data interaction with a user; the processor 101 and the memory 105 in the present invention can be set in the electronic device, and the electronic device calls the image processing device stored in the memory 105 through the processor 101, and executes the image processing method provided by the embodiment of the present invention.

[0055] See attached Figure 2 Based on the hardware device of the aforementioned embodiment, an embodiment of the present invention provides an image processing method, comprising the following steps:

[0056] S10: Expand the degradation model of two-dimensional images to obtain the spatial resolution degradation model of hyperspectral images.

[0057] In the specific implementation process, by extending the degradation model of two-dimensional images, the spatial resolution degradation model of hyperspectral images can be expressed as:

[0058] (1)

[0059] in, , They represent the hyperspectral image with low spatial resolution and the hyperspectral image to be restored respectively. and They are the downsampling operator and the blur operator respectively. By introducing The prior constraints and the least squares framework can be combined to obtain the following expression:

[0060] (2)

[0061] in, For authenticity, is a regularization term based on prior knowledge of latent space and spectrum, To balance the parameters of the above two items. Then, the problem is transformed into how to design appropriate spatial and spectral regularization terms from the potential hyperspectral image, such as total variation prior, spectral sparsity constraint and low-rank tensor expression. Although these methods have good performance, they ignore the linear and nonlinear structure of the joint spatial / spectral dimension, thus reducing the reconstruction accuracy of spatial and spectral information.

[0062] S20: Construct regularization term constraints based on linear constraints and nonlinear constraints on low-resolution hyperspectral images.

[0063] In the specific implementation process, based on the inspiration of the linear structure of hyperspectral images, considering the spatial local structure, spatial non-local similarity and spectral high correlation, a joint expression framework is proposed by combining linear and nonlinear constraints. That is, based on the linear constraints and nonlinear constraints on low-resolution hyperspectral images, regularization term constraints are constructed, including:

[0064] Based on the linear constraint on the low-resolution hyperspectral image, the first regularization term constraint is obtained;

[0065] Based on the nonlinear constraint on the low-resolution hyperspectral image, a second regularization term constraint is obtained;

[0066] A regular term constraint is constructed according to the first regular term constraint and the second regular term constraint.

[0067] For the construction of linear constraints: Based on the linear constraints of low-resolution hyperspectral images, the first regularization term constraints are obtained, including:

[0068] Segmentation is performed based on low-resolution hyperspectral images to obtain three-dimensional full-band image blocks;

[0069] Search for similar cubes of the target center cube on the full-band image block according to the k-nearest neighbor algorithm to obtain several similar cubes;

[0070] Stack similar cubes separately to obtain several fourth-order tensors;

[0071] Based on several fourth-order tensors, a low-rank regularization term is constructed to obtain the first regularization term constraint.

[0072] In the specific implementation process, the initialized hyperspectral image is segmented into three-dimensional full-band patches (FBP). For the central cube with spatial position k, the k-nearest neighbor (k-NN) algorithm is first used to search for the center cube in the local window (e.g. 70*70). Similar cubes are stacked to obtain a fourth-order tensor in order to avoid destroying the spatial spectral correlation. In this way, K fourth-order tensors can be constructed The fourth-order tensor constructed in this way can simultaneously preserve the spatial local structure, spatial non-local similarity and spectral high correlation. On this basis, a regularization term based on tensor rank minimization is proposed as follows:

[0073] (3)

[0074] in, and =1, is the kth fourth-order tensor rank, for The rank along the i-th dimension. In order to better approximate the rank of the tensor in each dimension, the MCP (minmax concave plus) relaxation term is used to characterize the low-rank structure of the tensor, as follows:

[0075] (4)

[0076] in, represents the MCP-based relaxation function, is the kth fourth-order tensor is a low-rank relaxed form of .

[0077] For the construction of nonlinear constraints: Based on the nonlinear constraints on low-resolution hyperspectral images, the second regularization term constraints are obtained, including:

[0078] Hypergraph representation of nonlinear structures based on low-resolution hyperspectral images;

[0079] According to the hypergraph form expression, a hypergraph matrix is ​​obtained;

[0080] The hypergraph matrix is ​​used to characterize the spatial and spectral manifold structures in low-resolution hyperspectral images, and the second regularization term constraint is obtained.

[0081] In the specific implementation process, from a mathematical point of view, the hypergraph form based on nonlinear structure is:

[0082] (5)

[0083] Among them, z i and z j Two points in the data, W ij is the weight of the edge connecting vertices i and j. Assume that the degree matrix in the graph structure is defined as D, which is a diagonal matrix, and the value of the i-th diagonal D ii Corresponding to all i The sum of the relevant similarities, i.e. Then, the expression of the hypergraph matrix is ​​derived as follows:

[0084] (6)

[0085] Among them, W is the degree matrix, L is the Laplas matrix of the hypergraph, and further, the optimization expression of the hypergraph is obtained as follows:

[0086] (7)

[0087] Where Tr represents the trace norm of the matrix, Z is the graph structure data, and considering the advantages of the hypergraph regularization term in describing the local piece-by-piece smoothness, formula (7) is used to describe the spatial and spectral manifold structure in the hyperspectral image, which is defined as follows:

[0088] (8)

[0089] in, Indicates that when j is 1, 2, or 3, three The weighted sum of represents the weighting parameter and greater than 0, is the graph structure matrix along two spatial and one spectral dimensions.

[0090] S30: Construct an image processing model based on the spatial resolution degradation model and regularization term constraints of hyperspectral images.

[0091] In the specific implementation process, by introducing formulas (4) and (8) into formula (2), a hyperspectral image super-resolution algorithm based on hypergraph manifold and non-local tensor low-rank constraint, that is, an image processing model, is obtained as follows:

[0092] (9)

[0093] Introducing auxiliary variables , so that , (9) can be equivalently expressed as:

[0094] (10)

[0095] S40: Obtaining an original image.

[0096] S50: Inputting the original image into the image processing model to obtain the target image; wherein the image processing model is constructed based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint, and the regularization term constraint is constructed based on the linear constraint and nonlinear constraint on the low-resolution hyperspectral image.

[0097] In the specific implementation process, the original image, that is, the low-resolution hyperspectral image to be processed, and the hyperspectral image, that is, the spectral resolution is 10 -2 Spectral images in the range of λ. Inspired by the linear structure of hyperspectral images, the non-local spatial cubes in the hyperspectral images are clustered by considering the spatial local structure, spatial non-local similarity and spectral high correlation, and the multi-dimensional linear advantage of the tensor is used to characterize its inherent low-dimensional space, and a regularization term of the non-local tensor low rank is proposed. In order to further finely encode the nonlinear structure of the hyperspectral image in the spatial and spectral dimensions, the hypergraph manifold learning method is introduced to characterize the manifold structure of the space, and the smoothness characteristics of the spatial and spectral dimensions are considered in the manifold construction, and the spatial-spectral information expression mechanism of the hypergraph manifold is proposed. Combined with linear and nonlinear constraints, a joint expression framework based on hypergraph manifold and non-local tensor low rank is proposed, and it is used as a regularization term to constrain the hyperspectral image super-resolution problem to achieve resolution enhancement and obtain the target image, that is, a high-resolution hyperspectral image.

[0098] In this embodiment, an original image with low spatial resolution is input into an image processing model, and a target image with high spatial resolution is output. Since the image processing model is constructed based on the spatial resolution degradation model and regularization term constraints of hyperspectral images, it is possible to mine the prior knowledge expression of latent space and spectrum. The regularization term constraints take into account the linear constraints and nonlinear constraints combined in the spatial or spectral dimensions, which not only characterize the low-dimensional space, but also represent the nonlinear structure in the spatial and spectral dimensions, and combine the linear and nonlinear constraints to improve the spatial resolution of the target image.

[0099] In one embodiment, after constructing an image processing model based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint, the image processing method further includes:

[0100] The image processing model is solved based on the alternating direction multiplier method to obtain the target image processing model.

[0101] In the specific implementation process, the alternating direction method of multipliers (ADMM) is a computational framework for solving separable convex optimization problems. Based on ADMM, the constraint problem in formula (10) is expressed as the following augmented Lagrangian function:

[0102] (11)

[0103] in, (i=1, K, 4) and is the Lagrange multiplier, is the penalty parameter. In the ADMM framework, each variable in (11) is solved alternately by fixing other variables.

[0104] (I) Optimization : Optimize when other variables remain unchanged The objective function is:

[0105] (12)

[0106] Formula (12) is equivalent to solving the following linear equation:

[0107] (13)

[0108] in and are the transposes of S and D, respectively. This linear equation can be solved efficiently using the preconditioned conjugate gradient method.

[0109] (II) Optimization : Optimize when other variables remain unchanged The objective function is:

[0110] (14)

[0111] This formula has a closed-form solution, which can be solved by the singular value soft threshold algorithm.

[0112] (III) Optimization : Optimize when other variables remain unchanged The objective function is:

[0113] (15)

[0114] For variables Taking the derivative of the above formula (15), we have:

[0115] (16)

[0116] Further, we get:

[0117] (17)

[0118] (IV) Update the multiplier (i=1, K, 4) and :

[0119] (18)

[0120] in, is the parameter iteration factor.

[0121] The super-resolution problem is solved by the ADMM algorithm, and the image processing model is optimized to obtain the target image processing model.

[0122] Based on the above steps, the original image is input into the image processing model to obtain the target image, including:

[0123] Input the original image into the target image processing model to obtain the target image.

[0124] In order to quantitatively and qualitatively evaluate the super-resolution effect, this paper selects NN (nearest neighbor interpolation), NARM (nonlocal autoregressive model), SSGS (spatial–spectral group sparsity) and LRTV (low-rank and total variation) super-resolution methods as comparison methods, using the same original image as shown in the attached figure. Figure 3As shown, these comparison methods are used to compare the super-resolution results of hyperspectral images with the results of the present invention. The present invention uses three quality evaluation indicators: peak signal-to-noise ratio PSNR (dB), structural similarity SSIM, and spectral angle matching SAM. The quantitative analysis results are shown in Table 1.

[0125] Table 1-Quantitative evaluation results under 2x super-resolution

[0126]

[0127] As can be seen from Table 1, under the 2-fold super-resolution condition, the algorithm of the present invention obtains larger PSNR and SSIM, which also proves the effectiveness of the method. In addition, the algorithm of the present invention obtains the smallest SAM value, which shows the inhibitory effect of the method on spectral distortion.

[0128] As attached Figure 4 As shown, a comparison diagram of the effects of the above super-resolution algorithms on the original image is given, where (b) is the effect diagram of the NN algorithm, (c) is the effect diagram of the NARM algorithm, (d) is the effect diagram of the SSGS algorithm, (e) is the effect diagram of the LRTV algorithm, (f) is the effect diagram of the algorithm of the present invention, and (g) is the high-resolution reference image of the original image. The above super-resolution algorithms can restore low-resolution images, but careful observation shows that NN, NARM and LRTV can only restore large-scale texture structures. Compared with NN, NARM and LRTV, SSGS can reconstruct small-scale details, but there is a more serious distortion of spectral information. Comprehensive comparison shows that the super-resolution algorithm of the present invention has the best restoration performance and is closer to the reference image.

[0129] See attached Figure 5 Based on the same inventive concept as in the above-mentioned embodiment, the embodiment of the present invention further provides an image processing device, comprising:

[0130] An acquisition module, which is used to obtain an original image;

[0131] The target module is used to input the original image into the image processing model to obtain the target image; wherein the image processing model is constructed based on the spatial resolution degradation model and regularization term constraints of the hyperspectral image, and the regularization term constraints are constructed based on the linear constraints and nonlinear constraints on the low-resolution hyperspectral image.

[0132] Those skilled in the art should understand that the division of the various modules in the embodiment is merely a division of logical functions, and in actual application, all or part of them can be integrated into one or more actual carriers, and these modules can be implemented entirely in the form of software called through a processing unit, or entirely in the form of hardware, or in the form of a combination of software and hardware. It should be noted that each module in the image processing device in this embodiment corresponds one-to-one to each step in the image processing method in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned image processing method, which will not be repeated here.

[0133] Based on the same inventive concept as in the aforementioned embodiment, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, the image processing method provided in the embodiment of the present invention is implemented.

[0134] Based on the same inventive concept as in the above-mentioned embodiment, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein:

[0135] The memory is used to store computer programs;

[0136] The processor is used to load and execute a computer program so that the electronic device executes the image processing method provided by the embodiment of the present invention.

[0137] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0138] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0139] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0140] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0143] In summary, the present invention provides an image processing method, device, storage medium and electronic device, the method comprising: obtaining an original image; inputting the original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, and the regularization term constraints are constructed based on linear constraints and nonlinear constraints on a low-resolution hyperspectral image. The present invention inputs an original image with low spatial resolution into an image processing model, and outputs a target image with high spatial resolution. Since the image processing model is constructed based on a spatial resolution degradation model and regularization term constraints of a hyperspectral image, it is able to mine the prior knowledge expression of the latent space and spectrum, wherein the regularization term constraints take into account the linear constraints and nonlinear constraints combined in the spatial or spectral dimensions, characterizing the low-dimensional space, and characterizing the nonlinear structure in the spatial and spectral dimensions, and combining linear and nonlinear constraints to improve the spatial resolution of the target image.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The following steps are involved: Get the original image; Inputting the original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model of a hyperspectral image and a regularization term constraint, and the regularization term constraint is constructed based on a linear constraint and a nonlinear constraint on the low-resolution hyperspectral image; Before inputting the original image into the image processing model to obtain the target image, the image processing method further includes: The regularization term constraint is constructed based on the linear constraint and the nonlinear constraint on the low-resolution hyperspectral image; the regularization term constraint is constructed based on the linear constraint and the nonlinear constraint on the low-resolution hyperspectral image, including: Based on the linear constraint on the low-resolution hyperspectral image, a first regularization term constraint is obtained; Based on the nonlinear constraint on the low-resolution hyperspectral image, a second regularization term constraint is obtained; According to the first regularization term constraint and the second regularization term constraint, the regularization term constraint is constructed; wherein: the first regularization term constraint is obtained based on the linear constraint on the low-resolution hyperspectral image, including: Segmenting the low-resolution hyperspectral image to obtain a three-dimensional full-band image block; Searching for similar cubes of the target central cube on the full-band image block according to a k-nearest neighbor algorithm to obtain a plurality of similar cubes; Stacking the similar cubes respectively to obtain a number of fourth-order tensors; Based on the plurality of fourth-order tensors, construct a low-rank regularization term to obtain a first regularization term constraint; The second regularization term constraint is obtained based on the nonlinear constraint on the low-resolution hyperspectral image, comprising: Based on the low-resolution hyperspectral image, a hypergraph expression of a nonlinear structure is performed; According to the hypergraph form expression, a hypergraph matrix is ​​obtained; The hypergraph matrix is ​​used to characterize the spatial and spectral manifold structure in the low-resolution hyperspectral image to obtain the second regularization term constraint; wherein the hypergraph form based on the nonlinear structure is: Among them, z i and z j Two points in the data, W ij is the weight of the edge connecting vertices i and j. Assume that the degree matrix in the graph structure is defined as D, which is a diagonal matrix. The value of the i-th diagonal is D ii Corresponding to all i The sum of the relevant similarities, i.e. , the expression of the hypergraph matrix is ​​derived as follows: Where W is the degree matrix, L is the Laplas matrix of the hypergraph, and the optimization expression of the hypergraph is obtained as follows: Among them, Tr represents the trace norm of the matrix, Z is the graph structure data, and the optimization expression of the hypergraph is used to characterize the spatial and spectral manifold structure in the hyperspectral image, which is defined as follows: in, Indicates that when j is 1, 2, or 3, three The weighted sum of represents the weighting parameter and greater than 0, is the graph structure matrix along two spatial and one spectral dimensions; The image processing model is constructed based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint.

2. The image processing method according to claim 1, characterized in that: Before constructing the image processing model based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint, the image processing method further includes: The degradation model of the two-dimensional image is expanded to obtain the spatial resolution degradation model of the hyperspectral image.

3. The image processing method according to claim 1, characterized in that: After constructing the image processing model based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint, the image processing method further includes: Solving the image processing model based on the alternating direction multiplier method to obtain a target image processing model; The step of inputting the original image into an image processing model to obtain a target image comprises: The original image is input into the target image processing model to obtain the target image.

4. An image processing device, characterized in that: include: An acquisition module, wherein the acquisition module is used to obtain an original image; A target module, wherein the target module is used to input the original image into an image processing model to obtain a target image; wherein the image processing model is constructed based on a spatial resolution degradation model of a hyperspectral image and a regularization term constraint, and the regularization term constraint is constructed based on a linear constraint and a nonlinear constraint on the low-resolution hyperspectral image; Before inputting the original image into the image processing model to obtain the target image, the method further includes: The regularization term constraint is constructed based on the linear constraint and the nonlinear constraint on the low-resolution hyperspectral image; the regularization term constraint is constructed based on the linear constraint and the nonlinear constraint on the low-resolution hyperspectral image, including: Based on the linear constraint on the low-resolution hyperspectral image, a first regularization term constraint is obtained; Based on the nonlinear constraint on the low-resolution hyperspectral image, a second regularization term constraint is obtained; According to the first regularization term constraint and the second regularization term constraint, the regularization term constraint is constructed; wherein: the first regularization term constraint is obtained based on the linear constraint on the low-resolution hyperspectral image, including: Segmenting the low-resolution hyperspectral image to obtain a three-dimensional full-band image block; Searching for similar cubes of the target central cube on the full-band image block according to a k-nearest neighbor algorithm to obtain a plurality of similar cubes; Stacking the similar cubes respectively to obtain a number of fourth-order tensors; Based on the plurality of fourth-order tensors, construct a low-rank regularization term to obtain a first regularization term constraint; The second regularization term constraint is obtained based on the nonlinear constraint on the low-resolution hyperspectral image, comprising: Based on the low-resolution hyperspectral image, a hypergraph expression of a nonlinear structure is performed; According to the hypergraph form expression, a hypergraph matrix is ​​obtained; The hypergraph matrix is ​​used to characterize the spatial and spectral manifold structure in the low-resolution hyperspectral image to obtain the second regularization term constraint; wherein the hypergraph form based on the nonlinear structure is: Among them, z i and z j Two points in the data, W ij is the weight of the edge connecting vertices i and j. Assume that the degree matrix in the graph structure is defined as D, which is a diagonal matrix. The value of the i-th diagonal is D ii Corresponding to all i The sum of the relevant similarities, i.e. , the expression of the hypergraph matrix is ​​derived as follows: Where W is the degree matrix, L is the Laplas matrix of the hypergraph, and the optimization expression of the hypergraph is obtained as follows: Among them, Tr represents the trace norm of the matrix, Z is the graph structure data, and the optimization expression of the hypergraph is used to characterize the spatial and spectral manifold structure in the hyperspectral image, which is defined as follows: in, Indicates that when j is 1, 2, or 3, three The weighted sum of represents the weighting parameter and greater than 0, is the graph structure matrix along two spatial and one spectral dimensions; The image processing model is constructed based on the spatial resolution degradation model of the hyperspectral image and the regularization term constraint.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by a processor, the image processing method according to any one of claims 1 to 3 is implemented.

6. An electronic device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the image processing method according to any one of claims 1 to 3.

Images