Multi-spectral image spectrum super-resolution method in self-supervision mode

Abstract

The invention relates to the technical field of multi-spectral remote sensing image spectral resolution enhancement, and provides a multi-spectral image spectral super-resolution method in a self-supervision mode, which comprises the following steps: respectively carrying out normalization processing on observation multi-spectral data # imgabs0 # and # imgabs1 # hyperspectral data # imgabs0 # and # imgabs1 #; inputting the normalized multispectral and hyperspectral data pairs # imgabs2 # and # imgabs3 # into the existing DiriNet to obtain space and spectrum degradation functions PSF # imgabs4 # and SRF # imgabs5 # between the data pairs, and carrying out space degradation processing on the normalized multispectral data # imgabs6 #; designing a U-shaped spectrum super-resolution network to improve the spectral resolution of the degradation result Xspad to the spectral resolution level of hyperspectral data; carrying out spectrum degradation treatment; under the supervision of # imgabs7 # and Xspad, parameter optimization of the U-shaped spectrum super-resolution network is realized; and inputting the normalized multispectral data # imgabs8 # into the U-shaped spectrum super-resolution network, and performing amplitude restoration to generate high-space and high-spectral-resolution data in a large view field, which is urgently needed in the field of remote sensing earth observation. The method is used for enhancing the spectral resolution of the multispectral image and providing effective data for the field of remote sensing earth observation.

Description

Technical Field

[0001] The present invention relates to the technical field of multispectral remote sensing image spectral resolution enhancement, which is a multispectral image spectral super-resolution method under a self-supervisory mode, and is used to enhance the spectral resolution of multispectral images and provide effective data for the remote sensing earth observation field. Background Art

[0002] With the rapid development of spectral imaging and remote sensing technology, hyperspectral remote sensing that can provide spectral characteristics for target identification and multispectral remote sensing that can provide fine spatial characteristics of targets have been widely used in earth observation fields such as mineral exploration, precision agriculture, military target identification, forest research, and environmental monitoring. Driven by a reasonable signal-to-noise ratio, narrow-band imaging hyperspectral images have the defects of low spatial resolution and small field of view. Although wide-band imaging multispectral images have relatively high spatial resolution and relatively large field of view, their spectral resolution is low. In order to generate remote sensing data with high spatial and spectral resolution to achieve fine observation of the ground, the fusion technology of hyperspectral and multispectral data captured in the same scene and at the same viewing angle has attracted much attention. At present, the mainstream hyperspectral and multispectral image fusion methods can be roughly divided into two categories: optimization and deep learning.

[0003] Generally speaking, optimization fusion methods embed one or more prior assumptions into the inverse problem of hyperspectral and multispectral image fusion, establish a prior regularized fusion model, and then design an effective optimization algorithm to solve the model. However, artificial priors are difficult to accurately characterize complex environmental features, and the optimization algorithm is highly complex, so that practical performance and real-time application have become huge challenges that optimization fusion cannot overcome. Since deep learning has excellent nonlinear feature extraction capabilities in high-dimensional data processing, and its data-driven model can transfer the computational burden to the training stage, it has attracted many scholars to introduce it into the field of hyperspectral and multispectral fusion. Existing deep learning fusion methods mainly include two modes: supervised and self-supervised. The fusion in the supervised mode directly learns the nonlinear reconstruction mapping of hyperspectral and multispectral images to the fusion target, and it is necessary to use high and multispectral sensor functions to construct sufficient high-resolution hyperspectral / low-resolution hyperspectral and high-resolution multispectral image group training networks. Existing supervised fusion methods are mostly based on public data sets and sensor functions to construct simulated sample sets. The scale of public data is small and the imaging conditions are ideal. It is difficult to fit the actual complex scenes, so its generalization performance is poor. Fusion in self-supervised mode is based on linear degradation theory and achieves fusion under the supervision of high and multi-spectral data themselves. There is no need to construct a simulated sample set with high-resolution hyperspectral images as labels. It has been very popular in recent years.

[0004] However, the above high-multispectral image fusion technology can only improve the spatial resolution of hyperspectral data within the field of view perceived by the hyperspectral sensor, and cannot meet the requirements of large-scene observation. Therefore, the multispectral image spectral super-resolution technology has emerged. This technology can effectively enhance the spectral resolution of multispectral data and generate high-spatial and high-spectral resolution data within the large field of view perceived by the multispectral sensor. Most existing methods are based on high- and multispectral data pairs with the same spatial resolution to estimate the spectral response function of the degradation of hyperspectral data to the corresponding multispectral data, and then use methods such as regularization modeling and spectral matching to improve the spectral resolution of multispectral data. However, the actual spatial resolution difference between high- and multispectral data is large. Simple upsampling operations such as interpolation of hyperspectral data are difficult to make up for the spatial difference between the two, resulting in inaccurate estimation of the spectral response function and unsatisfactory spectral super-resolution effect of multispectral data. Therefore, so far, there has been little research on the multispectral image spectral super-resolution technology. Summary of the Invention

[0005] In order to obtain high-spatial and high-spectral resolution remote sensing data under a large field of view, the present invention combines DiriNet and a U-shaped network based on a fully connected layer to propose a self-supervised multispectral image spectral super-resolution method. This technology inputs the observed high- and multispectral data pairs into DiriNet to estimate the SRF and PSF, uses the estimated PSF to spatially degrade the observed multispectral image, then designs a U-shaped spectral super-resolution network to perform spectral super-resolution on the degraded result, and uses the estimated SRF to spectrally degrade the spectral super-resolution result. Finally, the U-shaped spectral super-resolution network is optimized under the self-supervision of the observed hyperspectral data and the spatial degradation result of the multispectral data. Then, inputting the observed multispectral data into the optimized U-shaped spectral super-resolution network can achieve spectral super-resolution and generate high-spatial and high-spectral resolution data under a large field of view with the same field of view, spatial resolution as the multispectral data, and spectral resolution as the hyperspectral data. A self-supervised multispectral image spectral super-resolution method includes the following steps:

[0006] S1, respectively perform the following amplitude normalization processing on the observed multispectral data and hyperspectral data as follows:

[0007]

[0008] wherein, m, n, and B respectively refer to the number of rows, columns, and bands of the hyperspectral data, M = rm, N = rn, b << B respectively refer to the number of rows, columns, and bands of the multispectral data, and r represents the spatial resolution ratio between the high- and multispectral data, and Respectively represent the normalized multispectral and hyperspectral data; the subscript ijk represents the value of the element in the i-th row, j-th column, and k-th band in the tensor; for example, mean The pixel value of the k-th band in the i-th row and j-th column of ; max(·) represents the maximum value extraction operation.

[0009] S2, normalize the multi- and hyperspectral data and Input the existing DiriNet and get Degenerate to PSF with consistent spatial resolution And can Degenerate to SRF with consistent spectral resolution

[0010] S3, normalize the multispectral data using F estimated in step S2 Perform the following spatial degradation processing:

[0011]

[0012] in, yes The spatial resolution degrades to The spatial resolution of the results is consistent with that of r represents a shifted convolution with a stride of r.

[0013] S4, based on the fully connected layer, U-shaped network structure and skip layer connection, a U-shaped spectral super-resolution network is designed to transform the spatial degradation result of step S3 The spectral resolution of the data is improved to the level of the spectral resolution of the hyperspectral data, and the

[0014] S5, using R estimated in step S2 to perform spectral super-resolution on the result of step S4 Perform the following spectral degradation processing:

[0015]

[0016] in, yes The spectral resolution degrades to The result of the spectral resolution level, the subscript ×3 indicates the modulo 3 multiplication operation of the tensor and the matrix, R Τ is the transpose of R, It is equivalent to converting TR into its corresponding tensor form. yes The Casorati matrix.

[0017] S6, in and Under the supervision of , the parameter optimization of U-type spectral super-resolution network is achieved.

[0018] S7, normalize the multispectral data Input the U-type spectrum super-resolution network optimized in step S6, and then restore the amplitude to generate high spatial and high spectral resolution data with a large field of view, which is urgently needed in the field of remote sensing earth observation.

[0019]

[0020] Among them, U θ-trained (·) represents the U-type spectral super-resolution network with optimized parameters, and the symbol “·” represents the dot product operation.

[0021] Furthermore, step S4 specifically includes the following steps:

[0022] S401: Yes Perform matrix operations to obtain its Casorati matrix X spad ∈R mn×b :

[0023]

[0024] Here, reshape(·) represents the operation of reshaping the dimension of an array.

[0025] S402: X spad Input the U-type spectrum super-resolution network to increase its spectral resolution to the spectral resolution level of the observed hyperspectral data:

[0026] T=U θ (X spad ) (7)

[0027] in, Yes X spad Spectral super-resolution results, U θ (·) represents a U-type spectral super-resolution network.

[0028] S403: Convert T to its corresponding third-order tensor

[0029]

[0030] Furthermore, step S6 specifically includes the following steps:

[0031] S601: Calculate spectral super-resolution results With normalized hyperspectral data As well as the Euclidean distance between the spectral degradation results and the spatial degradation results, it is used to construct the loss function L for optimizing the U-type spectral super-resolution network:

[0032]

[0033] Among them, δ>0 is the weight factor.

[0034] S602: By minimizing the loss function L, the optimizer Adma is used to optimize the parameters of the U-type super-resolution network until they meet the threshold condition.

[0035] The present invention has the beneficial effects:

[0036] 1. The multispectral image spectral super-resolution method provided by the present invention adopts the latest function estimation method DiriNet to accurately estimate PSF and SRF based on high-resolution and multispectral data with large differences in spatial and spectral resolution, laying the premise for self-supervised spectral super-resolution of multispectral data.

[0037] 2. The multispectral image spectral super-resolution method provided by the present invention establishes a self-supervised spectral super-resolution strategy based on the linear degradation relationship between high- and multi-spectral data pairs, which effectively overcomes the problem of poor robustness of existing multispectral data spectral super-resolution technology to the spatial resolution difference between the two source data.

[0038] 3. The multispectral image spectral super-resolution method provided by the present invention designs a U-type spectral super-resolution network, which uses the superior nonlinear representation ability of neural networks to fit the inverse process of spectral degradation, enhance the spectral resolution of multispectral data, and obtain high spatial and high spectral resolution data under a large field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0041] The present invention provides a multispectral image spectral super-resolution method in a self-supervised mode, which can enhance the spectral resolution of multispectral data in a self-supervised manner based on high and multispectral data pairs with spatial and spectral resolution differences of several times, generate high spatial and high spectral data under a large field of view, and lay a data foundation for the field of large-scale earth observation. The multispectral remote sensing image self-supervised spectral super-resolution method provided by the present invention is as follows: Figure 1 shown.

[0042] The process of the present invention is described below with specific embodiments:

[0043] In this embodiment, the spatial resolution of the multispectral image is 8 times that of the hyperspectral image, and the spectral resolution of the hyperspectral image is more than 40 times that of the multispectral image. Taking two-source remote sensing data with large differences in spatial and spectral resolution as an example, the spectral super-resolution method provided by the present invention is specifically described. The spectral super-resolution method provided by the present invention is specifically as follows:

[0044] Step S1, observe the multispectral data and hyperspectral data Perform the following amplitude normalization process:

[0045]

[0046] Where m, n, and B refer to the number of rows, columns, and bands of hyperspectral data respectively, M = rm, N = rn, and b < < B refer to the number of rows, columns, and bands of multispectral data respectively, and r represents the spatial resolution ratio between hyperspectral and multispectral data. and Respectively represent the normalized multispectral and hyperspectral data; the subscript ijk represents the value of the element in the i-th row, j-th column, and k-th band in the tensor; for example, mean The pixel value of the k-th band in the i-th row and j-th column of ; max(·) represents the maximum value extraction operation.

[0047] Step S2: normalize the multi- and hyperspectral data and Input the existing DiriNet and get Degenerate to PSF with consistent spatial resolution And can Degenerate to SRF with consistent spectral resolution

[0048] Step S3, using the F estimated in step S2 to normalize the multispectral data Perform the following spatial degradation processing:

[0049]

[0050] in, yes The spatial resolution degrades to The spatial resolution of the results is consistent with that of r represents a shifted convolution with a stride of r.

[0051] Step S4, designing a U-type spectral super-resolution network based on a fully connected layer, a U-type network structure and a skip layer connection to convert the spatial degradation result of step S3 The spectral resolution of the data is improved to the level of the spectral resolution of the hyperspectral data, and the The specific steps include:

[0052] S401: Yes Perform matrix operations to obtain its Casorati matrix X spad ∈R mn×b :

[0053]

[0054] Here, reshape(·) represents the operation of reshaping the dimension of an array.

[0055] S402: X spad Input the U-type spectrum super-resolution network to increase its spectral resolution to the spectral resolution level of the observed hyperspectral data:

[0056] T=U θ (X spad ) (14)

[0057] in, Yes X spad Spectral super-resolution results, U θ (·) represents a U-type spectral super-resolution network.

[0058] S403: Convert T to its corresponding third-order tensor

[0059]

[0060] Step S5, using the R estimated in step S2 to perform spectral super-resolution on the result of step S4 Perform the following spectral degradation processing:

[0061]

[0062] in, yes The spectral resolution degrades to The result of the spectral resolution level, the subscript ×3 indicates the modulo 3 multiplication operation of the tensor and the matrix, R Τ is the transpose of R, It is equivalent to converting TR into its corresponding tensor form. yes The Casorati matrix.

[0063] Step S6, and Under the supervision of , the parameters of the U-type spectral super-resolution network are optimized. The specific steps include:

[0064] S601: Calculate spectral super-resolution results With normalized hyperspectral data As well as the Euclidean distance between the spectral degradation results and the spatial degradation results, it is used to construct the loss function L for optimizing the U-type spectral super-resolution network:

[0065]

[0066] Among them, δ>0 is the weight factor.

[0067] S602: By minimizing the loss function L, the optimizer Adma is used to optimize the parameters of the U-type super-resolution network until they meet the threshold condition.

[0068] Step S7, normalizing the multispectral data Input the U-type spectrum super-resolution network optimized in step S6, and then restore the amplitude to generate high spatial and high spectral resolution data with a large field of view, which is urgently needed in the field of remote sensing earth observation.

[0069]

[0070] Among them, U θ-trained (·) represents the U-type spectral super-resolution network with optimized parameters, and the symbol “·” represents the dot product operation.

Claims

1. A multispectral image super-resolution method in a self-supervised mode, characterized in that: The steps include: S1, respectively observe the multispectral data and hyperspectral data Perform the following amplitude normalization process: Wherein, m, n, and B respectively refer to the number of rows, columns, and bands of the hyperspectral data, M = rm, N = rn, and b << B respectively refer to the number of rows, columns, and bands of the multispectral data, r represents the spatial resolution ratio between the hyperspectral and multispectral data. and respectively represent the normalized multispectral and hyperspectral data; the subscript ijk represents the element value of the i-th row, j-th column, and k-th band in the tensor. means the pixel value of the i-th row, j-th column, and k-th band of; max(·) represents the operation of extracting the maximum value. S2, normalize the multi- and hyperspectral data and Input the existing DiriNet and get Degenerate to PSF with consistent spatial resolution And can Degenerate to SRF with consistent spectral resolution S3, normalize the multispectral data using F estimated in step S2 Perform the following spatial degradation processing: in, yes The spatial resolution degrades to The spatial resolution of the results is consistent with that of r represents a shifted convolution with a step size of r; S4, based on the fully connected layer, U-shaped network structure and skip layer connection, a U-shaped spectral super-resolution network is designed to transform the spatial degradation result of step S3 The spectral resolution of the data is improved to the level of the spectral resolution of the hyperspectral data, and the S5, using R estimated in step S2, performing the following spectral degradation processing on the spectral super-resolution result T of step S4: in, yes The spectral resolution degrades to The result of the spectral resolution level, the subscript ×3 indicates the modulo 3 multiplication operation of the tensor and the matrix, R Τ is the transpose of R, It is equivalent to converting TR into its corresponding tensor form. yes Casorati matrix; S6, in and Under the supervision of , the parameters of the U-type spectral super-resolution network are optimized; S7, normalize the multispectral data Input the U-type spectrum super-resolution network optimized in step S6, and then restore the amplitude to generate high spatial and high spectral resolution data with a large field of view, which is urgently needed in the field of remote sensing earth observation. Among them, U θ-trained (·) represents the U-type spectral super-resolution network with optimized parameters, and the symbol "·" represents the dot product operation.

2. The multispectral image super-resolution method in a self-supervised mode according to claim 1, characterized in that: Step S4 specifically includes the following steps: S401: Yes Perform matrix operations to obtain its Casorati matrix X spad ∈R mn×b : Among them, reshape(·) represents the dimension resetting operation of the array; S402: X spad Input the U-type spectrum super-resolution network to increase its spectral resolution to the spectral resolution level of the observed hyperspectral data: T=U θ (X spad ) (7) in, Yes X spad The spectral super-resolution results, U θ (·) represents a U-type spectral super-resolution network; S403: Convert T to its corresponding third-order tensor 3. The multispectral image super-resolution method in a self-supervised mode according to claim 1, characterized in that: Step S6 specifically includes the following steps: S601: Calculate spectral super-resolution results With normalized hyperspectral data As well as the Euclidean distance between the spectral degradation results and the spatial degradation results, it is used to construct the loss function L for optimizing the U-type spectral super-resolution network: Among them, δ>0 is the weight factor; S602: By minimizing the loss function L, the optimizer Adma is used to optimize the parameters of the U-type super-resolution network until they meet the threshold condition.

Images