Spectral image fusion super-resolution method, system and electronic device

Abstract

The application provides a spectral image fusion super-resolution method and system and electronic equipment, which performs spatial upsampling on an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image; performs spatial downsampling and spatial upsampling on an initial high-resolution RGB image in sequence to obtain a first high-resolution RGB image; extracts multi-order feature vectors of the initial high-resolution RGB image, the first high-resolution RGB image and the first high-resolution hyperspectral image respectively; and fuses all the multi-order feature vectors and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image. The application can use an existing high-resolution RGB image to perform super-resolution reconstruction on a low-resolution hyperspectral image, thereby obtaining a high-resolution hyperspectral image.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a super-resolution method, system and electronic device for fusion of spectral images. Background Technology

[0002] Due to limitations in spectral imaging principles, acquiring high spatial resolution hyperspectral images is extremely difficult. Currently, fusing high spatial resolution RGB images with low spatial resolution hyperspectral images has become the optimal method for obtaining high spatial resolution hyperspectral images. Existing fusion methods can be categorized into traditional optimization methods and deep learning methods.

[0003] The unified optimization algorithm fully utilizes the imaging principles of RGB cameras and hyperspectral imagers for modeling: Y = SX,X L =XH, where S represents the spectral response function (i.e., spectral degradation operation), and H represents the spatial degradation operation. Y represents a high spatial resolution RGB image, and X represents a high spatial resolution hyperspectral image of the target. L This represents a low spatial resolution hyperspectral image. Traditional optimization algorithms, through the study of prior knowledge, model one or more effective regularization terms, using these regularization terms as penalty functions to optimize the results for high spatial resolution hyperspectral images. The reconstruction accuracy of this type of method mainly depends on the effectiveness of the regularization terms modeled based on prior knowledge. These prior knowledge assumptions are often based on heuristic data distributions (e.g., low rank and sparsity) or shallow prior knowledge (e.g., extracting spatial structure from RGB images through operator filtering). This type of artificially defined prior knowledge is uncertain and not suitable for all fusion tasks, leading to failure when reconstructing certain complex spectral images. For example, current shallow priors always use total variation to make the target reconstructed image smoother in both the spectral and spatial domains. However, when facing images with complex structural information, this regularization term is clearly not entirely correct, thus limiting the application scope of traditional optimization methods.

[0004] In the field of deep learning methods, a method has been proposed that uses deep neural networks to learn and transfer features from high spatial resolution RGB images to low spatial resolution hyperspectral images, achieving better reconstruction accuracy. To better focus on image feature information, attention mechanisms have seen widespread application in the past two years. Current attention mechanisms mainly unfold the three-dimensional image tensor into a two-dimensional image matrix X along the spatial dimension. 2D Then, the image matrix is ​​divided into Q, K, V using a linear layer: Q = Linear(X 2D ), K = Linear(X) 2D V = Linear(X) 2D), where Linear(·) represents a linear layer; then attention is calculated on the extracted Q, K, V: Where d k This represents the number of pixels. However, the existing deep learning methods mentioned above are based on supervised learning training, which has a large number of parameters in its linear layers, making fitting difficult and requiring a long training time. Furthermore, it requires real high spatial resolution hyperspectral images as labels, and its application in reality is limited due to the difficulty in obtaining real labels. As for unsupervised learning methods, since there are no real labels, image features are difficult to extract correctly, and therefore they cannot be directly applied to super-resolution reconstruction. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, system and electronic device for fusion super-resolution of spectral images to alleviate the above-mentioned problems existing in the related art.

[0006] In a first aspect, embodiments of the present invention provide a fusion super-resolution method for spectral images, the method being applied to a pre-trained image feature extraction network; the image feature extraction network includes a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; the method includes: spatially upsampling an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image; sequentially spatially downsampling and spatially upsampling an initial high-resolution RGB image to obtain a first high-resolution RGB image; wherein the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image; and processing the initial high-resolution RGB image using the first deep convolutional neural network. Multi-level feature extraction is performed on the first high-resolution RGB image to obtain a first feature vector corresponding to each level of the initial high-resolution RGB image and a second feature vector corresponding to each level of the first high-resolution RGB image. Multi-level feature extraction is then performed on the first high-resolution hyperspectral image using a second deep convolutional neural network to obtain a third feature vector corresponding to each level of the first high-resolution hyperspectral image. Each level of feature extraction by the second deep convolutional neural network corresponds one-to-one with each level of feature extraction by the first deep convolutional neural network. All first feature vectors, all second feature vectors, all third feature vectors, and the first high-resolution hyperspectral image are fused to obtain a second high-resolution hyperspectral image.

[0007] Secondly, embodiments of the present invention also provide a super-resolution fusion system for spectral images, wherein the system is applied to a pre-trained image feature extraction network; the image feature extraction network includes a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; the system includes: a first processing subsystem for spatially upsampling an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image; a second processing subsystem for sequentially spatially downsampling and spatially upsampling an initial high-resolution RGB image to obtain the first high-resolution RGB image; wherein the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image; and a pre-trained first deep convolutional neural network for processing the initial high-resolution image. Multi-level feature extraction is performed on the initial high-resolution RGB image and the first high-resolution RGB image to obtain a first feature vector corresponding to each level of the initial high-resolution RGB image and a second feature vector corresponding to each level of the first high-resolution RGB image. A pre-trained second deep convolutional neural network is used to perform multi-level feature extraction on the first high-resolution hyperspectral image to obtain a third feature vector corresponding to each level of the first high-resolution hyperspectral image. Each level of feature extraction by the second deep convolutional neural network corresponds one-to-one with each level of feature extraction by the first deep convolutional neural network. A fusion subsystem is used to fuse all first feature vectors, all second feature vectors, all third feature vectors, and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image.

[0008] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-described spectral image fusion super-resolution method.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the above-described spectral image fusion super-resolution method.

[0010] This invention provides a method, system, and electronic device for super-resolution fusion of spectral images. The method involves spatially upsampling an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image; sequentially spatially downsampling and upsampling an initial high-resolution RGB image to obtain another high-resolution RGB image; performing multi-level feature extraction on both the initial high-resolution RGB image and the first high-resolution RGB image using a first deep convolutional neural network to obtain a first feature vector corresponding to each level in the initial high-resolution RGB image and a second feature vector corresponding to each level in the first high-resolution RGB image; performing multi-level feature extraction on the first high-resolution hyperspectral image using a second deep convolutional neural network to obtain a third feature vector corresponding to each level in the first high-resolution hyperspectral image; and fusing all first, second, and third feature vectors with the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image. By using this technique, a pre-trained convolutional neural network replaces the linear layer in the attention mechanism, allowing the attention operation to extract different levels of feature information from the image. This enables super-resolution reconstruction of low-resolution hyperspectral images using existing high-resolution RGB images, thereby obtaining high-resolution hyperspectral images.

[0011] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0012] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0013] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a spectral image fusion super-resolution method according to an embodiment of the present invention;

[0015] Figure 2 This is an example diagram illustrating the workflow of the image feature extraction network in an embodiment of the present invention;

[0016] Figure 3 This is an example diagram comparing the reconstructed spectrum with the true spectrum in an embodiment of the present invention;

[0017] Figure 4 This is a schematic diagram of the structure of a spectral image fusion super-resolution system according to an embodiment of the present invention;

[0018] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, fusing high spatial resolution RGB images with low spatial resolution hyperspectral images has become the optimal method for acquiring high spatial resolution hyperspectral images. Existing fusion methods can be divided into traditional optimization methods and deep learning methods.

[0021] The unified optimization algorithm fully utilizes the imaging principles of RGB cameras and hyperspectral imagers for modeling: Y = SX,X L =XH, where S represents the spectral response function (i.e., spectral degradation operation), and H represents the spatial degradation operation. Y represents a high spatial resolution RGB image, and X represents a high spatial resolution hyperspectral image of the target. L This represents a low spatial resolution hyperspectral image. Traditional optimization algorithms, through the study of prior knowledge, model one or more effective regularization terms, using these regularization terms as penalty functions to optimize the results for high spatial resolution hyperspectral images. The reconstruction accuracy of this type of method mainly depends on the effectiveness of the regularization terms modeled based on prior knowledge. These prior knowledge assumptions are often based on heuristic data distributions (e.g., low rank and sparsity) or shallow prior knowledge (e.g., extracting spatial structure from RGB images through operator filtering). This type of artificially defined prior knowledge is uncertain and not suitable for all fusion tasks, leading to failure when reconstructing certain complex spectral images. For example, current shallow priors always use total variation to make the target reconstructed image smoother in both the spectral and spatial domains. However, when facing images with complex structural information, this regularization term is clearly not entirely correct, thus limiting the application scope of traditional optimization methods.

[0022] In the field of deep learning methods, a method has been proposed that uses deep neural networks to learn and transfer features from high spatial resolution RGB images to low spatial resolution hyperspectral images, achieving better reconstruction accuracy. To better focus on image feature information, attention mechanisms have seen widespread application in the past two years. Current attention mechanisms mainly unfold the three-dimensional image tensor into a two-dimensional image matrix X along the spatial dimension. 2D Then, the image matrix is ​​divided into Q, K, V using a linear layer: Q = Linear(X 2D ), K = Linear(X) 2D V = Linear(X) 2D ), where Linear(·) represents a linear layer; then, attention scores are calculated and masks are superimposed on the extracted Q, K, and V: Where d k This represents the number of pixels. However, the existing deep learning methods mentioned above are based on supervised learning training, which has a large number of parameters in its linear layers, making fitting difficult and requiring a long training time. Furthermore, it requires real high spatial resolution hyperspectral images as labels, and its application in reality is limited due to the difficulty in obtaining real labels. As for unsupervised learning methods, since there are no real labels, image features are difficult to extract correctly, and therefore they cannot be directly applied to super-resolution reconstruction.

[0023] Based on this, the present invention provides a spectral image fusion super-resolution method, system, and electronic device that can alleviate the above-mentioned problems existing in related technologies.

[0024] To facilitate understanding of this embodiment, a spectral image fusion super-resolution method disclosed in this invention will first be described in detail. This method can be applied to a pre-trained image feature extraction network; the image feature extraction network may include a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; see [link to previous section]. Figure 1 As shown, the method may include the following steps:

[0025] Step S102: Spatial upsampling is performed on the initial low-resolution hyperspectral image to obtain the first high-resolution hyperspectral image.

[0026] Step S104: Spatial downsampling and spatial upsampling are performed sequentially on the initial high-resolution RGB image to obtain the first high-resolution RGB image; wherein the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image.

[0027] Both the spatial downsampling and spatial upsampling methods described above can be selected according to actual needs, and there are no restrictions on them.

[0028] Step S106: Multi-level feature extraction is performed on the initial high-resolution RGB image and the first high-resolution RGB image respectively through the first deep convolutional neural network to obtain the first feature vector of each level of the initial high-resolution RGB image and the second feature vector of each level of the first high-resolution RGB image.

[0029] Step S108: Multi-level feature extraction is performed on the first high-resolution hyperspectral image using a second deep convolutional neural network to obtain the third feature vector corresponding to each level of the first high-resolution hyperspectral image; wherein, each level of feature extraction by the second deep convolutional neural network corresponds one-to-one with each level of feature extraction by the first deep convolutional neural network.

[0030] Step S110: Fuse all first feature vectors, all second feature vectors, all third feature vectors and the first high-resolution hyperspectral image to obtain the second high-resolution hyperspectral image.

[0031] This invention provides a super-resolution fusion method for spectral images. The method involves spatially upsampling an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image; then spatially downsampling and upsampling an initial high-resolution RGB image to obtain another high-resolution RGB image; finally, a first deep convolutional neural network extracts multi-level features from both the initial high-resolution RGB image and the first high-resolution RGB image, obtaining a first feature vector corresponding to each level in the initial high-resolution RGB image and a second feature vector corresponding to each level in the first high-resolution RGB image; a second deep convolutional neural network extracts multi-level features from the first high-resolution hyperspectral image, obtaining a third feature vector corresponding to each level in the first high-resolution hyperspectral image; and finally, all first, second, and third feature vectors are fused with the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image. By using this technique, a pre-trained convolutional neural network replaces the linear layer in the attention mechanism, allowing the attention operation to extract different levels of feature information from the image. This enables super-resolution reconstruction of low-resolution hyperspectral images using existing high-resolution RGB images, thereby obtaining high-resolution hyperspectral images.

[0032] As one possible implementation, the aforementioned first deep convolutional neural network may include a first first-order feature extraction module, a first second-order feature extraction module, and a first third-order feature extraction module connected in sequence; the aforementioned step S106 (i.e., performing multi-order feature extraction on the initial high-resolution RGB image and the first high-resolution RGB image respectively through the first deep convolutional neural network to obtain the first feature vector corresponding to each order of the initial high-resolution RGB image and the second feature vector corresponding to each order of the first high-resolution RGB image) may include:

[0033] (11) Input the initial high-resolution RGB image into the first deep convolutional neural network, and output the first first-order first feature vector, the second-order first feature vector and the third-order first feature vector through the first first-order feature extraction module, the first second-order feature extraction module and the first third-order feature extraction module respectively.

[0034] For example, see Figure 2 As shown, Conv represents 2D convolution, ReLU represents the activation function; PixelShuffle represents subpixel convolution, which is a learnable upsampling operation; a high-resolution RGB image Y can be used as the initial high-resolution RGB image input to a pre-trained first deep convolutional neural network (i.e., Figure 2 In 'a', the pre-trained CNN1 is used as the first and second layers of CNN1 as the first first-order feature extraction module to output the first first-order feature vector V3. The third to fifth layers of CNN1 are used as the first second-order feature extraction module to output the second-order first feature vector V2. The sixth to eighth layers of CNN1 are used as the first third-order feature extraction module to output the third-order first feature vector V1. Features of different orders represent different information. For example, low-order information is often biased towards local texture features, while high-order features are more biased towards the features of a single target in the global context.

[0035] (12) Input the first high-resolution RGB image into the first deep convolutional neural network, and output the first-order second feature vector, the second-order second feature vector and the third-order second feature vector respectively through the first first-order feature extraction module, the first second-order feature extraction module and the first third-order feature extraction module.

[0036] For example, continuing from the previous example Figure 2 A high-resolution RGB image Y can be obtained by first upsampling the interpolation space and then downsampling the interpolation space to obtain a coarse-resolution RGB image (i.e., the first high-resolution RGB image). C Then the first high-resolution RGB image Y C Input to a pre-trained first deep convolutional neural network (i.e. Figure 2The pre-trained CNN1 in b first passes through the first and second layers of CNN1 to output the first-order second feature vector K3, then passes through the third to fifth layers of CNN1 to output the second-order second feature vector K2, and then passes through the sixth to eighth layers of CNN1 to output the third-order second feature vector K1.

[0037] As one possible implementation, the second deep convolutional neural network may include a second first-order feature extraction module, a second second-order feature extraction module, and a second third-order feature extraction module connected in sequence. The structure of the second first-order feature extraction module is different from that of the first first-order feature extraction module, the structure of the second second-order feature extraction module is the same as that of the first second-order feature extraction module, and the structure of the second third-order feature extraction module is the same as that of the first third-order feature extraction module. The above step S108 (i.e., performing multi-order feature extraction on the first high-resolution hyperspectral image through the second deep convolutional neural network to obtain the third feature vector corresponding to each order of the first high-resolution hyperspectral image) may include: inputting the first high-resolution hyperspectral image into the second deep convolutional neural network, and outputting the first-order third feature vector, the second-order third feature vector, and the third-order third feature vector through the second first-order feature extraction module, the second second-order feature extraction module, and the second third-order feature extraction module, respectively.

[0038] For example, continuing from the previous example Figure 2 It can X-ray a low-resolution hyperspectral image L As the initial low-resolution hyperspectral image, and based on this initial low-resolution hyperspectral image X L Interpolation space upsampling produces a coarse high-resolution hyperspectral image (i.e., the first high-resolution hyperspectral image) with the same spatial resolution as the initial high-resolution RGB. C Then the first high-resolution hyperspectral image X C Input to a pre-trained second deep convolutional neural network (i.e. Figure 2 The pre-trained CNN2 in c is different from the first layer of the pre-trained CNN1. The first layer of the pre-trained CNN2 supports the number of spectral bands as input channels. The second to eighth layers of the pre-trained CNN2 are the same as the second to eighth layers of the pre-trained CNN1. The first and second layers of CNN2 are used as the second first-order feature extraction module to output the first-order third feature vector Q3. The third to fifth layers of CNN2 are used as the second second-order feature extraction module to output the second-order third feature vector Q2. The sixth to eighth layers of CNN2 are used as the second third-order feature extraction module to output the third-order third feature vector Q1.

[0039] As one possible implementation, the image feature extraction network described above may further include a pre-trained fusion module; step S110 (i.e., fusing all first feature vectors, all second feature vectors, all third feature vectors, and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image) may include:

[0040] (21) Perform corresponding attention feature encoding on the first feature vector, second feature vector and third feature vector of each order respectively to obtain the first feature map of the first order, the first feature map of the second order and the first feature map of the third order.

[0041] (22) The first feature map of the second order and the third feature vector of the second order are concatenated to form the second feature map of the second order, and the first feature map of the third order and the third feature vector of the third order are concatenated to form the second feature map of the third order.

[0042] (23) The feature residuals of the first-order first feature map, the second-order second feature map and the third-order second feature map are calculated by the fusion module to obtain the corresponding feature residuals.

[0043] The above-mentioned fusion module can adopt Figure 2 The U-shaped network shown in c or other residual networks (such as ResNet) can be selected according to actual needs, and there are no restrictions on this.

[0044] (24) The obtained feature residuals are superimposed on the first high-resolution hyperspectral image to obtain the second high-resolution hyperspectral image.

[0045] For example, continuing from the previous example Figure 2 After obtaining V1, V2, V3, K1, K2, and K3 through the pre-trained CNN1 output and Q1, Q2, and Q3 through the pre-trained CNN2 output, attention feature encoding is performed on the first, second, and third feature vectors of each order using the following formula: Among them, Attention(Q i ,K i V i ) represents the first feature map of order i, d k Q represents the number of pixels. i Let K represent the third eigenvector of order i. i V represents the second eigenvector of order i. iLet $i$ represent the first feature vector of order $i$, where $i = 1, 2, 3$. This yields three sets of feature maps of different orders (i.e., the first feature map A3 of order $i$, the first feature map A2 of order $i$, and the first feature map A1 of order $i$). The spatial dimensions of $A3$, $A2$, and $A1$ decrease sequentially, while the channel dimensions of $A3$, $A2$, and $A1$ increase sequentially. Then, the second-order first feature map A2 and the third-order third feature vector $Q2$ are concatenated to form the second-order second feature map B2. Similarly, the third-order first feature map A1 and the third-order third feature vector $Q1$ are concatenated to form the second-order second feature map B1. Next, subpixel convolutional upsampling is performed on B2 to obtain C2, which has the same spatial dimension as A3. Subpixel convolutional upsampling is also performed on B1 to obtain C1, which has the same spatial dimension as A3. Finally, A3, C2, and C1 are input into a fusion module (e.g., ...). Figure 2 The U-shaped network with skip connections in c is used to calculate the feature residuals corresponding to A3, C2, and C1 through a fusion module; all the obtained feature residuals are then compared with the first high-resolution hyperspectral image X. C The images are superimposed to obtain the second high-resolution hyperspectral image X.

[0046] As one possible implementation, training the image feature extraction network described above may include the following steps:

[0047] (31) Input the pre-acquired second high-resolution RGB image and second low-resolution hyperspectral image into the initial image feature extraction network to be trained, and output the third high-resolution hyperspectral image through the output of the initial image feature extraction network.

[0048] The structure of the initial image feature extraction network is the same as that of the image feature extraction network.

[0049] (32) Perform spectral degradation on the third high-resolution hyperspectral image to obtain the third high-resolution RGB image.

[0050] For example, the spectral degradation operation on the third high-resolution hyperspectral image can be performed using the following formula: Y″=SX′, where Y″ represents the third high-resolution RGB image, X′ represents the third high-resolution hyperspectral image, and S represents the spectral response function (i.e., the spectral degradation operation).

[0051] (33) Spatial degradation is performed on the third high-resolution hyperspectral image to obtain the third low-resolution hyperspectral image.

[0052] For example, continuing from the previous example, the spatial degradation operation can be performed on the third high-resolution hyperspectral image using the following formula: X L "=X′H, where X L "" indicates the third low-resolution hyperspectral image, and H indicates the spatial degradation operation.

[0053] (34) Based on the second high-resolution RGB image, the second low-resolution hyperspectral image, the third high-resolution hyperspectral image, the third high-resolution RGB image and the third low-resolution hyperspectral image, construct the loss function of the initial image feature extraction network.

[0054] For example, continuing from the previous example, given the second high-resolution RGB image Y′ and the second low-resolution hyperspectral image X′... L The third high-resolution hyperspectral image X′, the third high-resolution RGB image Y″, and the third low-resolution hyperspectral image X L Afterwards, the loss function of the initial image feature extraction network can be established using the following formula: Loss = |Y″ - Y′| + |X L "-X" L ′|, where Loss is the loss function of the initial image feature extraction network.

[0055] (35) Based on the loss function, the initial image feature extraction network is iteratively trained by the backpropagation algorithm, and the image feature extraction network is obtained after training.

[0056] For example, the loss function of the initial image feature extraction network can be used to iteratively train and update the trainable layers of the initial image feature extraction network through the backpropagation algorithm until the loss function is stable or the training ends after a preset number of iterations, and finally the above-mentioned image feature extraction network is obtained.

[0057] To facilitate understanding, the workflow of the above image feature extraction network is described below using a specific application example:

[0058] See Figure 2 As shown, the high-resolution RGB image Y has dimensions of 512×512×3, corresponding to a low-resolution hyperspectral image X with a wavelength range of 400nm to 700nm and a spectral interval of 10nm. L The dimensions are 16×16×31, and the Y and X dimensions need to be fused using the aforementioned spectral image fusion super-resolution method. L To achieve a 32x super-resolution of low-resolution hyperspectral images, a high-resolution hyperspectral image X of 512×512×31 is obtained. Taking a pre-trained VGG16 as an example, the outputs of layers 6, 15, and 21 of VGG16 can be used as the output layers of multi-order feature vectors to build a pre-trained CNN1. Then, the first layer of the pre-trained CNN1 is replaced with a convolutional layer that supports the number of spectral bands as input channels to build a pre-trained CNN2. That is, the first layer of the pre-trained CNN2 supports the number of spectral bands as input channels. Figure 2 In c, a U-shaped network with skip connections is used as the fusion module.

[0059] Based on this, the above-mentioned super-resolution method for fusion of spectral images can be performed according to the following steps:

[0060] The first step is to use linear interpolation to interpolate the low-resolution hyperspectral image X. L Upsampling was performed to obtain a coarse, high-resolution hyperspectral image X with a resolution of 512×512. c :X c =Bicubic(X) L (32), where Bicubic(·,f) represents bicubic interpolation, and f represents the scaling factor; then, the high-resolution RGB image Y is first downsampled and then upsampled to obtain a coarse high-resolution RGB image Y. c :

[0061] The second step involves using the pre-trained CNN1 to calculate V3, V2, and V1 corresponding to the output Y; using the pre-trained CNN1 to calculate the output Y... C The corresponding K3, K2, and K1.

[0062] The third step is to calculate the output X using the pre-trained CNN1. c The corresponding Q3, Q2, and Q1.

[0063] Fourth step: Based on V3, V2, V1, K3, K2, K1, Q3, Q2, and Q1, attention feature encoding is performed using the following formula to obtain three sets of feature maps A3, A2, and A1 of different orders:

[0064] Fifth step, concatenate A2 and Q2 to form the second feature map B2 of the second order, and concatenate A1 and Q1 to form B1; then, perform subpixel convolution on B2 and B1 respectively to upsample to C2 and C1 with the same spatial dimension as A3.

[0065] Step 6: Combine A3, C2, and C1 into a multi-order feature vector group M, and feed M into the fusion module to calculate the feature residuals. Then, compare the calculated feature residuals with X. c The high-resolution hyperspectral image X is obtained by superimposing the images.

[0066] The specific calculation process can be performed according to the following formula: X = f Θ (M)+X c , where f Θ (·) represents a parameterized fusion module, and Θ represents the parameters of the fusion module (specifically, the weights of the fusion module).

[0067] The training process for the aforementioned pre-trained CNN1, pre-trained CNN2, and fusion module may include:

[0068] Step 1: First, acquire a high-resolution RGB image Y' with dimensions of 512×512×3 and a low-resolution hyperspectral image X with a wavelength range of 400nm~700nm, a spectral interval of 10nm, and dimensions of 16×16×31. L Then, following a similar procedure to steps one through six above, a high-resolution hyperspectral image X of 512×512×31 is calculated.

[0069] Step 2: Perform spectral degradation on X′ using the formula Y″ = SX′ to obtain Y″, and use the formula X′... L "=X′H performs a spatial degradation operation on X′ to obtain X. L ″.

[0070] Step 3, use the formula Loss = |Y″ - Y′| + |X L "-X" L Establish a loss function Loss, and iterate the training layers and update their parameters using the backpropagation algorithm based on Loss until the loss is stable or the preset number of iterations is reached to complete the training.

[0071] Specifically, the initial learning rate can be set to 8×10. -3 The algorithm uses an iteration cycle of 100 times, employs a learning rate decay strategy where the learning rate decreases to 0.99 every 100 iterations, uses the Adam optimizer for gradient descent, and has an iteration step size of 3000.

[0072] After reconstructing the high-resolution hyperspectral image X from the first to the sixth steps described above, X can be evaluated by calculating the average structural similarity (ASSIM), peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), and relative dimensionless global error (ERGAS) between X and existing real high-resolution hyperspectral images. Among these, ASSIM and PSNR can be used to evaluate the spatial reconstruction structure, while SAM and ERGAS can be used to evaluate the spectral reconstruction quality.

[0073] As shown in Table 1, the calculated ASSIM, PSNR, SAM and ERGAS are 0.999, 43.64, 0.085 and 1.35, respectively, indicating that the first to sixth steps above can complete the task of 32 times super-resolution of low-resolution hyperspectral images.

[0074] Table 1 Evaluation results of super-resolution hyperspectral images

[0075] Beer 0.999 43.64 0.085 1.35

[0076] See Figure 3 As shown, Figure 3The example image shows a comparison between the reconstructed spectrum (i.e., the spectrum in the reconstructed high-resolution hyperspectral image X) and the real spectrum (i.e., the spectrum in the existing real high-resolution hyperspectral image). It can be seen that the reconstructed spectrum and the real spectrum are basically consistent.

[0077] The aforementioned spectral image fusion super-resolution method overcomes the shortcomings of supervised training and proposes a feature attention mechanism that can be used for unsupervised training. This mechanism replaces the linear layer in the attention mechanism with a pre-trained convolutional neural network, thereby enabling the attention operation to extract feature information of different orders from the image. In addition, the aforementioned spectral image fusion super-resolution method does not rely on the real label, which can reduce training parameters and training time.

[0078] Based on the above-described spectral image fusion super-resolution method, this embodiment of the invention also provides a spectral image fusion super-resolution system, which can be applied to a pre-trained image feature extraction network; the image feature extraction network may include a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; see [link to previous document]. Figure 4 As shown, the system may include the following components:

[0079] The first processing subsystem 402 can be used to spatially upsample the initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image.

[0080] The second processing subsystem 404 can be used to sequentially perform spatial downsampling and spatial upsampling on the initial high-resolution RGB image to obtain a first high-resolution RGB image; wherein the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image.

[0081] The first feature extraction subsystem 406 can be used to perform multi-level feature extraction on the initial high-resolution RGB image and the first high-resolution RGB image respectively through the first deep convolutional neural network, to obtain the first feature vector of the initial high-resolution RGB image corresponding to each level and the second feature vector of the first high-resolution RGB image corresponding to each level.

[0082] The second feature extraction subsystem 408 can be used to perform multi-level feature extraction on the first high-resolution hyperspectral image through the second deep convolutional neural network to obtain the third feature vector of each level corresponding to the first high-resolution hyperspectral image; wherein, each level of feature extraction of the second deep convolutional neural network corresponds one-to-one with each level of feature extraction of the first deep convolutional neural network.

[0083] The fusion subsystem 410 can be used to fuse all first feature vectors, all second feature vectors, all third feature vectors and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image.

[0084] This invention provides a super-resolution fusion system for spectral images. The system performs spatial upsampling on an initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image. It then sequentially performs spatial downsampling and upsampling on an initial high-resolution RGB image to obtain another high-resolution RGB image. A first deep convolutional neural network extracts multi-level features from both the initial high-resolution RGB image and the first high-resolution RGB image, obtaining a first feature vector corresponding to each level in the initial high-resolution RGB image and a second feature vector corresponding to each level in the first high-resolution RGB image. A second deep convolutional neural network extracts multi-level features from the first high-resolution hyperspectral image, obtaining a third feature vector corresponding to each level in the first high-resolution hyperspectral image. Finally, all first, second, and third feature vectors are fused with the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image. By using this technique, a pre-trained convolutional neural network replaces the linear layer in the attention mechanism, allowing the attention operation to extract different levels of feature information from the image. This enables super-resolution reconstruction of low-resolution hyperspectral images using existing high-resolution RGB images, thereby obtaining a high-resolution hyperspectral image.

[0085] See Figure 4 As shown, the system may also include:

[0086] The training subsystem 412 can be used to input a pre-acquired second high-resolution RGB image and a second low-resolution hyperspectral image into the image feature extraction network, and output a third high-resolution hyperspectral image through the image feature extraction network; perform spectral degradation on the third high-resolution hyperspectral image to obtain a third high-resolution RGB image; perform spatial degradation on the third high-resolution hyperspectral image to obtain a third low-resolution hyperspectral image; construct a loss function for the image feature extraction network based on the second high-resolution RGB image, the second low-resolution hyperspectral image, the third high-resolution hyperspectral image, the third high-resolution RGB image, and the third low-resolution hyperspectral image; and iteratively train the image feature extraction network using a backpropagation algorithm based on the loss function.

[0087] The aforementioned first deep convolutional neural network may include a first first-order feature extraction module, a first second-order feature extraction module, and a first third-order feature extraction module connected in sequence; the aforementioned first feature extraction subsystem 406 may also be used to: input the initial high-resolution RGB image into the first deep convolutional neural network, and output a first-order first feature vector, a second-order first feature vector, and a third-order first feature vector through the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module, respectively; input the first high-resolution RGB image into the first deep convolutional neural network, and output a first-order second feature vector, a second-order second feature vector, and a third-order second feature vector through the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module, respectively.

[0088] The aforementioned second deep convolutional neural network may include a second first-order feature extraction module, a second second-order feature extraction module, and a second third-order feature extraction module connected in sequence. The structure of the second first-order feature extraction module is different from that of the first first-order feature extraction module, the structure of the second second-order feature extraction module is the same as that of the first second-order feature extraction module, and the structure of the second third-order feature extraction module is the same as that of the first third-order feature extraction module. The aforementioned second feature extraction subsystem 408 may also be used to: input the first high-resolution hyperspectral image into the second deep convolutional neural network, and output a first-order third feature vector, a second-order third feature vector, and a third-order third feature vector through the second first-order feature extraction module, the second second-order feature extraction module, and the second third-order feature extraction module, respectively.

[0089] The image feature extraction network described above may further include a pre-trained fusion module; the fusion subsystem 410 may also be used to: perform corresponding attention feature encoding on the first feature vector, second feature vector, and third feature vector of each order respectively to obtain a first feature map of order one, a first feature map of order two, and a first feature map of order three; concatenate the first feature map of order two and the third feature vector of order two to form a second feature map of order two, and concatenate the first feature map of order three and the third feature vector of order three to form a second feature map of order three; perform feature residual calculation on the first feature map of order one, the second feature map of order two, and the second feature map of order three through the fusion module to obtain the corresponding feature residuals; and superimpose all the obtained feature residuals with the first high-resolution hyperspectral image to obtain the second high-resolution hyperspectral image.

[0090] The aforementioned fusion subsystem 410 can also be used to: encode attention features for each of the first, second, and third feature vectors of each order using the following formula: Among them, Attention(Q i,K i V i ) represents the first feature map of order i, d k Q represents the number of pixels. i Let K represent the third eigenvector of order i. i V represents the second eigenvector of order i. i Let i represent the first eigenvector of order i, where i = 1, 2, 3.

[0091] The spectral image fusion super-resolution system provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned spectral image fusion super-resolution method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0092] This invention also provides an electronic device, such as... Figure 5 The diagram shows the structure of the electronic device 100, which includes a processor 51 and a memory 50. The memory 50 stores computer-executable instructions that can be executed by the processor 51. The processor 51 executes the computer-executable instructions to implement the above-mentioned spectral image fusion super-resolution method.

[0093] exist Figure 5 In the illustrated embodiment, the electronic device further includes a bus 52 and a communication interface 53, wherein the processor 51, the communication interface 53, and the memory 50 are connected via the bus 52.

[0094] The memory 50 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 53 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 52 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0095] Processor 51 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 51 or by instructions in software form. Processor 51 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 51 reads the information in the memory and, in conjunction with its hardware, completes the steps of the spectral image fusion super-resolution method of the aforementioned embodiment.

[0096] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0097] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0099] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A super-resolution method for fusion of spectral images, characterized in that, The method is applied to a pre-trained image feature extraction network; the image feature extraction network includes a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; the method includes: Spatial upsampling is performed on the initial low-resolution hyperspectral image to obtain the first high-resolution hyperspectral image; The initial high-resolution RGB image is sequentially spatially downsampled and spatially upsampled to obtain a first high-resolution RGB image; wherein the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image; The first deep convolutional neural network is used to perform multi-level feature extraction on the initial high-resolution RGB image and the first high-resolution RGB image respectively, to obtain the first feature vector of the initial high-resolution RGB image corresponding to each level and the second feature vector of the first high-resolution RGB image corresponding to each level; The first high-resolution hyperspectral image is subjected to multi-level feature extraction by the second deep convolutional neural network to obtain the third feature vector corresponding to each level of the first high-resolution hyperspectral image; wherein, each level of feature extraction by the second deep convolutional neural network corresponds one-to-one with each level of feature extraction by the first deep convolutional neural network. The first feature vector, all second feature vectors, the third feature vector, and the first high-resolution hyperspectral image are fused to obtain the second high-resolution hyperspectral image; The first deep convolutional neural network includes a first first-order feature extraction module, a first second-order feature extraction module, and a first third-order feature extraction module connected in sequence. The step of performing multi-level feature extraction on the initial high-resolution RGB image and the first high-resolution RGB image respectively using the first deep convolutional neural network to obtain the first feature vector corresponding to each level of the initial high-resolution RGB image and the second feature vector corresponding to each level of the first high-resolution RGB image includes: The initial high-resolution RGB image is input into the first deep convolutional neural network, and the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module respectively output the first-order first feature vector, the second-order first feature vector, and the third-order first feature vector. The first high-resolution RGB image is input into the first deep convolutional neural network, and the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module respectively output the first-order second feature vector, the second-order second feature vector, and the third-order second feature vector. The training of the image feature extraction network includes: The pre-acquired second high-resolution RGB image and second low-resolution hyperspectral image are input into the initial image feature extraction network to be trained, and the third high-resolution hyperspectral image is obtained by the output of the initial image feature extraction network. The third high-resolution hyperspectral image is subjected to spectral degradation to obtain a third high-resolution RGB image; Spatial degradation is performed on the third high-resolution hyperspectral image to obtain a third low-resolution hyperspectral image; Based on the second high-resolution RGB image, the second low-resolution hyperspectral image, the third high-resolution hyperspectral image, the third high-resolution RGB image, and the third low-resolution hyperspectral image, the loss function of the initial image feature extraction network is constructed; Based on the loss function, the initial image feature extraction network is iteratively trained using the backpropagation algorithm, and the image feature extraction network is obtained after training.

2. The method according to claim 1, characterized in that, The second deep convolutional neural network includes a second first-order feature extraction module, a second second-order feature extraction module, and a second third-order feature extraction module connected in sequence; the structure of the second first-order feature extraction module is different from the structure of the first first-order feature extraction module, the structure of the second second-order feature extraction module is the same as the structure of the first second-order feature extraction module, and the structure of the second third-order feature extraction module is the same as the structure of the first third-order feature extraction module. The step of extracting multi-order features from the first high-resolution hyperspectral image using the second deep convolutional neural network to obtain the third feature vector corresponding to each order of the first high-resolution hyperspectral image includes: The first high-resolution hyperspectral image is input into the second deep convolutional neural network, and the second first-order feature extraction module, the second second-order feature extraction module, and the second third-order feature extraction module respectively output the first-order third feature vector, the second-order third feature vector, and the third-order third feature vector.

3. The method according to claim 2, characterized in that, The image feature extraction network also includes a pre-trained fusion module; The steps of fusing all first feature vectors, all second feature vectors, all third feature vectors, and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image include: The first, second, and third feature vectors of each order are respectively subjected to attention feature encoding to obtain the first feature map of the first order, the first feature map of the second order, and the first feature map of the third order. The first feature map of the second order and the third feature vector of the second order are concatenated to form the second feature map of the second order, and the first feature map of the third order and the third feature vector of the third order are concatenated to form the second feature map of the third order. The fusion module calculates the feature residuals of the first-order first feature map, the second-order second feature map, and the third-order second feature map to obtain the corresponding feature residuals. The obtained feature residuals are superimposed on the first high-resolution hyperspectral image to obtain the second high-resolution hyperspectral image.

4. The method according to claim 3, characterized in that, The steps for performing attention feature encoding on the first, second, and third feature vectors of each order to obtain the first-order, second-order, and third-order first feature maps include: Attention feature encoding is performed on the first, second, and third feature vectors of each order using the following formula: in, Represents the first feature map of order i. Indicates the number of pixels. Let i represent the third eigenvector of order i. This represents the second eigenvector of order i. Let i represent the first eigenvector of order i, where i = 1, 2, 3.

5. A super-resolution system for fusion of spectral images, characterized in that, The system is applied to a pre-trained image feature extraction network; the image feature extraction network includes a pre-trained first deep convolutional neural network and a pre-trained second deep convolutional neural network; the system includes: The first processing subsystem is used to spatially upsample the initial low-resolution hyperspectral image to obtain a first high-resolution hyperspectral image. The second processing subsystem is used to sequentially perform spatial downsampling and spatial upsampling on the initial high-resolution RGB image to obtain a first high-resolution RGB image; wherein, the spatial resolution of the initial high-resolution RGB image is the same as the spatial resolution of the first high-resolution hyperspectral image; The first feature extraction subsystem is used to perform multi-level feature extraction on the initial high-resolution RGB image and the first high-resolution RGB image respectively through the first deep convolutional neural network, to obtain the first feature vector of the initial high-resolution RGB image corresponding to each level and the second feature vector of the first high-resolution RGB image corresponding to each level. The second feature extraction subsystem is used to perform multi-order feature extraction on the first high-resolution hyperspectral image through the second deep convolutional neural network to obtain the third feature vector of the first high-resolution hyperspectral image corresponding to each order; wherein, each order of feature extraction of the second deep convolutional neural network corresponds one-to-one with each order of feature extraction of the first deep convolutional neural network. The fusion subsystem is used to fuse all first feature vectors, all second feature vectors, third feature vectors and the first high-resolution hyperspectral image to obtain a second high-resolution hyperspectral image. The first deep convolutional neural network includes a first first-order feature extraction module, a first second-order feature extraction module, and a first third-order feature extraction module connected in sequence. The first feature extraction subsystem is further configured to: input the initial high-resolution RGB image into the first deep convolutional neural network, and output a first-order first feature vector, a second-order first feature vector, and a third-order first feature vector through the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module, respectively; input the first high-resolution RGB image into the first deep convolutional neural network, and output a first-order second feature vector, a second-order second feature vector, and a third-order second feature vector through the first first-order feature extraction module, the first second-order feature extraction module, and the first third-order feature extraction module, respectively. The system also includes: A training subsystem is used to input a pre-acquired second high-resolution RGB image and a second low-resolution hyperspectral image into an initial image feature extraction network to be trained, and to output a third high-resolution hyperspectral image through the initial image feature extraction network; to perform spectral degradation on the third high-resolution hyperspectral image to obtain a third high-resolution RGB image; to perform spatial degradation on the third high-resolution hyperspectral image to obtain a third low-resolution hyperspectral image; to construct a loss function for the initial image feature extraction network based on the second high-resolution RGB image, the second low-resolution hyperspectral image, the third high-resolution hyperspectral image, the third high-resolution RGB image, and the third low-resolution hyperspectral image; and to iteratively train the initial image feature extraction network using a backpropagation algorithm based on the loss function, thereby obtaining the image feature extraction network after training.

6. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 4.

Images