A four-branch neural network remote sensing image fusion method and computer readable medium

Abstract

The method of the present application proposes a four-branch neural network remote sensing image fusion method and a computer readable medium. The present application carries out spectral dimension downsampling processing on each original hyperspectral image to obtain a multispectral image with low spectral resolution; carries out fuzzy processing and downsampling processing on the spatial dimension of each original hyperspectral image to obtain a hyperspectral image with low spatial resolution; constructs a four-branch neural network model, combines each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution for iterative optimization to obtain a trained four-branch neural network model; real-time acquisition of hyperspectral images, combining the real-time acquired hyperspectral images with the trained four-branch neural network model for fusion to obtain each predicted real-time hyperspectral image. The present application can fully extract spatial features and spectral features from hyperspectral and multispectral images, and has great advantages in terms of computational complexity and spatial complexity.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing, and specifically relates to a four-branch neural network remote sensing image fusion method and a computer-readable medium. Background Technology

[0002] Multispectral images (MSI) typically encompass several bands within the visible and near-infrared wavelength ranges, such as RGB color images. Due to the trade-off between spectral and spatial resolution, MSI images generally possess lower spectral resolution and higher spatial resolution. In contrast, hyperspectral images (HSI) typically contain 100 to 200 spectral bands, exhibiting higher spectral resolution and lower spatial resolution. Therefore, fusing these two types of images to obtain a dual-high resolution image (high spectral and spatial resolution) has significant practical implications.

[0003] Currently, methods for fusing hyperspectral and multispectral images can be broadly categorized into two types: model-based traditional methods and network-based deep learning methods. Since the fusion problem can be viewed as an inverse underdetermined problem, model-based traditional methods typically make prior assumptions about the hyperspectral and multispectral images as regularization terms. Common priors include sparse priors, Gaussian priors, and low-rank priors. Although this method has a solid mathematical foundation, the linear observation models it employs struggle to adequately describe the nonlinear relationships between data, thus affecting the fusion results. Benefiting from powerful feature extraction capabilities, deep learning (DL) methods can fully describe the nonlinear relationships between data. Common network structures include convolutional neural networks (CNNs), residual networks, and 3D CNNs. The convolutional kernels in CNNs typically can only move in the spatial direction, thus failing to fully extract the spatial-spectral information of remote sensing images. 3D CNNs can solve this problem; their kernels can slide in the spatial and spectral dimensions, enabling the extraction of spatial-spectral information. However, the output of 3D CNNs leads to huge storage requirements and computational complexity, placing significant performance pressure on computers. In conclusion, it is of great significance to design a feature extraction module that can fully extract spatial spectral features while having low computational and spatial complexity. Summary of the Invention

[0004] To address the above problems, this invention proposes a four-branch neural network remote sensing image fusion method and a computer-readable medium.

[0005] Traditional two-dimensional convolutional neural networks focus excessively on spatial information while neglecting spectral information. Three-dimensional convolutional neural networks solve this problem by using convolutional kernels that can slide in three dimensions, but their high computational and spatial complexity places enormous performance pressure on computers.

[0006] The technical solution of this invention is a four-branch neural network remote sensing image fusion method, characterized in that:

[0007] Each original hyperspectral image is downsampled in the spectral dimension to obtain a low-spectral-resolution multispectral image.

[0008] Each original hyperspectral image is subjected to spatial dimension blurring and downsampling to obtain a low spatial resolution hyperspectral image.

[0009] A four-branch neural network model is constructed by cascading a feature extraction module, a feature fusion module, and an image restoration module. The model is then iteratively optimized using the ADAM optimizer by combining each low-spectral-resolution multispectral image and each low-spatial-resolution hyperspectral image to obtain the trained four-branch neural network model.

[0010] Each low-spectral-resolution multispectral image and low-spatial-resolution hyperspectral image is acquired in real time and fused using a trained four-branch neural network model to obtain a predicted real-time hyperspectral image.

[0011] The specific steps of the method of the present invention are as follows:

[0012] Step 1: Import multiple original hyperspectral images, and perform spectral dimension downsampling processing on each original hyperspectral image to obtain a low-spectral-resolution multispectral image for each image.

[0013] Step 2: Perform spatial blurring and downsampling on each original hyperspectral image to obtain each low spatial resolution hyperspectral image;

[0014] Step 3: Construct a four-branch neural network model. Input each low-spectral-resolution multispectral image and each low-spatial-resolution hyperspectral image into the four-branch neural network model for fusion to obtain each predicted hyperspectral image. Combine each original hyperspectral image to construct a network loss function. Iteratively optimize the model using the ADAM optimizer to obtain the trained four-branch neural network model.

[0015] Step 4: Acquire each low-spectral-resolution multispectral image and low-spatial-resolution hyperspectral image in real time, and fuse them through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image.

[0016] Preferably, the four-branch neural network model described in step 3 includes:

[0017] Feature extraction module, feature fusion module, and image restoration module;

[0018] The feature extraction module, feature fusion module, and image restoration module are cascaded in sequence.

[0019] The feature extraction module is composed of a first feature extraction submodule, a second feature extraction submodule, a third feature extraction submodule, and a fourth feature extraction submodule connected in parallel;

[0020] The feature fusion module includes a first feature fusion submodule, a second feature subfusion module, and a third feature fusion submodule;

[0021] The first feature extraction submodule, the second feature extraction submodule, and the first feature fusion submodule are connected;

[0022] The third feature extraction submodule, the fourth feature extraction submodule, and the second feature fusion submodule are connected;

[0023] The first feature fusion submodule, the second feature fusion submodule, and the third feature fusion submodule are connected;

[0024] The third feature fusion submodule is connected to the multi-scale module;

[0025] The image restoration module includes a multi-scale sub-module and a multi-level sub-module;

[0026] The multi-scale submodules and multi-level submodules are connected;

[0027] The first feature extraction submodule takes each low-spectral-resolution multispectral image as input and performs feature extraction on each low-spectral-resolution multispectral image to obtain the spatial features of each low-spectral-resolution multispectral image.

[0028] The second feature extraction submodule takes each low-spectral-resolution multispectral image as input, performs feature extraction on each low-spectral-resolution multispectral image, and obtains the spectral features of each low-spectral-resolution multispectral image.

[0029] The third feature extraction submodule takes each low spatial resolution hyperspectral image as input and performs feature extraction on each low spatial resolution hyperspectral image to obtain the spatial features of each low spatial resolution hyperspectral image.

[0030] The fourth feature extraction submodule takes each low spatial resolution hyperspectral image as input and performs feature extraction on each low spatial resolution hyperspectral image to obtain the spectral features of each low spatial resolution hyperspectral image.

[0031] The t-th feature extraction submodule includes the (t-1)*4+1-th multispectral image spatial feature extraction module, the (t-1)*4+2-th multispectral image spatial feature extraction module, the (t-1)*4+3-th multispectral image spatial feature extraction module, and the (t-1)*4+4-th multispectral image spatial feature extraction module, which are cascaded together.

[0032] The (t-1)*4+1 multispectral image spatial feature extraction module, the (t-1)*4+2 multispectral image spatial feature extraction module, the (t-1)*4+3 multispectral image spatial feature extraction module, and the (t-1)*4+4 multispectral image spatial feature extraction module each contain a depthwise separable convolutional layer and a PReLU activation layer cascaded in sequence.

[0033] The (t-1)*4+1 multispectral image spatial feature extraction module takes each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+1 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+2 multispectral image spatial feature extraction module takes the (t-1)*4+1 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+2 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+3 multispectral image spatial feature extraction module takes the (t-1)*4+2 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+3 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+4 multispectral image spatial feature extraction module takes the (t-1)*4+3 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction.

[0034] If t=1, the spatial features of each low-spectral-resolution multispectral image are obtained;

[0035] If t=2, the spectral characteristics of each low-spectral-resolution multispectral image are obtained;

[0036] If t=3, the spatial features of each low spatial resolution hyperspectral image are obtained;

[0037] If t=4, the spectral characteristics of each low spatial resolution hyperspectral image are obtained;

[0038] The spatial features and spectral features of each low-spectral-resolution multispectral image are output to the first feature fusion submodule.

[0039] The spatial features and spectral features of each low spatial resolution hyperspectral image are output to the second feature fusion submodule for connection.

[0040] The first feature fusion submodule takes the spatial features and spectral features of each low-spectral-resolution multispectral image as input, performs feature fusion, obtains the spatial-spectral features of each low-spectral-resolution multispectral image, and outputs them to the third feature fusion submodule.

[0041] The second feature fusion submodule takes the spatial features and spectral features of each low spatial resolution hyperspectral image as input, performs feature fusion, obtains the spatial-spectral features of each low spatial resolution hyperspectral image, and outputs them to the third feature fusion submodule.

[0042] The third feature fusion submodule takes the spatial spectral features of each low-spectral-resolution multispectral image and the spatial spectral features of each low-spatial-resolution hyperspectral image as input, performs feature fusion, obtains the spatial spectral features of each predicted image, and outputs them to the multi-scale submodule.

[0043] The first feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer;

[0044] The second feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer;

[0045] The third feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer.

[0046] The multi-scale submodule takes the spatial-spectral features of each predicted image as input, performs feature extraction, and obtains the multi-scale spatial-spectral features of each predicted hyperspectral image.

[0047] The multi-scale submodule includes a first hole feature extraction module, a second hole feature extraction module, a third hole feature extraction module, a fourth hole feature extraction module, a channel connection module, and a 1×1 convolutional layer, all connected in parallel.

[0048] The first, second, third, and fourth hole feature extraction modules each take the spatial spectral features of each predicted image as input and perform feature extraction to obtain the first, second, third, and fourth multi-scale spatial spectral features of each predicted hyperspectral image, respectively.

[0049] The first, second, third, and fourth hole feature extraction modules each contain a cascaded dilated convolutional layer and a PReLU activation layer.

[0050] The third feature fusion submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module, respectively.

[0051] The multi-scale channel connection module in the multi-scale submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module;

[0052] The multi-scale 1×1 convolutional layer in the multi-scale submodule is connected to the channel connection module in the multi-scale submodule;

[0053] The multi-level sub-modules take the multi-scale spatial-spectral features of each predicted image as input, perform feature extraction, and obtain each predicted hyperspectral image.

[0054] The multi-level sub-module includes cascaded multi-level 1×1 convolutional layers and multi-level channel connection modules;

[0055] The network loss function model described in step 3 is defined as follows:

[0056]

[0057] Where K represents the number of images, H represents the width of the image, W represents the height of the image and the band index, S represents the number of bands, and G represents the number of bands. k (i,j,s) represents the pixel value in the i-th row and j-th column of the s-th band in the k-th original hyperspectral image. This represents the pixel value in the i-th row and j-th column of the s-th band in the k-th predicted hyperspectral image;

[0058] The present invention also provides a computer-readable medium storing a computer program executed by an electronic device, wherein when the computer program is run on the electronic device, it executes the steps of the four-part neural network model remote sensing image fusion method.

[0059] This invention provides a remote sensing image fusion method using a four-part neural network model, proposing a two-dimensional spatial feature extraction submodule and a two-dimensional spectral feature extraction submodule, which can fully extract spatial and spectral features from hyperspectral and multispectral images; moreover, this method can extract spectral features more fully and shows great advantages in terms of computational and spatial complexity. Attached Figure Description

[0060] Figure 1 : Flowchart of the method according to an embodiment of the present invention.

[0061] Figure 2 : A schematic diagram of the method for obtaining training data according to an embodiment of the present invention.

[0062] Figure 3 Comparison diagram of the method of this invention with other remote sensing image fusion methods. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0065] Since the spatial and spectral resolutions of remote sensing images are mutually constrained, fusing low-spectral-resolution images with low-spatial-resolution hyperspectral images to obtain high-spatial-resolution hyperspectral images has significant practical implications. This embodiment demonstrates the training of a four-branch neural network using Python 3.7.1, PyTorch 1.10.2, Windows 10, a 64GB CPU, and a GeForce GTX 3090Ti GPU. The learning rate for the four-branch neural network is 0.0001.

[0066] The following is combined with Figure 1-3 The technical solution of this invention is a four-branch network remote sensing image fusion method, as detailed below:

[0067] like Figure 1 The diagram shown is a flowchart of the method of the present invention.

[0068] Step 1: Import multiple original hyperspectral images, and perform spectral dimension downsampling processing on each original hyperspectral image to obtain a low-spectral-resolution multispectral image for each image.

[0069] The low-spectral-resolution multispectral image described in step 1 uses 25 randomly selected images from the CAVE dataset as the training set. During each training iteration, one image is randomly selected from these 25 images, and a random image patch of size 128×128×32 is chosen from this image as the original hyperspectral image. This random selection is repeated 10,000 times. In summary, a low-spectral-resolution multispectral image in the training set can be represented as:

[0070] M = G * T

[0071] Where M represents a matrix-form multispectral image with dimensions HW×s, where H=128, W=128 and s=3 represent the width, height and number of bands of the multispectral image, respectively; G represents a matrix-form original hyperspectral image with dimensions HW×S, where S=32 represents the number of bands of the dual hyperspectral image; T represents a spectral response function with dimensions S×s, which, when multiplied by matrix T, allows for downsampling of the image in the spectral bands.

[0072] Step 2: Perform spatial blurring and downsampling on each original hyperspectral image to obtain each low spatial resolution hyperspectral image;

[0073] A low spatial resolution hyperspectral image described in step 2 can be represented as:

[0074] H = (B5(G))↓8

[0075] Where H represents a hyperspectral image in matrix form with dimensions hw×S, where h=32 and w=32 represent the width and height of the hyperspectral image, respectively; B5(.) represents a standard Gaussian blur function with a kernel size of 5×5; (.)↓8 represents an 8-fold downsampling function.

[0076] Step 3: Construct a four-branch neural network model. Input each low-spectral-resolution multispectral image and each low-spatial-resolution hyperspectral image into the four-branch neural network model for fusion to obtain each predicted hyperspectral image. Combine each original hyperspectral image to construct a network loss function. Iteratively optimize the model using the ADAM optimizer to obtain the trained four-branch neural network model.

[0077] The four-branch neural network model described in step 3 includes:

[0078] Feature extraction module, feature fusion module, and image restoration module;

[0079] The feature extraction module, feature fusion module, and image restoration module are cascaded in sequence.

[0080] The feature extraction module is composed of a first feature extraction submodule, a second feature extraction submodule, a third feature extraction submodule, and a fourth feature extraction submodule connected in parallel;

[0081] The feature fusion module includes a first feature fusion submodule, a second feature subfusion module, and a third feature fusion submodule;

[0082] The first feature extraction submodule, the second feature extraction submodule, and the first feature fusion submodule are connected;

[0083] The third feature extraction submodule, the fourth feature extraction submodule, and the second feature fusion submodule are connected;

[0084] The first feature fusion submodule, the second feature fusion submodule, and the third feature fusion submodule are connected;

[0085] The third feature fusion submodule is connected to the multi-scale module;

[0086] The image restoration module includes a multi-scale sub-module and a multi-level sub-module;

[0087] The multi-scale submodules and multi-level submodules are connected;

[0088] The first feature extraction submodule takes each low-spectral-resolution multispectral image as input and performs feature extraction on each low-spectral-resolution multispectral image to obtain the spatial features of each low-spectral-resolution multispectral image.

[0089] The second feature extraction submodule takes each low-spectral-resolution multispectral image as input, performs feature extraction on each low-spectral-resolution multispectral image, and obtains the spectral features of each low-spectral-resolution multispectral image.

[0090] The third feature extraction submodule takes each low spatial resolution hyperspectral image as input and performs feature extraction on each low spatial resolution hyperspectral image to obtain the spatial features of each low spatial resolution hyperspectral image.

[0091] The fourth feature extraction submodule takes each low spatial resolution hyperspectral image as input and performs feature extraction on each low spatial resolution hyperspectral image to obtain the spectral features of each low spatial resolution hyperspectral image.

[0092] The t-th feature extraction submodule includes the (t-1)*4+1-th multispectral image spatial feature extraction module, the (t-1)*4+2-th multispectral image spatial feature extraction module, the (t-1)*4+3-th multispectral image spatial feature extraction module, and the (t-1)*4+4-th multispectral image spatial feature extraction module, which are cascaded together.

[0093] The (t-1)*4+1 multispectral image spatial feature extraction module, the (t-1)*4+2 multispectral image spatial feature extraction module, the (t-1)*4+3 multispectral image spatial feature extraction module, and the (t-1)*4+4 multispectral image spatial feature extraction module each contain a depthwise separable convolutional layer and a PReLU activation layer cascaded in sequence.

[0094] The (t-1)*4+1 multispectral image spatial feature extraction module takes each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+1 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+2 multispectral image spatial feature extraction module takes the (t-1)*4+1 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+2 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+3 multispectral image spatial feature extraction module takes the (t-1)*4+2 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the (t-1)*4+3 spatial feature of each low-spectral-resolution multispectral image; the (t-1)*4+4 multispectral image spatial feature extraction module takes the (t-1)*4+3 spatial feature of each low-spectral-resolution multispectral image as input and performs spatial feature extraction.

[0095] If t=1, the spatial features of each low-spectral-resolution multispectral image are obtained. The processing procedure of this branch for each low-spectral-resolution multispectral image can be expressed as:

[0096] Φ A (M ten )=SPA1(SPA2(SPA3(SPA4(M ten ))))

[0097] Where M ten A low-spectral-resolution multispectral image in tensor form, with dimensions of 128×128×3; SPA x1 (.) represents the x1-th multispectral image spatial feature extraction module (x1 = {1,2,3,4}), which includes a depthwise separable convolutional layer and a PReLU activation layer. The kernel size of the separable convolutional layer is 5, and the number of kernels is 64; Φ A (M ten ) represents the spatial features obtained after processing the low-spectral-resolution multispectral image by the first feature extraction submodule, with a size of 128×128×64;

[0098] If t = 2, the spectral characteristics of each low-spectral-resolution multispectral image are obtained. The processing procedure of this branch for each low-spectral-resolution multispectral image can be expressed as:

[0099] Φ A (M ten ) = SPE5(SPE6(SPE7(SPE8(M ten ))))

[0100] SPE x2 (.) represents the x2-th multispectral image spectral feature extraction module (x2 = {5, 6, 7, 8}), which includes one 1×1 convolutional layer and one PReLU activation layer, with a total of 64 convolutional kernels. Φ E (M ten ) represents the spectral features obtained after processing the low-spectral-resolution multispectral image by the second feature extraction submodule, with a size of 128×128×64;

[0101] If t=3, the spatial features of each low spatial resolution hyperspectral image are obtained. The processing procedure of this branch for each low spatial resolution hyperspectral image can be expressed as:

[0102] Φ A (H ten ) = SPA9 (SPA 10 (SPA 11 (SPA 12 (H ten ))))

[0103] Where H ten This represents a low spatial resolution hyperspectral image in tensor form, with dimensions of 128×128×3; SPA x3 (.) represents the x3rd hyperspectral image spatial feature extraction module (x3 = {9, 10, 11, 12}), which includes a depthwise separable convolutional layer and a PReLU activation layer. The kernel size of the separable convolutional layer is 5, and the number of kernels is 64; Φ A (H ten ) represents the spatial features obtained after processing a low spatial resolution hyperspectral image by the third feature extraction submodule, with a size of 128×128×64;

[0104] If t = 4, the spectral characteristics of each low spatial resolution hyperspectral image are obtained. The processing procedure of this branch for each low spatial resolution hyperspectral image can be expressed as:

[0105] Φ A (H ten ) = SPE 13 (SPE 14 (SPE 15(SPE 16 (H ten ))))

[0106] SPE x4 (.) represents the x4th hyperspectral image spectral feature extraction module (x4 = {13, 14, 15, 16}), which includes one 1×1 convolutional layer and one PReLU activation layer, with a total of 64 convolutional kernels. Φ E (H ten ) represents the spectral features obtained after each low spatial resolution hyperspectral image is processed by the Dessian feature extraction submodule, and its size is 128×128×64;

[0107] The spatial features and spectral features of each low-spectral-resolution multispectral image are output to the first feature fusion submodule.

[0108] The spatial features and spectral features of each low spatial resolution hyperspectral image are output to the second feature fusion submodule for connection.

[0109] The first feature fusion submodule takes the spatial features and spectral features of each low-spectral-resolution multispectral image as input, performs feature fusion to obtain the spatial-spectral features of each low-spectral-resolution multispectral image, and outputs them to the third feature fusion submodule; the fusion process can be represented as:

[0110] Φ(M ten )=ξ[Φ A (M ten )⊕Φ E (M ten )]

[0111] Where ⊕ represents the spectral dimension connection operation, ξ[.] represents a 1×1 convolution with 64 kernels; Φ(M ten ) represents the spatial spectral features of the low-spectral-resolution multispectral image obtained after fusion, with a size of 128×128×64;

[0112] The second feature fusion submodule takes the spatial features and spectral features of each low-spatial-resolution hyperspectral image as input, performs feature fusion to obtain the spatial-spectral features of each low-spatial-resolution hyperspectral image, and outputs them to the third feature fusion submodule; the fusion process can be represented as:

[0113] Φ(H ten )=ξ[Φ A (H ten )⊕Φ E (Hten )]

[0114] Where Φ(H) ten ) represents the spatial spectral features of the low spatial resolution hyperspectral image obtained after fusion, with a size of 128×128×64;

[0115] The third feature fusion submodule takes the spatial-spectral features of each low-spectral-resolution multispectral image and the spatial-spectral features of each low-spatial-resolution hyperspectral image as input, performs feature fusion to obtain the spatial-spectral features of each predicted image, and outputs them to the multi-scale submodule; this process can be represented as:

[0116] Φ(M ten H ten )=ξ[Φ(M ten )⊕Φ(H ten )]

[0117] Where Φ(M) ten H ten () represents the spatial spectral features of multispectral and hyperspectral data, with a size of 128×128×64;

[0118] The first feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer;

[0119] The second feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer;

[0120] The third feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer.

[0121] The multi-scale submodule takes the spatial-spectral features of each predicted image as input, performs feature extraction, and obtains the multi-scale spatial-spectral features of each predicted hyperspectral image.

[0122] The multi-scale submodule includes a first hole feature extraction module, a second hole feature extraction module, a third hole feature extraction module, a fourth hole feature extraction module, a channel connection module, and a 1×1 convolutional layer, all connected in parallel.

[0123] The first, second, third, and fourth hole feature extraction modules each take the spatial spectral features of each predicted image as input and perform feature extraction to obtain the first, second, third, and fourth multi-scale spatial spectral features of each predicted hyperspectral image, respectively.

[0124] The first, second, third, and fourth hole feature extraction modules each contain a cascaded dilated convolutional layer and a PReLU activation layer.

[0125] The third feature fusion submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module, respectively.

[0126] The multi-scale channel connection module in the multi-scale submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module;

[0127] The multi-scale 1×1 convolutional layer in the multi-scale submodule is connected to the channel connection module in the multi-scale submodule;

[0128] The purpose of the multi-scale submodule is to handle complex features with large size differences in remote sensing imagery and to train robust convolutional layers, Φ(M ten H ten The first step is to process the data using a multi-scale fusion strategy, which can be represented as:

[0129]

[0130] in This represents the multi-scale fusion processing operation, where r1=1, r2=2, r3=4, r4=6 represent the internal parameters of the multi-scale fusion processing operation, and G... ten-ms This represents the output of the multi-scale fusion processing, with dimensions of 128×128×64.

[0131] The multi-level submodules take the multi-scale spatial-spectral features of each predicted image as input to perform feature extraction, thereby obtaining each predicted hyperspectral image; in order to better recover spatial detail information, Φ(M ten H ten This will be processed by a multi-level fusion strategy, which can be represented as:

[0132] G ten-es =ξ(G ten-ms ⊕SPA2(SPA1(G ten ))⊕SPE6(SPE5(G ten ))

[0133] ⊕SPA9(SPA 10 (H ten ))⊕SPE 13 (SPE 14 (H ten ))

[0134] Among them G ten-es This is the final recovered vector-form predicted hyperspectral image, with a size of 128×128×32.

[0135] The multi-level sub-module includes cascaded multi-level 1×1 convolutional layers and multi-level channel connection modules;

[0136] Furthermore, the loss function model described in step 3 is defined as follows:

[0137]

[0138] Where K = 10000 represents the number of images, H = 128 represents the width of the image, W = 128 represents the height and band index of the image, S = 32 represents the number of bands, and G... k (i,j,s) represents the pixel value in the i-th row and j-th column of the s-th band in the k-th original hyperspectral image. This represents the pixel value in the i-th row and j-th column of the s-th band in the k-th predicted hyperspectral image;

[0139] Step 4: Acquire each low-spectral-resolution multispectral image and low-spatial-resolution hyperspectral image in real time, and fuse them through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image.

[0140] The test dataset in step 4 consists of the remaining 7 images selected in steps 1 and 2, each with a size of 512×512×32. Figure 2 The method described above can obtain each low spatial resolution hyperspectral image and each low spectral resolution multispectral image, with the former having a size of 128×128×32 and the latter having a size of 512×512×3. Then, each low spatial resolution hyperspectral image and each low spectral resolution multispectral image are input into the trained network to obtain the predicted hyperspectral image.

[0141] Appendix Figure 3The results of this invention's algorithm were compared with those of other remote sensing image fusion methods. For easier observation, magnified images of some regions are provided. In the magnified images, compared to the original hyperspectral image, the outputs of the 3D-CNN, HSR-Net, and LTTR methods are significantly blurred, indicating that spatial information was not fully recovered. Furthermore, the outputs of the CNMF, TF-Net, and LTTR methods show problems of excessive and insufficient brightness, respectively, compared to the original hyperspectral image, indicating that spectral information was not fully recovered. In contrast, the four-branch network proposed in this invention closely approximates the original hyperspectral image in terms of spatial detail and image brightness, demonstrating that this invention successfully achieves the fusion of low spatial resolution hyperspectral images and low spectral resolution multispectral images.

[0142] A specific embodiment of the present invention also provides a computer-readable medium.

[0143] The computer-readable medium is a server workstation;

[0144] The server workstation stores the computer program executed by the electronic device. When the computer program runs on the electronic device, it causes the electronic device to execute the steps of the four-branch network remote sensing image fusion method of the present invention.

[0145] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0146] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A four-branch neural network remote sensing image fusion method, characterized in that: Each original hyperspectral image is downsampled in the spectral dimension to obtain a low-spectral-resolution multispectral image. Each original hyperspectral image is subjected to spatial dimension blurring and downsampling to obtain a low spatial resolution hyperspectral image. A four-branch neural network model is constructed by cascading a feature extraction module, a feature fusion module, and an image restoration module. The trained four-branch neural network model is obtained by iteratively optimizing each low-spectral-resolution multispectral image and each low-spatial-resolution hyperspectral image using the ADAM optimizer. The four-branch neural network model includes a feature extraction module, a feature fusion module, and an image restoration module, all cascaded sequentially. The feature extraction module consists of four parallel feature extraction sub-modules: a first feature extraction sub-module, a second feature extraction sub-module, a third feature extraction sub-module, and a fourth feature extraction sub-module. The feature fusion module includes a first feature fusion sub-module, a second feature fusion sub-module, and a third feature fusion sub-module. The first, second, and first feature extraction sub-modules are connected. The third, fourth, and second feature extraction sub-modules are connected. The first, second, and third feature fusion sub-modules are connected. The third feature fusion sub-module is connected to a multi-scale module. The image restoration module includes a multi-scale sub-module and a multi-level sub-module. The multi-scale sub-module and the multi-level sub-module are connected. The first feature extraction submodule takes each low-spectral-resolution multispectral image as input and extracts features from each low-spectral-resolution multispectral image to obtain the spatial features of each low-spectral-resolution multispectral image. The second feature extraction submodule takes each low-spectral-resolution multispectral image as input and extracts features from each low-spectral-resolution multispectral image to obtain the spectral features of each low-spectral-resolution multispectral image. The third feature extraction submodule takes each low-spatial-resolution hyperspectral image as input and extracts features from each low-spatial-resolution hyperspectral image to obtain the spatial features of each low-spatial-resolution hyperspectral image. The fourth feature extraction submodule takes each low-spatial-resolution hyperspectral image as input and extracts features from each low-spatial-resolution hyperspectral image to obtain the spectral features of each low-spatial-resolution hyperspectral image. The first feature fusion submodule takes the spatial features and spectral features of each low-spectral-resolution multispectral image as input, performs feature fusion to obtain the spatial-spectral features of each low-spectral-resolution multispectral image, and outputs them to the third feature fusion submodule. The second feature fusion submodule takes the spatial features and spectral features of each low-spatial-resolution hyperspectral image as input, performs feature fusion to obtain the spatial-spectral features of each low-spatial-resolution hyperspectral image, and outputs them to the third feature fusion submodule. The third feature fusion submodule takes the spatial features and spectral features of each low-spatial-resolution hyperspectral image as input, performs feature fusion to obtain the spatial-spectral features of each low-spatial-resolution hyperspectral image, and outputs them to the third feature fusion submodule. The spatial spectral features of the multispectral image at high spatial resolution and the spatial spectral features of each low-spatial-resolution hyperspectral image are taken as input, and feature fusion is performed to obtain the spatial spectral features of each predicted image, which are then output to the multi-scale submodule. The first feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer. The second feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer. The third feature fusion submodule includes a channel connection module and a 1×1 convolutional layer, wherein the channel connection module is connected to the 1×1 convolutional layer. The multi-scale submodule takes the spatial-spectral features of each predicted image as input and performs feature extraction to obtain the multi-scale spatial-spectral features of each predicted hyperspectral image. The multi-scale submodule includes a first hole feature extraction module, a second hole feature extraction module, a third hole feature extraction module, a fourth hole feature extraction module, a channel connection module, and a 1×1 convolutional layer, all connected in parallel. The first, second, third, and fourth hole feature extraction modules take the spatial-spectral features of each predicted image as input and perform feature extraction to obtain the first, second, third, and fourth multi-scale spatial-spectral features of each predicted hyperspectral image, respectively. The feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module each contain a cascaded dilated convolutional layer and a PReLU activation layer; the third feature fusion submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module, respectively; the multi-scale channel connection module in the multi-scale submodule is connected to the first hole feature extraction module, the second hole feature extraction module, the third hole feature extraction module, and the fourth hole feature extraction module; the multi-scale 1×1 convolutional layer in the multi-scale submodule is connected to the channel connection module in the multi-scale submodule; the multi-level submodule uses the multi-scale spatial spectral features of each predicted image as input to perform feature extraction, thereby obtaining each predicted hyperspectral image; the multi-level submodule includes cascaded multi-level 1×1 convolutional layers and multi-level channel connection modules. Each low-spectral-resolution multispectral image and low-spatial-resolution hyperspectral image is acquired in real time and fused using a trained four-branch neural network model to obtain a predicted real-time hyperspectral image.

2. The four-branch neural network remote sensing image fusion method according to claim 1, characterized in that, Includes the following steps: Step 1: Import multiple original hyperspectral images, and perform spectral dimension downsampling processing on each original hyperspectral image to obtain a low-spectral-resolution multispectral image for each image. Step 2: Perform spatial blurring and downsampling on each original hyperspectral image to obtain each low spatial resolution hyperspectral image; Step 3: Construct a four-branch neural network model by cascading the feature extraction module, feature fusion module, and image restoration module. Input each low-spectral-resolution multispectral image and each low-spatial-resolution hyperspectral image into the four-branch neural network model for fusion to obtain each predicted hyperspectral image. Combine each original hyperspectral image to construct the network loss function. Iteratively optimize the model using the ADAM optimizer to obtain the trained four-branch neural network model. Step 4: Acquire each low-spectral-resolution multispectral image and low-spatial-resolution hyperspectral image in real time, and fuse them through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image.

3. The four-branch neural network remote sensing image fusion method according to claim 1, characterized in that: The t-th feature extraction submodule includes cascaded t-th features. Multispectral image spatial feature extraction module, the first Multispectral image spatial feature extraction module, the first Multispectral image spatial feature extraction module, the first Multispectral image spatial feature extraction module, t∈[1,4]; The first Multispectral image spatial feature extraction module, the first Multispectral image spatial feature extraction module, the first Multispectral image spatial feature extraction module, the first The multispectral image spatial feature extraction module contains a depthwise separable convolutional layer and a PReLU activation layer cascaded in sequence. The first The multispectral image spatial feature extraction module takes each low-spectral-resolution multispectral image as input and performs spatial feature extraction to obtain the first feature of each low-spectral-resolution multispectral image. Spatial features; the first The multispectral image spatial feature extraction module extracts features from each low-spectral-resolution multispectral image. Spatial features are used as input to extract spatial features, resulting in the first spatial feature of each low-spectral-resolution multispectral image. Spatial features; the first The multispectral image spatial feature extraction module extracts features from each low-spectral-resolution multispectral image. Spatial features are used as input to extract spatial features, resulting in the first spatial feature of each low-spectral-resolution multispectral image. Spatial features; the first The multispectral image spatial feature extraction module extracts features from each low-spectral-resolution multispectral image. Spatial features are used as input for spatial feature extraction: If t=1, the spatial features of each low-spectral-resolution multispectral image are obtained; If t=2, the spectral characteristics of each low-spectral-resolution multispectral image are obtained; If t=3, the spatial features of each low spatial resolution hyperspectral image are obtained; If t=4, the spectral characteristics of each low spatial resolution hyperspectral image are obtained; The spatial features and spectral features of each low-spectral-resolution multispectral image are output to the first feature fusion submodule. The spatial features and spectral features of each low spatial resolution hyperspectral image are output to the second feature fusion submodule.

4. The four-branch neural network remote sensing image fusion method according to claim 2, characterized in that: The network loss function model mentioned in step 3 is as follows: in, K Indicates the number of images. H Indicates the width of the image. W Indicates the image's height and band index. S G represents the number of bands. k ( i , j , s ) indicates the first k The first original hyperspectral image s The first band i Line number j Column pixel values, Indicates the first k The first predicted hyperspectral image s The first band i Line number j The pixel values ​​of the column.

5. A computer-readable medium, characterized in that, It stores a computer program executed by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method as described in any one of claims 1-4.

Images