A method and system for recovering spatial and spectral resolution of remote sensing images

Abstract

The present application relates to a kind of remote sensing image space and spectral resolution recovery method and system, comprising: obtaining panchromatic image and carrying out pre-processing, obtain the panchromatic image after degradation;Based on the feature extraction network of multiple branches to the panchromatic image feature extraction, obtain four feature maps;The feature map is aggregated, and feature aggregation map is obtained;Based on feature reconstruction network to the feature aggregation map feature recovery, obtain feature reconstruction map;The feature reconstruction map is Lab three channel image;The color space of the Lab three channel image is converted into RGB color space, and the final image of space and spectral resolution recovered is obtained.The present application proposes an end-to-end remote sensing image space and spectral resolution recovery method, only input panchromatic image can generate high spatial resolution color image, while recovering the spatial resolution and spectral resolution of remote sensing image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for restoring the spatial and spectral resolution of remote sensing images. Background Technology

[0002] High-quality remote sensing imagery plays a crucial role in remote sensing applications such as urban planning, environmental monitoring, resource exploration, and target identification. Existing remote sensing satellites typically acquire a pair of panchromatic (PAN) and multispectral (MS) images. PAN images have high spatial resolution but lack spectral information, while MS images have spectral information but low spatial resolution. Spatial resolution is the ability to distinguish spatial details in a remote sensing image; high spatial resolution is beneficial for target identification and visual interpretation. Spectral resolution is the ability to distinguish spectral details of ground features in a remote sensing image; high spectral resolution is important for the classification and identification of ground features in remote sensing images. Due to equipment limitations, it is difficult to acquire remote sensing images with both high spatial and high spectral resolution simultaneously, and the quality of the acquired remote sensing images deteriorates during image generation and transmission. Image colorization methods can restore the spectral resolution of remote sensing images, while image super-resolution methods can restore the spatial resolution.

[0003] Image colorization aims to transform grayscale images into color images to improve their visual appeal. Existing image colorization methods are mainly divided into user-guided and unguided methods. User-guided methods rely on information provided by the user, such as doodles, reference images, and semantic information, to achieve image colorization. This method requires not only extensive user interaction but also a certain level of user expertise; otherwise, satisfactory colorization results are difficult to achieve. In recent years, deep learning has made significant strides in image processing, and applying deep learning to image colorization can overcome the limitations of user-guided image colorization methods. Deep learning can be broadly categorized into supervised and unsupervised learning based on whether the training dataset contains labeled information. Grayscale image colorization algorithms based on Convolutional Neural Networks (CNNs) belong to supervised learning, using a large number of color images as labels for training to achieve image colorization. Generative Adversarial Networks (GANs) can be considered unsupervised learning networks, consisting of a generator and a discriminator. The generator produces colored images, and the discriminator distinguishes whether the input image is a real image or an image generated by the generator. GAN-based image colorization methods can generate rich and varied color images. Existing research shows that deep learning plays an important role in image colorization tasks, but existing image colorization methods still have certain limitations, such as the inability to solve multimodal problems, reliance on color distribution or object detection algorithms, and the need for large amounts of data for training.

[0004] Early image colorization methods were user-guided, such as using sketches and doodles to generate realistic colors for cars, bedrooms, and faces. Alternatively, they were based on sketching to colorize line art, with models generating realistic colors from illustrations with sparse lines and colored brushstrokes. User-guided image colorization methods require additional guidance from the user, which has limitations. Applying deep learning to image colorization tasks can overcome these limitations. Deep learning-based methods use color images as labels to train neural network models, enabling fully automatic colorization of grayscale images without human intervention. Existing techniques have also proposed transforming image colorization tasks into self-supervised learning tasks, making progress in the field of automatic image colorization. They build models to address the multimodal problem of image colorization and encourage the exploration of color diversity. However, panchromatic image colorization requires the colorization model to generate colors consistent with reality, rather than generating different colors for the same panchromatic image. CNNs are gradually becoming the main force in solving various image prediction tasks, including image colorization. CNNs minimize loss through learning; although the learning process is automated, a significant amount of human effort is needed to design an effective loss function. GANs can automatically design loss functions suitable for specific objectives (such as generating images indistinguishable from real images), thereby reducing the human effort required to design loss functions. Existing technologies use conditional GANs to propose a general image conversion network that can convert grayscale images to color images. While deep learning-based automatic image colorization techniques have made significant progress, these methods require large amounts of training data, and their colorization performance remains poor with limited sample sizes. Existing technologies also propose a colorization memory model to address the limited sample size problem, generating high-quality color images with limited data. The proposed ChromaGAN uses semantic cues as conditions to infer the chromaticity of a given grayscale image, eliminating impossible colors, such as a blue apple. However, ChromaGAN is only suitable for semantically clear images and not for remote sensing images with complex backgrounds and ambiguous semantic information. Some existing image colorization methods perform poorly on remote sensing images, exhibiting color overflow and failure to colorize rare colors. Therefore, image colorization remains a challenging problem requiring further exploration.

[0005] Image super-resolution aims to generate high spatial resolution images from low spatial resolution images. Improving the spatial resolution of images is an important image processing technique in computer vision. Super-resolution also helps with image compression; we can use smaller images for transmission or storage, and then use larger images for viewing, thus reducing the amount of data transmitted and stored. Deep learning-based image super-resolution maps low-spatial-resolution images to high-spatial-resolution images by training a neural network model. During training, the high-spatial-resolution image serves as the ground truth label, and the low-spatial-resolution image serves as the input.

[0006] In recent years, deep learning-based image super-resolution has made significant progress. Among numerous methods, the SRCNN model was the first application of deep learning in image super-resolution, using CNNs to achieve end-to-end mapping between low-spatial-resolution and high-spatial-resolution images. The FSRCNN proposed in existing techniques enhances SRCNN, accelerating model inference and improving the quality of the output image. Many methods optimize image super-resolution performance from a model perspective, such as introducing channel attention mechanisms or dense convolutional modules. Image super-resolution is an ill-posed problem; the same high-spatial-resolution image can be obtained by enlarging low-spatial-resolution images of different sizes. Models trained using mean squared error or mean absolute error as loss functions often produce blurred images. To address the problems of the aforementioned image super-resolution methods, further research is needed on suitable models and loss functions for image super-resolution. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a method and system for restoring the spatial and spectral resolution of remote sensing images.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] A method for restoring the spatial and spectral resolution of remote sensing images, comprising:

[0010] The panchromatic image is acquired and preprocessed to obtain the degraded panchromatic image;

[0011] The panchromatic image is used to extract features based on a multi-branch feature extraction network, resulting in four feature maps.

[0012] The feature map is then aggregated to obtain a feature aggregate map;

[0013] The feature aggregation map is restored based on the feature reconstruction network to obtain the feature reconstruction map; the feature reconstruction map is a Lab three-channel image.

[0014] The color space of the Lab three-channel image is converted to RGB color space to obtain the final image with restored spatial and spectral resolution.

[0015] Preferably, acquiring the panchromatic image based on the remote sensing image includes:

[0016] Acquire the remote sensing image;

[0017] The remote sensing image is format converted and downsampled using bicubic interpolation to obtain the degraded panchromatic image.

[0018] Preferably, the feature extraction of the panchromatic image based on the multi-branch feature extraction network yields four feature maps, including:

[0019] The channel correlation of the features of the panchromatic image is enhanced based on a feature channel hybridization network to obtain an enhanced feature map;

[0020] The enhanced feature map is input into the VGG-19 network to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map; the shapes of the first feature map, the second feature map, the third feature map, and the fourth feature map are all different;

[0021] The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively input into four branches for upsampling and strided convolution to obtain four feature maps with the same shape.

[0022] Preferably, the formula for the feature channel hybrid network is:

[0023]

[0024] out (p,q) =S(Φ(C(v1,v2,…,v c ),W (p,q) ),X (p,q) );

[0025] Among them, I (i,j) This refers to the j-th part of the i-th channel of the feature map of the input panchromatic image, where i takes the value [1, c], c is the number of channels in the feature map, and j takes the value 1 or 2; out (p,q) Let W(p,q) represent the value of the output feature map of the feature channel hybrid network at pixel (p,q), K(.) represent a two-stream self-convolutional network, Φ(.) represent a convolution operation, C(.) represent a aggregation operation, W(p,q) represent the weight of the convolution operation at pixel (p,q), X(p,q) represent the value of the input feature at pixel (p,q), and S(.) represent a two-stream attention network.

[0026] Preferably, the step of performing feature aggregation on the feature map to obtain a feature aggregation map includes:

[0027] The four feature maps are processed using a sub-pixel upconvolutional network and a convolutional network to obtain a fused feature map;

[0028] The fused feature map is enhanced by applying a polynomial filter to obtain the enhanced feature aggregate map.

[0029] Preferably, the formula for the polynomial filter is:

[0030]

[0031] in, Let F(i,j) represent the value of the polynomial filter at pixel (i,j), J(.) represent the network operations of convolution, pooling, and fully connected operations, F(i,j) represent the value of the input feature of the polynomial at pixel (i,j), W(i,j) represent the weights of the J(.) operation, and P(X,Y) represent a cubic polynomial, where X is an H×H matrix, Y is a W×W matrix, and H and W are the height and width of the input feature map of the polynomial filter, respectively.

[0032] Preferably, the feature reconstruction based network performs feature recovery on the feature aggregation map to obtain a feature reconstruction map, including:

[0033] The feature aggregation map is enlarged using two self-convolutional networks to obtain an enlarged feature aggregation map; the self-convolutional network includes inner convolution, sub-pixel upward convolution, and two standard convolutions.

[0034] The number of channels in the enlarged feature aggregation map is adjusted using two branches that use standard convolution operations to obtain the first channel feature map and the second channel feature map, respectively.

[0035] The first channel feature map and the second channel feature map are stitched together using a stitching operation in the channel dimension to obtain the feature reconstruction map.

[0036] Preferably, the step of converting the color space of the Lab three-channel image to the RGB color space to obtain the final image with restored spatial and spectral resolution includes:

[0037] The Lab three-channel image is converted from the Lab color space to the RGB color space to obtain the converted data;

[0038] The converted data is converted and saved to obtain a generated image; the generated image is the final image with restored spatial and spectral resolution.

[0039] Preferably, after converting the color space of the Lab three-channel image to the RGB color space to obtain the final image with restored spatial and spectral resolution, the method further includes:

[0040] Based on multiple objective evaluation indicators, the generated image is compared with the original panchromatic image and multispectral image to analyze the spatial and spectral resolution restoration effect of the generated image.

[0041] A remote sensing image spatial and spectral resolution restoration system includes:

[0042] The acquisition module is used to acquire the panchromatic image and perform preprocessing to obtain the degraded panchromatic image;

[0043] The feature extraction module is used to extract features from the panchromatic image based on a multi-branch feature extraction network to obtain four feature maps;

[0044] The aggregation module is used to aggregate the features in the feature map to obtain a feature aggregation map;

[0045] The reconstruction module is used to perform feature recovery on the feature aggregation map based on the feature reconstruction network to obtain a feature reconstruction map; the feature reconstruction map is a Lab three-channel image;

[0046] The conversion module is used to convert the color space of the Lab three-channel image into the RGB color space to obtain the final image with restored spatial and spectral resolution.

[0047] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] This invention provides a method and system for restoring the spatial and spectral resolution of remote sensing images, comprising: acquiring a panchromatic image and preprocessing it to obtain a degraded panchromatic image; extracting features from the panchromatic image based on a multi-branch feature extraction network to obtain four feature maps; aggregating the feature maps to obtain a feature aggregation map; restoring features from the feature aggregation map based on a feature reconstruction network to obtain a feature reconstruction map; the feature reconstruction map is a Lab three-channel image; and converting the color space of the Lab three-channel image to RGB color space to obtain the final image with restored spatial and spectral resolution. This invention proposes an end-to-end method for restoring the spatial and spectral resolution of remote sensing images, which can generate a high spatial resolution color image by inputting only a panchromatic image, while simultaneously restoring the spatial and spectral resolution of the remote sensing image. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of the remote sensing image spatial and spectral resolution restoration method provided in the embodiments of the present invention;

[0051] Figure 2 A flowchart of the recovery process provided in the embodiments of the present invention;

[0052] Figure 3 This is a schematic diagram of the feature channel mixing module in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the structure of the DInv block in the embodiments provided by the present invention;

[0054] Figure 5 This is a schematic diagram of the structure of the DSE block in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of the polynomial filter in the embodiments provided by the present invention;

[0056] Figure 7 This is a schematic diagram of the neural network architecture in the embodiments provided by the present invention;

[0057] Figure 8 The spatial resolution and spectral resolution restoration effect diagrams provided in the embodiments of the present invention;

[0058] Figure 9 A comparison of the spectral resolution recovery results of the method provided in the embodiments of the present invention with other algorithms;

[0059] Figure 10 The objective evaluation index data comparison results of the method in the embodiments of the present invention with other spectral resolution recovery algorithms;

[0060] Figure 11 A comparison of the spatial resolution restoration effect of the method in the embodiments of the present invention with that of other algorithms is shown in the figure.

[0061] Figure 12 The objective evaluation index data of the method in the embodiments of the present invention are compared with those of other spatial resolution restoration algorithms. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, including a series of steps, processes, methods, etc., is not limited to the steps listed, but may optionally include steps not listed, or may optionally include other steps inherent to these processes, methods, products, or devices.

[0065] The purpose of this invention is to provide a method and system for restoring the spatial and spectral resolution of remote sensing images, which can restore the spatial and spectral resolution of remote sensing images end-to-end. It can generate a high spatial resolution color image by inputting only a panchromatic image, and simultaneously restore the spatial and spectral resolution of the remote sensing image.

[0066] This embodiment proposes a multi-branch remote sensing image spatial and spectral resolution restoration network (MPCSRnet) based on convolutional neural networks to simultaneously restore the spatial and spectral resolution of remote sensing images. MPCSRnet employs a multi-branch structure to extract rich features and utilizes Feature Channel Mixing Blocks (FCMB) to enhance the interaction between adjacent channels of the features. Feature aggregation in MPCSR-Net is used to learn more effective features from each branch, and then a multinomial filter is used to enhance the aggregated features. After obtaining rich features, this invention uses a U-shaped structure as the main structure of the restoration architecture to generate images with high spatial and high spectral resolution.

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Figure 1A flowchart of the remote sensing image spatial and spectral resolution restoration method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the present invention provides a method for restoring the spatial and spectral resolution of remote sensing images, including:

[0069] Step 100: Acquire the panchromatic image and perform preprocessing to obtain the degraded panchromatic image;

[0070] Step 200: Extract features from the panchromatic image based on a multi-branch feature extraction network to obtain four feature maps;

[0071] Step 300: Perform feature aggregation on the feature map to obtain a feature aggregation map;

[0072] Step 400: Perform feature recovery on the feature aggregation map based on the feature reconstruction network to obtain the feature reconstruction map; the feature reconstruction map is a Lab three-channel image;

[0073] Step 500: Convert the color space of the Lab three-channel image to RGB color space to obtain the final image with restored spatial and spectral resolution.

[0074] Preferably, step 100 includes:

[0075] Acquire the remote sensing image;

[0076] The remote sensing image is format converted and downsampled using bicubic interpolation to obtain the degraded panchromatic image.

[0077] Figure 2 A flowchart of the recovery process in the embodiments provided by the present invention, such as Figure 2 As shown, the first step in this embodiment is to downsample the panchromatic image using bicubic interpolation to obtain a degraded panchromatic image, which is then used as the input to MPCSR-Net.

[0078] Preferably, step 200 includes:

[0079] The channel correlation of the features of the panchromatic image is enhanced based on a feature channel hybridization network to obtain an enhanced feature map;

[0080] The enhanced feature map is input into the VGG-19 network to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map; the shapes of the first feature map, the second feature map, the third feature map, and the fourth feature map are all different;

[0081] The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively input into four branches for upsampling and strided convolution to obtain four feature maps with the same shape.

[0082] Preferably, the formula for the feature channel hybrid network is:

[0083]

[0084] out (p,q) =S(Φ(C(v1,v2,…,v c ),W (p,q) ),X (p,q) );

[0085] Among them, I (i,j) This refers to the j-th part of the i-th channel of the feature map of the input panchromatic image, where i takes the value [1, c], c is the number of channels in the feature map, and j takes the value 1 or 2; out (p,q) Let W(p,q) represent the value of the output feature map of the feature channel hybrid network at pixel (p,q), K(.) represent a two-stream self-convolutional network, Φ(.) represent a convolution operation, C(.) represent a aggregation operation, W(p,q) represent the weight of the convolution operation at pixel (p,q), X(p,q) represent the value of the input feature at pixel (p,q), and S(.) represent a two-stream attention network.

[0086] Specifically, the second step in this embodiment is to extract features using a multi-branch feature extraction module. Before branching, a feature channel mixing module (FCMB) is used to enhance the channel correlation of the input features, and then VGG-19 is used as the backbone network to extract four feature maps. These four feature maps are then input into four branches. Each branch begins with an upsampling block. The first branch uses four 3×3 convolutions with a stride of 2, combined with the features extracted by the FCMB module; the second branch uses three 3×3 convolutions with a stride of 2; the third branch uses two 3×3 convolutions with a stride of 2; and the fourth branch uses one 3×3 convolution with a stride of 2. Since a stride of 2 convolution reduces the size of the feature map, the number of convolution operations in each branch is different to ensure that the final output size of the feature map obtained from each branch remains consistent.

[0087] Furthermore, the second step of this embodiment first uses the Feature Channel Mixing Module (FCMB) of this invention to enhance the correlation between channels. The two-stream self-convolutional block (DInv) in FCMB assigns weights to the features of adjacent channels to increase the correlation between them. The FCMB structure is as follows: Figure 3 As shown, FCMB is implemented in five steps. The first step is to divide the input image into C feature maps along the channel dimension (C is the number of channels in the input image), and label these feature maps as u. i The second step is to... i The feature u is obtained by splitting the parts in half.i1 and u i2 The third step is to use the DInv block to obtain the mixed feature, denoted as v, based on the characteristics between adjacent channels. i Fourth step, use the concatenation operation (concat) to combine all the v i The features are combined; in the fifth step, weights are assigned to the input features and the features obtained in the fourth step through a two-stream attention block (DSE).

[0088] Secondly, the input to the DInv block is two feature maps, denoted as x and y respectively. The structure of the DInv block is as follows: Figure 4 As shown, x is processed by average pooling and two 1×1 convolutions to obtain a k×k convolution kernel. The obtained convolution kernel is then used to convolve y to obtain the output feature map.

[0089] Secondly, the input to the DSE block is two feature maps, denoted as x and y, and the structure of the DSE block is as follows: Figure 5 As shown, x is weighted through an average pooling layer and a fully connected layer, and the resulting weights are assigned to y to obtain the output feature map.

[0090] Preferably, step 300 includes:

[0091] The four feature maps are processed using a sub-pixel upconvolutional network and a convolutional network to obtain a fused feature map;

[0092] The fused feature map is enhanced by applying a polynomial filter to obtain the enhanced feature aggregate map.

[0093] Preferably, the formula for the polynomial filter is:

[0094]

[0095] in, Let F(i,j) represent the value of the polynomial filter at pixel (i,j), J(.) represent the network operations of convolution, pooling, and fully connected operations, F(i,j) represent the value of the input feature of the polynomial at pixel (i,j), W(i,j) represent the weights of the J(.) operation, and P(X,Y) represent a cubic polynomial, where X is an H×H matrix, Y is a W×W matrix, and H and W are the height and width of the input feature map of the polynomial filter, respectively.

[0096] Specifically, in this embodiment, the third step is to use sub-pixel upward convolution to aggregate the four features of the same size extracted in S2 together to increase the size of the feature map, and then use a polynomial filter to enhance the aggregated features.

[0097] Furthermore, the purpose of the third step in this embodiment is to first use the features of the polynomial filter to enhance important information and suppress unimportant information, that is, to assign different weights to the features. The structure of the polynomial filter is as follows: Figure 6 As shown, the input features are processed through four overlapping convolution operations, three max pooling operations, a global pooling operation, and a fully connected operation to obtain a vector. This vector is then multiplied by a polynomial constructed from the width and height of the input features to obtain a weight matrix. Finally, the input features are multiplied by the weight matrix to obtain the output feature map.

[0098] Preferably, step 400 includes:

[0099] The feature aggregation map is enlarged using two self-convolutional networks to obtain an enlarged feature aggregation map; the self-convolutional network includes inner convolution, sub-pixel upward convolution, and two standard convolutions.

[0100] The number of channels in the enlarged feature aggregation map is adjusted using two branches that use standard convolution operations to obtain the first channel feature map and the second channel feature map, respectively.

[0101] The first channel feature map and the second channel feature map are stitched together using a stitching operation in the channel dimension to obtain the feature reconstruction map.

[0102] Specifically, in this embodiment, the fourth step inputs the feature map obtained in the third step into the feature reconstruction module for feature recovery. The feature reconstruction module consists of two self-convolutional blocks (SCBs), six convolution operations, and one concatenation operation.

[0103] Furthermore, in this embodiment, the SCB in the fourth step consists of two standard convolutional layers, one inner convolutional layer, and one sub-pixel up convolutional layer. The SCB utilizes the inner convolutional layer to reduce kernel redundancy across channels. Standard convolutional kernels possess two significant characteristics: spatial agnosticness and channel specificity. Inner convolution has the opposite characteristics: spatial specificity and channel agnosticness. That is, the inner convolutional kernels differ spatially but are shared across the channel. In the feature recovery stage, increasing the feature map size is unavoidable; therefore, this embodiment utilizes a sub-pixel convolutional layer to expand the feature map size.

[0104] Next, the six convolution operations are divided into two groups to obtain the outputs of the L channel (1 channel) and the ab channel (2 channels).

[0105] The outputs of the L channel and the ab channel are stitched together again using the stitching operation to obtain a three-channel (L, a, b channel) output image.

[0106] Preferably, step 500 includes:

[0107] The Lab three-channel image is converted from the Lab color space to the RGB color space to obtain the converted data;

[0108] The converted data is converted and saved to obtain a generated image; the generated image is the final image with restored spatial and spectral resolution.

[0109] Specifically, in this embodiment, the fifth step is to convert the output image from the fourth step from the Lab color space to the RGB color space to obtain the final image that has restored the spatial and spectral resolution.

[0110] Preferably, after step 500, the method further includes:

[0111] Based on multiple objective evaluation indicators, the generated image is compared with the original panchromatic image and multispectral image to analyze the spatial and spectral resolution restoration effect of the generated image.

[0112] Figure 7 This is a schematic diagram of the neural network architecture in the embodiments provided by the present invention, such as... Figure 7 As shown, this embodiment also includes the following steps:

[0113] S1: Use bicubic interpolation to degenerate a 256×256×1 panchromatic image into a 128×128×1 image. Convert the image into NumPy data format and copy it twice in the channel dimension to obtain 128×128×3 data, denoted as S1.

[0114] S2: Input the data obtained in S1 into the multi-branch feature extraction module.

[0115] S2-1: Obtain a feature map with a shape of 128×128×3 through FCMB blocks. The formula for FCMB is as follows:

[0116]

[0117] out (p,q) =S(Φ(C(v1,v2,…,v c ),W (p,q) ),X (p,q) );

[0118] Among them, I (i,j) This is the j-th part of the i-th channel of the input feature map, where i takes the value [1, c], c is the number of channels in the feature map, and j takes the value 1 or 2. (p,q)Let represent the value of the output feature map of the FCMB module at pixel (p,q), K(.) represent a two-stream self-convolutional block, Φ(.) represent a convolution operation, C(.) represent a aggregation operation, W(p,q) represent the weight of the convolution operation at pixel (p,q), X(p,q) represent the value of the input feature at pixel (p,q), and S(.) represent a two-stream attention block. Through these operations, the shape of the output feature map is made consistent with the shape of the input feature map.

[0119] S2-2: Input the feature map obtained in S2-1 into the VGG-19 network to obtain four features with shapes of 128×128×64, 64×64×128, 32×32×256 and 16×16×512 respectively.

[0120] S2-3: Input the four feature maps obtained in S2-2 into four branches. Each branch first uses an upsampling operation to enlarge the feature map, resulting in feature maps with shapes of 256×256×64, 128×128×128, 64×64×256, and 32×32×512, respectively. Further, each branch uses a different number of strided convolutions with a stride of 2. A strided convolution with a stride of 2 can reduce the size of the feature map by half. The four branches use four, three, two, and one strided convolution with a stride of 2, respectively, to obtain four feature maps with a shape of 16×16×192.

[0121] S3: Input the feature maps extracted from each branch in S2 into the feature aggregation module.

[0122] S3-1: Four 16×16×192 feature maps are convolved upwards by a subpixel to obtain a 32×32×192 feature map. A convolution operation is then performed to adjust the number of channels, resulting in a 32×32×160 feature map. Another subpixel-level convolution is then performed upwards to obtain a 64×64×160 feature map. Finally, a convolution operation is performed to adjust the number of channels, resulting in a 64×64×128 feature map.

[0123] S3-2: Use a polynomial filter to assign weights to the features to enhance them. The formula for polynomial filtering is as follows:

[0124]

[0125] in Let F(i,j) represent the value of the polynomial filter at pixel (i,j), J(.) represent the convolution, pooling, and fully connected operation modules, F(i,j) represent the value of the input feature of the polynomial at pixel (i,j), W(i,j) represent the weights of the J(.) operation, and P(X,Y) represent a cubic polynomial, where X is an H×H matrix, Y is a W×W matrix, and H and W are the height and width of the input feature map of the polynomial filter, respectively. The polynomial filter yields an enhanced feature map with a shape of 64×64×128.

[0126] S4: Input the feature map obtained in S3 into the feature reconstruction module.

[0127] S4-1: The feature map is expanded using two self-convolutional blocks. Each self-convolutional block consists of two standard convolutions, one inner convolution, and one subpixel up convolution. The standard convolution is used to adjust the number of channels in the feature map, the inner convolution is used to achieve channel sharing, and the subpixel up convolution is used to expand the size of the feature map. The two self-convolutional blocks result in a feature map with a shape of 256×256×64.

[0128] S4-2: The number of channels is adjusted using two branches that employ standard convolution operations, resulting in two feature maps with shapes of 256×256×1 and 256×256×2, denoted as... and

[0129] S4-3: Use a concatenation operation in the channel dimension to concatenate the two feature maps from S4-2 to obtain a feature map with a shape of 256×256×3, denoted as...

[0130] S4-4: Compared to the original panchromatic image (denoted as S) PAN Calculate L1 loss. With the ab channels of the multispectral image (denoted as Y) MS_ab Calculate L1 loss. Images stitched together with panchromatic and multispectral images (denoted as C(S)) PAN ,Y MS_ab Calculate the structural similarity loss. The loss function is set as follows:

[0131]

[0132] in:

[0133]

[0134]

[0135] S5: Take the result from S4 Convert from Lab color space to RGB color space.

[0136] S6: Compile the data generated in S5 into an image and save it to obtain the final high spatial resolution color image.

[0137] S7: By comparing the generated image with the original panchromatic image and multispectral image using various objective evaluation indicators, the spatial and spectral resolution restoration effect is analyzed.

[0138] See Figure 7 As shown, this embodiment also includes the following steps:

[0139] S1: Acquire remote sensing images and convert them into NumPy data format. The shape of the panchromatic image data is 256×256×1, and the shape of the multispectral image data is 64×64×3. Copy the panchromatic image data twice along the channel dimension to obtain panchromatic image data with a shape of 256×256×3, denoted as Y.

[0140] S2: Input Y into the feature extraction module to obtain four feature maps with a shape of 32×32×192.

[0141] S3: Input the four feature maps obtained in S2 into the feature aggregation module to obtain a feature map with a shape of 128×128×128.

[0142] S4: Input the feature map obtained in S3 into the feature reconstruction module to obtain data with a shape of 512×512×3.

[0143] S5: Convert the data obtained in S4 from the Lab color space to the RGB color space.

[0144] S6: Compile the data generated in S5 into an image and save it to obtain the final high spatial resolution color image.

[0145] S7: By comparing the generated image with the original panchromatic image and multispectral image using various objective evaluation indicators, the spatial resolution and spectral resolution restoration effect are analyzed.

[0146] Figure 8 The spatial resolution and spectral resolution restoration effect diagrams provided in the embodiments of the present invention are as follows: Figure 8 As shown, (a) is an MS image, (b) is the input image of the present invention, and (c) is the output image of the present invention. Figure 8 It is understood that the present invention can restore spatial resolution and spectral resolution for remote sensing images of different sizes.

[0147] Figure 9This section compares the spectral resolution restoration performance of the method of this invention with that of other algorithms. (a) is a MS image magnified 4 times; (b) is the input image; (c)-(g) represent the methods proposed by Iizuka et al., Zhang et al., Isola et al., Yoo et al., and Vitoria et al., respectively; and (h) represents the method proposed in this invention. Figure 9 It can be seen that: 1) Compared with other methods, the spectral information of the image generated by the method of the present invention is closer to that of a multispectral image. For example, Figure 9 The first line shows that the method proposed in this invention correctly colors rooftops, land, and vegetation, while other methods produce some incorrect colors (e.g., the rooftop in the upper left corner is gray, but blue in the multispectral image; the land within the red rectangle is green, but brown in the multispectral image). 2) Our model is particularly good at coloring rare instances. For example, Figure 9 In the fourth row, the MS image shows green vegetation within a red box; only our method colors this vegetation green, while all other methods color it brown. 3) Our method does not have the problem of color overflow. For example, Figure 8 In the last row, the road in (d) is shown as blue, and the roads in (f) and (g) are shown as green, but neither blue nor green is the color of a road. However, the method of the present invention generates the same road colors as the multispectral image.

[0148] To better analyze and compare the results of image spectral resolution restoration, this invention uses multiple image quality evaluation metrics (PSNR, SSIM, CC, UIQI, MSE, SAM, and ERGAS) to objectively evaluate the images after spectral resolution restoration. The objective evaluation metrics data are as follows: Figure 10 As shown. By Figure 10 As can be seen, the method of this invention is optimal in all metrics except ERGAS. This is because our multi-branch structure model can extract rich features, resulting in high-quality final images. In summary, the method of this invention outperforms the comparative algorithms in both visual effects and objective evaluation metrics.

[0149] Figure 11 The spatial resolution restoration results of the method provided in the embodiments of the present invention are compared with those of other algorithms, as shown in the figure. Figure 11The image shows a comparison of the spatial resolution restoration performance of the method of this invention with other algorithms. (a) is the input image, and (b)-(h) represent the images from SRCNN proposed by Dong et al., FSRCNN proposed by Dong et al., SRGAN proposed by Ledig et al., EDSR proposed by Lim et al., LapSR proposed by Lai et al., MSRN proposed by Li et al., and RDN proposed by Zhang et al., respectively. (i) represents the method proposed in this invention. Figure 11 As can be seen, the images generated by the spatial resolution restoration method of this invention are visually comparable to those generated by other methods. In particular, our method does not produce noticeable artifacts.

[0150] To better analyze and compare the spatial resolution restoration results of images, this invention uses multiple image quality evaluation metrics (PSNR, SSIM, CC, UIQI, MSE, SAM, and ERGAS) to objectively evaluate the spatially restored images. The objective evaluation metrics data are as follows: Figure 12 As shown. By Figure 12 As can be seen from the method of the present invention, our method is superior to other methods, which further illustrates the effectiveness of the multi-branch model proposed in this invention.

[0151] This embodiment also provides a remote sensing image spatial and spectral resolution restoration system, including:

[0152] The acquisition module is used to acquire the panchromatic image and perform preprocessing to obtain the degraded panchromatic image;

[0153] The feature extraction module is used to extract features from the panchromatic image based on a multi-branch feature extraction network to obtain four feature maps;

[0154] The aggregation module is used to aggregate the features in the feature map to obtain a feature aggregation map;

[0155] The reconstruction module is used to perform feature recovery on the feature aggregation map based on the feature reconstruction network to obtain a feature reconstruction map; the feature reconstruction map is a Lab three-channel image;

[0156] The conversion module is used to convert the color space of the Lab three-channel image into the RGB color space to obtain the final image with restored spatial and spectral resolution.

[0157] The beneficial effects of this invention are as follows:

[0158] (1) This invention proposes an end-to-end method for restoring the spatial and spectral resolution of remote sensing images. It can generate a high spatial resolution color image by inputting only a panchromatic image, and at the same time restore the spatial and spectral resolution of the remote sensing image.

[0159] (2) This invention uses a multi-branch structure to extract features in multiple dimensions and aggregates the features extracted from each branch to improve feature extraction capability.

[0160] (3) The present invention designs a Feature Channel Mixing Module (FCMB) to utilize the correlation between channels.

[0161] (4) Inspired by the cubic filter, this invention designs a polynomial filter to enhance features.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0163] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for restoring the spatial and spectral resolution of remote sensing images, characterized in that, include: The panchromatic image is acquired and preprocessed to obtain the degraded panchromatic image; Feature extraction is performed on the panchromatic image using a multi-branch feature extraction network to obtain four feature maps, including: The channel correlation of features in the panchromatic image is enhanced using a feature channel fusion network to obtain an enhanced feature map. The formula for the feature channel fusion network is as follows: out(p, q)=S(Φ(C(v1, v2,..., vc), W(p, q)), X(p, q)); Where I(i,j) is the j-th part of the i-th channel of the feature map of the input panchromatic image, i takes the value [1,c], c is the number of channels of the feature map, and j takes the value 1 or 2; out(p,q) represents the value of the output feature map of the feature channel mixing network at pixel (p,q), K(.) represents a two-stream self-convolutional network, Φ(.) represents a convolution operation, C(.) represents a aggregation operation, W(p,q) represents the weight of the convolution operation at pixel (p,q), X(p,q) represents the value of the input feature at pixel (p,q), and S(.) represents a two-stream attention network; The enhanced feature map is input into the VGG-19 network to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map; the shapes of the first feature map, the second feature map, the third feature map, and the fourth feature map are all different; The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively input into four branches for upsampling and strided convolution to obtain four feature maps with the same shape; The feature map is then aggregated to obtain a feature aggregate map; The feature aggregation map is restored based on the feature reconstruction network to obtain the feature reconstruction map; the feature reconstruction map is a Lab three-channel image. The color space of the Lab three-channel image is converted to RGB color space to obtain the final image with restored spatial and spectral resolution.

2. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 1, characterized in that, Obtain a panchromatic image based on remote sensing images, including: Acquire the remote sensing image; The remote sensing image is format converted and downsampled using bicubic interpolation to obtain the degraded panchromatic image.

3. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 1, characterized in that, The feature map is then subjected to feature aggregation to obtain... Feature aggregation map, including: The four feature maps are processed using a sub-pixel upconvolutional network and a convolutional network to obtain a fused feature map; The fused feature map is enhanced by applying a polynomial filter to obtain the enhanced feature aggregate map.

4. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 3, characterized in that, The formula for the polynomial filter is: in, Let F(i,j) represent the value of the polynomial filter at pixel (i,j), J(.) represent the network operations of convolution, pooling, and fully connected operations, F(i,j) represent the value of the input feature of the polynomial at pixel (i,j), W(i,j) represent the weights of the J(.) operation, and P(X,Y) represent a cubic polynomial, where X is an H×H matrix, Y is a W×W matrix, and H and W are the height and width of the input feature map of the polynomial filter, respectively.

5. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 1, characterized in that, The feature reconstruction network performs feature recovery on the feature aggregation map to obtain a feature reconstruction map, including: The feature aggregation map is enlarged using two self-convolutional networks to obtain an enlarged feature aggregation map; the self-convolutional network includes inner convolution, sub-pixel upward convolution, and two standard convolutions. The number of channels in the enlarged feature aggregation map is adjusted using two branches that use standard convolution operations to obtain the first channel feature map and the second channel feature map, respectively. The first channel feature map and the second channel feature map are stitched together using a stitching operation in the channel dimension to obtain the feature reconstruction map.

6. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 1, characterized in that, The process of converting the color space of the Lab three-channel image to the RGB color space to obtain the final image with restored spatial and spectral resolution includes: The Lab three-channel image is converted from the Lab color space to the RGB color space to obtain the converted data; The converted data is converted and saved to obtain a generated image; the generated image is the final image with restored spatial and spectral resolution.

7. The method for restoring the spatial and spectral resolution of remote sensing images according to claim 6, characterized in that, After converting the color space of the Lab three-channel image to the RGB color space to obtain the final image with restored spatial and spectral resolution, the process further includes: Based on multiple objective evaluation indicators, the generated image is compared with the original panchromatic image and multispectral image to analyze the spatial and spectral resolution restoration effect of the generated image.

8. A remote sensing image spatial and spectral resolution restoration system, characterized in that, include: The acquisition module is used to acquire the panchromatic image and perform preprocessing to obtain the degraded panchromatic image; The feature extraction module is used to extract features from the panchromatic image based on a multi-branch feature extraction network, resulting in four feature maps, including: The channel correlation of features in the panchromatic image is enhanced using a feature channel fusion network to obtain an enhanced feature map. The formula for the feature channel fusion network is as follows: out(p, q)=S(Φ(C(v1, v2,..., vc), W(p, q)), X(p, q)); Where I(i,j) is the j-th part of the i-th channel of the feature map of the input panchromatic image, i takes the value [1,c], c is the number of channels of the feature map, and j takes the value 1 or 2; out(p,q) represents the value of the output feature map of the feature channel mixing network at pixel (p,q), K(.) represents a two-stream self-convolutional network, Φ(.) represents a convolution operation, C(.) represents a aggregation operation, W(p,q) represents the weight of the convolution operation at pixel (p,q), X(p,q) represents the value of the input feature at pixel (p,q), and S(.) represents a two-stream attention network; The enhanced feature map is input into the VGG-19 network to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map; the shapes of the first feature map, the second feature map, the third feature map, and the fourth feature map are all different; The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively input into four branches for upsampling and strided convolution to obtain four feature maps with the same shape; The aggregation module is used to aggregate the features in the feature map to obtain a feature aggregation map; The reconstruction module is used to perform feature recovery on the feature aggregation map based on the feature reconstruction network to obtain the feature reconstruction map; The reconstructed feature image is a Lab three-channel image; The conversion module is used to convert the color space of the Lab three-channel image into the RGB color space to obtain the final image with restored spatial and spectral resolution.

Images