Satellite remote sensing image super-resolution method, device, equipment, medium and product

Abstract

The application relates to the technical field of remote sensing image processing, in particular to a satellite remote sensing image super-resolution method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring an original image of target satellite remote sensing; enlarging the original image to a target size to obtain an enlarged image; extracting a shallow feature image and a deep feature image of the original image, performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; and generating a super-resolution reconstruction image according to the super-resolution residual image and the enlarged image. The method can generate an image with higher resolution, richer information and higher accuracy by using linearly increasing calculation amount and a larger receptive field. Thus, the problems of exponential expansion of calculation amount, limited receptive field and detail restoration in the process of processing satellite remote sensing multispectral images by using a traditional attention mechanism super-resolution algorithm are solved.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method, device, electronic device and storage medium for super-resolution of satellite remote sensing images. Background Art

[0002] In the current field of satellite remote sensing image processing, using deep learning to obtain super-resolution images is one of the most popular directions. Neural networks are usually used to obtain higher-resolution generated images from low-resolution original images that meet usage requirements and are consistent with real scenes. This method has the characteristics of low cost and easy implementation.

[0003] Correspondingly, super-resolution algorithms based on the self-attention mechanism have been a research hotspot in recent years. As an alternative to convolutional neural networks, emerging self-attention neural networks achieve excellent performance by calculating the autocorrelation of each element within the global receptive field. However, direct application of such methods to satellite remote sensing image processing has the following shortcomings:

[0004] (1) The traditional self-attention mechanism for ordinary image super-resolution algorithms with few channel dimensions (such as red, green, and blue channels) mostly operates from the spatial dimension. This cannot fully utilize the multi-channel information of satellite multispectral remote sensing images, limits the potential of attention mechanism feature extraction, and cannot effectively restore more details of satellite remote sensing images.

[0005] (2) The multi-head attention operation strategy adopted by the traditional self-attention mechanism requires dividing the channel into multiple unrelated parts, which indirectly reduces the receptive field of the channel attention algorithm and affects the accuracy of the super-resolution algorithm.

[0006] (3) The rank of the autocorrelation matrix of the channel attention mechanism is relatively smaller than that of the spatial attention mechanism, resulting in the feature diversity obtained by the channel attention feature extraction module being lower than that of the spatial attention algorithm, which limits the expressive power of the algorithm based on the channel attention mechanism. Summary of the Invention

[0007] The present application provides a satellite remote sensing image super-resolution method, device, electronic device and storage medium to solve the problems of exponential expansion of computational complexity, limited receptive field and detail restoration in the traditional attention mechanism super-resolution algorithm during the satellite remote sensing multispectral image processing process.

[0008] A first aspect of the present application provides a satellite remote sensing image super-resolution method, comprising the following steps: acquiring an original image of a target satellite remote sensing image; enlarging the original image to a target size to obtain an enlarged image; extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; and generating a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

[0009] Optionally, extracting shallow feature images and deep feature images of the original image includes: inputting the original image into a shallow feature extraction module, the shallow feature extraction module outputting the shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; inputting the shallow feature image into a deep feature extraction module, the deep feature extraction module outputting the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolution layer.

[0010] 可选地,浅层特征图像的表达式为:

[0011] F0=H SF (I LR )

[0012] 其中,F0表示浅层特征图像,H SF 表示浅层特征提取模块,H SF (·) indicates that the image is processed using a shallow feature extraction module, I LR 表示原始图像, 其中H、W以及C in Represent the length, width and number of channels of the original image respectively.

[0013] 深层特征图像的表达式为:

[0014] F DF =H CONV (F N )

[0015] Among them, F DF 表示深层特征图像, 表示第i个通道变换注意力模块,F N represents the feature image generated by the Nth channel transformation attention module. Specifically, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·)表示一个卷积层;

[0016] F DF =H DF (F0)。

[0017] 其中,F0表示浅层特征图像,HDF 表示深层特征提取模块。

[0018] Optionally, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, and each channel transformation attention module is:

[0019]

[0020] F i,out =H CONV (F i,L )+F i,0

[0021] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out Indicates the feature image finally obtained by the channel transformation attention module, H CONV (F i,L )表示通过卷积层生成的特征图,H CONV (F i,L ) and F i,0 通过一个残差跳跃链接获得F i,out .

[0022] Optionally, each channel-transformed attention layer includes a channel-transformed self-attention operation unit and a feature diversity supplementation unit, and the function of the channel-transformed attention layer is:

[0023]

[0024] Y=MLP(LN(X N ))+X N

[0025] Among them, X N represents the channel-transformable attention unit generating the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel-transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0026] Optionally, the expression of the super-resolution reconstructed image is:

[0027] I RSR =H Rec (F0+FDF )+H Int (I LR )

[0028] Among them, I RSR 表示最终的超分辨率图像,H Rec (·)表示图像超分模块,H Int (·)表示原始图像插值运算。

[0029] The second aspect of the present application provides a satellite remote sensing image super-resolution device, including: an acquisition module for acquiring an original image of a target satellite remote sensing; an amplification module for amplifying the original image to a target size to obtain an amplified image; an extraction module for extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; and a reconstruction module for generating a super-resolution reconstructed image based on the super-resolution residual image and the amplified image.

[0030] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the satellite remote sensing image super-resolution method of the first aspect.

[0031] The fourth embodiment of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the satellite remote sensing image super-resolution method of the first aspect is implemented.

[0032] The fifth embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed, implements the satellite remote sensing image super-resolution method of the first aspect.

[0033] 由此,本申请包括如下有益效果:

[0034] The embodiment of the present application can obtain the original image of the target satellite remote sensing and amplify it, extract the shallow feature image and deep feature image of the original image, and perform super-resolution processing on the shallow feature image and deep feature image to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the amplified image. This method can generate a higher resolution, more information-rich, and more accurate image with a linearly increasing computational load and a larger receptive field. This solves the problems of exponential computational expansion, limited receptive field, and detail restoration in the traditional attention mechanism super-resolution algorithm in the process of satellite remote sensing multispectral image processing.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a process for super-resolution of satellite remote sensing images provided according to an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a super-resolution image reconstruction process according to an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a flow chart of a satellite remote sensing image super-resolution method based on a channel transformation attention mechanism provided in one embodiment of the present application;

[0040] Figure 4 A schematic diagram of a deep feature extraction module provided in one embodiment of the present application;

[0041] Figure 5 A schematic diagram of the channel transformation attention layer structure provided for one embodiment of the present application;

[0042] Figure 6 A schematic diagram of channel conversion provided in one embodiment of the present application;

[0043] Figure 7 This is a block diagram of an exemplary device for super-resolution of satellite remote sensing images according to an embodiment of the present application;

[0044] Figure 8 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0046] The following describes the satellite remote sensing image super-resolution method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology of the traditional attention mechanism super-resolution algorithm in the process of satellite remote sensing multispectral image processing, such as exponential expansion of computational complexity, limited receptive field and detail restoration, the present application provides a satellite remote sensing image super-resolution method. In this method, the original image of the target satellite remote sensing can be obtained and amplified, the shallow feature image and deep feature image of the original image can be extracted, and the shallow feature image and deep feature image can be super-resolution processed to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the amplified image. The above method can generate a higher resolution, richer information and more accurate image with linearly growing computational complexity and a larger receptive field. Thus, the problems of exponential expansion of computational complexity, limited receptive field and detail restoration in the process of satellite remote sensing multispectral image processing of the traditional attention mechanism super-resolution algorithm are solved.

[0047] Specifically, Figure 1 A schematic flow chart of a satellite remote sensing image super-resolution method provided in an embodiment of the present application.

[0048] like Figure 1 As shown, the satellite remote sensing image super-resolution method includes the following steps:

[0049] In step S101, an original remote sensing image of a target satellite is acquired.

[0050] Among them, the original images of target satellite remote sensing are hyperspectral images, which can capture the fine spectral information of ground objects, have a large spectral range and rich band information, but high spectral resolution and high spatial resolution cannot be obtained at the same time.

[0051] It is understandable that the embodiments of the present application can obtain the original hyperspectral image of the target satellite remote sensing and convert it into a high-resolution image through the following method.

[0052] In step S102 , the original image is enlarged to a target size to obtain an enlarged image.

[0053] The method of enlarging the original image is interpolation enlargement; the target size is a size value set based on actual needs and is not specifically limited here.

[0054] It is understandable that the embodiment of the present application can enlarge the original image to the required target size through the interpolation enlargement method to obtain the enlarged image.

[0055] In step S103, a shallow feature image and a deep feature image of the original image are extracted, and super-resolution processing is performed on the shallow feature image and the deep feature image to obtain a super-resolution residual image.

[0056] Among them, the shallow feature image is an image obtained after shallow feature extraction is performed on the original image, and the deep feature image is an image obtained after deep feature extraction is performed on the shallow feature image.

[0057] It can be understood that the embodiment of the present application can perform shallow feature extraction on the original image to obtain a shallow feature image, then perform deep feature extraction on the shallow feature image to obtain a deep feature image, and finally perform super-resolution processing on the two to obtain a super-resolution residual image.

[0058] In an embodiment of the present application, a shallow feature image and a deep feature image of an original image are extracted, including: inputting the original image into a shallow feature extraction module, the shallow feature extraction module outputting the shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; inputting the shallow feature image into a deep feature extraction module, the deep feature extraction module outputting the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolutional layer.

[0059] It can be understood that the embodiment of the present application performs shallow feature extraction on the original image through a shallow feature extraction module, and the shallow feature extraction module includes a convolutional neural network unit. The original image is input into the shallow feature extraction module, and the shallow feature extraction module can output a shallow feature image; after obtaining the shallow feature image, the deep feature extraction module is used to perform deep feature extraction on the shallow feature image. The deep feature extraction module includes a channel transformation attention module and a convolution layer. The shallow feature image is input into the deep feature extraction module, and the deep feature extraction module outputs a deep feature image.

[0060] In the embodiment of the present application, the expression of the shallow feature image is:

[0061] F0=H SF (I LR )

[0062] 其中,F0表示浅层特征图像,H SF 表示浅层特征提取模块,H SF (·) indicates that the image is processed using a shallow feature extraction module, I Lr 表示原始图像, 其中H、W以及C in Represent the length, width and number of channels of the original image respectively.

[0063] 深层特征图像的表达式为:

[0064] F DF =H CoNV (F N )

[0065] Among them, FDF 表示深层特征图像, 表示第i个通道变换注意力模块,F N represents the feature image generated by the Nth channel transformation attention module. Specifically, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·)表示一个卷积层;

[0066] F DF =H Df (F0)。

[0067] 其中,F0表示浅层特征图像,H DF 表示深层特征提取模块。

[0068] It can be understood that, in the embodiment of the present application, the original image I LR 经过浅层特征提取模块H SF , obtain the shallow feature image F0, and pass the shallow feature image F0 through the deep feature extraction module H DF 处理,获取深层特征图像F DF ,并且,深层特征图像F DF The feature image F generated by the Nth channel transformation attention module N 经过卷积层处理得到。

[0069] In the embodiment of the present application, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, and each channel transformation attention module is:

[0070]

[0071] F i,out =H CONV (F i,L )+F i,0

[0072] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out Indicates the feature image finally obtained by the channel transformation attention module, H CoNV (F i,L )表示通过卷积层生成的特征图,H CONV (F i,L ) and F i,0 通过一个残差跳跃链接获得F i,out .

[0073] It can be understood that each channel transformation attention module in the deep extraction module includes multiple channel transformation attention layers and a convolution layer, wherein the image input to the channel transformation attention layer module is F i,0 , the feature image F generated by the channel transformation attention layer module i,out is the feature map H generated by the convolution layer COMV (F i,L ) and F i,0 Obtained via a residual skip link.

[0074] In the embodiment of the present application, each channel transformation attention layer includes a channel-transformable self-attention operation unit and a feature diversity supplement unit. The function of the channel transformation attention layer is:

[0075]

[0076] Y=MLP(LN(X N ))+X N

[0077] Among them, X N represents the channel-transformable attention unit generating the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel-transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0078] It can be understood that in the embodiment of the present application, each channel-transformed attention layer includes a channel-transformed self-attention operation unit (S)C-MSA(·) and a feature diversity supplementation unit DWC(·). After processing by the above units and the multi-layer perceptron MLP(·), the channel-transformed attention operation unit generates an image X, which can be obtained from the above expression. N , and the feature image Y finally obtained by the channel transformation attention layer.

[0079] In step S104, a super-resolution reconstructed image is generated according to the super-resolution residual image and the enlarged image.

[0080] The super-resolution residual image is an image obtained by performing super-resolution processing on the shallow feature image and the deep feature image.

[0081] It is understandable that the embodiment of the present application can perform super-resolution processing on the shallow feature image and the deep feature image obtained in the above steps to obtain a super-resolution residual image, and generate a super-resolution reconstructed image with the enlarged image.

[0082] In the embodiment of the present application, the expression of the super-resolution reconstructed image is:

[0083] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0084] Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0085] It can be understood that, according to the above expression, the embodiment of the present application can use the image super-resolution module H Rec (·) For the shallow feature image F0 and the deep feature image F DF Perform super-resolution processing to obtain a super-resolution residual image, and compare it with the image interpolation operation H Int (·) The original image after enlargement I LR Reconstruct and obtain the final super-resolution image I RSR .

[0086] According to the satellite remote sensing image super-resolution method proposed in the embodiment of the present application, the original image of the target satellite remote sensing can be obtained and enlarged, the shallow feature image and the deep feature image of the original image can be extracted, and the shallow feature image and the deep feature image are super-resolution processed to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the enlarged image. Through the above method, a linearly growing computational amount and a larger receptive field can be used to generate images with higher resolution, richer information and greater accuracy.

[0087] The satellite remote sensing image super-resolution method is further described below through a specific embodiment.

[0088] This example uses the Venusstar-1 satellite as an example and proposes a super-resolution method for satellite remote sensing images based on a channel-transformed attention mechanism. Since Venusstar-1 can acquire 32-band hyperspectral images, it can capture detailed spectral information of ground objects, with a wide spectral range and rich band information. However, it is impossible to simultaneously achieve high spectral resolution and high spatial resolution. To further expand the use cases of Venusstar-1 satellite hyperspectral images, effective methods are needed to increase the resolution of the acquired images to facilitate subsequent visual interpretation, ground object classification, and other tasks. Super-resolution will be performed on Venusstar-1 multispectral data to improve the ability to extract and analyze image information, effectively solving the problems of artifact removal and detail restoration in the super-resolution process of Venusstar-1 satellite hyperspectral images.

[0089] This embodiment includes a shallow feature extraction module, a deep feature extraction module, and an image super-resolution module. The shallow feature extraction module consists of a convolutional neural network unit, which uses a 3×3 convolution kernel to perform shallow feature extraction on the original image to obtain a shallow feature image, and then directly transmits the shallow feature image to the image super-resolution module. The deep feature extraction module consists of multiple channel transformation attention modules and convolutional layers. Each channel transformation attention module contains multiple channel transformation attention layers and a convolutional layer. The channel transformation attention layer uses a channel transformable self-attention unit to realize intra-window autocorrelation calculation and cross-window interaction. The image super-resolution module is implemented using a convolutional neural network unit and a PixelShuffle function to process the shallow feature image and deep feature image obtained above, ultimately obtaining a high-quality super-resolution image.

[0090] like Figure 2 As shown, this embodiment first interpolates and amplifies the collected Venusstar-1 original image to the required size; then performs shallow feature extraction on the original image to obtain a shallow feature image; then performs deep feature extraction on the shallow feature image to obtain a deep feature image; super-resolution processing is performed on the obtained shallow feature image and deep feature image to obtain a super-resolution residual image; finally, the super-resolution residual image and the interpolated image of the original image are combined to obtain a super-resolution reconstructed image.

[0091] This embodiment generates a super-resolution residual image by using the feature information extracted by the channel transformation attention neural network and combines it with the original image to generate a super-resolution image. It can extract more detail information from the original image, better restore image details, and eliminate the generation of various artifacts, significantly improving the super-resolution reconstruction effect of low-resolution input images of satellite remote sensing.

[0092] Specifically, Figure 3 is a flow chart of the satellite remote sensing image super-resolution method based on the channel transformation attention mechanism provided in this embodiment. Figure 3As shown, the method includes the following steps:

[0093] Step 201: Acquire an original image of a target satellite remote sensing image, and enlarge the original image to a target size to obtain an enlarged image.

[0094] The original image is upscaled by bicubic linear interpolation according to a specified multiple using a specified function in the pytorch software to obtain an upscaled image. Specifically, this embodiment uses the nn.Upsample() function in the pytorch software to upscale the low-resolution original image by a specified multiple using bicubic linear interpolation.

[0095] Step 202: extract shallow features from the original image to obtain a shallow feature image;

[0096] Specifically, the original image (H, W and C in Representing the length, width and number of channels of the original image respectively) are input into the trained neural network model. First, the image enters the shallow feature extraction module H SF , this module uses a convolutional neural network unit with a convolution kernel of 3×3 to generate a shallow feature image to obtain the low-dimensional feature information of the original image. The objective function is:

[0097] F0=H SF (I LR )

[0098] F0 is the shallow feature image obtained (H, W and C represent the length, width and number of channels of the shallow feature image respectively).

[0099] It should be noted that the shallow feature extraction module uses a common convolutional neural network unit with a convolution kernel of 3X3 to generate a shallow feature extraction image, and the shallow feature image will be used as the input of the deep feature extraction module.

[0100] Step 203: extracting deep features from the shallow feature image to obtain a deep feature image;

[0101] Specifically, if Figure 4 As shown, the shallow feature image F0 is input into the deep feature extraction module H DF , through this module, a deep feature image F can be generated DF , obtain the high-dimensional feature information of the original image, and the objective function is:

[0102] F DF =H DF (F0)

[0103] This module consists of multiple channel transformation attention modules and a convolutional layer HCONV (·) is composed of the shallow feature image first entering the N channel transformation attention module in sequence, and finally passing through a convolution layer to generate the final deep feature image. Its function is expressed as:

[0104]

[0105] F DF =H CONV (F N )

[0106] in, represents the i-th channel transformation attention module, F i represents the feature image generated by the i-th channel transformation attention module.

[0107] The image input to the deep feature module first enters the channel transformation attention module for processing. Each channel transformation attention module is composed of multiple channel transformation attention layers and a convolution layer. This cascade structure can more effectively reuse multi-level features. The function expression of each channel transformation attention module is:

[0108]

[0109] F i,out =H CONV (F i,L )+F i,0

[0110] in represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,j Represents the feature image generated by the channel transformation attention layer. The channel transformation attention module finally obtains the feature image F i,out The input image F of the channel-transformed attention module is generated by the feature map generated by the convolution layer i,0 It is obtained through a residual skip link.

[0111] like Figure 5 As shown in the figure, the channel transformation attention layer contains a basic channel transformation self-attention layer, and adds a feature diversity unit to the channel self-attention operation unit to obtain the primary channel transformation attention feature map, and then regularizes and reconstructs the image to obtain the final channel transformation attention feature map. The function expression of each channel transformation attention layer is:

[0112]

[0113] Y=MLP(LN(X N ))+X N

[0114] Among them, LN(·) represents regularization processing, (S)C-MSA(·) represents that the channel can be transformed into an attention unit, such as Figure 6 As shown in Figure 2, two adjacent channel transformation attention layers form a channel transformation attention layer pair, in which the self-attention operation unit in the second channel transformation attention layer performs channel transformation operations and restores them after the operation to achieve cross-window interaction. DWC(·) represents the feature diversity unit. After obtaining the channel attention feature map and the feature diversity supplementary map, they are linked to the input image through residual jumps to generate a primary channel transformation attention feature map. The generated image is then regularized and placed in a multi-layer perceptron MLP(·) for reconstruction. X represents the input image, X represents the input image, and N Indicates that the spatial attention unit generates an image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0115] The deep feature extraction module constructed in this embodiment includes N channel transformation attention modules and a 3×3 convolutional layer. Each channel transformation attention module also includes M channel transformation attention layers and a 3×3 convolutional layer. The entire deep feature extraction module uses neural network layers with multiple receptive fields and a cascade structure to better extract feature information from the original image at multiple levels and restore more image details.

[0116] The channel-transformed attention layer consists of a channel-transformed spatial self-attention operation unit and a feature diversity supplementation unit. To transfer information across windows, two adjacent channel-transformed attention layers are paired together. The second channel-transformed attention layer performs a channel-transformation operation, and after the calculation is complete, the window is restored, effectively increasing the receptive field of the channel-transformed attention mechanism. When using channel-transformed attention for feature extraction, self-attention is first used to extract features from the channel dimension. Finally, the generated channel feature image and the input image are aggregated through residual skip links, regularized, and reconstructed to generate a deep feature image.

[0117] During the spatial self-attention operation, P is generated by inputting the image Q ,P K and P V Three matrices are used to calculate the autocorrelation between each pixel and all other pixels to generate a new image. Each pixel in the new image is composed of all other pixels related to it. The higher the similarity between two pixels, the greater the proportion of the new pixel. The value image obtained when the input image is self-attention calculated is supplemented with feature diversity from the spatial dimension. When performing feature diversity supplementation operation, the value image obtained by P is used. VThe generated value image is input into the depthwise separable convolutional layer to obtain a feature-diverse complementary image from the spatial dimension.

[0118] Step 204: Super-resolution processing is performed on the shallow feature image and the deep feature image to obtain a super-resolution residual image, and the super-resolution residual image is combined with the magnified image to obtain a super-resolution reconstructed image.

[0119] The shallow feature image and the deep feature image are aggregated through the residual jump link and input into the image super-resolution module to obtain the super-resolution residual image, which is then merged with the original image interpolation image to obtain the final super-resolution image I RSR . Its function is:

[0120] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0121] Among them H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0122] Next, a satellite remote sensing image super-resolution device according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0123] Figure 7 4 is a block diagram of a satellite remote sensing image super-resolution device according to an embodiment of the present application.

[0124] like Figure 7 As shown, the satellite remote sensing image super-resolution device 10 includes: an acquisition module 301 , an amplification module 302 , an extraction module 303 and a reconstruction module 304 .

[0125] Among them, the acquisition module 301 is used to obtain the original image of the target satellite remote sensing; the enlargement module 302 is used to enlarge the original image to the target size to obtain the enlarged image; the extraction module 303 is used to extract the shallow feature image and the deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; the reconstruction module 304 is used to generate a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

[0126] In the embodiment of the present application, the extraction module 303 is further configured to input the original image into a shallow feature extraction module, and output a shallow feature image from the shallow feature extraction module, wherein the shallow feature extraction module comprises a convolutional neural network unit; and input the shallow feature image into a deep feature extraction module, and output a deep feature image from the deep feature extraction module, wherein the deep feature extraction module comprises a channel transformation attention module and a convolutional layer.

[0127] In the embodiment of the present application, the expression of the shallow feature image is as follows:

[0128] F0=H SF LR

[0129] wherein F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) represents processing the image using the shallow feature extraction module, I LR represents the original image, wherein H, W and C in respectively represent the length, width and channel number of the original image.

[0130] The expression of the deep feature image is as follows:

[0131] F DF =H CONV (F N )

[0132] wherein F DF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the N-th channel transformation attention module, and specifically, F i represents the feature image generated by the i-th channel transformation attention module, H CONV (·) represents a convolutional layer;

[0133] F DF =H DF (F0)。

[0134] wherein F0 represents the shallow feature image, H DF represents the deep feature extraction module.

[0135] In the embodiment of the present application, each channel transformation attention module comprises a plurality of channel transformation attention layers and a convolutional layer, and each channel transformation attention module is as follows:

[0136]

[0137] F​​i,out =H CONV (F i,L )+F i,0

[0138] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out Indicates the feature image finally obtained by the channel transformation attention module, H CONV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

[0139] In the embodiment of the present application, each channel transformation attention layer includes a channel-transformable self-attention operation unit and a feature diversity supplement unit. The function of the channel transformation attention layer is:

[0140]

[0141] Y=MLP(LN(X N ))+X N

[0142] Among them, X N represents the channel-transformable attention unit generating the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel-transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0143] In the embodiment of the present application, the expression of the super-resolution reconstructed image is:

[0144] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0145] Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0146] It should be noted that the aforementioned explanation of the embodiment of the satellite remote sensing image super-resolution method is also applicable to the satellite remote sensing image super-resolution device of this embodiment, and will not be repeated here.

[0147] According to the satellite remote sensing image super-resolution device proposed in the embodiment of the present application, through the coordinated action of the acquisition module, the amplification module, the extraction module and the reconstruction module, the original image of the target satellite remote sensing is acquired and amplified, the shallow feature image and the deep feature image of the original image are extracted, and the shallow feature image and the deep feature image are super-resolution processed to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the amplified image. Through the above method, a linearly growing computational amount and a larger receptive field can be used to generate images with higher resolution, richer information and greater accuracy.

[0148] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0149] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0150] When the processor 402 executes the program, the satellite remote sensing image super-resolution method provided in the above embodiment is implemented.

[0151] Furthermore, the electronic device further includes:

[0152] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0153] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0154] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0155] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0156] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0157] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0158] An embodiment of the present application further provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the above-mentioned satellite remote sensing image super-resolution method is implemented.

[0159] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed, implements the above-mentioned satellite remote sensing image super-resolution method.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0161] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0162] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0163] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the method: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0164] A person skilled in the art may understand that all or part of the steps carried out in the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0165] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A satellite remote sensing image super-resolution method, characterized in that: The following steps are involved: Obtain original images of target satellite remote sensing; Enlarging the original image to a target size to obtain an enlarged image; Extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image, wherein extracting the deep feature image of the original image includes inputting the shallow feature image into a deep feature extraction module, and the deep feature extraction module outputs the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolution layer, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, each channel transformation attention layer includes a channel-transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is: Y=MLP(LN(X N ))+x N Among them, X N represents the channel-transformable attention unit generating the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel-transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer; A super-resolution reconstructed image is generated according to the super-resolution residual image and the enlarged image.

2. The satellite remote sensing image super-resolution method according to claim 1, characterized in that: The extracting of the shallow feature image and the deep feature image of the original image includes: The original image is input into a shallow feature extraction module, and the shallow feature extraction module outputs the shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit.

3. The satellite remote sensing image super-resolution method according to claim 1 or 2, characterized in that: The expression of the shallow feature image is: F0=H SF (I LR ) Among them, F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) indicates that the image is processed using a shallow feature extraction module, I LR represents the original image, Among them, W, H and C in Respectively represent the length, width and number of channels of the original image; The expression of the deep feature image is: F DF =H CONV (F N ) Among them, F DF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the Nth channel transformation attention module, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·) represents a convolutional layer; F DF =H DF (F0). Among them, F0 represents the shallow feature image, H DF Represents the deep feature extraction module.

4. The satellite remote sensing image super-resolution method according to claim 1, characterized in that: The per-channel transformation attention module is: F i,out =H CoNV (F i,L )+F i,0 Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out Indicates the feature image finally obtained by the channel transformation attention module, H CONV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

5. The satellite remote sensing image super-resolution method according to claim 1, characterized in that: The expression of the super-resolution reconstructed image is: I RSR =H Rec (F0+F DF )+H Int (I LR ) Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, c Int (·) represents the original image interpolation operation.

6. A satellite remote sensing image super-resolution device, characterized in that: include: Acquisition module, used to obtain the original image of target satellite remote sensing; an enlargement module, configured to enlarge the original image to a target size to obtain an enlarged image; An extraction module is used to extract a shallow feature image and a deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image, wherein extracting the deep feature image of the original image includes inputting the shallow feature image into a deep feature extraction module, and the deep feature extraction module outputs the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolution layer, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, each channel transformation attention layer includes a channel-transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is: Y=MLP(LN(X N ))+X N Among them, X N represents the channel-transformable attention unit generating the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel-transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer; A reconstruction module is used to generate a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the satellite remote sensing image super-resolution method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the satellite remote sensing image super-resolution method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the satellite remote sensing image super-resolution method according to any one of claims 1 to 5 is implemented.

Images