Satellite remote sensing image super-resolution method, device, equipment, medium and product

Abstract

The invention relates to the technical field of remote sensing image processing, in particular to a satellite remote sensing image super-resolution method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an original remote sensing image of a target satellite; amplifying the original image to a target size to obtain an amplified image; extracting a shallow-layer feature image and a deep-layer feature image of the original image, and performing super-resolution processing on the shallow-layer feature image and the deep-layer feature image to obtain a super-resolution residual image; and generating a super-resolution reconstructed image according to the super-resolution residual image and the amplified image. According to the method, the calculated amount of linear growth can be utilized, and the image with higher resolution and richer and more accurate information can be obtained. Therefore, the problems of calculated amount index expansion, receptive field limitation, detail reduction and the like of a traditional attention mechanism super-resolution algorithm in a satellite remote sensing multispectral image processing process are solved.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a satellite remote sensing image super-resolution method, device, electronic equipment and storage medium. Background Art

[0002] In the current field of satellite remote sensing image processing, using deep learning to obtain super-resolution images is one of the most popular directions. Neural networks are usually used to obtain higher-resolution generated images from low-resolution original images that meet usage requirements and are consistent with real scenes. This method has the characteristics of low cost and easy implementation.

[0003] Correspondingly, super-resolution algorithms based on self-attention mechanisms have been a hot topic in recent years. As an alternative to convolutional neural networks, the emerging self-attention mechanism neural network has achieved excellent performance by calculating the autocorrelation of each element in the global receptive field. However, this type of method has the following shortcomings when directly applied to the field of satellite remote sensing image processing:

[0004] (1) Traditional self-attention mechanisms for common image super-resolution algorithms with few channel dimensions (such as red, green, and blue channels) mostly operate from the spatial dimension. This cannot fully utilize the multi-channel information of satellite multispectral remote sensing images, limits the potential of feature extraction of the attention mechanism, and cannot effectively restore more details of satellite remote sensing images.

[0005] (2) The multi-head attention operation strategy adopted by the traditional self-attention mechanism requires dividing the channel into multiple unrelated parts, which in turn reduces the receptive field of the channel attention algorithm and affects the accuracy of the super-resolution algorithm.

[0006] (3) The rank of the autocorrelation matrix of the channel attention mechanism is smaller than that of the spatial attention mechanism, which results in the feature diversity obtained by the channel attention feature extraction module being lower than that of the spatial attention algorithm, limiting the expressive power of the algorithm based on the channel attention mechanism. Summary of the invention

[0007] The present application provides a satellite remote sensing image super-resolution method, device, electronic device and storage medium to solve the problems of exponential expansion of computational complexity, limited receptive field and detail restoration in the traditional attention mechanism super-resolution algorithm during the satellite remote sensing multispectral image processing process.

[0008] The first aspect of the present application provides a satellite remote sensing image super-resolution method, comprising the following steps: acquiring an original image of a target satellite remote sensing; enlarging the original image to a target size to obtain an enlarged image; extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; and generating a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

[0009] Optionally, extracting a shallow feature image and a deep feature image of an original image includes: inputting the original image into a shallow feature extraction module, the shallow feature extraction module outputting the shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; inputting the shallow feature image into a deep feature extraction module, the deep feature extraction module outputting the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolutional layer.

[0010] Optionally, the expression of the shallow feature image is:

[0011] F0=H SF (I LR )

[0012] Among them, F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) indicates that the image is processed using a shallow feature extraction module, I LR represents the original image, Among them, H, W and C in Respectively represent the length, width and number of channels of the original image.

[0013] The expression of deep feature image is:

[0014] F DF =H CONV (F N )

[0015] Among them, F DF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the Nth channel transformation attention module. Specifically, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·) indicates a convolutional layer;

[0016] F DF =H DF (F0).

[0017] Among them, F0 represents the shallow feature image, HDF Represents the deep feature extraction module.

[0018] Optionally, each channel transform attention module includes multiple channel transform attention layers and a convolution layer, and each channel transform attention module is:

[0019]

[0020] F i,out =H CONV (F i,L )+F i,0

[0021] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out It indicates that the channel transformation attention module finally obtains the feature image, H CONV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

[0022] Optionally, each channel transformation attention layer includes a channel transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is:

[0023]

[0024] Y = MLP(LN(X N ))+X N

[0025] Among them, X N represents the channel transformable attention unit to generate the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0026] Optionally, the expression of the super-resolution reconstructed image is:

[0027] I RSR =H Rec (F0+FDF )+H Int (I LR )

[0028] Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0029] The second aspect of the present application provides a satellite remote sensing image super-resolution device, including: an acquisition module, used to acquire the original image of the target satellite remote sensing; an enlargement module, used to enlarge the original image to a target size to obtain an enlarged image; an extraction module, used to extract a shallow feature image and a deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; a reconstruction module, used to generate a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

[0030] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the satellite remote sensing image super-resolution method of the first aspect.

[0031] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed, the satellite remote sensing image super-resolution method of the first aspect is implemented.

[0032] The fifth aspect of the present application provides a computer program product, including a computer program or instructions, which, when executed, implements the satellite remote sensing image super-resolution method of the first aspect.

[0033] Therefore, this application includes the following beneficial effects:

[0034] The embodiment of the present application can obtain the original image of the target satellite remote sensing and enlarge it, extract the shallow feature image and deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the enlarged image. The above method can utilize the linearly growing amount of calculation and a larger receptive field to generate a higher resolution, more information-rich and accurate image. Thus, the problems of exponential expansion of the amount of calculation, limited receptive field and detail restoration in the traditional attention mechanism super-resolution algorithm in the process of satellite remote sensing multispectral image processing are solved.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a process for super-resolution of satellite remote sensing images provided according to an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a processing process of a super-resolution image reconstruction process provided by an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a flow chart of a satellite remote sensing image super-resolution method based on a channel transformation attention mechanism provided in one embodiment of the present application;

[0040] Figure 4 A schematic diagram of a deep feature extraction module provided for one embodiment of the present application;

[0041] Figure 5 A schematic diagram of a channel transformation attention layer structure provided for an embodiment of the present application;

[0042] Figure 6 A schematic diagram of channel conversion provided for an embodiment of the present application;

[0043] Figure 7 This is a block diagram of an exemplary device for super-resolution of satellite remote sensing images provided according to an embodiment of the present application;

[0044] Figure 8 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0046] The following describes the satellite remote sensing image super-resolution method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the problems of exponential expansion of computational amount, limited receptive field and detail restoration in the traditional attention mechanism super-resolution algorithm mentioned in the above background technology in the process of satellite remote sensing multispectral image processing, the present application provides a satellite remote sensing image super-resolution method, in which the original image of the target satellite remote sensing can be obtained and enlarged, the shallow feature image and the deep feature image of the original image can be extracted, and the shallow feature image and the deep feature image can be super-resolution processed to obtain a super-resolution residual image, and finally a super-resolution reconstructed image is generated according to the super-resolution residual image and the enlarged image. The above method can generate a higher resolution, richer information and more accurate image by using a linearly growing computational amount and a larger receptive field. As a result, the problems of exponential expansion of computational amount, limited receptive field and detail restoration in the traditional attention mechanism super-resolution algorithm in the process of satellite remote sensing multispectral image processing are solved.

[0047] Specifically, Figure 1 A schematic diagram of a flow chart of a satellite remote sensing image super-resolution method provided in an embodiment of the present application.

[0048] like Figure 1 As shown, the satellite remote sensing image super-resolution method includes the following steps:

[0049] In step S101, an original image of a target satellite remote sensing is acquired.

[0050] Among them, the original images of target satellite remote sensing are hyperspectral images, which can capture the fine spectral information of ground objects, have a large spectral range and rich band information, but high spectral resolution and high spatial resolution cannot be obtained at the same time.

[0051] It can be understood that the embodiments of the present application can obtain the original hyperspectral image of the target satellite remote sensing and convert it into a high-resolution image through the following method.

[0052] In step S102, the original image is enlarged to a target size to obtain an enlarged image.

[0053] The method of enlarging the original image is interpolation enlargement; the target size is a size value set based on actual needs and is not specifically limited here.

[0054] It is understandable that the embodiment of the present application can enlarge the original image to the required target size through the interpolation enlargement method to obtain the enlarged image.

[0055] In step S103, a shallow feature image and a deep feature image of the original image are extracted, and super-resolution processing is performed on the shallow feature image and the deep feature image to obtain a super-resolution residual image.

[0056] Among them, the shallow feature image is an image obtained after shallow feature extraction is performed on the original image, and the deep feature image is an image obtained after deep feature extraction is performed on the shallow feature image.

[0057] It can be understood that the embodiment of the present application can perform shallow feature extraction on the original image to obtain a shallow feature image, then perform deep feature extraction on the shallow feature image to obtain a deep feature image, and finally perform super-resolution processing on the two to obtain a super-resolution residual image.

[0058] In an embodiment of the present application, a shallow feature image and a deep feature image of an original image are extracted, including: inputting the original image into a shallow feature extraction module, the shallow feature extraction module outputs a shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; inputting the shallow feature image into a deep feature extraction module, the deep feature extraction module outputs a deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolutional layer.

[0059] It can be understood that the embodiment of the present application performs shallow feature extraction on the original image through a shallow feature extraction module, and the shallow feature extraction module includes a convolutional neural network unit. The original image is input into the shallow feature extraction module, and the shallow feature extraction module can output a shallow feature image; after obtaining the shallow feature image, the deep feature extraction module is used to perform deep feature extraction on the shallow feature image. The deep feature extraction module includes a channel transformation attention module and a convolution layer. The shallow feature image is input into the deep feature extraction module, and the deep feature extraction module outputs a deep feature image.

[0060] In the embodiment of the present application, the expression of the shallow feature image is:

[0061] F0=H SF (I LR )

[0062] Among them, F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) indicates that the image is processed using a shallow feature extraction module, I Lr represents the original image, Among them, H, W and C in Respectively represent the length, width and number of channels of the original image.

[0063] The expression of deep feature image is:

[0064] F DF =H CoNV (F N )

[0065] Among them, FDF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the Nth channel transformation attention module. Specifically, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·) indicates a convolutional layer;

[0066] F DF =H Df (F0).

[0067] Among them, F0 represents the shallow feature image, H DF Represents the deep feature extraction module.

[0068] It can be understood that, in the embodiment of the present application, the original image I LR After the shallow feature extraction module H SF , obtain the shallow feature image F0, and pass the shallow feature image F0 through the deep feature extraction module H DF Processing, obtain deep feature image F DF , and the deep feature image F DF The feature image F generated by the Nth channel transformation attention module N It is obtained through convolutional layer processing.

[0069] In the embodiment of the present application, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, and each channel transformation attention module is:

[0070]

[0071] F i,out =H CONV (F i,L )+F i,0

[0072] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out It indicates that the channel transformation attention module finally obtains the feature image, H CoNV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

[0073] It can be understood that each channel transformation attention module in the deep extraction module includes multiple channel transformation attention layers and a convolution layer, wherein the image input to the channel transformation attention layer module is F i,0 , the feature image F generated by the channel transformation attention layer module i,out is the feature map H generated by the convolutional layer COMV (F i,L ) and F i,0 Obtained via a residual skip link.

[0074] In the embodiment of the present application, each channel transformation attention layer includes a channel transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is:

[0075]

[0076] Y = MLP(LN(X N ))+X N

[0077] Among them, X N represents the channel transformable attention unit to generate the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0078] It can be understood that in the embodiment of the present application, each channel transformation attention layer includes a channel transformable self-attention operation unit (S)C-MSA(·) and a feature diversity supplement unit DWC(·). After being processed by the above units and the multi-layer perceptron MLP(·), the channel transformable attention operation unit generates an image X, which can be obtained from the above expression. N , and the feature image Y finally obtained by the channel transformation attention layer.

[0079] In step S104, a super-resolution reconstructed image is generated according to the super-resolution residual image and the enlarged image.

[0080] The super-resolution residual image is an image obtained by performing super-resolution processing on the shallow feature image and the deep feature image.

[0081] It is understandable that the embodiment of the present application can perform super-resolution processing on the shallow feature image and the deep feature image obtained in the above steps to obtain a super-resolution residual image, and generate a super-resolution reconstructed image with the enlarged image.

[0082] In the embodiment of the present application, the expression of super-resolution reconstructed image is:

[0083] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0084] Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0085] It can be understood that, according to the above expression, the embodiment of the present application can use the image super-resolution module H Rec (·) For the shallow feature image F0 and the deep feature image F DF Perform super-resolution processing to obtain a super-resolution residual image, and compare it with the image interpolation operation H Int (·) The original image after enlargement I LR Reconstruct and obtain the final super-resolution image I RSR .

[0086] According to the satellite remote sensing image super-resolution method proposed in the embodiment of the present application, the original image of the target satellite remote sensing can be obtained and enlarged, the shallow feature image and the deep feature image of the original image can be extracted, and the shallow feature image and the deep feature image are super-resolution processed to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the enlarged image. Through the above method, a linearly growing amount of calculation and a larger receptive field can be used to generate images with higher resolution, richer information and more accuracy.

[0087] The satellite remote sensing image super-resolution method is further described below through a specific embodiment.

[0088] This embodiment takes the Venus-1 satellite as an example and proposes a satellite remote sensing image super-resolution method based on the channel transformation attention mechanism. Since Venus-1 can obtain 32-band hyperspectral images, it can capture the fine spectral information of ground objects, with a large spectral range and rich band information. However, it is impossible to obtain high spectral resolution and high spatial resolution at the same time. In order to further expand the use scenarios of the hyperspectral images of the Venus-1 satellite, an effective method is needed to improve the resolution of the images obtained, which is conducive to subsequent visual interpretation, ground object classification and other tasks. Super-resolution will be performed on the multispectral data of Venus-1, which can improve the ability of image information extraction and analysis, and effectively solve the problems of artifact removal and detail restoration in the super-resolution process of the hyperspectral images of the Venus-1 satellite.

[0089] This embodiment includes a shallow feature extraction module, a deep feature extraction module and an image super-resolution module. Among them, the shallow feature extraction module is composed of a convolutional layer neural network unit, and uses a 3×3 convolution kernel to perform shallow feature extraction on the original image to obtain a shallow feature image, and the shallow feature image is directly transmitted to the image super-resolution module; the deep feature extraction module is composed of multiple channel transformation attention modules and convolutional layers, each channel transformation attention module contains multiple channel transformation attention layers and a convolutional layer, and the channel transformation attention layer uses a channel transformable self-attention unit to realize the autocorrelation calculation within the window and the cross-window interaction; the image super-resolution module is implemented using a convolutional layer neural network unit and a PixelShuffle function, and the shallow feature image and deep feature image obtained previously are processed to finally obtain a high-quality super-resolution image.

[0090] like Figure 2 As shown, this embodiment first interpolates and amplifies the collected Venusstar-1 original image to the required size; then performs shallow feature extraction on the original image to obtain a shallow feature image; then performs deep feature extraction on the shallow feature image to obtain a deep feature image; super-resolution is performed on the obtained shallow feature image and deep feature image to obtain a super-resolution residual image; finally, the super-resolution residual image and the interpolated image of the original image are combined to obtain a super-resolution reconstructed image.

[0091] This embodiment generates a super-resolution residual image by using the feature information extracted by the channel transformation attention neural network and combines the original image to generate a super-resolution image. It can extract more detail information from the original image, better restore the image details, and eliminate the generation of various artifacts, which significantly improves the super-resolution reconstruction effect of satellite remote sensing low-resolution input images.

[0092] Specifically, Figure 3 is a flow chart of a satellite remote sensing image super-resolution method based on a channel transformation attention mechanism provided in this embodiment. Figure 3As shown, the method comprises the following steps:

[0093] Step 201: Acquire the original image of the target satellite remote sensing, and enlarge the original image to the target size to obtain the enlarged image.

[0094] The specified function in the pytorch software is used to enlarge the original image by bicubic linear interpolation according to the specified multiple to obtain the enlarged image. Specifically, this embodiment uses the nn.Upsample() function in the pytorch software to enlarge the low-resolution original image by bicubic linear interpolation at a specified multiple.

[0095] Step 202: extract shallow features from the original image to obtain a shallow feature image;

[0096] Specifically, the original image (H, W and C in Representing the length, width and number of channels of the original image respectively) are input into the trained neural network model. First, the image enters the shallow feature extraction module H SF , this module uses a convolutional neural network unit with a convolution kernel of 3×3 to generate a shallow feature image to obtain the low-dimensional feature information of the original image. The objective function is:

[0097] F0=H SF (I LR )

[0098] F0 is the shallow feature image obtained (H, W and C represent the length, width and number of channels of the shallow feature image respectively).

[0099] It should be noted that the shallow feature extraction module uses a common convolutional neural network unit with a convolution kernel of 3X3 to generate a shallow feature extraction image, and the shallow feature image will be used as the input of the deep feature extraction module.

[0100] Step 203: extracting deep features from the shallow feature image to obtain a deep feature image;

[0101] Specifically, Figure 4 As shown, the shallow feature image F0 is input into the deep feature extraction module H DF , through this module, a deep feature image F can be generated DF , obtain the high-dimensional feature information of the original image, and the objective function is:

[0102] F DF =H DF (F0)

[0103] This module consists of multiple channel transformation attention modules and a convolutional layer HCONV (·), the shallow feature image first enters the N channel transformation attention module in sequence, and finally passes through a convolution layer to generate the final deep feature image. Its function is expressed as:

[0104]

[0105] F DF =H CONV (F N )

[0106] in, represents the i-th channel transformation attention module, F i Denotes the feature image generated by the i-th channel transformation attention module.

[0107] The image input to the deep feature module first enters the channel transformation attention module for processing. Each channel transformation attention module is composed of multiple channel transformation attention layers and a convolution layer. This cascade structure can more effectively reuse multi-level features. The function expression of each channel transformation attention module is:

[0108]

[0109] F i,out =H CONV (F i,L )+F i,0

[0110] in represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,j Represents the feature image generated by the channel transformation attention layer. The channel transformation attention module finally obtains the feature image F i,out The input image F of the channel-transformed attention module is composed of the feature map generated by the convolution layer and the channel-transformed attention module. i,0 It is obtained through a residual skip link.

[0111] like Figure 5 As shown in the figure, the channel transformation attention layer contains a basic channel transformable self-attention layer, and adds a feature diversity unit to the channel self-attention operation unit to obtain the primary channel transformation attention feature map, and then regularizes and reconstructs the image to obtain the final channel transformation attention feature map. The function expression of each channel transformation attention layer is:

[0112]

[0113] Y = MLP(LN(X N ))+X N

[0114] Among them, LN(·) represents regularization processing, (S)C-MSA(·) means that the channel can be transformed from the attention unit, such as Figure 6 As shown in Figure 1, two adjacent channel transformation attention layers form a channel transformation attention layer pair, in which the self-attention operation unit in the second channel transformation attention layer performs channel transformation operations and restores them after the operation to achieve cross-window interaction. DWC(·) represents the feature diversity unit. After obtaining the channel attention feature map and the feature diversity supplementary map, they are linked to the input image through residual skipping to generate a primary channel transformation attention feature map. The generated image is then regularized and placed in a multi-layer perceptron MLP(·) for reconstruction. X represents the input image, X represents the input image, and N represents the spatial attention unit generating an image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0115] The deep feature extraction module constructed in this embodiment includes N channel transformation attention modules and a 3X3 convolution layer. Each channel transformation attention module includes M channel transformation attention layers and a 3X3 convolution layer. The entire deep feature extraction module uses a variety of neural network layers and cascade structures with a variety of receptive field ranges to better extract the feature information of the original image at multiple levels and restore more image details.

[0116] The channel transformation attention layer is composed of a channel-transformable spatial self-attention operation unit and a feature diversity supplement unit. In order to transfer information across windows, two adjacent channel transformation attention layers form a pair, and the channel transformation operation is performed in the second channel transformation attention layer, and the window is restored after the calculation is completed, thereby effectively increasing the receptive field of the channel attention mechanism. When using channel transformation attention for feature extraction, features are first extracted from the channel dimension through self-attention, and finally the generated channel feature image and the input image are aggregated through residual jump links, regularized and reconstructed to generate a deep feature image.

[0117] In the spatial self-attention operation, P is generated by inputting the image Q ,P K and P V Three matrices are used to calculate the autocorrelation of each pixel with all other pixels to generate a new image. Each pixel in the new image is composed of all other pixels related to it. The higher the similarity between two pixels, the greater the proportion of the new pixel. Then, the value image obtained when the input image is self-attention calculated is supplemented with feature diversity from the spatial dimension. When performing feature diversity supplementation operations, the P VThe generated value image is input into a deep separable convolutional layer to obtain a feature-diverse complementary image from the spatial dimension.

[0118] Step 204: Perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image, and combine the super-resolution residual image with the enlarged image to obtain a super-resolution reconstructed image.

[0119] The shallow feature image and the deep feature image are aggregated through residual jump links and input into the image super-resolution module to obtain a super-resolution residual image, which is then merged with the original image interpolation image to obtain the final super-resolution image I RSR . Its function is:

[0120] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0121] Among them, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0122] Next, the satellite remote sensing image super-resolution device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0123] Figure 7 It is a block diagram of a satellite remote sensing image super-resolution device according to an embodiment of the present application.

[0124] like Figure 7 As shown, the satellite remote sensing image super-resolution device 10 includes: an acquisition module 301, an amplification module 302, an extraction module 303 and a reconstruction module 304.

[0125] Among them, the acquisition module 301 is used to acquire the original image of the target satellite remote sensing; the enlargement module 302 is used to enlarge the original image to the target size to obtain the enlarged image; the extraction module 303 is used to extract the shallow feature image and the deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; the reconstruction module 304 is used to generate a super-resolution reconstructed image based on the super-resolution residual image and the enlarged image.

[0126] In an embodiment of the present application, the extraction module 303 is further used to: input the original image into a shallow feature extraction module, the shallow feature extraction module outputs a shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; input the shallow feature image into a deep feature extraction module, the deep feature extraction module outputs a deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolutional layer.

[0127] In the embodiment of the present application, the expression of the shallow feature image is:

[0128] F0=H SF (I LR )

[0129] Among them, F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) indicates that the image is processed using a shallow feature extraction module, I LR represents the original image, Among them, H, W and C in Respectively represent the length, width and number of channels of the original image.

[0130] The expression of deep feature image is:

[0131] F DF =H CONV (F N )

[0132] Among them, F DF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the Nth channel transformation attention module. Specifically, F i Denotes the feature image generated by the i-th channel transformation attention module, H CONV (·) indicates a convolutional layer;

[0133] F DF =H DF (F0).

[0134] Among them, F0 represents the shallow feature image, H DF Represents the deep feature extraction module.

[0135] In the embodiment of the present application, each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer, and each channel transformation attention module is:

[0136]

[0137] Fi,out =H CONV (F i,L )+F i,0

[0138] Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out It indicates that the channel transformation attention module finally obtains the feature image, H CONV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

[0139] In the embodiment of the present application, each channel transformation attention layer includes a channel transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is:

[0140]

[0141] Y = MLP(LN(X N ))+X N

[0142] Among them, X N represents the channel transformable attention unit to generate the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

[0143] In the embodiment of the present application, the expression of super-resolution reconstructed image is:

[0144] I RSR =H Rec (F0+F DF )+H Int (I LR )

[0145] Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

[0146] It should be noted that the aforementioned explanation of the satellite remote sensing image super-resolution method embodiment is also applicable to the satellite remote sensing image super-resolution device of this embodiment, and will not be repeated here.

[0147] According to the satellite remote sensing image super-resolution device proposed in the embodiment of the present application, through the coordinated action of the acquisition module, the amplification module, the extraction module and the reconstruction module, it is possible to acquire and amplify the original image of the target satellite remote sensing, extract the shallow feature image and the deep feature image of the original image, and perform super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image. Finally, a super-resolution reconstructed image is generated based on the super-resolution residual image and the amplified image. Through the above method, a linearly growing amount of calculation and a larger receptive field can be used to generate images with higher resolution, richer information and more accuracy.

[0148] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0149] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0150] When the processor 402 executes the program, the satellite remote sensing image super-resolution method provided in the above embodiment is implemented.

[0151] Furthermore, the electronic device also includes:

[0152] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0153] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0154] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0155] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0156] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0157] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0158] An embodiment of the present application also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the above-mentioned satellite remote sensing image super-resolution method is implemented.

[0159] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed, implements the above-mentioned satellite remote sensing image super-resolution method.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0161] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0162] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0163] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0164] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0165] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A satellite remote sensing image super-resolution method, characterized in that: The following steps are involved: Obtain original images of target satellite remote sensing; Enlarging the original image to a target size to obtain an enlarged image; Extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; A super-resolution reconstructed image is generated according to the super-resolution residual image and the enlarged image.

2. The satellite remote sensing image super-resolution method according to claim 1, characterized in that: The step of extracting the shallow feature image and the deep feature image of the original image comprises: Inputting the original image into a shallow feature extraction module, the shallow feature extraction module outputs the shallow feature image, wherein the shallow feature extraction module includes a convolutional neural network unit; The shallow feature image is input into a deep feature extraction module, and the deep feature extraction module outputs the deep feature image, wherein the deep feature extraction module includes a channel transformation attention module and a convolution layer.

3. The satellite remote sensing image super-resolution method according to claim 1 or 2, characterized in that: The expression of the shallow feature image is: F0=H SF (I LR ) Among them, F0 represents the shallow feature image, H SF represents the shallow feature extraction module, H SF (·) indicates that the image is processed using a shallow feature extraction module, I LR represents the original image, Among them, H, W and C in Respectively represent the length, width and number of channels of the original image. The expression of the deep feature image is: F DF =H CONV (F N ) Among them, F DF represents the deep feature image, represents the i-th channel transformation attention module, F N represents the feature image generated by the Nth channel transformation attention module, F i represents the feature image generated by the i-th channel transformation attention module, H CONV (·) indicates a convolutional layer; F DF =H DF (F0). Among them, F0 represents the shallow feature image, H DF Represents the deep feature extraction module.

4. The satellite remote sensing image super-resolution method according to claim 2, characterized in that: Each channel transformation attention module includes multiple channel transformation attention layers and a convolution layer. Each channel transformation attention module is: F i,out =H CONV (F i,L )+F i,0 Among them, F i,j represents the feature image generated by the channel transformation attention layer, represents the jth channel transformation attention layer in the i-th channel transformation attention module, F i,0 represents the input image of the channel transformation attention module, F i,out It indicates that the channel transformation attention module finally obtains the feature image, H CoNV (F i,L ) represents the feature map generated by the convolutional layer, H CONV (F i,L ) and F i,0 Obtain F through a residual skip link i,out .

5. The satellite remote sensing image super-resolution method according to claim 4, characterized in that: Each channel transformation attention layer includes a channel transformable self-attention operation unit and a feature diversity supplement unit, and the function of the channel transformation attention layer is: Y=MLP(LN(X N ))+X N Among them, X N represents the channel transformable attention unit to generate the image, LN(·) represents the regularization process, (S)C-MSA(·) represents the channel transformable self-attention unit, DWC(·) represents the feature diversity supplement unit, MLP(·) represents the multi-layer perceptron, X represents the input image, Represents X N The value image generated by the value matrix, Y represents the feature image obtained by the channel transformation attention layer.

6. The satellite remote sensing image super-resolution method according to claim 1, characterized in that: The expression of the super-resolution reconstructed image is: I RSR =H Rec (F0+F DF )+H Int (I LR ) Among them, I RSR represents the final super-resolution image, H Rec (·) represents the image super-resolution module, H Int (·) represents the original image interpolation operation.

7. A satellite remote sensing image super-resolution device, characterized in that: include: An acquisition module is used to acquire the original image of the target satellite remote sensing; An enlargement module, used for enlarging the original image to a target size to obtain an enlarged image; An extraction module, used for extracting a shallow feature image and a deep feature image of the original image, and performing super-resolution processing on the shallow feature image and the deep feature image to obtain a super-resolution residual image; A reconstruction module is used to generate a super-resolution reconstructed image according to the super-resolution residual image and the enlarged image.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the satellite remote sensing image super-resolution method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the satellite remote sensing image super-resolution method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the satellite remote sensing image super-resolution method according to any one of claims 1 to 6 is implemented.

Images