A scale-frequency dual-phase modulation remote sensing image continuous super-resolution method and device

Abstract

The present application relates to a kind of scale frequency biphase modulation remote sensing image continuous super-resolution method and device, belong to remote sensing image processing technical field.The method is fused with the shallow feature that encoder extracts and scale information by scale modulation feature fusion module, to obtain the feature of different levels.Then the feature that scale modulation feature fusion module outputs is recovered as the super-resolution image of specified scale by frequency modulation reconstruction module through frequency domain information modulation.

Description

Technical Field

[0001] This invention relates to a method and apparatus for continuous super-resolution of scale-frequency biphase modulation remote sensing images, belonging to the field of remote sensing image processing technology. Background Technology

[0002] Due to limitations in atmospheric conditions and imaging equipment, remote sensing images acquired directly from satellites often have very low resolution. Therefore, super-resolution technologies that can reconstruct high-resolution remote sensing images are becoming increasingly important. As is well known, change detection technology plays a crucial role in smart agriculture, emergency management, public safety, and resource exploration.

[0003] Traditional super-resolution methods operate at a fixed and single scale, limiting their applicability. First, from the perspective of remote sensing image acquisition, image signals in nature are continuous, while the imaging process of remote sensing images is actually discrete sampling of continuous signals. Therefore, super-resolution techniques used for remote sensing image reconstruction should also be continuous. Second, the demands of downstream tasks for different remote sensing image resolutions render single-scale super-resolution models insufficient. Furthermore, remote sensing images with different resolutions can depict the texture and structure of objects at different levels, and continuous-scale super-resolution can acquire image information at different levels. Therefore, developing a remote sensing image super-resolution network capable of continuous and arbitrary-scale super-resolution is crucial. The application of continuous-scale remote sensing image super-resolution methods is becoming increasingly widespread.

[0004] Currently, deep learning algorithms, due to their powerful feature representation capabilities, have been widely applied in remote sensing image super-resolution tasks. Existing deep learning-based continuous-scale super-resolution methods mainly employ image super-resolution methods based on implicit neural representations. Implicit neural representation-based methods can achieve continuous arbitrary-scale super-resolution using a single network. They convert the image into a continuous representation in the latent space, and use coordinate sampling methods to obtain images of different resolutions according to the different scale requirements of the super-resolution task. Current continuous-scale super-resolution methods mainly consist of two modules: a feature extraction module and an upsampling reconstruction module. The feature extraction module takes the pixel coordinates and nearby pixel information in the two-dimensional spatial context as input to extract deep feature representations. The upsampling reconstruction module predicts and outputs the RGB values ​​corresponding to specific coordinate positions, thereby achieving continuous arbitrary-scale image super-resolution. The vast majority of current continuous-scale super-resolution methods adopt this architecture.

[0005] However, current methods treat feature extraction and upsampling reconstruction in isolation, failing to organically integrate these two modules within the network. On one hand, current methods fail to recognize the different levels of feature requirements at different scales in the super-resolution process, neglecting the crucial role of scale factors in feature extraction. They only use scale factors in upsampling reconstruction, employing the same feature extraction module for different scales, thus failing to extract different levels of information based on the scale of the super-resolution task, limiting further performance improvements in continuous-scale super-resolution tasks. On the other hand, methods based on implicit neural representations use MLPs to learn the continuous representation of images in the implicit space. Due to the global optimization strategy of MLPs, they tend to obtain the most average result during reconstruction, losing a significant amount of high-frequency information. This results in overly smoothed and blurred super-resolution images, losing texture and edge information. Compared to natural images, remote sensing images have more complex distributions and richer details and textures, making the utilization of high-frequency information particularly important. Current methods do not acknowledge the crucial role of frequency domain information. To obtain a more effective continuous-scale super-resolution model for remote sensing images, these key issues need to be addressed. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of existing technologies, namely the rigidity of feature extraction methods and the loss of high-frequency information in current continuous scale super-resolution methods for remote sensing images, and to propose a scale-frequency biphase modulation method for continuous super-resolution of remote sensing images.

[0007] The technical solution for realizing the present invention is as follows:

[0008] In a first aspect, this application provides a scale-frequency biphasic modulation remote sensing image super-resolution method. The applicable network includes an encoder, a scale-modulation feature fusion module, a frequency-modulation reconstruction module, and a convolutional layer. The specific process of this method is as follows:

[0009] The first step is to extract low-resolution remote sensing image I using an encoder. LR The encoded features θ0 are then converted into network-aware scale information s based on the set super-resolution scale factor. embedded ;

[0010] The second step involves using a scale modulation feature fusion module to process the encoded features θ0 and scale information s. embedded Multi-scale convolution and fusion are performed to obtain a feature map θ′ containing scale information. Pooling is then performed on θ′ to obtain channel information θ of the multi-level features. c Then the scale information s embedded As a weight for channel modulation and the channel information θ c The fusion is performed to obtain Θ;

[0011] The third step involves using a frequency modulation reconstruction module to extract features from Θ through two 3x3 convolutional branches and performing coordinate-based upsampling to obtain features Θ0 and Θ0 containing corresponding positional information. f Incorporate the location information of nearby coordinates into Θ f Θ′ is obtained from f ; for the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm ; to the Θ′ f After periodic activation, then by Θ fm Modulation is performed to obtain the feature Θ after frequency domain information modulation. F The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M ;

[0012] Fourth step, regarding the Θ M And Θ, using convolutional layers to aggregate information within cells to obtain the final super-resolution remote sensing image I. SR ;

[0013] Fifth step, based on the high-resolution reference image I HR and super-resolution remote sensing images I SR The loss function is calculated and used to update the network parameters. After the network training is completed, the trained network is used to generate super-resolution remote sensing images.

[0014] Furthermore, in the second step of this invention, the scale information is used as a weight for channel modulation along with the channel information θ. c The result of fusion is Θ as follows:

[0015] θ s =L(s) embedded )

[0016] Θ=f 3d→d (θ′⊙(θ c ⊙θ s ))+s embedded

[0017] Where, θ s f represents the scale information obtained through the linear layer L for channel modulation. 3d→d Θ represents the convolutional layer for channel-dimensional feature fusion, Θ represents the output of the scale-modulated feature fusion module, and ⊙ represents element-wise feature multiplication.

[0018] Furthermore, in the second step of this invention, the encoded features θ0 and scale information s are... embeddedMulti-scale convolution and fusion are performed to obtain a feature map θ′ containing scale information. The specific process is as follows:

[0019] The scale modulation feature fusion module uses multi-scale depth convolutional layers with kernel sizes of 3, 5, and 7 to perform convolutions, and then concatenates the convolution results, as follows:

[0020] θ3=f 3×3 (θ0+s embedded )

[0021] θ5=f 5×5 (θ0+s embedded )

[0022] θ7=f 7×7 (θ0+s embedded )

[0023] θ′=Concat(θ3, θ5, θ7)

[0024] Where θ3, θ5, and θ7 represent the features extracted from different levels by convolutional layers with kernel sizes of 3, 5, and 7, respectively, f 3×3 f 5×5 and f 7×7 θ represents a convolutional layer with kernel sizes of 3, 5, and 7. Concat means connecting features at different levels along the channel dimension, and θ′ represents a feature map containing scale information.

[0025] Furthermore, in the third step, the present invention utilizes the nearest latent vectors to predict the RGB values ​​at a specified coordinate position, incorporating the positional information of the nearest coordinates into the Θ. f Θ′ is obtained from f :

[0026] Θ′ f =Θ f +f FC (cell)

[0027] Where cell represents the location information of neighboring coordinates, f FC This represents a fully connected layer that extracts neighboring latent vectors, Θ′ f This indicates a feature that includes information about neighboring coordinates.

[0028] Furthermore, in the third step of the present invention, the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm The specific process is as follows:

[0029] For Θ′ f Perform a Fourier transform to obtain Θ fft ;

[0030] Through pooling operations and linear layers, Θ fft Convert to a concise representation suitable for modulation:

[0031] Θ fm =L′(Pooling(Θ) fft ))

[0032] Where, Θ fm The weights used for frequency modulation are represented by Pooling(), which represents the pooling operation, and L′ represents the linear layer.

[0033] Furthermore, in the third step of this invention, the feature Θ after frequency domain information modulation is obtained. F for:

[0034]

[0035] Where, Θ F This represents the features after modulation by frequency domain information. period() represents the periodic activation function, Real() represents the operation of extracting the real part, Imag() represents the operation of extracting the imaginary part, and Concat represents the concatenation operation.

[0036] Furthermore, in the third step of this invention, the modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​are predicted using MLP, as detailed below:

[0037] Θ′ F =f 2d→d (Θ F )

[0038] Θ m =Θ0⊙Θ′ F

[0039] Θ M =f mlp (Θ m )

[0040] Among them, f 2d→d This represents the dimensionality reduction of the modulated channel, Θ′ F Represents the features after dimensionality reduction, Θ m f represents a feature containing initial information and frequency modulation information. mlp Used to learn the continuous representation of remote sensing images in implicit space, Θ M This indicates the preliminary forecast result.

[0041] Furthermore, the fourth step of this invention utilizes convolutional layers to aggregate information within cells to obtain the final super-resolution remote sensing image I. SR for:

[0042]

[0043] Where UP(Θ) represents the reconstruction result using the most original features as reference constraints, f final This represents a 3x3 convolutional layer, where S represents the area enclosed by the four nearest neighboring coordinate points. cell This represents the area corresponding to each coordinate point.

[0044] Furthermore, the loss function described in this invention is:

[0045]

[0046] Where r represents the super-resolution scale, and H and W represent the height and width of the input image, respectively. Represents super-resolution remote sensing image I SR In (x i y j ) pixel value, Represents high-resolution reference image I HR In (x i y j The pixel value of ).

[0047] Secondly, an embodiment of this application provides a scale-frequency biphasic modulation remote sensing image super-resolution device, comprising an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module, and a convolutional layer.

[0048] Encoder for extracting low-resolution remote sensing images I LR The encoded features θ0 are then converted into network-aware scale information s based on the set super-resolution scale factor. embedded ;

[0049] The scale modulation feature fusion module is used to fuse the encoded features θ0 and scale information s. embedded Multi-scale convolution and fusion are performed to obtain a feature map θ′ containing scale information. Pooling is then performed on θ′ to obtain channel information θ of the multi-level features. c Then the scale information s embedded As a weight for channel modulation and the channel information θ c The fusion is performed to obtain Θ;

[0050] The frequency modulation reconstruction module is used to extract features from Θ through two 3*3 convolutional branches and perform coordinate-based upsampling to obtain features Θ0 and Θ0 containing corresponding position information. f Incorporate the location information of nearby coordinates into Θ f Θ′ is obtained from f ; for the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm; to the Θ′ f After periodic activation, then by Θ fm Modulation is performed to obtain the feature Θ after frequency domain information modulation. F The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M ;

[0051] Convolutional layers, used for Θ M The final super-resolution remote sensing image I is obtained by aggregating information within the Θ cell. SR .

[0052] Beneficial effects:

[0053] First, this invention first fuses the shallow features extracted by the encoder with scale information through a scale modulation feature fusion module to obtain features at different levels. Then, a frequency modulation reconstruction module uses frequency domain information modulation to restore the features output by the scale modulation feature fusion module to a super-resolution image at a specified scale. The scale modulation feature fusion module deeply integrates scale information into the feature learning process, enabling the network to obtain corresponding intermediate features based on different scale factors. The frequency modulation reconstruction module uses frequency domain information obtained through Fourier transform to adaptively modulate the image reconstruction process, thereby preserving a large amount of high-frequency information and improving the clarity of the super-resolution image.

[0054] Second, extensive experiments have shown that the method of this invention achieves superior results compared to other continuous super-resolution methods for remote sensing images. The method proposed in this invention provides an effective technical means for remote sensing image super-resolution. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 Here is a diagram illustrating the overall framework of a scale-frequency biphase modulation method for continuous super-resolution of remote sensing images:

[0057] Figure 2 Schematic diagram of scale modulation feature fusion module:

[0058] Figure 3 This is a schematic diagram of the frequency modulation reconstruction module. Detailed Implementation

[0059] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0060] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0061] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0062] The design concept of this invention is as follows: A scale-modulation feature fusion module integrates scale factors with a multi-level feature extraction network, enabling the network to perceive super-resolution scale information. It extracts features at different levels according to the requirements of different scale factors and performs adaptive weighted fusion along the channel dimension, allowing the network to extract intermediate features aligned with specific scale factors. Furthermore, a frequency modulation reconstruction module incorporates frequency domain information into the upsampling reconstruction process through Fourier transform, enabling the network to learn frequency domain information and modulate features after periodic activation. This ensures that the network fully preserves high-frequency information during upsampling reconstruction, resulting in a super-resolution image with clear texture and edges.

[0063] This application's embodiment describes a scale-frequency biphase modulation remote sensing image continuous super-resolution method. Taking the practical application scenario of continuous super-resolution of remote sensing images as an example, the overall framework diagram of this method is shown below. Figure 1 As shown, the steps of this method include:

[0064] The first step is to extract low-resolution remote sensing image I using an encoder. LR The encoded features θ0 are then converted into network-aware scale information s based on the set super-resolution scale factor. embedded .

[0065] This step first requires encoding the low-resolution image into a form that the network can understand. The low-resolution remote sensing image is fed into the encoder to extract shallow features for subsequent processing. The encoder uses the RCAN encoder part, and the corresponding encoded feature map is shown below:

[0066] θ0=Φencoder (I LR )

[0067] Among them, I LR Φ represents the input low-resolution remote sensing image. encoder The encoder represents the initial processing of the low-resolution image input, and θ0 represents the encoded feature map output by the encoder. The encoder has 12 RCABs. Afterwards, to incorporate the scale factor into the network, an embedding operation is needed. Sinusoidal embedding can convert scale information into a continuous and bounded representation. Due to its excellent performance in learnability and periodicity, sinusoidal embedding is chosen for embedding scale information, as follows:

[0068] s embedded =Φ embedding (Sinu(r))

[0069] Where r represents the scale factor of the super-resolution, Sinu represents the sinusoidal embedding, and Φ embedding This represents the initial processing of the embedded scale information, consisting of a three-layer fully connected network, s embedded This represents the embedded scale information that the network can understand.

[0070] In this step, the calculation method for the sinusoidal embedding method is as follows:

[0071]

[0072]

[0073] Where d represents the dimension of the scale modulation feature fusion module, and Sinou(r) (i) This represents the value of Sinu(r) at the i-th position, and N is set to 10000.

[0074] The second step involves using a scale modulation feature fusion module to process the encoded features θ0 and scale information s. embedded Multi-scale convolution and fusion are performed to obtain a feature map θ′ containing scale information. Pooling is then performed on θ′ to obtain channel information θ of the multi-level features. c Then the scale information s embedded As a weight for channel modulation and the channel information θ c The Θ is obtained by fusion.

[0075] like Figure 2 As shown, this step will combine the encoded low-resolution remote sensing image feature map θ0 obtained in the first step with the scale information s that the network can perceive. embeddedThe data is fed into the scale-modulated feature fusion module. This module adaptively learns weights modulated by a scale factor to embed scale information into feature learning, fusing features at different levels. Therefore, this module can extract features at different levels for super-resolution tasks with varying scale requirements. Because neural networks of different depths contain feature information at different levels, convolutional kernels of different sizes with different receptive fields can extract feature information at different levels. Furthermore, super-resolution tasks at different scales require the use of features at different levels. To achieve this, the scale-modulated feature fusion module employs multi-scale depth convolutions with kernel sizes of 3, 5, and 7, and concatenates them as follows:

[0076] θ3=f 3×3 (θ0+s embedded )

[0077] θ5=f 5×5 (θ0+s embedded )

[0078] θ7=f 7×7 (θ0+s embedded )

[0079] θ′=Concat(θ3, θ5, θ7)

[0080] Where θ3, θ5, and θ7 represent the features extracted from different levels by convolutional layers with kernel sizes of 3, 5, and 7, respectively, f 3×3 f 5×5 and f 7×7 This indicates convolutional layers with kernel sizes of 3, 5, and 7. `Concat` represents connecting features at different levels along the channel dimension, and `θ′` represents a feature map containing scale information. In this embodiment, this step will be repeated three times to fully extract information at different levels according to the scale factor requirements.

[0081] Perform average pooling on θ′ to obtain multi-level feature channel information:

[0082] θ c =AvgPooling(θ′)

[0083] Where, θ c This represents the channel information of the feature map aggregation, and AvgPooling represents the average pooling operation.

[0084] Since the scale factor does not contain any spatial information, it is unnecessary for it to participate extensively in pixel-level operations. Scale information operations are maintained at the channel level, and the scale factor is deeply integrated into the network as a weight for channel modulation, as follows:

[0085] θ s =L(s)embedded )

[0086] Θ=f 3d→d (θ′⊙(θ c ⊙θ s ))+s embedded

[0087] Where, θ s f represents the scale information obtained through the linear layer L for channel modulation. 3d→d Θ represents the convolutional layer for channel-dimensional feature fusion, and Θ represents the output of the scale-modulated feature fusion module. Through the above operations, the scale factor is fully integrated into the feature learning process, enabling the network to comprehensively perceive scale information and modulate the weights of feature fusion at different levels according to the requirements of the super-resolution task.

[0088] The third step involves using a frequency modulation reconstruction module to extract features from Θ through two 3x3 convolutional branches and performing coordinate-based upsampling to obtain features Θ0 and Θ0 containing corresponding positional information. f Incorporate the location information of nearby coordinates into Θ f Θ′ is obtained from f ; for the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm ; to the Θ′ f After periodic activation, then by Θ fm Modulation is performed to obtain the feature Θ after frequency domain information modulation. F The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M .

[0089] like Figure 3 As shown, the output Θ and super-resolution coordinates from the scale-modulation feature fusion module obtained in the second step are fed into the frequency-modulation reconstruction module. First, a convolutional layer with a kernel of 3 is used to distinguish the features for subsequent operations. Then, the features are upsampled based on the super-resolution coordinates to obtain features containing corresponding location information, as detailed below:

[0090] Θ0=UP grid (f 3×3 1 (Θ), coord)

[0091] Θ f =UP grid (f 3×3 2 (Θ), coord)

[0092] Among them, f3×3 1 f represents the convolutional layer used for preprocessing the backbone branch to preserve the original information. 3×3 2 This represents the convolutional layer used for frequency modulation branch preprocessing, where Θ0 represents the initial features containing coordinate information, and Θ f Indicates the feature that is about to undergo Fourier modulation, UP grid This indicates upsampling based on a super-resolution coordinate grid.

[0093] A local ensemble strategy is employed to avoid checkerboard artifacts by using neighboring latent vectors to predict the RGB values ​​at specified coordinates. Therefore, local spatial information is incorporated into the Θ function. f middle:

[0094] Θ′ f =Θ f +f FC (cell)

[0095] Where cell represents the location information of neighboring coordinates, f FC This represents a fully connected layer that extracts neighboring latent vectors.

[0096] In order to obtain the frequency domain information of the features, for Θ′ f Perform a Fourier transform to obtain Θ fft Since the features after Fourier transform are completely different from the original features at the pixel level, the feature frequency domain information is used for modulation in the channel dimension.

[0097] In this step, Θ′ f The calculation process for performing the Fourier transform is as follows:

[0098]

[0099] Where, Θ i Let (x, y) and (u, v) represent the features of the i-th channel, respectively. fft The point in the middle.

[0100] Through average pooling and two linear layers, Θ fft It is converted into a concise representation suitable for modulation:

[0101] Θ fm =L′(AvgPooling(Θ) fft ))

[0102] Where, Θ fm This represents the weights used for frequency modulation. Using a periodic activation function helps the network learn high-frequency information, therefore Θ′ is... f After periodic activation, then by Θ fmModulation is performed. Periodic activation functions are chosen to be cosine and sinine, and modulated separately by the real and imaginary parts of the Fourier transform, as follows:

[0103]

[0104] Where, Θ F This represents the features modulated by frequency domain information. The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M .

[0105] Θ′ F =f 2d→d (Θ F )

[0106] Θ m =Θ0⊙Θ′ F

[0107] Θ M =f mlp (Θ m )

[0108] Among them, f 2d→d This represents the dimensionality reduction of the modulated channel, Θ′ F Represents the features after dimensionality reduction, Θ m f represents a feature containing initial information and frequency modulation information. mlp Used to learn the continuous representation of remote sensing images in implicit space, Θ M This indicates the preliminary forecast result.

[0109] Fourth step, regarding the Θ M And Θ, using convolutional layers to aggregate information within cells to obtain the final super-resolution remote sensing image I. SR ;

[0110]

[0111] Where UP(Θ) represents the reconstruction result using the most original features as reference constraints, f final This represents a 3x3 convolutional layer, where S represents the area enclosed by the four nearest neighboring coordinate points. cell This represents the area corresponding to each coordinate point.

[0112] Fifth, during network training, calculate the loss function and update the network parameters. The loss function chosen is L1 loss.

[0113]

[0114] After the network training is completed, the trained network is used to generate super-resolution remote sensing images.

[0115] In another embodiment of this application, a scale-frequency biphase modulation remote sensing image super-resolution device includes an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module, and a convolutional layer.

[0116] Encoder for extracting low-resolution remote sensing images I HR The encoded features θ0 are then converted into network-aware scale information s based on the set super-resolution scale factor. embedded ;

[0117] The scale modulation feature fusion module is used to fuse the encoded features θ0 and scale information s. embedded Multi-scale convolution and fusion are performed to obtain a feature map θ′ containing scale information. Pooling is then performed on θ′ to obtain channel information θ of the multi-level features. c Then the scale information s embedded As a weight for channel modulation and the channel information θ c The fusion is performed to obtain Θ;

[0118] The frequency modulation reconstruction module is used to extract features from Θ through two 3*3 convolutional branches and perform coordinate-based upsampling to obtain features Θ0 and Θ0 containing corresponding position information. f Incorporate the location information of nearby coordinates into Θ f Θ′ is obtained from f ; for the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm ; to the Θ′ f After periodic activation, then by Θ fm Modulation is performed to obtain the feature Θ after frequency domain information modulation. F The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M ;

[0119] Convolutional layers, used for Θ M The final super-resolution remote sensing image I is obtained by aggregating information within the Θ cell. SR .

[0120] Taking the practical application scenario of continuous super-resolution of remote sensing images as an example, on a publicly available remote sensing dataset, a comparison of various existing continuous super-resolution methods is shown in Tables 1 and 2. It can be seen that the method proposed in the embodiments has better accuracy.

[0121] Table 1 compares various existing continuous super-resolution methods and implementations on the WHU-RS19 dataset (intra-training scale).

[0122]

[0123] Table 2 compares various existing continuous super-resolution methods and implementation examples on the WHU-RS19 dataset (outside of training scale).

[0124]

[0125]

[0126] The specific embodiments of the invention are described herein, but the scope of protection of the invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should be included within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A scale-frequency dual-phase modulation remote sensing image super-resolution method, characterized in that, The specific process is as follows: First, the encoder is used to extract the coding features θ0 of the low-resolution remote sensing image I LR , and convert the set super-resolution scale factor into network perceptible scale information s embedded ; Second step, using scale modulation feature fusion module to encode features θ0and scale information s embedded Multi-scale convolution and fusion are performed to obtain a feature map θ' containing scale information, and a pooling operation is performed on θ' to obtain channel information θ of multi-level features c The scale information s embedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ; The third step involves using a frequency modulation reconstruction module to extract features from Θ through two 3x3 convolutional branches and performing coordinate-based upsampling to obtain features Θ0 and Θ0 containing corresponding positional information. f Incorporate the location information of nearby coordinates into Θ f Θ′ is obtained from f ; for the Θ′ f Perform a Fourier transform to obtain Θ fft And based on this, the weight Θ used for frequency modulation is calculated. fm ; to the Θ′ f After periodic activation, then by Θ fm Modulation is performed to obtain the feature Θ after frequency domain information modulation. F The modulated features are incorporated into the original features as an additional information enhancement branch, and then the RGB values ​​Θ are predicted using MLP. M ; In the third step, the modulated features are added to the original features as additional information enhancement branches, and then the RGB values are predicted through the MLP, as follows: Θ' F = f 2d→d (Θ F ) Θ m = Θ0Θ' F Θ M = f mlp (Θ m ) where f 2d→d represents the modulated channel dimension reduction, Θ′ F represents the reduced dimension feature, Θ m represents the feature containing initial information and frequency modulation information, f mlp for learning the continuous expression of remote sensing images in the implicit space, Θ M represents the result of preliminary prediction; In the fourth step, the position information of the adjacent coordinates in the cell is aggregated by using the convolution layer to obtain the final super-resolution remote sensing image I M and Θ, and the information in the cell is aggregated by using the convolution layer to obtain the final super-resolution remote sensing image I SR ; In the fifth step, based on the high-resolution reference image I HR and the super-resolution remote sensing image I SR , a loss function is calculated, and the network parameter is updated by using the loss function; and when the network training is completed, the super-resolution remote sensing image is generated by using the trained network.

2. The scale-frequency dual-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that, In the second step the scale information is used as a weight for modulating the channels of the channel information Θ c The fusion is done to get Θ as: θ s = L(s embedded ) Θ = f 3d→d (θ' ⊙ (θ c ⊙ θ s ))+ s embedded where θ s denotes the scale information for channel modulation obtained by the linear layer L, f 3d→d denotes the convolution layer for channel dimension feature fusion, Θ denotes the output of the scale modulation feature fusion module, and ⊙ denotes element-wise multiplication of features.

3. The method of claim 1, wherein the method further comprises: In the second step, the encoded feature θ0 and the scale information s embedded Multi-scale convolution and fusion are performed to obtain a feature map θ' containing scale information, and the specific process is as follows: The scale modulation feature fusion module adopts multi-scale deep convolution layers with convolution kernel sizes of 3, 5, and 7 for convolution, and the convolution results are connected, as follows: θ3 = f 3×3 (θ0 + s embedded ) θ5 = f 5×5 (θ0 + s embedded ) θ7 = f 7×7 (θ0 + s embedded ) θ' = Concat(θ3, θ5, θ7) wherein θ3, θ5 and θ7 represent different levels of features extracted by convolutional layers with kernel size of 3, 5 and 7 respectively, f 3×3 , f 5×5 and f 7×7 represent convolutional layers with kernel size of 3, 5 and 7 respectively, Concat represents connecting different levels of features in the channel dimension, and θ' represents a feature map containing scale information.

4. The method of claim 1, wherein, In the third step, the RGB value of the specified coordinate position is predicted using the neighboring latent vectors, and the position information of the neighboring coordinates is incorporated into Θ f to obtain Θ' f : Θ' f = Θ f + f FC (cell) where cell denotes the position information of the neighboring coordinates, f FC denotes a fully connected layer that extracts the neighboring potential vectors, Θ′ f denotes a feature that contains the neighboring coordinate information.

5. The method of claim 1, wherein the method further comprises: In the third step, the Θ' is calculated based on the Θ f The Fourier transform is performed to obtain Θ fft And based on it, the weight Θ for frequency modulation is calculated fm The specific process is as follows: Θ' f Fourier transform of Θ fft ; Θ is converted to a succinct representation suitable for modulation by a pooling operation and linear layers: fft Θ = Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ + Θ Θ fm = L'(Pooling(Θ fft )) where Θ fm denotes the weights used for frequency modulation, Pooling() denotes a pooling operation, and L' denotes a linear layer.

6. The method of claim 1, wherein the method further comprises: In the third step, the feature Θ modulated by the frequency domain information is obtained F is: wherein Θ F denotes the feature modulated by the frequency domain information, period() denotes a periodic activation function, Real() denotes a real part operation, Imag() denotes an imaginary part operation, and Concat denotes a concatenation operation.

7. The method of claim 1, wherein the method further comprises: The fourth step aggregates information within the cell using a convolutional layer to obtain the final super-resolution remote sensing image I SR is: wherein UP(Θ) represents the reconstruction result constrained by the most original features, f final represents a 3*3 convolution layer, S represents the area surrounded by four adjacent coordinate points, S cell represents the area corresponding to each coordinate point.

8. The method of claim 1, wherein the method further comprises: The loss function is as follows: where r denotes the super-resolution scale, H and W resolution denote the height and width of the input image, denotes the super-resolved remote sensing image I SR at the pixel value of the point (x i ,y j ), denotes the high-resolution reference image I HR at the pixel value of the point (x i ,y j ).

9. A scale-frequency dual-phase modulation remote sensing image super-resolution device, characterized in that, It includes an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module, and a convolution layer. an encoder configured to extract a coding feature θ0 of a low-resolution remote sensing image I LR and convert a set super-resolution scale factor into network perceptible scale information s embedded ; Scale a modulation feature fusion module, configured to modulate the coding feature θ0 and the scale information s embedded perform multi-scale convolution and fusion to obtain a feature map θ' containing scale information, and perform a pooling operation on the θ' to obtain channel information θ of a multi-level feature c , and then modulate the scale information s embedded as a weight of channel modulation and the channel information θ c to obtain Θ; a frequency modulation reconstruction module, configured to extract features of the Θ through two 3*3 convolution branches and perform coordinate position-based up-sampling to obtain features Θ0 and Θ containing corresponding position information f , and the position information of the adjacent coordinates is included in the Θ f to obtain Θ' f ; Fourier transform is performed on the Θ' f to obtain Θ fft , and a weight Θ for frequency modulation is calculated based on the Θ fm ; the Θ' f is modulated by the Θ fm after periodic activation to obtain features Θ modulated in the frequency domain F ; the modulated features are included in the original features as additional information enhancement branches, and the RGB value Θ is predicted through the MLP M ; The modulated features are added to the original features as additional information enhancement branches, and then the RGB values are predicted through the MLP, as follows: Θ' F = f 2d→d (Θ F ) Θ m = Θ0Θ' F Θ M = f mlp (Θ m ) where f 2d→d represents the modulated channel dimension reduction, Θ′ F represents the reduced dimension feature, Θ m represents the feature containing initial information and frequency modulation information, f mlp is used to learn the continuous expression of remote sensing images in the implicit space, Θ M represents the result of preliminary prediction; a convolutional layer for aggregating the location information of the neighboring coordinates for Θ M and Θ to obtain the final super-resolution remote sensing image I SR .

Images