Scale frequency biphase modulation remote sensing image continuous super-resolution method and device

Abstract

The invention relates to a scale frequency biphase modulation remote sensing image continuous super-resolution method and device, and belongs to the technical field of remote sensing image processing. According to the method, shallow layer features extracted by an encoder are fused with scale information through a scale modulation feature fusion module, so that features of different levels are obtained. And then, a frequency modulation reconstruction module is used for recovering the features output by the scale modulation feature fusion module into a super-resolution image with a specified scale through frequency domain information modulation.

Description

Technical Field

[0001] The invention relates to a scale-frequency bi-phase modulation remote sensing image continuous super-resolution method and device, belonging to the technical field of remote sensing image processing. Background Art

[0002] Due to the limitations of the atmospheric environment and imaging equipment, the resolution of remote sensing images directly obtained from satellites is often very low. Therefore, super-resolution technology that can restore high-resolution remote sensing images has become increasingly important. As we all know, change detection technology has played an important role in smart agriculture, emergency management, public safety, and resource exploration.

[0003] Traditional super-resolution methods are performed at a fixed and single scale, which limits their scope of application. First, from the perspective of remote sensing image acquisition, image signals in nature are continuous, and the imaging process of remote sensing images is actually discrete sampling of continuous signals. Therefore, super-resolution techniques for remote sensing image reconstruction should also be continuous. Secondly, the demand for different remote sensing image resolutions in downstream tasks makes single-scale remote sensing image super-resolution models insufficient. In addition, remote sensing images with different resolutions can depict the texture and structure of objects at different levels in the image, and continuous-scale super-resolution can obtain image information at different levels. Therefore, it is crucial to develop a remote sensing image super-resolution network that can perform continuous and arbitrary-scale super-resolution. The application of continuous-scale remote sensing image super-resolution methods is becoming more and more common.

[0004] At present, deep learning algorithms have been widely used in remote sensing image super-resolution tasks due to their powerful feature representation capabilities. Existing continuous-scale super-resolution methods based on deep learning mainly use image super-resolution methods based on implicit neural representation. The method based on implicit neural representation can achieve continuous arbitrary-scale super-resolution through a single network. It converts the image into a continuous representation of the latent space. According to the different scale requirements of the super-resolution task, the coordinate sampling method is used to obtain images of different resolutions. The current continuous-scale super-resolution method mainly consists of two modules: a feature extraction module and an upsampling reconstruction module. The feature extraction module takes the coordinates of the pixel and the nearby pixel information in the two-dimensional space context as input to extract deep feature representation. The upsampling reconstruction module predicts and outputs the RGB value corresponding to the specific coordinate position, thereby achieving continuous arbitrary-scale image super-resolution. Most of the current continuous-scale super-resolution methods use this architecture.

[0005] However, current methods view the two modules of feature extraction and upsampling reconstruction in isolation, without organically combining the two modules in the network. On the one hand, current methods do not recognize that the super-resolution process of different scale factors has different levels of feature requirements, ignore the key role of scale factors in feature extraction, only use scale factors in upsampling reconstruction, and use the same feature extraction module for different scale factors. It is impossible to extract different levels of information according to different scales of super-resolution tasks, which limits the further improvement of the performance of continuous-scale super-resolution tasks. On the other hand, the implicit neural expression-based method uses MLP to learn the continuous expression of images in implicit space. Due to the global optimization strategy of MLP, it tends to obtain the most average results in the reconstruction process, and a large amount of high-frequency information is lost, resulting in the obtained super-resolution result image being overly smoothed and blurred, and texture and edge information is lost. Compared with natural images, remote sensing images have more complex distributions and richer details and textures, so the use of high-frequency information is particularly important. Current methods do not pay attention to the important role of frequency domain information. In order to obtain a better continuous-scale super-resolution model for remote sensing images, the above key issues need to be solved. Summary of the invention

[0006] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, namely the rigid feature extraction method and high-frequency information loss problems existing in the current remote sensing image continuous scale super-resolution method, and to propose a scale-frequency bi-phase modulation remote sensing image continuous super-resolution method.

[0007] The technical solutions to realize the present invention are as follows:

[0008] In the first aspect, the embodiment of the present application provides a scale-frequency bi-phase modulation remote sensing image super-resolution method, the network applicable to the method includes an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module and a convolution layer, and the specific process of the method is:

[0009] The first step is to use the encoder to extract the low-resolution remote sensing image I LR The encoding feature θ 0 , and convert the set super-resolution scale factor into network-perceivable scale information s embedded ;

[0010] In the second step, the scale modulation feature fusion module is used to fuzzy the encoding feature θ 0 and scale information embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s embedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ;

[0011] In the third step, the frequency modulation reconstruction module is used to extract features of the θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain the feature θ containing the corresponding position information. 0 and θ f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the said Θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulation F ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M ;

[0012] The fourth step is to M And Θ, the convolutional layer is used to aggregate the information in the cell to obtain the final super-resolution remote sensing image I SR ;

[0013] Step 5: Based on the high-resolution reference image I HR and super-resolution remote sensing images I SR , calculate the loss function, and use the loss function to update the network parameters; when the network training is completed, use the trained network to generate super-resolution remote sensing images.

[0014] Furthermore, in the second step of the present invention, the scale information is used as the weight of the channel modulation and the channel information θ c After fusion, we get Θ as:

[0015] θ s =L(s embedded )

[0016] Θ=f 3d→d (θ′⊙(θ c ⊙θ s ))+s embedded

[0017] Among them, θ s represents the scale information for channel modulation obtained by the linear layer L, f 3d→d represents the convolutional layer of channel dimension feature fusion, Θ represents the output of the scale modulation feature fusion module, and ⊙ represents the element-by-element multiplication of features.

[0018] Furthermore, in the second step of the present invention, the encoding feature θ 0and scale information embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information. The specific process is:

[0019] The scale modulation feature fusion module uses a multi-scale deep convolution layer with convolution kernel sizes of 3, 5, and 7 for convolution, and connects the convolution results, as follows:

[0020] θ 3 =f 3×3 (θ 0 +s embedded )

[0021] θ 5 =f 5×5 (θ 0 +s embedded )

[0022] θ 7 =f 7×7 (θ 0 +s embedded )

[0023] θ′=Concat(θ 3 ,θ 5 ,θ 7 )

[0024] Among them, θ 3 ,θ 5 and θ 7 They represent the features of different levels extracted by the convolutional layers with convolution kernel sizes of 3, 5, and 7, respectively. 3×3 、f 5×5 and f 7×7 It represents the convolutional layers with kernel sizes of 3, 5, and 7. Concat means connecting features of different levels in the channel dimension. θ′ represents the feature map containing scale information.

[0025] Furthermore, in the third step, the present invention uses the adjacent potential vector to predict the RGB value of the specified coordinate position, and incorporates the position information of the adjacent coordinates into θ f Get Θ′ f :

[0026] Θ′ f =Θ f +f FC (cell)

[0027] Among them, cell represents the location information of the adjacent coordinates, f FC represents the fully connected layer that extracts the adjacent latent vector, Θ′ f Represents a feature that contains nearby coordinate information.

[0028] Furthermore, in the third step of the present invention, the θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm , the specific process is:

[0029] For θ′ f Perform Fourier transform to get Θ fft ;

[0030] Through the pooling operation and the linear layer, Θ fft Convert to a compact representation suitable for modulation:

[0031] Θ fm =L′(Pooling(Θ fft ))

[0032] Among them, Θ fm represents the weight used for frequency modulation, Pooling() represents the pooling operation, and L′ represents the linear layer.

[0033] Further, in the third step of the present invention, the characteristic θ modulated by the frequency domain information is obtained. F for:

[0034]

[0035] Among them, Θ F It represents the features after frequency domain information modulation, period() represents the periodic activation function, Real() represents the real part operation, Imag() represents the imaginary part operation, and Concat represents the concatenation operation.

[0036] Furthermore, in the third step of the present invention, the modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​are predicted by MLP, as follows:

[0037] Θ′ F =f 2d→d (Θ F )

[0038] Θ m =Θ 0 ⊙Θ′ F

[0039] Θ M =f mlp (Θ m )

[0040] Among them, f 2d→d represents the channel dimension reduction after modulation, Θ′ F represents the features after dimensionality reduction, Θ m represents the feature containing the initial information and the frequency modulation information, fmlp Used to learn the continuous expression of remote sensing images in implicit space, Θ M Represents the results of preliminary predictions.

[0041] Furthermore, the fourth step of the present invention uses the convolutional layer to aggregate the information in the cell to obtain the final super-resolution remote sensing image I SR for:

[0042]

[0043] Among them, UP(Θ) means that the most original features are used as the reference constraint reconstruction result, f final represents a 3*3 convolutional layer, S represents the area enclosed by four adjacent coordinate points, S cell Indicates the area corresponding to each coordinate point.

[0044] Furthermore, the loss function of the present invention is:

[0045]

[0046] Where r represents the super-resolution scale, H and W represent the height and width of the input image. Represents the super-resolution remote sensing image I SR In (x i ,y j ) pixel value, Represents the high-resolution reference image I HR In (x i ,y j )’s pixel value.

[0047] In a second aspect, an embodiment of the present application provides a scale-frequency bi-phase modulation remote sensing image super-resolution device, including an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module, and a convolution layer;

[0048] Encoder for extracting low-resolution remote sensing images I LR The encoding feature θ 0 , and convert the set super-resolution scale factor into network-perceivable scale information s embedded ;

[0049] The scale modulation feature fusion module is used to encode the feature θ 0 and scale information embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s embedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ;

[0050] The frequency modulation reconstruction module is used to extract features of the θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain a feature θ containing the corresponding position information. 0 and θ f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the said Θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulation F ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M ;

[0051] Convolutional layer for Θ M The information in the cell is aggregated with Θ to obtain the final super-resolution remote sensing image I SR .

[0052] Beneficial effects:

[0053] First, the present invention first fuses the shallow features extracted by the encoder with the scale information through a scale modulation feature fusion module to obtain features at different levels. Then, the frequency modulation reconstruction module is used to restore the features output by the scale modulation feature fusion module to a super-resolution image of a specified scale through frequency domain information modulation. The scale modulation feature fusion module deeply integrates the scale information into the feature learning process, so that the network can obtain corresponding intermediate features according to different scale factors. The frequency modulation reconstruction module uses the frequency domain information obtained by Fourier transform to adaptively modulate the image reconstruction process, thereby retaining a large amount of high-frequency information and improving the clarity of the super-resolution result image.

[0054] Second, after a large number of experiments, the results show that the method of the present invention achieves better results than other remote sensing image continuous super-resolution methods. The method proposed in the present invention provides an effective technical means for remote sensing image super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is the overall framework diagram of a scale-frequency bi-phase modulation remote sensing image continuous super-resolution method:

[0057] Figure 2 This is a schematic diagram of the scale modulation feature fusion module:

[0058] Figure 3 Schematic diagram of the frequency modulation reconstruction module. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0060] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0061] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0062] The design idea of ​​the present invention is: the scale factor is fused with the multi-level feature extraction network through the scale modulation feature fusion module, so that the network can perceive the scale information of super-resolution, extract different levels of features according to the requirements of different scale factors, and perform adaptive weighted fusion in the channel dimension, so that the network can extract intermediate features aligned with specific scale factors. In addition, the frequency modulation reconstruction module incorporates frequency domain information into the upsampling reconstruction process through Fourier transform, so that the network can learn frequency domain information and modulate features after periodic activation, so that the network can fully retain high-frequency information during the upsampling reconstruction process and obtain super-resolution images with clear textures and edges.

[0063] In the embodiment of the present application, a method for continuous super-resolution of remote sensing images using scale-frequency bi-phase modulation is used. Taking the actual application scenario of continuous super-resolution of remote sensing images as an example, the overall framework of the method is shown in FIG. Figure 1 As shown, the steps of the method include:

[0064] The first step is to use the encoder to extract the low-resolution remote sensing image ILR The encoding feature θ 0 , and convert the set super-resolution scale factor into network-perceivable scale information s embedded .

[0065] This step first requires encoding the low-resolution image into a form that the network can understand. The low-resolution remote sensing image is sent to the encoder to extract the shallow features in the image for subsequent processing. The encoder uses the encoder part of RCAN, and the corresponding encoding feature graph can be obtained as follows:

[0066] θ 0 =Φ encoder (I LR )

[0067] Among them, I LR represents the input low-resolution remote sensing image, Φ encoder represents the encoder that initially processes the low-resolution image input, θ 0 Represents the encoded feature map output by the encoder. The encoder has 12 RCABs. Afterwards, in order to integrate the scale factor into the network, the scale factor needs to be embedded. Sinusoidal embedding can convert scale information into a continuous and bounded representation. Due to its excellent performance in learnability and periodicity, sinusoidal embedding is selected to embed scale information, as follows:

[0068] s embedded =Φ embedding (Sinu(r))

[0069] Where r represents the scale factor of super-resolution, Sinu represents the sinusoidal embedding, and Φ embedding It represents the initial processing of the embedded scale information and consists of a three-layer fully connected network. embedded Represents the embedded scale information that the network can understand.

[0070] In this step, the calculation method of the sinusoidal embedding method is as follows:

[0071]

[0072]

[0073] Where d represents the dimension of the scale modulation feature fusion module, Sinu(r) (i) Represents the value of Sinu(r) at the i-th position, and the value of N is set to 10000.

[0074] In the second step, the scale modulation feature fusion module is used to fuzzy the encoding feature θ 0 and scale informationembedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s embedded As the weight of channel modulation and the channel information θ c The fusion is performed to obtain Θ.

[0075] like Figure 2 As shown in the figure, this step converts the encoded low-resolution remote sensing image feature map θ obtained in the first step into 0 and the scale information s that the network can perceive embedded It is sent to the scale modulation feature fusion module. This module can adaptively learn the weights modulated by the scale factor to embed the scale information into the feature learning and fuse the features at different levels. Therefore, this module can extract features at different levels for super-resolution tasks with different scale requirements. Because neural networks of different depths contain feature information at different levels, correspondingly, convolution kernels of different sizes with different sizes of receptive fields can extract feature information at different levels. In addition, super-resolution tasks of different scales need to utilize features at different levels. To achieve this, the scale modulation feature fusion module uses multi-scale deep convolutions with convolution kernel sizes of 3, 5, and 7, respectively, and connects them as follows:

[0076] θ 3 =f 3×3 (θ 0 +s embedded )

[0077] θ 5 =f 5×5 (θ 0 +s embedded )

[0078] θ 7 =f 7×7 (θ 0 +s embedded )

[0079] θ′=Concat(θ 3 ,θ 5 ,θ 7 )

[0080] Among them, θ 3 ,θ 5 and θ 7 They represent the features of different levels extracted by the convolutional layers with convolution kernel sizes of 3, 5, and 7, respectively. 3×3 、f 5×5 and f 7×7represents the convolutional layers with kernel sizes of 3, 5, and 7, Concat represents connecting features of different levels in the channel dimension, and θ′ represents a feature map containing scale information. In this embodiment, this step will be repeated 3 times to fully extract information of different levels according to the requirements of the scale factor.

[0081] Perform an average pooling operation on θ′ to obtain multi-level feature channel information:

[0082] θ c =AvgPooling(θ′)

[0083] Among them, θ c It represents the channel information of feature map aggregation, and AvgPooling represents the average pooling operation.

[0084] Considering that the scale factor does not contain any spatial information, it is not necessary to participate too much in pixel-level operations. The operation of scale information is kept at the channel level, and the scale factor is deeply integrated into the network as the weight of channel modulation, as follows:

[0085] θ s =L(s embedded )

[0086] Θ=f 3d→d (θ′⊙(θ c ⊙θ s ))+s embedded

[0087] Among them, θ s represents the scale information for channel modulation obtained by the linear layer L, f 3d→d represents the convolutional layer of channel dimension feature fusion, and Θ represents the output of the scale modulation feature fusion module. After the above operations, the scale factor is fully integrated into the feature learning process, so that the network can fully perceive the scale information and modulate the weights of feature fusion at different levels according to the requirements of the super-resolution task.

[0088] In the third step, the frequency modulation reconstruction module is used to extract features of the θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain the feature θ containing the corresponding position information. 0 and θ f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the said Θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulationF ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M .

[0089] like Figure 3 As shown in the figure, the scale modulation feature fusion module output θ and the super-resolution coordinate coord obtained in the second step are sent to the frequency modulation reconstruction module. First, a convolution layer with a convolution kernel of 3 is used to distinguish the features for subsequent operations, and then the features are upsampled according to the super-resolution coordinates to obtain features containing corresponding position information, as follows:

[0090] Θ 0 =UP grid (f 3×3 1 (θ), coord)

[0091] Θ f =UP grid (f 3×3 2 (θ), coord)

[0092] Among them, f 3×3 1 represents the convolutional layer of the trunk branch preprocessing used to retain the original information, f 3×3 2 represents the convolutional layer for frequency modulation branch preprocessing, Θ 0 represents the initial features containing coordinate information, Θ f Indicates the characteristics of the upcoming Fourier modulation, UP grid Represents upsampling based on a super-resolution coordinate grid.

[0093] A local integration strategy is used to avoid checkerboard artifacts, and the RGB value of the specified coordinate position is predicted using the nearby potential vector, thus incorporating local spatial information into Θ f middle:

[0094] Θ′ f =Θ f +f FC (cell)

[0095] Among them, cell represents the location information of the adjacent coordinates, f FC Represents a fully connected layer that extracts nearby latent vectors.

[0096] In order to obtain the frequency domain information of the feature, Θ′ f Perform Fourier transform to get Θ fft Since the features after Fourier transformation are completely different from the original features in the pixel-level sense, the feature frequency domain information is used for modulation in the channel dimension.

[0097] In this step, Θ′ f The calculation process for Fourier transform is:

[0098]

[0099] Among them, Θ i represents the features of the i-th channel, (x, y) and (u, v) represent Θ and Θ respectively. fft The point in .

[0100] Through the average pooling operation and 2 linear layers, Θ fft is converted into a compact representation suitable for modulation:

[0101] Θ fm = L′(AvgPooling(Θ fft ))

[0102] Among them, Θ fm Represents the weight used for frequency modulation. Using a periodic activation function helps the network learn high-frequency information, so Θ′ f After the cycle is activated, it is then fm The periodic activation function selects cosine and sinine, and is modulated by the real and imaginary parts after Fourier transformation, as follows:

[0103]

[0104] Among them, Θ F Represents the features after frequency domain information modulation. The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB value Θ is predicted by MLP. M .

[0105] Θ′ F =f 2d→d (Θ F )

[0106] Θ m =Θ 0 ⊙Θ′ F

[0107] Θ M =f mlp (Θ m )

[0108] Among them, f 2d→d represents the channel dimension reduction after modulation, Θ′ F represents the features after dimensionality reduction, Θ m represents the feature containing the initial information and the frequency modulation information, f mlp Used to learn the continuous expression of remote sensing images in implicit space, ΘM Represents the results of preliminary predictions.

[0109] The fourth step is to M And Θ, the convolutional layer is used to aggregate the information in the cell to obtain the final super-resolution remote sensing image I SR ;

[0110]

[0111] Among them, UP(Θ) means that the most original features are used as the reference constraint reconstruction result, f final represents a 3*3 convolutional layer, S represents the area enclosed by four adjacent coordinate points, S cell Indicates the area corresponding to each coordinate point.

[0112] The fifth step is to calculate the loss function and update the network parameters during the network training process. The loss function selects L1 loss:

[0113]

[0114] After the network training is completed, the trained network is used to generate super-resolution remote sensing images.

[0115] Another embodiment of the present application is a scale-frequency bi-phase modulation remote sensing image super-resolution device, comprising an encoder, a scale modulation feature fusion module, a frequency modulation reconstruction module, and a convolution layer;

[0116] Encoder for extracting low-resolution remote sensing images I HR The encoding feature θ 0 , and convert the set super-resolution scale factor into network-perceivable scale information s embedded ;

[0117] The scale modulation feature fusion module is used to encode the feature θ 0 and scale information embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s embedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ;

[0118] The frequency modulation reconstruction module is used to extract features of the θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain a feature θ containing the corresponding position information. 0 and θ f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the said Θ′f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulation F ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M ;

[0119] Convolutional layer for Θ M The information in the cell is aggregated with Θ to obtain the final super-resolution remote sensing image I SR .

[0120] Taking the actual application scenario of continuous super-resolution of remote sensing images as an example, by comparing a variety of existing continuous super-resolution methods on a public remote sensing dataset, as shown in Tables 1 and 2, it can be seen that the method proposed in the embodiment has better accuracy.

[0121] Table 1 Comparison of various existing continuous super-resolution methods and embodiments on the WHU-RS19 dataset (training internal scale)

[0122]

[0123] Table 2 Comparison of various existing continuous super-resolution methods and embodiments on the WHU-RS19 dataset (training outer scale)

[0124]

[0125]

[0126] The specific implementation methods of the invention are described in detail below, but the protection scope of the invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the invention should be included in the protection scope of the invention. Therefore, the protection scope of the invention should be based on the protection scope of the claims.

Claims

1. A scale-frequency bi-phase modulation remote sensing image super-resolution method, characterized in that: The specific process is: The first step is to use the encoder to extract the low-resolution remote sensing image I LR The encoding feature θ0 is converted into the network-perceivable scale information s embedded ; In the second step, the scale modulation feature fusion module is used to fusion the encoding feature θ0 and the scale information s embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s enbedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ; In the third step, the frequency modulation reconstruction module is used to extract features of Θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain features Θ0 and Θ1 containing corresponding position information. f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the Θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulation F ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M ; The fourth step is to calculate the θM 和 Θ, use the convolutional layer to aggregate the information in the cell to obtain the final super-resolution remote sensing image I SR ; Step 5: Based on the high-resolution reference image I HR and super-resolution remote sensing images I SR , calculate the loss function, and use the loss function to update the network parameters; when the network training is completed, use the trained network to generate super-resolution remote sensing images.

2. According to the scale-frequency bi-phase modulation remote sensing image super-resolution method of claim 1, it is characterized in that: In the second step, the scale information is used as the weight of channel modulation and the channel information θ c After fusion, we get Θ as: i s =L(s embedded ) Θ=f 3d→d (θ′☉(θ c ☉)θ s ))+s embedded Among them, θ s represents the scale information for channel modulation obtained by the linear layer L, f 3d→d represents the convolutional layer of channel dimension feature fusion, Θ represents the output of the scale modulation feature fusion module, and ⊙ represents the element-by-element multiplication of features.

3. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: In the second step, the encoding feature θ0 and scale information s embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information. The specific process is: The scale modulation feature fusion module uses a multi-scale deep convolution layer with convolution kernel sizes of 3, 5, and 7 for convolution, and connects the convolution results, as follows: θ3=f 3×3 (θ0+s embedded ) θ5=f 5×5 (θ0+s embedded ) θ7=f 7×7 (θ0+s embedded ) θ′=Concat(θ3,θ5,θ7) Among them, θ3, θ5 and θ7 represent the features of different levels extracted by the convolutional layers with convolution kernel sizes of 3, 5 and 7 respectively, and f 3×3 、f 5×5 and f 7×7 It represents the convolutional layers with kernel sizes of 3, 5, and 7. Concat means connecting features of different levels in the channel dimension. θ′ represents the feature map containing scale information.

4. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: In the third step, the RGB value of the specified coordinate position is predicted using the nearby latent vector, and the position information of the nearby coordinates is incorporated into θ f Get Θ′ f : I' f =Θ f +f FC (cell) Among them, cell represents the location information of the adjacent coordinates, f FC represents the fully connected layer that extracts the adjacent latent vector, Θ′ f Represents a feature that contains nearby coordinate information.

5. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: In the third step, the θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm , the specific process is: For θ′ f Perform Fourier transform to get Θ fft ; Through the pooling operation and the linear layer, Θ fft Convert to a compact representation suitable for modulation: I fm =L(Pooling(Θ fft )) Among them, Θ fm represents the weight used for frequency modulation, Pooling() represents the pooling operation, and L′ represents the linear layer.

6. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: In the third step, the feature θ modulated by the frequency domain information is obtained F for: Among them, Θ F Represents the features after frequency domain information modulation, period() represents the periodic activation function, Real() represents the real part operation, Imag() represents the imaginary part operation, and Concat represents the concatenation operation.

7. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: In the third step, the modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​are predicted through MLP, as follows: I' F =f 2d→d (I F ) I m =Θ0☉Θ′ F I M =f mlp (I m ) Among them, f 2d→d represents the channel dimension reduction after modulation, Θ′ F represents the features after dimensionality reduction, Θ m represents the feature containing the initial information and the frequency modulation information, f mlp Used to learn the continuous expression of remote sensing images in implicit space, Θ M Represents the results of preliminary predictions.

8. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: The fourth step uses the convolutional layer to aggregate the information in the cell to obtain the final super-resolution remote sensing image I SR for: Among them, UP(Θ) means that the most original features are used as the reference constraint reconstruction result, f final represents a 3*3 convolutional layer, S represents the area surrounded by four adjacent coordinate points, S cell Indicates the area corresponding to each coordinate point.

9. The scale-frequency bi-phase modulation remote sensing image super-resolution method according to claim 1, characterized in that: The loss function is: Where r represents the super-resolution scale, H and W represent the height and width of the input image. Represents the super-resolution remote sensing image I SR In (x i ,y j ) pixel value, Represents the high-resolution reference image I HR In (x i ,y j )’s pixel value.

10. A scale-frequency dual-phase modulation remote sensing image super-resolution device, characterized in that: It includes encoder, scale modulation feature fusion module, frequency modulation reconstruction module and convolution layer; Encoder for extracting low-resolution remote sensing images I LR The encoding feature θ0 is converted into the network-perceivable scale information s embedded ; scale Modulation feature fusion module is used to encode the feature θ0 and scale information s embedded Perform multi-scale convolution and fusion to obtain a feature map θ′ containing scale information, and perform a pooling operation on the θ′ to obtain the channel information θ of the multi-level features. c , and then the scale information s embedded As the weight of channel modulation and the channel information θ c Fusion is performed to obtain Θ; The frequency modulation reconstruction module is used to extract features of the Θ through two 3*3 convolution branches and perform upsampling based on the coordinate position to obtain features Θ0 and Θ1 containing corresponding position information. f , incorporate the position information of the adjacent coordinates into Θ f Get Θ′ f ; for the Θ′ f Perform Fourier transform to get Θ fft And based on it, the weight θ for frequency modulation is calculated fm ; The said Θ′ f After the cycle is activated, it is then fm Modulate and obtain the feature Θ after frequency domain information modulation F ; The modulated features are incorporated into the original features as additional information enhancement branches, and then the RGB values ​​Θ are predicted through MLP M ; Convolutional layer for Θ M The information in the cell is aggregated with Θ to obtain the final super-resolution remote sensing image I SR .

Images