Remote sensing image super-resolution method, system and equipment based on implicit neural representation

Abstract

The invention discloses a remote sensing image super-resolution method, system and device based on implicit neural representation, and belongs to the technical field of deep learning and remote sensing image processing. The method comprises the following steps: firstly, acquiring a data set suitable for a remote sensing image, training a remote sensing image super-resolution model based on the data set, then taking a to-be-reconstructed low-resolution remote sensing image and a pre-constructed query array as input, and performing multi-scale feature extraction by a hierarchical feature pyramid extraction module; the double-path implicit attention feature fusion module carries out fusion, and finally the implicit high-resolution reconstruction module carries out the reconstruction of a high-resolution image. According to the method, the feature pyramid, the attention mechanism and the like are combined with implicit neural representation, the features of different levels are fused together, the problems of multi-scale feature fusion and efficient information extraction in the remote sensing image super-resolution process are solved, the remote sensing image super-resolution precision can be improved, and meanwhile the remote sensing image super-resolution efficiency is improved. And the target of continuous scale super-resolution of the remote sensing image is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and remote sensing image processing, and in particular, relates to a remote sensing image super-resolution method, system and device based on implicit neural representation. Background Art

[0002] Remote sensing technology obtains images and data of the earth's surface through platforms such as satellites, aircraft or drones. It has a wide range of applications in environmental monitoring, natural disaster assessment, urban planning and other fields. However, the spatial resolution of remote sensing images is limited by the physical limitations of sensors and it is difficult to meet the growing demand for high precision. Therefore, improving the spatial resolution of remote sensing images has become an important research topic.

[0003] Super-resolution (SR) technology aims to convert low-resolution images into high-resolution images through algorithmic means. This method can be divided into single image SR (SISR) and multiple image SR (MISR). Traditional SR methods have some limitations, such as interpolation algorithms easily lead to image blur, and reconstruction-based methods are computationally intensive and sensitive to noise. With the rise of deep learning, especially the continuous development of convolutional neural networks (CNNs), super-resolution methods based on deep learning have gradually become a research hotspot. Compared with traditional methods, deep learning methods can extract high-dimensional features of images through learning a large number of samples, thereby significantly improving the super-resolution reconstruction effect. Although these methods can improve the details and clarity of images, the magnification is fixed or combined with several fixed magnifications. If a certain random scale of magnification is required, the model architecture may need to be readjusted.

[0004] Remote sensing images are an important source of geographic information. They are different from natural images in that they have wide coverage, large observation range, and rich spectral information. Different spatial resolutions may reveal different levels and structures of surface objects. At the same time, there will be multi-source and multi-resolution image data in a task. In the process of analysis, the resolution of the data needs to be unified, which involves the problem of super-resolution at different scales. Currently, many methods use multi-path learning at a specific scale to cope with the multi-scale super-resolution of a single network. Usually, the upsampling module is designed to use the scale factor as input to dynamically adjust the result. Such methods are difficult to obtain optimal performance and may require a large number of parameters for calculation.

[0005] Implicit Neural Representations (INR) refers to a method of representing input signals such as images, audio, and point clouds as functions through neural networks, which has achieved good results in 3D reconstruction. The key idea is to represent an object as a function that maps coordinates to corresponding signals. In deep learning, a trained network is used to approximate this function, and the network is an implicit neural representation. INR provides a new idea for SR technology, which converts low-resolution images into high-dimensional information representations in coordinate space, and obtains high-resolution images by querying the features of pixels based on coordinates.

[0006] Most current INR-based SR methods are extensions of the Local Implicit Image Function (LIIF), which simply connects the periodic transformation of coordinates and local features and predicts RGB values ​​using a multi-layer perceptron (MLP) network. These methods mainly focus on local representation and face problems such as high computational complexity, high memory requirements, and high time overhead in the model feature extraction network. Therefore, it is necessary to construct a new type of continuous-scale super-resolution method for remote sensing images based on implicit neural representation. Summary of the invention

[0007] The purpose of the present invention is to provide a remote sensing image super-resolution method, system and device based on implicit neural representation to solve the problems of low generalization performance and poor feature reconstruction effect of existing methods. The present invention aims to solve the need for continuous scale super-resolution in the process of super-resolution reconstruction of remote sensing images by utilizing the powerful representation ability of implicit neural representation network, which belongs to the cutting-edge technology of cross-application of computer vision and deep learning.

[0008] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a remote sensing image super-resolution method based on implicit neural representation, which comprises the following steps:

[0010] S1. Acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset comprises a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0011] S2. The low-resolution remote sensing image to be reconstructed and the pre-built query array are input into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed. Then, the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. The remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0012] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0013] As a preferred embodiment of the first aspect, in step S1, the remote sensing image dataset is constructed as follows:

[0014] S11, presetting the initial size of the low-resolution remote sensing image pixel during the sampling process, and performing a rounding operation on the result of multiplying the initial size by the random magnification, removing the decimal part of the multiplication result, and obtaining the target size of the super-resolution remote sensing image;

[0015] S12, according to the calculated target size of the super-resolution remote sensing image, using a random cropping method to crop the acquired original remote sensing image to obtain a cropped remote sensing image;

[0016] S13, performing an image enhancement operation on the cropped remote sensing image to obtain a remote sensing image after image enhancement processing and use it as a high-resolution target remote sensing image;

[0017] S14, applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then perform a noise addition process to generate a low-resolution remote sensing image;

[0018] S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

[0019] As a preferred embodiment of the first aspect, in step S2, the query array is constructed as follows:

[0020] AS21. Using a low-resolution remote sensing image as an input image and a super-resolution remote sensing image as an output image, dividing a two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the input image, obtaining a multidimensional array formed by the coordinates of the grid center points of the input image, recorded as a first multidimensional array; dividing the two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the output image, obtaining a multidimensional array formed by the coordinates of the grid center points of the output image, recorded as a second multidimensional array;

[0021] AS22, perform linear interpolation on the first multidimensional array to obtain a multidimensional array with the same shape as the second multidimensional array, recorded as a third multidimensional array;

[0022] AS23. Subtract the third multidimensional array from the second multidimensional array to obtain a relative coordinate multidimensional array.

[0023] AS24, broadcast the random magnifications to obtain a magnification array with the same shape as the second multidimensional array;

[0024] AS25, concatenate the second multidimensional array, the relative coordinate multidimensional array, and the magnification array in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as query arrays.

[0025] As a preferred embodiment of the first aspect, in step AS21, the first multidimensional array and the second multidimensional array are obtained as follows:

[0026]

[0027] i in =0,1,…,H in -1

[0028] j in =0,1,…,W in -1

[0029]

[0030] i out =0,1,…,H out -1

[0031] j out =0,1,…,H out -1

[0032] Among them, the size of the input image is (H in ,W in ), H in Indicates the width of the input image, W in Indicates the height of the input image; i in and j inRespectively represent the row and column index of the input image; P in (i in ,j in ) represents the coordinates of the grid center point of the input image, from which the first multidimensional array is generated; the size of the output image is (H out ,W out ), H out Indicates the width of the output image, W out Indicates the height of the output image; i out and i out Respectively represent the row and column index of the output image; P out (i out ,i out ) represents the grid center point coordinates of the output image, from which the second multidimensional array is generated.

[0033] As a preferred embodiment of the first aspect, in step S2, the specific processing process in the hierarchical feature pyramid extraction module is as follows:

[0034] BS21. First, the feature map output by the encoder is input into a parallel three-way sub-network. The first sub-network consists of one spatial channel downsampling block, the second sub-network consists of two spatial channel downsampling blocks, and the third sub-network consists of three spatial channel downsampling blocks, wherein each spatial channel downsampling block is cascaded in sequence by a first convolutional layer, a ReLU activation function, a second convolutional layer, and a ReLU activation function; the convolution kernel size of the first convolutional layer is 3×3, the number of convolution kernels is 64, and the step size is 1; the convolution kernel size of the second convolutional layer is 3×3, the number of convolution kernels is 64, and the step size is 2;

[0035] BS22, input the feature map output by the first sub-network into the third convolutional layer to obtain a first feature map; perform a nearest neighbor interpolation operation on the feature map output by the third sub-network and input it into the fourth convolutional layer to obtain a second feature map; wherein the convolution kernel size of the third convolutional layer is 3×3, and the convolution kernel size of the fourth convolutional layer is 1×1;

[0036] BS23, passing the feature map output by the encoder through the fifth convolution layer to obtain a third feature map, passing the third feature map through three processing branches respectively, each processing branch performs a maximum pooling operation and a nearest neighbor interpolation on the third feature map, and the output results of the three processing branches are spliced ​​in the channel dimension and then passed through the sixth convolution layer to obtain a spatial pyramid pooling feature map; wherein the convolution kernel size of the fifth convolution layer is 1×1, and the convolution kernel size of the sixth convolution layer is 1×1;

[0037] BS24, the first feature map, the feature map output by the second sub-network, the second feature map and the spatial pyramid pooling feature map are spliced ​​in the channel dimension and then passed through the seventh convolution layer to obtain a multi-scale fusion feature map; wherein the convolution kernel size of the seventh convolution layer is 1×1.

[0038] As a preferred embodiment of the first aspect, in step S2, the specific processing process in the dual-path implicit attention feature fusion module is as follows:

[0039] CS21, apply the implicit attention mechanism to the feature map output by the encoder and the query array after bilinear interpolation to obtain the fourth feature map;

[0040] CS22, after bilinear interpolation of the multi-scale fusion feature map, apply the implicit attention mechanism together with the query array to obtain the fifth feature map;

[0041] CS23. After concatenating the fourth feature map, the fifth feature map and the query array, the implicit attention fusion feature is obtained.

[0042] As a preferred embodiment of the above-mentioned first aspect, the specific processing process in the implicit attention mechanism is as follows: the input feature map passes through the eighth convolution layer to obtain the sixth feature map, the input feature map passes through the ninth convolution layer to obtain the seventh feature map, the input query array passes through the tenth convolution layer and is multiplied element-by-element with the sixth feature map and undergoes a SoftMax normalization operation to obtain the eighth feature map, the eighth feature map is multiplied element-by-element with the seventh feature map to obtain the feature map output by the implicit attention mechanism; wherein, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

[0043] As a preferred embodiment of the above-mentioned first aspect, the implicit high-resolution reconstruction module consists of an eleventh convolutional layer, three 3×3 convolutional layers and an output layer, each 3×3 convolutional layer adopts dense connection and ReLU function activation, the convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

[0044] In a second aspect, the present invention provides a remote sensing image super-resolution system based on implicit neural representation, comprising:

[0045] A data acquisition module is used to acquire a remote sensing image data set suitable for a super-resolution task, wherein the remote sensing image data set includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0046] The result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features, and the implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image; wherein, the remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0047] In a third aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0048] The memory is used to store computer programs;

[0049] The processor is used to implement the remote sensing image super-resolution method based on implicit neural representation as described in any scheme of the first aspect when executing the computer program.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The method of the present invention only requires a unified remote sensing image super-resolution model, without pre-defining a specific magnification, and can perform super-resolution reconstruction at continuous scales, which makes the remote sensing image super-resolution model extremely flexible and adaptable, and can generate high-resolution images of different scales according to actual needs. The method of the present invention combines deep neural networks with implicit neural representations to accurately capture and reconstruct image details and significantly improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0053] Figure 2 It is an overall schematic diagram of the remote sensing image super-resolution model of the present invention;

[0054] Figure 3 A schematic diagram of the network structure of the remote sensing image super-resolution model of the present invention;

[0055] Figure 4 It is a structural diagram of the hierarchical feature pyramid extraction module in the present invention;

[0056] Figure 5Schematic diagram of a dual-path implicit attention feature fusion module in the present invention;

[0057] Figure 6 A schematic diagram of a reconstruction result with a magnification of 2.5 provided in an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of a reconstruction result with a magnification of 5.5 provided in an embodiment of the present invention;

[0059] Figure 8 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0061] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0062] like Figure 1 As shown, in a preferred implementation of the present invention, the remote sensing image super-resolution method based on implicit neural representation includes the following steps S1 to S2. The specific implementation process is described below.

[0063] S1. Obtain a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image.

[0064] It should be noted that in the present invention, the remote sensing image dataset is used for training the remote sensing image super-resolution model based on implicit neural representation. In the training process of the remote sensing image super-resolution model, the high-resolution target remote sensing image can be regarded as a label (Ground Truth), and the training goal is to make the super-resolution remote sensing image reconstructed by the remote sensing image super-resolution model as close as possible to the high-resolution target remote sensing image. When the remote sensing image super-resolution model is inferring, it is only necessary to obtain the low-resolution remote sensing image to be reconstructed, and the trained remote sensing image super-resolution model outputs the reconstructed super-resolution remote sensing image.

[0065] It should be noted that in step S1 of the present invention, the method for constructing the remote sensing image dataset is as follows:

[0066] S11. Preset the initial size of the low-resolution remote sensing image pixel during the sampling process, and round the result of multiplying the initial size by the random magnification factor to an integer, remove the decimal part of the multiplication result, and obtain the target size of the super-resolution remote sensing image.

[0067] It should be noted that step S11 of the present invention is a low-resolution sampling design. In the process of obtaining the random magnification, as a preferred method, this embodiment uses a uniform distribution sampling function to generate the random magnification.

[0068] S12. According to the calculated target size of the super-resolution remote sensing image, the acquired original remote sensing image is cropped using a random cropping method to obtain a cropped remote sensing image.

[0069] It should be noted that in step S12 of the present invention, the cropping range is selected from a preset random area within the original remote sensing image to ensure the diversity and randomness of the high-resolution target remote sensing image.

[0070] S13, performing an image enhancement operation on the cropped remote sensing image to obtain a remote sensing image after image enhancement processing and use it as a high-resolution target remote sensing image.

[0071] It should be noted that in step S13 of the embodiment of the present invention, the image enhancement operation includes but is not limited to: horizontal flipping, upside down flipping, random rotation operations with random probability, and introducing random noise to enhance the complexity of the cropped remote sensing image. Among them, the random probability is used to determine whether the cropped remote sensing image is rotated. When the cropped remote sensing image needs to be rotated, there is an equal probability of rotating 90°, 180°, and 270°.

[0072] S14. Applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then perform noise processing to generate a low-resolution remote sensing image.

[0073] It should be noted that, in step S14 of the present invention, image noise is introduced to make the generated low-resolution remote sensing image closer to the actual data distribution.

[0074] S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

[0075] S2. The low-resolution remote sensing image to be reconstructed and the pre-built query array are input into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map θ output by the encoder is input into the hierarchical feature pyramid extraction module to extract multi-scale features of the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then the feature map θ output by the encoder and the multi-scale fusion feature map The pre-built query array is input into the dual-path implicit attention feature fusion module to obtain the implicit attention fusion feature, and the implicit attention fusion feature is input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain the reconstructed super-resolution remote sensing image; wherein the above-mentioned remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0076] It should be noted that, in step S2 of the present invention, the method for constructing the above query array is as follows:

[0077] AS21. Use the low-resolution remote sensing image as the input image and the super-resolution remote sensing image as the output image. Divide the two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the input image, and obtain a multidimensional array formed by the coordinates of the grid center points of the input image, which is recorded as the first multidimensional array lr. coord , divide the two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the output image, and obtain the multidimensional array formed by the coordinates of the grid center points of the output image, which is recorded as the second multidimensional array hr coord .

[0078] It should be noted that in step AS21 of the present invention, the second multidimensional array hr coord Right now Figure 2 and Figure 5 The super-resolution coordinate information referred to in .

[0079] It should be noted that in step AS21 of the present invention, the first multidimensional array lr is obtained. coord and the second multidimensional array hr coord The way is as follows:

[0080]

[0081] i in =0,1,…,H in -1

[0082] j in =0,1,…,W in -1

[0083]

[0084] i out =0,1,…,H out -1

[0085] j out =0,1,…,H out -1

[0086] Among them, the size of the input image is (H in ,W in ), H in Indicates the width of the input image, W in Indicates the height of the input image; i in and j in Respectively represent the row and column index of the input image; P in (i in ,j in ) represents the grid center point coordinates of the input image, from which the first multidimensional array lr is generated coord ; The size of the output image is (H out ,W out ), H out Indicates the width of the output image, W out Indicates the height of the output image; i out and i out Respectively represent the row and column index of the output image; P out (i out ,i out ) represents the grid center point coordinates of the output image, from which the second multidimensional array hr is generated coord .

[0087] AS22, the first multidimensional array lr coord Perform linear interpolation to obtain the value that is consistent with the second multidimensional array hr coord A multidimensional array of the same shape, denoted as the third multidimensional array

[0088] AS23, using the second multidimensional array hr coord Subtract the third multidimensional array Get the relative coordinate multidimensional array rel coord .

[0089] It should be noted that in step AS23 of the present invention, the relative coordinate multidimensional array rel coord Right now Figure 2 and Figure 5 The relative coordinate information referred to in .

[0090] AS24. Broadcast the random magnification to obtain the value corresponding to the second multidimensional array hr. coord An array of magnifications of the same shape s map .

[0091] It should be noted that in step AS24 of the present invention, the magnification array s map Right now Figure 2 and Figure 5 The magnification information indicated in the figure.

[0092] AS25, the second multidimensional array hr coord , relative coordinate multidimensional array rel coord and the magnification array s map Perform concatenation in the channel dimension to obtain an array of coordinates and magnifications and use it as the query array cs map .

[0093] It should be noted that in step AS25 of the present invention, the second multidimensional array hr coord , relative coordinate multidimensional array rel coord and the magnification array s map All three arrays have the same width and height.

[0094] It should be noted that, in the present invention, the feature map refers to the result obtained by convolving the input image with a series of convolution kernels, which can be regarded as an abstract representation of the original image, in which each pixel represents some specific features.

[0095] It should be noted that in step S2 of the present invention, the encoder adopts a conventional super-resolution model to preliminarily extract multi-scale features, and its structure is not limited in the present invention. The feature map output by the encoder is denoted as θ.

[0096] It should be noted that if Figure 4 As shown, the specific processing process in the hierarchical feature pyramid extraction module in step S2 of the present invention is as follows:

[0097] BS21. First, the feature map θ output by the encoder is input into the three parallel sub-networks. The first sub-network consists of one spatial channel downsampling block, the second sub-network consists of two spatial channel downsampling blocks, and the third sub-network consists of three spatial channel downsampling blocks, where each spatial channel downsampling block is composed of the first convolutional layer, the ReLU activation function, the second convolutional layer and the ReLU activation function cascaded in sequence.

[0098] It should be noted that in step BS21 of the present invention, all parameters of the three sub-networks are not shared. The first sub-network reduces the spatial dimension of the feature map θ output by the encoder, reduces the width and height of the feature map θ to 1 / 2 of the original size, and increases the number of channels to twice the original. The feature map output by the first sub-network is recorded as The second sub-network reduces the width and height of the feature map θ to 1 / 4 of the original, but increases the number of channels to four times the original. The feature map output by the second sub-network is recorded as The third sub-network reduces the width and height of the feature map θ to 1 / 8 of the original, but increases the number of channels to eight times the original. The feature map output by the third sub-network is recorded as in, The size of (b x ,c x ×2,w x / 2,h x / 2), The size of (b x ,c x ×4,w x / 4,h x / 4), The size of (b x ,c x ×8,w x / 8,h x / 8), b x represents the batch size of low-resolution remote sensing images; c x Indicates the number of low-resolution remote sensing image channels; w x The number of pixels representing the width of low-resolution remote sensing images; h x The number of pixels representing the height of low-resolution remote sensing images.

[0099] It should be noted that in the spatial channel downsampling block of step BS21 of the present invention, the convolution kernel size of the first convolution layer is 3×3, the number of convolution kernels is 64, and the step length is 1, which is used to keep the width and height of the feature map θ unchanged, and the number of channels is increased to twice the original; the convolution kernel size of the second convolution layer is 3×3, the number of convolution kernels is 64, and the step length is 2, which is used to keep the number of channels of the feature map θ unchanged. Here, the first convolution layer and the second convolution layer are used to extract fine-grained features, and the middle grid is a receptive field center that can capture feature changes up, down, left, right, and diagonally.

[0100] In the three-way sub-network in parallel in step BS21 of the embodiment of the present invention, the feature extraction process can be expressed as:

[0101]

[0102]

[0103] Among them, Conv 3×3 Represents a convolutional layer containing a 3×3 convolution kernel.

[0104] BS22, the feature map output by the first sub-network Input to the third convolutional layer to change the feature map Width and height, get the first feature map Feature map of the third sub-network output After the nearest neighbor interpolation operation, it is input to the fourth convolutional layer to change the feature map Width and height, get the second feature map Among them, the convolution kernel size of the third convolution layer is 3×3, and the convolution kernel size of the fourth convolution layer is 1×1.

[0105] It should be noted that in step BS22 of the present invention, the first feature map And the second feature map The width and height of same.

[0106] BS23, the feature map output by the encoder passes through the fifth convolution layer to obtain the third feature map, the third feature map passes through three processing branches respectively, each processing branch performs maximum pooling operation and nearest neighbor interpolation on the third feature map, the output results of the three processing branches are spliced ​​in the channel dimension and then pass through the sixth convolution layer to obtain the spatial pyramid pooling feature map Among them, the convolution kernel size of the fifth convolution layer is 1×1, and the convolution kernel size of the sixth convolution layer is 1×1.

[0107] It should be noted that, in step BS23 of the present invention, the fifth convolutional layer is used to capture multi-scale features and enhance the robustness of feature size changes at different levels.

[0108] It should be noted that in step BS23 of the present invention, the spatial pyramid pooling feature map Width and height same.

[0109] BS24, the first feature map Feature map of the second sub-network output Second feature map Spatial pyramid pooling feature map After splicing in the channel dimension, it passes through the seventh convolution layer to obtain a multi-scale fusion feature map Among them, the convolution kernel size of the seventh convolutional layer is 1×1.

[0110] It should be noted that the implicit neural representation in the image field can be expressed as:

[0111] f INR :R 2 →R 3 ,f INR (x INR ,y INR )=(r INR , g INR,b INR )

[0112] Among them, R 2 Represents two-dimensional space; R 3 Represents three-dimensional space; (x INR ,y INR ) represents the coordinates of any pixel point in the image plane; (r INR ,g INR ,b INR ) represents the RGB value of the image at that point. There exists a function f INR Able to map pixel coordinates to RGB values.

[0113] Based on the above theory, Figure 5 As shown, the dual-path implicit attention feature fusion module in step S2 of the present invention is used to optimize the transmission and fusion of feature information, wherein the specific processing process is as follows:

[0114] CS21, perform bilinear interpolation on the feature map θ output by the encoder and compare it with the query array cs map Apply the implicit attention mechanism together to get the fourth feature map

[0115] CS22, multi-scale fusion feature map After bilinear interpolation and query array cs map Apply the implicit attention mechanism together to get the fifth feature map

[0116] CS23, the fourth feature map Fifth characteristic diagram And query array cs map After concatenation, the implicit attention fusion features are obtained.

[0117] It should be noted that in steps CS21-CS22 of the present invention, different implicit attention mechanisms are applied to different feature maps, and parameters are not shared. In this embodiment, the implicit attention mechanism adopts non-local attention. In the implicit attention mechanism, the input feature map is the feature map output by the encoder after bilinear interpolation or the multi-scale fusion feature map after bilinear interpolation. Specifically, Figure 5 As shown, the input feature map passes through the eighth convolution layer to obtain the sixth feature map k map , the input feature map passes through the ninth convolution layer to obtain the seventh feature map v map , the input query array passes through the tenth convolutional layer and is combined with the sixth feature map k map After multiplication and SoftMax normalization, the eighth feature map is obtained. The eighth feature map is combined with the seventh feature map v mapMultiply them to obtain the feature map output by the implicit attention mechanism; among them, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

[0118] Furthermore, in the implicit attention mechanism, the 1×1 convolution layer uses 64 convolution kernels, the purpose of which is to change the number of channels of the input feature map to increase the depth, and subsequently connect to a nonlinear activation function so that the remote sensing image super-resolution model can learn more complex features.

[0119] It should be noted that, in step S2 of the present invention, Figure 3 As shown in Figure 1, the implicit high-resolution reconstruction module consists of the eleventh convolutional layer, three 3×3 convolutional layers, and an output layer. Each 3×3 convolutional layer uses dense connections and ReLU function activation. The convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

[0120] Furthermore, in the present invention, an implicit high-resolution reconstruction module is used to reconstruct a high-resolution image of the implicit attention fusion features, and finally a reconstructed super-resolution remote sensing image is obtained through a decoding operation, wherein the convolution operation does not reduce the resolution of the feature map and does not change the number of channels of the feature map.

[0121] The present invention will now use a specific example to demonstrate the application effect of the remote sensing image super-resolution method based on implicit neural representation described in S1 to S2 in the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0122] Example

[0123] In this embodiment, the public remote sensing image dataset UC Merced Land Use Dataset is used as source data to implement the remote sensing image super-resolution method based on implicit neural representation described in S1 to S2 above. The specific implementation process of the method is as described above and will not be repeated here.

[0124] First, obtain the original remote sensing image set from the official website. This dataset contains 21 types of land use images, each class has 100 images, the pixel resolution is 1 foot, and the image size is 256×256 pixels. Select (1.0,4.0] as the magnification range, and the low-resolution remote sensing image size is selected as 32×32 pixels. The randomly cropped remote sensing images are flipped horizontally with a probability of 0.5, flipped up and down with a probability of 0.5, and randomly rotated with a probability of 0.5, and Gaussian noise is added. Then, bicubic interpolation is downsampled to 32×32 pixels, and Gaussian noise is added after obtaining low-resolution remote sensing images, and finally 2100 image groups are obtained. Randomly divide 80% of the data in the image group into training data and 20% into verification data for subsequent training and verification of remote sensing image super-resolution models.

[0125] When training the remote sensing image super-resolution model, the input data will randomly select a batch of sample pairs from the image group each time, and the sample pairs have the same magnification. The initial learning rate is 0.001, the optimizer is the Adam optimizer, the betas parameter is (0.9, 0.999), the maximum number of iterations is 2000, and the batch size is 16. The remote sensing image super-resolution model is trained using the aforementioned training data until the remote sensing image super-resolution model converges, and then the performance of the remote sensing image super-resolution model is verified using the aforementioned verification data.

[0126] The encoder can adopt other basic super-resolution models. In this embodiment, an enhanced deep residual network (EDSR) is adopted, which is derived from the related paper "Enhanced Deep Residual Networks for Single Image Super-Resolution". The enhanced deep residual network specifically uses 16 residual blocks, and the convolution adopts 3×3 convolution with 64 channels without changing the image size, and removes the upsampling module.

[0127] In this embodiment, in order to evaluate the performance of the remote sensing image super-resolution model obtained by the above training, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), relative dimensionless global error (ERGAS) and learned perceptual image patch similarity (LPIPS) are used to evaluate the remote sensing image super-resolution model. In this embodiment, the above indicators are introduced as follows.

[0128] Peak signal-to-noise ratio (PSNR): It is usually used to evaluate the similarity between an image and the original image. The higher the PSNR value, the higher the similarity between the two images and the better the quality. The calculation formula is:

[0129]

[0130] Among them, I HR represents the original image, I SR Represents the generated super-resolution result image, Max(I HR ) represents the operation of taking the maximum value among all pixels of the original image, MSE(I HR ,I SR ) represents the pixel mean square error between the original image and the generated super-resolution result image, log 10 Represents the logarithm operation with base 10.

[0131] Structural Similarity Index SSIM: An index used to measure the similarity between two images. It not only considers brightness and contrast, but also structural information. The value range is [0,1]. The larger the SSIM value, the smaller the image distortion. SSIM is a combination of three different factors: brightness, contrast, and structure. Here we choose a general simplified calculation formula, which is:

[0132]

[0133] Among them, μ SR and μ HR Represent the average values ​​of the generated super-resolution image and the original image, σ SR and σ HR Represent the standard deviation of the generated super-resolution image and the original image respectively; σ HR , SR is the covariance between the generated super-resolution image and the original image; C1 and C2 are constants, usually C1 = (Ks1 P s ) 2 , C2=(K s2 P s ) 2 , P s is the maximum possible pixel value, K s1 and K s2 is a constant coefficient. In this embodiment, K s1 and K s2 0.01 and 0.03, respectively, P s is 255.

[0134] Relative dimensionless global error ERGAS: It is used to measure the relative global accuracy between the estimated image and the real image. It combines the spatial spectrum performance and the spectral performance to more comprehensively evaluate the quality of remote sensing images. The closer its value is to 0, the better the image quality. Its calculation formula is:

[0135]

[0136] Among them, l e and h e Represents the spatial resolution (pixel size) of low-resolution and high-resolution images respectively; N e Indicates the number of bands of remote sensing images; is the kth e The root mean square error of each band; It is the high-resolution image k e The average value of the bands.

[0137] Learning Perceptual Patch Similarity (LPIPS): Uses a pre-trained feature extraction network to measure the perceptual similarity between super-resolution results and real high-resolution images. LPIPS is more consistent with human perception than traditional methods. The lower the LPIPS value, the more similar the two images are. Its calculation formula can be expressed as:

[0138]

[0139] Among them, I SR Represents the generated super-resolution result image; I HR represents the original image; l represents the lth layer network; φ l represents the feature space of the lth layer of a pre-trained deep convolutional neural network; N l Represents the number of elements of the feature map obtained by the network layer l; ω l represents the learned weight vector; ⊙ represents the vector dot product of the channel dimension. Here, the pre-trained network selects the Alex network.

[0140] The method of the present invention is compared with the bicubic interpolation method on the above indicators, and the results are shown in Table 1. In the embodiment of the present invention, the reconstruction results of the method of the present invention and the bicubic interpolation method are compared under the condition of a magnification of 2.5. Figure 6 The reconstruction results of the method of the present invention and the bicubic interpolation method are compared under the condition of a magnification of 5.5. Figure 7 As shown in Table 1, all parameters of the remote sensing image super-resolution model at each magnification are better than the bicubic interpolation method, indicating that the remote sensing image super-resolution model of the present invention has good accuracy.

[0141] Table 1. Super-resolution reconstruction results at different magnifications

[0142]

[0143] In view of the deficiencies in the prior art, the present invention proposes a remote sensing image super-resolution method based on implicit neural representation. In the method of the present invention, a remote sensing image super-resolution model is first constructed, and different modules work together. Then, through training, the remote sensing image super-resolution model is enabled to recover high-resolution images from low-resolution images. The method provided by the present invention significantly improves the clarity and visual effect of the image, has high flexibility, supports super-resolution reconstruction of remote sensing images of continuous scales, and can freely select the magnification according to actual application requirements to meet the needs of images with different resolutions.

[0144] It should also be noted that the remote sensing image super-resolution method based on implicit neural representation in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a remote sensing image super-resolution system based on implicit neural representation corresponding to the remote sensing image super-resolution method based on implicit neural representation provided in the above embodiment, such as Figure 8 As shown, it includes:

[0145] A data acquisition module is used to acquire a remote sensing image data set suitable for a super-resolution task, wherein the remote sensing image data set includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0146] The result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features, and the implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image; wherein, the remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0147] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the remote sensing image super-resolution method based on implicit neural representation provided in the above embodiment, which includes a memory and a processor;

[0148] The memory is used to store computer programs;

[0149] The processor is used to implement the remote sensing image super-resolution method based on implicit neural representation in the above embodiment when executing the computer program.

[0150] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0151] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0153] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A remote sensing image super-resolution method based on implicit neural representation, characterized in that: The following steps are involved: S1. Acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset comprises a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image; S2. The low-resolution remote sensing image to be reconstructed and the pre-built query array are input into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed. Then, the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. The remote sensing image super-resolution model is trained on the remote sensing image dataset.

2. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, characterized in that: In step S1, the remote sensing image dataset is constructed as follows: S11, presetting the initial size of the low-resolution remote sensing image pixel during the sampling process, and performing a rounding operation on the result of multiplying the initial size by the random magnification, removing the decimal part of the multiplication result, and obtaining the target size of the super-resolution remote sensing image; S12, according to the calculated target size of the super-resolution remote sensing image, using a random cropping method to crop the acquired original remote sensing image to obtain a cropped remote sensing image; S13, performing an image enhancement operation on the cropped remote sensing image to obtain a remote sensing image after image enhancement processing and use it as a high-resolution target remote sensing image; S14, applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then perform a noise addition process to generate a low-resolution remote sensing image; S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

3. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, characterized in that: In step S2, the query array is constructed as follows: AS21. Using a low-resolution remote sensing image as an input image and a super-resolution remote sensing image as an output image, dividing a two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the input image, obtaining a multidimensional array formed by the coordinates of the grid center points of the input image, recorded as a first multidimensional array; dividing the two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the output image, obtaining a multidimensional array formed by the coordinates of the grid center points of the output image, recorded as a second multidimensional array; AS22, perform linear interpolation on the first multidimensional array to obtain a multidimensional array with the same shape as the second multidimensional array, recorded as a third multidimensional array; AS23. Subtract the third multidimensional array from the second multidimensional array to obtain a relative coordinate multidimensional array. AS24, broadcast the random magnifications to obtain a magnification array with the same shape as the second multidimensional array; AS25, concatenate the second multidimensional array, the relative coordinate multidimensional array, and the magnification array in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as query arrays.

4. The remote sensing image super-resolution method based on implicit neural representation according to claim 3, characterized in that: In step AS21, the first multidimensional array and the second multidimensional array are obtained as follows: i in =0,1,…,H in -1 j in =0,1,…,W in -1 i out =0,1,…,H out -1 j out =0,1,…,H out -1 Among them, the size of the input image is (H in ,W in ), H in Indicates the width of the input image, W in Indicates the height of the input image; i in and j in Respectively represent the row and column index of the input image; P in (i in ,j in ) represents the coordinates of the grid center point of the input image, from which the first multidimensional array is generated; the size of the output image is (H out ,W out ), H out Indicates the width of the output image, W out Indicates the height of the output image; i out and i out Respectively represent the row and column index of the output image; P out (i out ,i out ) represents the grid center point coordinates of the output image, from which the second multidimensional array is generated.

5. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, characterized in that: In step S2, the specific processing process in the hierarchical feature pyramid extraction module is as follows: BS21. First, the feature map output by the encoder is input into a parallel three-way sub-network. The first sub-network consists of one spatial channel downsampling block, the second sub-network consists of two spatial channel downsampling blocks, and the third sub-network consists of three spatial channel downsampling blocks, wherein each spatial channel downsampling block is cascaded in sequence by a first convolutional layer, a ReLU activation function, a second convolutional layer, and a ReLU activation function; the convolution kernel size of the first convolutional layer is 3×3, the number of convolution kernels is 64, and the step size is 1; the convolution kernel size of the second convolutional layer is 3×3, the number of convolution kernels is 64, and the step size is 2; BS22, input the feature map output by the first sub-network into the third convolutional layer to obtain a first feature map; perform a nearest neighbor interpolation operation on the feature map output by the third sub-network and input it into the fourth convolutional layer to obtain a second feature map; wherein the convolution kernel size of the third convolutional layer is 3×3, and the convolution kernel size of the fourth convolutional layer is 1×1; BS23, passing the feature map output by the encoder through the fifth convolution layer to obtain a third feature map, passing the third feature map through three processing branches respectively, each processing branch performs a maximum pooling operation and a nearest neighbor interpolation on the third feature map, and the output results of the three processing branches are spliced ​​in the channel dimension and then passed through the sixth convolution layer to obtain a spatial pyramid pooling feature map; wherein the convolution kernel size of the fifth convolution layer is 1×1, and the convolution kernel size of the sixth convolution layer is 1×1; BS24, the first feature map, the feature map output by the second sub-network, the second feature map and the spatial pyramid pooling feature map are spliced ​​in the channel dimension and then passed through the seventh convolution layer to obtain a multi-scale fusion feature map; wherein the convolution kernel size of the seventh convolution layer is 1×1.

6. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, characterized in that: In step S2, the specific processing process in the dual-path implicit attention feature fusion module is as follows: CS21, apply the implicit attention mechanism to the feature map output by the encoder and the query array after bilinear interpolation to obtain the fourth feature map; CS22, after bilinear interpolation of the multi-scale fusion feature map, apply the implicit attention mechanism together with the query array to obtain the fifth feature map; CS23. After concatenating the fourth feature map, the fifth feature map and the query array, the implicit attention fusion feature is obtained.

7. The remote sensing image super-resolution method based on implicit neural representation according to claim 6, characterized in that: The specific processing process in the implicit attention mechanism is as follows: the input feature map passes through the eighth convolution layer to obtain the sixth feature map, the input feature map passes through the ninth convolution layer to obtain the seventh feature map, the input query array passes through the tenth convolution layer and is multiplied element-by-element with the sixth feature map and subjected to the SoftMax normalization operation to obtain the eighth feature map, the eighth feature map is multiplied element-by-element with the seventh feature map to obtain the feature map output by the implicit attention mechanism; wherein, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

8. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, characterized in that: The implicit high-resolution reconstruction module consists of an eleventh convolutional layer, three 3×3 convolutional layers and an output layer. Each 3×3 convolutional layer adopts dense connection and ReLU function activation. The convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

9. A remote sensing image super-resolution system based on implicit neural representation, characterized in that: include: A data acquisition module is used to acquire a remote sensing image data set suitable for a super-resolution task, wherein the remote sensing image data set includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image; The result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features, and the implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image; wherein, the remote sensing image super-resolution model is trained on the remote sensing image dataset.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the remote sensing image super-resolution method based on implicit neural representation as described in any one of claims 1 to 8 when executing the computer program.

Images