Remote sensing image super-resolution method, system and device based on implicit neural representation

Abstract

The application discloses a remote sensing image super-resolution method, system and device based on implicit neural representation, and belongs to the technical field of deep learning and remote sensing image processing. The method of the application firstly acquires a remote sensing image data set suitable for remote sensing image data set and trains a remote sensing image super-resolution model based on the remote sensing image data set, then takes a low-resolution remote sensing image to be reconstructed and a query array constructed in advance as input, performs multi-scale feature extraction by a hierarchical feature pyramid extraction module, performs fusion by a double-path implicit attention feature fusion module, and finally performs high-resolution image reconstruction by an implicit high-resolution reconstruction module. The application combines a feature pyramid, an attention mechanism and the like with implicit neural representation, fuses features at different levels together, solves the problems of multi-scale feature fusion and efficient information extraction in the remote sensing image super-resolution process, can improve the remote sensing image super-resolution precision, and realizes the target of continuous scale remote sensing image super-resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and remote sensing image processing, and in particular relates to a remote sensing image super-resolution method, system and equipment based on implicit neural representation. Background Art

[0002] Remote sensing technology, which captures images and data of the Earth's surface through platforms such as satellites, aircraft, and drones, has a wide range of applications in environmental monitoring, natural disaster assessment, and urban planning. However, the spatial resolution of remote sensing images is limited by the physical limitations of sensors, making it difficult to meet the growing demand for high precision. Therefore, improving the spatial resolution of remote sensing images has become a crucial research topic.

[0003] Super-resolution (SR) technology aims to convert low-resolution images into high-resolution images through algorithmic means. This method can be divided into single-image SR (SISR) and multi-image SR (MISR). Traditional SR methods have some limitations, such as interpolation algorithms that easily lead to image blur, and reconstruction-based methods that are computationally intensive and sensitive to noise. With the rise of deep learning, especially the continuous development of convolutional neural networks (CNNs), super-resolution methods based on deep learning have gradually become a research hotspot. Compared with traditional methods, deep learning methods can extract high-dimensional features of images by learning from a large number of samples, thereby significantly improving super-resolution reconstruction results. Although these methods can improve image detail and clarity, the magnification factor is fixed or a combination of several fixed magnification factors. If a random magnification scale is required, the model architecture may need to be restructured.

[0004] Remote sensing imagery is an important source of geographic information. Unlike natural imagery, it boasts wide coverage, a large observation range, and rich spectral information. Different spatial resolutions can reveal different layers and structures of surface objects. Furthermore, a single task may involve multi-source, multi-resolution image data. The data resolution needs to be unified during analysis, which presents the problem of super-resolution at different scales. Currently, many methods employ multi-path learning at specific scales to address multi-scale super-resolution in a single network. These methods typically use a scale factor as input in the upsampling module design to dynamically adjust the results. However, such methods struggle to achieve optimal performance and may require a large number of parameters for calculation.

[0005] Implicit Neural Representations (INR) are a method that uses neural networks to represent input signals such as images, audio, and point clouds as functions. They have achieved impressive results in 3D reconstruction. The key idea is to represent an object as a function that maps coordinates to corresponding signals. In deep learning, this function is approximated using a trained network, which is known as an implicit neural representation. INR offers a new approach to SR technology, transforming low-resolution images into high-dimensional information representations in a coordinate space. High-resolution images can then be obtained by querying pixel features based on coordinates.

[0006] Most current INR-based SR methods are extensions of the Local Implicit Image Function (LIIF), which simply concatenates periodic transformations of coordinates and local features and uses a multilayer perceptron (MLP) network to predict RGB values. These methods primarily focus on local representations, but suffer from high computational complexity, high memory requirements, and high time overhead in the model feature extraction network. Therefore, it is necessary to construct a novel continuous-scale super-resolution method for remote sensing imagery based on implicit neural representations. Summary of the Invention

[0007] The purpose of this invention is to provide a remote sensing image super-resolution method, system, and device based on implicit neural representation to address the low generalization performance and poor feature reconstruction of existing methods. This invention aims to address the need for continuous-scale super-resolution in remote sensing image super-resolution reconstruction by leveraging the powerful representation capabilities of implicit neural representation networks. This technology represents a cutting-edge application at the intersection of computer vision and deep learning.

[0008] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a remote sensing image super-resolution method based on implicit neural representation, which comprises the following steps:

[0010] S1. Acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0011] S2. The low-resolution remote sensing image to be reconstructed and the pre-constructed query array are input together into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed. Then, the feature map output by the encoder is input into the hierarchical feature pyramid extraction module, and multi-scale feature extraction is performed on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-constructed query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. wherein, the remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0012] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0013] As a preferred embodiment of the first aspect, in step S1, the remote sensing image dataset is constructed as follows:

[0014] S11, presetting the initial size of the low-resolution remote sensing image pixel during the sampling process, and performing a rounding operation on the result of multiplying the initial size by the random magnification, removing the decimal part of the multiplication result, to obtain the target size of the super-resolution remote sensing image;

[0015] S12, cropping the acquired original remote sensing image using a random cropping method according to the calculated target size of the super-resolution remote sensing image to obtain a cropped remote sensing image;

[0016] S13, performing an image enhancement operation on the cropped remote sensing image to obtain an image-enhanced remote sensing image and using it as a high-resolution target remote sensing image;

[0017] S14, applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then performing a noise addition operation to generate a low-resolution remote sensing image;

[0018] S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

[0019] As a preferred embodiment of the first aspect, in step S2, the query array is constructed as follows:

[0020] AS21. Use a low-resolution remote sensing image as an input image and a super-resolution remote sensing image as an output image. Divide a two-dimensional rectangular coordinate system into corresponding grids based on the size of the input image to obtain a multidimensional array of coordinates of the grid centers of the input image, denoted as a first multidimensional array. Divide the two-dimensional rectangular coordinate system into corresponding grids based on the size of the output image to obtain a multidimensional array of coordinates of the grid centers of the output image, denoted as a second multidimensional array.

[0021] AS22. Perform linear interpolation on the first multidimensional array to obtain a multidimensional array with the same shape as the second multidimensional array, which is recorded as the third multidimensional array;

[0022] AS23. Subtract the third multidimensional array from the second multidimensional array to obtain a relative coordinate multidimensional array.

[0023] AS24. Broadcast the random magnifications to obtain a magnification array with the same shape as the second multidimensional array.

[0024] AS25. Concatenate the second multidimensional array, the relative coordinate multidimensional array, and the magnification array in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as query arrays.

[0025] As a preferred embodiment of the first aspect, in step AS21, the first multidimensional array and the second multidimensional array are obtained as follows:

[0026]

[0027] i in =0,1,…,H in -1

[0028] j in =0,1,…,W in -1

[0029]

[0030] i out =0,1,…,H out -1

[0031] j out =0,1,…,H out -1

[0032] Among them, the size of the input image is (H in ,W in ), H in Indicates the width of the input image, W in Indicates the height of the input image; i in and j inRepresents the row and column index of the input image respectively; P in (i in ,j in ) represents the coordinates of the grid center point of the input image, from which the first multidimensional array is generated; the size of the output image is (H out ,W out ), H out Indicates the width of the output image, W out Indicates the height of the output image; i out and i out Respectively represent the row and column index of the output image; P out (i out ,i out ) represents the coordinates of the grid center point of the output image, from which the second multidimensional array is generated.

[0033] As a preferred embodiment of the first aspect, in step S2, the specific processing process in the hierarchical feature pyramid extraction module is as follows:

[0034] BS21. First, the feature map output by the encoder is input into a parallel three-way sub-network. The first way network consists of one spatial channel downsampling block, the second way network consists of two spatial channel downsampling blocks, and the third way network consists of three spatial channel downsampling blocks. Each spatial channel downsampling block is composed of a first convolutional layer, a ReLU activation function, a second convolutional layer, and a ReLU activation function cascaded in sequence; the convolution kernel size of the first convolutional layer is 3×3, the number of convolution kernels is 64, and the stride is 1. The convolution kernel size of the second convolutional layer is 3×3, the number of convolution kernels is 64, and the stride is 2.

[0035] BS22: Input the feature map output by the first sub-network into the third convolutional layer to obtain a first feature map; perform a nearest neighbor interpolation operation on the feature map output by the third sub-network and input it into the fourth convolutional layer to obtain a second feature map; wherein the convolution kernel size of the third convolutional layer is 3×3, and the convolution kernel size of the fourth convolutional layer is 1×1;

[0036] BS23: Pass the feature map output by the encoder through the fifth convolutional layer to obtain a third feature map. The third feature map is passed through three processing branches respectively. Each processing branch performs a maximum pooling operation and nearest neighbor interpolation on the third feature map. The output results of the three processing branches are spliced ​​in the channel dimension and then passed through the sixth convolutional layer to obtain a spatial pyramid pooling feature map. The convolution kernel size of the fifth convolutional layer is 1×1, and the convolution kernel size of the sixth convolutional layer is 1×1.

[0037] BS24: The first feature map, the feature map output by the second sub-network, the second feature map, and the spatial pyramid pooling feature map are concatenated in the channel dimension and then passed through the seventh convolutional layer to obtain a multi-scale fusion feature map; among them, the convolution kernel size of the seventh convolutional layer is 1×1.

[0038] As a preferred embodiment of the first aspect, in step S2, the specific processing process in the dual-path implicit attention feature fusion module is as follows:

[0039] CS21, bilinearly interpolate the feature map output by the encoder and apply the implicit attention mechanism together with the query array to obtain the fourth feature map;

[0040] CS22, after bilinear interpolation of the multi-scale fusion feature map, apply the implicit attention mechanism together with the query array to obtain the fifth feature map;

[0041] CS23. After concatenating the fourth feature map, the fifth feature map, and the query array, the implicit attention fusion feature is obtained.

[0042] As a preferred embodiment of the above-mentioned first aspect, the specific processing process in the implicit attention mechanism is as follows: the input feature map passes through the eighth convolution layer to obtain the sixth feature map, the input feature map passes through the ninth convolution layer to obtain the seventh feature map, the input query array passes through the tenth convolution layer and is multiplied element-by-element with the sixth feature map and undergoes a SoftMax normalization operation to obtain the eighth feature map, the eighth feature map is multiplied element-by-element with the seventh feature map to obtain the feature map output by the implicit attention mechanism; wherein, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

[0043] As a preferred embodiment of the above-mentioned first aspect, the implicit high-resolution reconstruction module consists of an eleventh convolutional layer, three 3×3 convolutional layers and an output layer. Each 3×3 convolutional layer adopts dense connection and ReLU function activation. The convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

[0044] In a second aspect, the present invention provides a remote sensing image super-resolution system based on implicit neural representation, comprising:

[0045] A data acquisition module is used to acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0046] The result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. The feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are then input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. The remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0047] In a third aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0048] The memory is used to store computer programs;

[0049] The processor is configured to implement the remote sensing image super-resolution method based on implicit neural representation as described in any one of the solutions of the first aspect above when executing the computer program.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The method of the present invention requires only a unified remote sensing image super-resolution model, eliminating the need for predefined magnification factors and enabling super-resolution reconstruction at a continuous scale. This makes the remote sensing image super-resolution model extremely flexible and adaptable, enabling the generation of high-resolution images at different scales based on actual needs. By combining deep neural networks with implicit neural representations, the method of the present invention accurately captures and reconstructs image details, significantly improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0053] Figure 2 Schematic diagram of the remote sensing image super-resolution model of the present invention;

[0054] Figure 3 Schematic diagram of the network structure of the remote sensing image super-resolution model of the present invention;

[0055] Figure 4 This is a structural diagram of the hierarchical feature pyramid extraction module in the present invention;

[0056] Figure 5Schematic diagram of the dual-path implicit attention feature fusion module in the present invention;

[0057] Figure 6 This is a schematic diagram of a reconstruction result with a magnification of 2.5 provided in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of a reconstruction result with a magnification of 5.5 provided in an embodiment of the present invention;

[0059] Figure 8 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0061] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0062] like Figure 1 As shown, in a preferred implementation of the present invention, the remote sensing image super-resolution method based on implicit neural representation includes the following steps S1 to S2. The specific implementation process is described below.

[0063] S1. Obtain a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes low-resolution remote sensing images and corresponding high-resolution target remote sensing images.

[0064] It should be noted that in the present invention, remote sensing image datasets are used to train remote sensing image super-resolution models based on implicit neural representations. During the training process of the remote sensing image super-resolution model, the high-resolution target remote sensing image can be regarded as the label (ground truth). The training goal is to make the super-resolution remote sensing image reconstructed by the remote sensing image super-resolution model as close as possible to the high-resolution target remote sensing image. When the remote sensing image super-resolution model performs inference, only the low-resolution remote sensing image to be reconstructed needs to be obtained, and the trained remote sensing image super-resolution model outputs the reconstructed super-resolution remote sensing image.

[0065] It should be noted that, in step S1 of the present invention, the method for constructing the remote sensing image dataset is as follows:

[0066] S11. Preset the initial size of the low-resolution remote sensing image pixel during the sampling process, round the result of multiplying the initial size by the random magnification, remove the decimal part of the multiplication result, and obtain the target size of the super-resolution remote sensing image.

[0067] It should be noted that step S11 of the present invention is a low-resolution sampling design. In the process of obtaining the random magnification, as a preferred method, this embodiment uses a uniform distribution sampling function to generate the random magnification.

[0068] S12. According to the calculated target size of the super-resolution remote sensing image, the obtained original remote sensing image is cropped using a random cropping method to obtain a cropped remote sensing image.

[0069] It should be noted that, in step S12 of the present invention, the cropping range is selected from a preset random area within the original remote sensing image to ensure the diversity and randomness of the high-resolution target remote sensing image.

[0070] S13. Performing an image enhancement operation on the cropped remote sensing image to obtain an enhanced remote sensing image and using it as a high-resolution target remote sensing image.

[0071] It should be noted that in step S13 of the embodiment of the present invention, the image enhancement operations include, but are not limited to, performing horizontal flipping, upside down flipping, and random rotation operations with random probabilities, and introducing random noise to enhance the complexity of the cropped remote sensing image. The random probability is used to determine whether the cropped remote sensing image is rotated. If the cropped remote sensing image requires rotation, there is an equal probability of performing 90°, 180°, and 270° rotations.

[0072] S14. Applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then performing noise processing to generate a low-resolution remote sensing image.

[0073] It should be noted that, in step S14 of the present invention, image noise is introduced to make the generated low-resolution remote sensing image closer to the actual data distribution.

[0074] S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

[0075] S2. The low-resolution remote sensing image to be reconstructed and the pre-built query array are input into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map θ output by the encoder is input into the hierarchical feature pyramid extraction module to extract multi-scale features of the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then the feature map θ output by the encoder and the multi-scale fusion feature map The pre-built query array is input into the dual-path implicit attention feature fusion module to obtain the implicit attention fusion feature, and the implicit attention fusion feature is input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image; wherein the above-mentioned remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0076] It should be noted that, in step S2 of the present invention, the method for constructing the query array is as follows:

[0077] AS21. Use the low-resolution remote sensing image as the input image and the super-resolution remote sensing image as the output image. Divide the two-dimensional rectangular coordinate system into corresponding grids according to the size of the input image. Obtain a multidimensional array formed by the coordinates of the grid center points of the input image, which is recorded as the first multidimensional array lr. coord , divide the two-dimensional plane rectangular coordinate system into corresponding grids according to the size of the output image, and obtain the multidimensional array formed by the coordinates of the grid center points of the output image, which is recorded as the second multidimensional array hr coord .

[0078] It should be noted that in step AS21 of the present invention, the second multidimensional array hr coord Right now Figure 2 and Figure 5 The super-resolution coordinate information referred to in .

[0079] It should be noted that, in step AS21 of the present invention, the first multidimensional array lr is obtained. coord and the second multidimensional array hr coord The way is as follows:

[0080]

[0081] i in =0,1,…,H in -1

[0082] j in =0,1,…,W in -1

[0083]

[0084] i out =0,1,…,H out -1

[0085] j out =0,1,…,H out -1

[0086] Among them, the size of the input image is (H in ,W in ), H in Indicates the width of the input image, W in Indicates the height of the input image; i in and j in Represents the row and column index of the input image respectively; P in (i in ,j in ) represents the grid center coordinates of the input image, from which the first multidimensional array lr is generated coord ; The size of the output image is (H out ,W out ), H out Indicates the width of the output image, W out Indicates the height of the output image; i out and i out Respectively represent the row and column index of the output image; P out (i out ,i out ) represents the grid center coordinates of the output image, from which the second multidimensional array hr is generated coord .

[0087] AS22, the first multidimensional array lr coord Perform linear interpolation to obtain the same value as the second multidimensional array hr coord A multidimensional array of the same shape, denoted as the third multidimensional array

[0088] AS23, using the second multidimensional array hr coord Subtract the third multidimensional array Get the relative coordinate multidimensional array rel coord .

[0089] It should be noted that in step AS23 of the present invention, the relative coordinate multidimensional array rel coord Right now Figure 2 and Figure 5 The relative coordinate information referred to in .

[0090] AS24. Broadcast the random magnification to obtain the value corresponding to the second multidimensional array hr. coord An array s of magnifications of the same shape map .

[0091] It should be noted that in step AS24 of the present invention, the magnification array s map Right now Figure 2 and Figure 5 The magnification information indicated in the figure.

[0092] AS25, the second multidimensional array hr coord , relative coordinate multidimensional array rel coord and the magnification array s map Perform splicing operations in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as query array cs map .

[0093] It should be noted that in step AS25 of the present invention, the second multidimensional array hr coord , relative coordinate multidimensional array rel coord and the magnification array s map All three arrays have the same width and height.

[0094] It should be noted that, in the present invention, the feature map refers to the result obtained by convolution operation of the input image by a series of convolution kernels, which can be regarded as an abstract representation of the original image, in which each pixel represents some specific features.

[0095] It should be noted that in step S2 of the present invention, the encoder uses a conventional super-resolution model to initially extract multi-scale features, and its structure is not limited in the present invention. The feature map output by the encoder is denoted as θ.

[0096] It should be noted that if Figure 4 As shown, the specific processing process in the hierarchical feature pyramid extraction module in step S2 of the present invention is as follows:

[0097] BS21. First, the feature map θ output by the encoder is input into a parallel three-way sub-network. The first sub-network consists of one spatial channel downsampling block, the second sub-network consists of two spatial channel downsampling blocks, and the third sub-network consists of three spatial channel downsampling blocks. Each spatial channel downsampling block is composed of the first convolutional layer, ReLU activation function, second convolutional layer and ReLU activation function cascaded in sequence.

[0098] It should be noted that in step BS21 of the present invention, all parameters of the three sub-networks are not shared. The first sub-network reduces the spatial dimension of the feature map θ output by the encoder, reducing the width and height of the feature map θ to 1 / 2 of the original size, and at the same time increasing the number of channels to twice the original size. The feature map output by the first sub-network is recorded as The second sub-network reduces the width and height of the feature map θ to 1 / 4 of the original, but increases the number of channels to four times the original. The feature map output by the second sub-network is recorded as The third sub-network reduces the width and height of the feature map θ to 1 / 8 of the original, but increases the number of channels to eight times the original. The feature map output by the third sub-network is recorded as in, The size of (b x ,c x ×2,w x / 2,h x / 2), The size of (b x ,c x ×4,w x / 4,h x / 4), The size of (b x ,c x ×8,w x / 8,h x / 8), b x represents the batch size of low-resolution remote sensing images; c x Indicates the number of low-resolution remote sensing image channels; w x The number of pixels representing the width of low-resolution remote sensing images; h x The number of pixels representing the height of low-resolution remote sensing images.

[0099] It should be noted that in the spatial channel downsampling block in step BS21 of the present invention, the first convolutional layer has a 3×3 kernel size, 64 kernels, and a stride of 1, which is used to maintain the width and height of the feature map θ unchanged while doubling the number of channels. The second convolutional layer has a 3×3 kernel size, 64 kernels, and a stride of 2, which is used to maintain the number of channels in the feature map θ unchanged. Here, the first and second convolutional layers are used to extract fine-grained features, and the center grid is the center of the receptive field, which can capture feature changes in the vertical, horizontal, and diagonal directions.

[0100] In the three-way parallel sub-network of step BS21 in the embodiment of the present invention, the feature extraction process can be expressed as:

[0101]

[0102]

[0103] Among them, Conv 3×3 Represents a convolutional layer containing a 3×3 convolution kernel.

[0104] BS22, the feature map output by the first sub-network Input to the third convolutional layer, changing the feature map Width and height, get the first feature map Feature map of the third sub-network output After the nearest neighbor interpolation operation is performed, it is input into the fourth convolutional layer to change the feature map Width and height, get the second feature map Among them, the convolution kernel size of the third convolution layer is 3×3, and the convolution kernel size of the fourth convolution layer is 1×1.

[0105] It should be noted that in step BS22 of the present invention, the first feature map and the second feature map The width and height of same.

[0106] BS23, the feature map output by the encoder passes through the fifth convolution layer to obtain the third feature map, the third feature map passes through three processing branches respectively, each processing branch performs maximum pooling operation and nearest neighbor interpolation on the third feature map, the output results of the three processing branches are spliced ​​in the channel dimension and then pass through the sixth convolution layer to obtain the spatial pyramid pooling feature map Among them, the convolution kernel size of the fifth convolution layer is 1×1, and the convolution kernel size of the sixth convolution layer is 1×1.

[0107] It should be noted that, in step BS23 of the present invention, the fifth convolutional layer is used to capture multi-scale features and enhance the robustness to changes in the size of features at different levels.

[0108] It should be noted that in step BS23 of the present invention, the spatial pyramid pooling feature map Width and height same.

[0109] BS24, the first feature map Feature map output by the second sub-network Second feature map Spatial pyramid pooling feature map After splicing in the channel dimension and then passing through the seventh convolution layer, a multi-scale fusion feature map is obtained. Among them, the convolution kernel size of the seventh convolutional layer is 1×1.

[0110] It should be noted that the implicit neural representation in the image field can be expressed as:

[0111] f INR :R 2 →R 3 ,f INR (x INR ,y INR )=(r INR , g INR,b INR )

[0112] Among them, R 2 Represents two-dimensional space; R 3 Represents three-dimensional space; (x INR ,y INR ) represents the coordinates of any pixel point in the image plane; (r INR ,g INR ,b INR ) represents the RGB value of the image at that point, and there exists a function f INR Able to map pixel coordinates to RGB values.

[0113] Based on the above theory, Figure 5 As shown, the dual-path implicit attention feature fusion module in step S2 of the present invention is used to optimize the transmission and fusion of feature information, wherein the specific processing process is as follows:

[0114] CS21, perform bilinear interpolation on the feature map θ output by the encoder and compare it with the query array cs map Apply the implicit attention mechanism together to obtain the fourth feature map

[0115] CS22, multi-scale fusion feature map After bilinear interpolation and query array cs map Apply the implicit attention mechanism together to obtain the fifth feature map

[0116] CS23, the fourth feature map Fifth characteristic map And query array cs map After splicing, the implicit attention fusion feature is obtained.

[0117] It should be noted that in steps CS21-CS22 of the present invention, different implicit attention mechanisms are applied to different feature maps, and parameters are not shared. In this embodiment, the implicit attention mechanism adopts non-local attention. In the implicit attention mechanism, the input feature map is the feature map output by the encoder after bilinear interpolation or the multi-scale fusion feature map after bilinear interpolation. Specifically, Figure 5 As shown, the input feature map passes through the eighth convolution layer to obtain the sixth feature map k map , the input feature map passes through the ninth convolution layer to obtain the seventh feature map v map , the input query array passes through the tenth convolution layer and is combined with the sixth feature map k map After multiplication and SoftMax normalization, the eighth feature map is obtained. The eighth feature map is combined with the seventh feature map v mapMultiply them to obtain the feature map output by the implicit attention mechanism; among them, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

[0118] Furthermore, in the implicit attention mechanism, the 1×1 convolution layer uses 64 convolution kernels. Its purpose is to change the number of channels of the input feature map to increase the depth. The subsequent access to nonlinear activation functions enables the remote sensing image super-resolution model to learn more complex features.

[0119] It should be noted that, in step S2 of the present invention, Figure 3 As shown in Figure 1, the implicit high-resolution reconstruction module consists of the eleventh convolutional layer, three 3×3 convolutional layers, and an output layer. Each 3×3 convolutional layer uses dense connections and ReLU function activation. The convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

[0120] Furthermore, in the present invention, the implicit high-resolution reconstruction module is used to reconstruct a high-resolution image of the implicit attention fusion features, and finally a reconstructed super-resolution remote sensing image is obtained through a decoding operation, wherein the convolution operation does not reduce the resolution of the feature map and does not change the number of channels of the feature map.

[0121] The present invention will now use a specific example to demonstrate the application effect of the remote sensing image super-resolution method based on implicit neural representation described in S1 to S2 of the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0122] Example

[0123] In this embodiment, the public remote sensing image dataset UC Merced Land Use Dataset is used as the source data to implement the remote sensing image super-resolution method based on implicit neural representation described in S1 to S2 above. The specific implementation process of this method is as described above and will not be repeated here.

[0124] First, we obtained a set of original remote sensing images from the official website. This dataset contains 21 land use image categories, with 100 images per category, a pixel resolution of 1 foot, and an image size of 256×256 pixels. We selected the magnification range (1.0, 4.0) and the low-resolution remote sensing image size of 32×32 pixels. We then flipped the randomly cropped remote sensing images horizontally with a probability of 0.5, flipped them upside down with a probability of 0.5, and randomly rotated them with a probability of 0.5. We then added Gaussian noise to the images and then downsampled them to 32×32 pixels using bicubic interpolation. After adding Gaussian noise to the low-resolution remote sensing images, we finally obtained 2,100 image groups. We randomly divided 80% of the data in the image group into training data and 20% into validation data for subsequent training and validation of the remote sensing image super-resolution model.

[0125] When training the remote sensing image super-resolution model, the input data is randomly selected from the image set, with the same magnification. The initial learning rate is 0.001, the Adam optimizer is used, the beta parameter is (0.9, 0.999), the maximum number of iterations is 2000, and the batch size is 16. The remote sensing image super-resolution model is trained using the aforementioned training data until convergence. The performance of the remote sensing image super-resolution model is then verified using the aforementioned validation data.

[0126] The encoder can adopt other basic super-resolution models. In this embodiment, the enhanced deep residual network (EDSR) is adopted, which is derived from the related paper "Enhanced Deep Residual Networks for Single Image Super-Resolution". The enhanced deep residual network specifically uses 16 residual blocks, and the convolution adopts 3×3 convolution with 64 channels without changing the image size, and removes the upsampling module.

[0127] In this embodiment, in order to evaluate the performance of the remote sensing image super-resolution model obtained by the above training, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), relative dimensionless global error (ERGAS) and learned perceptual image patch similarity (LPIPS) are used to evaluate the remote sensing image super-resolution model. In this embodiment, the above indicators are introduced as follows.

[0128] Peak Signal-to-Noise Ratio (PSNR): This is commonly used to evaluate the similarity between an image and the original image. The higher the PSNR value, the higher the similarity between the two images and the better the quality. The calculation formula is:

[0129]

[0130] Among them, I HR represents the original image, I SR Represents the generated super-resolution result image, Max(I HR ) represents the operation of taking the maximum value of all pixels in the original image, MSE(I HR ,I SR ) represents the pixel mean square error between the original image and the generated super-resolution result image, log 10 Represents the base 10 logarithm operation.

[0131] Structural Similarity Index (SSIM): This is a metric used to measure the similarity between two images. It takes into account not only brightness and contrast but also structural information. The value range is [0, 1]. A larger SSIM value indicates less image distortion. SSIM is a combination of three different factors: brightness, contrast, and structure. Here, a simplified general calculation formula is used:

[0132]

[0133] Among them, μ SR and μ HR Represent the average values ​​of the generated super-resolution image and the original image, σ SR and σ HR Represents the standard deviation of the generated super-resolution image and the original image respectively; σ HR , SR is the covariance between the generated super-resolution image and the original image; C1 and C2 are constants, usually C1=(Ks1 P s ) 2 , C2=(K s2 P s ) 2 , P s is the maximum possible pixel value, K s1 and K s2 is a constant coefficient. In this embodiment, K s1 and K s2 0.01 and 0.03, respectively, P s is 255.

[0134] Relative dimensionless global error (ERGAS): This is used to measure the relative global accuracy between the estimated image and the true image. It combines spatial spectrum performance and spectral performance to more comprehensively evaluate the quality of remote sensing images. The closer its value is to 0, the better the image quality. Its calculation formula is:

[0135]

[0136] Among them, l e and h e Represents the spatial resolution (pixel size) of low-resolution and high-resolution images respectively; N e Indicates the number of bands of remote sensing images; is the kth e The root mean square error of each band; It is the high-resolution image k e The average value of the bands.

[0137] Learning Perceptual Patch Similarity (LPIPS): This method uses a pre-trained feature extraction network to measure the perceptual similarity between the super-resolution result and the true high-resolution image. LPIPS is more consistent with human perception than traditional methods. Lower LPIPS values ​​indicate greater similarity between the two images. Its calculation formula can be expressed as:

[0138]

[0139] Among them, I SR Represents the generated super-resolution result image; I HR represents the original image; l represents the lth layer network; φ l Represents the feature space of the lth layer of a pre-trained deep convolutional neural network; N l Indicates the number of elements in the feature map obtained by the network layer l; ω l Here, the pre-trained network is the Alex network.

[0140] The method of the present invention is compared with the bicubic interpolation method on the above indicators, and the results are shown in Table 1. In the embodiment of the present invention, the reconstruction results of the method of the present invention and the bicubic interpolation method are compared under the condition of a magnification of 2.5. Figure 6 The reconstruction results of the method of the present invention and the bicubic interpolation method are compared under the condition of a magnification of 5.5. Figure 7 As shown in Table 1, all parameters of the remote sensing image super-resolution model at each magnification are better than those of the bicubic interpolation method, indicating that the remote sensing image super-resolution model of the present invention has good accuracy.

[0141] Table 1. Super-resolution reconstruction results at different magnifications

[0142]

[0143] In response to the shortcomings of the existing technology, the present invention proposes a remote sensing image super-resolution method based on implicit neural representation. In the method of the present invention, a remote sensing image super-resolution model is first constructed, and different modules work together. Then, through training, the remote sensing image super-resolution model is enabled to recover high-resolution images from low-resolution images. The method provided by the present invention significantly improves the clarity and visual effect of the image, has high flexibility, supports super-resolution reconstruction of remote sensing images of continuous scales, and can freely select the magnification according to actual application requirements to meet the needs of images with different resolutions.

[0144] It should also be noted that the remote sensing image super-resolution method based on implicit neural representation in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a remote sensing image super-resolution system based on implicit neural representation corresponding to the remote sensing image super-resolution method based on implicit neural representation provided in the above embodiment, such as Figure 8 As shown, it includes:

[0145] A data acquisition module is used to acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image;

[0146] The result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. The feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are then input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. The remote sensing image super-resolution model is trained on the remote sensing image dataset.

[0147] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the remote sensing image super-resolution method based on implicit neural representation provided in the above embodiment, which includes a memory and a processor;

[0148] The memory is used to store computer programs;

[0149] The processor is configured to implement the remote sensing image super-resolution method based on implicit neural representation in the above embodiment when executing the computer program.

[0150] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0151] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0152] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0153] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A remote sensing image super-resolution method based on implicit neural representation, characterized in that: The following steps are involved: S1. Acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image; S2. Inputting the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model, firstly encoding the input low-resolution remote sensing image to be reconstructed by the encoder, then inputting the feature map output by the encoder into the hierarchical feature pyramid extraction module, performing multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map, then inputting the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features, and inputting the implicit attention fusion features into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image; wherein, the remote sensing image super-resolution model is trained on the remote sensing image dataset; In step S2, the query array is constructed as follows: AS21. Use a low-resolution remote sensing image as an input image and a super-resolution remote sensing image as an output image. Divide a two-dimensional rectangular coordinate system into corresponding grids based on the size of the input image to obtain a multidimensional array of coordinates of the grid centers of the input image, denoted as a first multidimensional array. Divide the two-dimensional rectangular coordinate system into corresponding grids based on the size of the output image to obtain a multidimensional array of coordinates of the grid centers of the output image, denoted as a second multidimensional array. AS22. Perform linear interpolation on the first multidimensional array to obtain a multidimensional array with the same shape as the second multidimensional array, which is recorded as the third multidimensional array; AS23. Subtract the third multidimensional array from the second multidimensional array to obtain a relative coordinate multidimensional array. AS24. Broadcast the random magnifications to obtain a magnification array with the same shape as the second multidimensional array. AS25. Concatenate the second multidimensional array, the relative coordinate multidimensional array, and the magnification array in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as the query array. In step AS21, the first multidimensional array and the second multidimensional array are obtained as follows: ； ； ； ； ； ； The size of the input image is , Indicates the width of the input image. Indicates the height of the input image; and Represent the row and column indices of the input image respectively; Represents the grid center point coordinates of the input image, from which the first multidimensional array is generated; the size of the output image is , Indicates the width of the output image. Indicates the height of the output image; and Represent the row and column indices of the output image respectively; The coordinates of the grid center points of the output image, from which the second multidimensional array is generated.

2. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, wherein: In step S1, the remote sensing image dataset is constructed as follows: S11, presetting the initial size of the low-resolution remote sensing image pixel during the sampling process, and performing a rounding operation on the result of multiplying the initial size by the random magnification, removing the decimal part of the multiplication result, to obtain the target size of the super-resolution remote sensing image; S12, cropping the acquired original remote sensing image using a random cropping method according to the calculated target size of the super-resolution remote sensing image to obtain a cropped remote sensing image; S13, performing an image enhancement operation on the cropped remote sensing image to obtain an image-enhanced remote sensing image and using it as a high-resolution target remote sensing image; S14, applying a bicubic interpolation method to the remote sensing image after image enhancement processing to perform a downsampling operation and then performing a noise addition operation to generate a low-resolution remote sensing image; S15. Pair the generated low-resolution remote sensing image with the corresponding high-resolution target remote sensing image to form a super-resolution training sample pair, and then randomly select a certain proportion of super-resolution training sample pairs from the constructed super-resolution training sample pairs to construct a remote sensing image dataset.

3. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, wherein: In step S2, the specific processing process in the hierarchical feature pyramid extraction module is as follows: BS21. First, the feature map output by the encoder is input into a parallel three-way sub-network. The first way network consists of one spatial channel downsampling block, the second way network consists of two spatial channel downsampling blocks, and the third way network consists of three spatial channel downsampling blocks. Each spatial channel downsampling block is composed of a first convolutional layer, a ReLU activation function, a second convolutional layer, and a ReLU activation function cascaded in sequence; the convolution kernel size of the first convolutional layer is 3×3, the number of convolution kernels is 64, and the stride is 1. The convolution kernel size of the second convolutional layer is 3×3, the number of convolution kernels is 64, and the stride is 2. BS22: Input the feature map output by the first sub-network into the third convolutional layer to obtain a first feature map; perform a nearest neighbor interpolation operation on the feature map output by the third sub-network and input it into the fourth convolutional layer to obtain a second feature map; wherein the convolution kernel size of the third convolutional layer is 3×3, and the convolution kernel size of the fourth convolutional layer is 1×1; BS23: Pass the feature map output by the encoder through the fifth convolutional layer to obtain a third feature map. The third feature map is passed through three processing branches respectively. Each processing branch performs a maximum pooling operation and nearest neighbor interpolation on the third feature map. The output results of the three processing branches are spliced ​​in the channel dimension and then passed through the sixth convolutional layer to obtain a spatial pyramid pooling feature map. The convolution kernel size of the fifth convolutional layer is 1×1, and the convolution kernel size of the sixth convolutional layer is 1×1. BS24: The first feature map, the feature map output by the second sub-network, the second feature map, and the spatial pyramid pooling feature map are concatenated in the channel dimension and then passed through the seventh convolutional layer to obtain a multi-scale fusion feature map; among them, the convolution kernel size of the seventh convolutional layer is 1×1.

4. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, wherein: In step S2, the specific processing process in the dual-path implicit attention feature fusion module is as follows: CS21, bilinearly interpolate the feature map output by the encoder and apply the implicit attention mechanism together with the query array to obtain the fourth feature map; CS22, after bilinear interpolation of the multi-scale fusion feature map, apply the implicit attention mechanism together with the query array to obtain the fifth feature map; CS23. After concatenating the fourth feature map, the fifth feature map, and the query array, the implicit attention fusion feature is obtained.

5. The remote sensing image super-resolution method based on implicit neural representation according to claim 4, characterized in that: The specific processing process in the implicit attention mechanism is as follows: the input feature map passes through the eighth convolution layer to obtain the sixth feature map, the input feature map passes through the ninth convolution layer to obtain the seventh feature map, the input query array passes through the tenth convolution layer and is multiplied element-wise with the sixth feature map and subjected to the SoftMax normalization operation to obtain the eighth feature map, the eighth feature map is multiplied element-wise with the seventh feature map to obtain the feature map output by the implicit attention mechanism; among them, the convolution kernel size of the eighth convolution layer is 1×1, the convolution kernel size of the ninth convolution layer is 1×1, and the convolution kernel size of the tenth convolution layer is 1×1.

6. The remote sensing image super-resolution method based on implicit neural representation according to claim 1, wherein: The implicit high-resolution reconstruction module consists of the eleventh convolutional layer, three 3×3 convolutional layers and an output layer. Each 3×3 convolutional layer adopts dense connection and ReLU function activation. The convolution kernel size of the eleventh convolutional layer is 1×1, and the convolution kernel size of the output layer is 1×1.

7. A remote sensing image super-resolution system based on implicit neural representation, characterized in that: include: A data acquisition module is used to acquire a remote sensing image dataset suitable for a super-resolution task, wherein the remote sensing image dataset includes a low-resolution remote sensing image and a corresponding high-resolution target remote sensing image; A result acquisition module is used to input the low-resolution remote sensing image to be reconstructed and the pre-built query array into the trained remote sensing image super-resolution model. First, the encoder encodes the input low-resolution remote sensing image to be reconstructed, and then the feature map output by the encoder is input into the hierarchical feature pyramid extraction module to perform multi-scale feature extraction on the low-resolution remote sensing image to obtain a multi-scale fusion feature map. Then, the feature map output by the encoder, the multi-scale fusion feature map and the pre-built query array are input into the dual-path implicit attention feature fusion module to obtain implicit attention fusion features. The implicit attention fusion features are input into the implicit high-resolution reconstruction module to reconstruct the high-resolution image to obtain a reconstructed super-resolution remote sensing image. The remote sensing image super-resolution model is trained on the remote sensing image dataset. In the result acquisition module, the query array is constructed as follows: AS21. Use a low-resolution remote sensing image as an input image and a super-resolution remote sensing image as an output image. Divide a two-dimensional rectangular coordinate system into corresponding grids based on the size of the input image to obtain a multidimensional array of coordinates of the grid centers of the input image, denoted as a first multidimensional array. Divide the two-dimensional rectangular coordinate system into corresponding grids based on the size of the output image to obtain a multidimensional array of coordinates of the grid centers of the output image, denoted as a second multidimensional array. AS22. Perform linear interpolation on the first multidimensional array to obtain a multidimensional array with the same shape as the second multidimensional array, which is recorded as the third multidimensional array; AS23. Subtract the third multidimensional array from the second multidimensional array to obtain a relative coordinate multidimensional array. AS24. Broadcast the random magnifications to obtain a magnification array with the same shape as the second multidimensional array. AS25. Concatenate the second multidimensional array, the relative coordinate multidimensional array, and the magnification array in the channel dimension to obtain arrays of relevant coordinates and magnifications and use them as the query array. In step AS21, the first multidimensional array and the second multidimensional array are obtained as follows: ； ； ； ； ； ； The size of the input image is , Indicates the width of the input image. Indicates the height of the input image; and Represent the row and column indices of the input image respectively; Represents the grid center point coordinates of the input image, from which the first multidimensional array is generated; the size of the output image is , Indicates the width of the output image. Indicates the height of the output image; and Represent the row and column indices of the output image respectively; The coordinates of the grid center points of the output image, from which the second multidimensional array is generated.

8. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the remote sensing image super-resolution method based on implicit neural representation as described in any one of claims 1 to 6 when executing the computer program.

Images