A digital elevation model super-resolution reconstruction method based on self-similarity and parameterized convolution

Abstract

The application discloses a kind of based on self-similarity and parameterized convolution digital elevation model super-resolution reconstruction method, belong to computer image processing field, including: first, high-resolution DEM data is preprocessed, generates high, low-resolution image data set. Then, wavelet transform is carried out to the convolution base in the isometric convolution network based on partial differential operator, low-frequency and high-frequency information is extracted, and the base is reconstructed by inverse wavelet transform. The reconstructed base is input into the isometric convolution network for training, and a model with translation, rotation and scale isometricity is constructed. Finally, the low-resolution DEM image is input into the trained model, and the super-resolution reconstruction of high-resolution DEM image is realized. The method fully utilizes the self-similarity of DEM data, combines the characteristics of isometric convolution network, and effectively improves the accuracy and efficiency of DEM super-resolution reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing, and in particular relates to a super-resolution reconstruction method for digital elevation models based on self-similarity and parameterized convolution. Background Technology

[0002] Self-similarity specifically refers to the correlation and similarity between the whole and its parts in a complex system, where certain parts of the whole can reflect certain characteristics of the whole. In natural geography, watershed morphology, terrain texture, and contour line patterns exhibit strong self-similarity. Similarly, image textures also possess self-similarity. Because image textures exhibit self-similarity, image self-similarity has been used in experiments reconstructing super-resolution images to study the mapping relationship between lower-resolution and higher-resolution images.

[0003] Research on super-resolution reconstruction models of Digital Elevation Models (DEMs) has a long history. With the development of computer technology and artificial intelligence, it has become increasingly clear that deep learning-based methods can effectively utilize high-performance computer hardware to abstractly learn the mapping relationship between high-resolution and low-resolution data, thereby improving DEM accuracy.

[0004] For DEM super-resolution tasks, the inaccuracies arising from applying filters from the continuous domain to the discrete domain in current filter parameterization methods often affect the performance of corresponding equivariant convolutions. Especially in low-level image processing tasks, the rotational equivariance of local features is crucial. The concepts of filter parameterization and equivariant convolution are significant for terrain data with self-similarity. However, how to fully utilize the relationship between self-similarity and equivariance in DEM super-resolution reconstruction models, through translational equivariance, rotational equivariance, and scale equivariance, to efficiently and accurately complete DEM super-resolution reconstruction has not been fully and accurately explored. A super-resolution reconstruction method that can effectively utilize the self-similarity of DEMs and the translational, rotational, and scale equivariance of equivariant convolutional networks is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for super-resolution reconstruction of digital elevation models based on self-similarity and parameterized convolution, comprising:

[0006] Data preprocessing is performed on high-resolution digital elevation model data to obtain high-resolution images and corresponding low-resolution images, and the data is divided into training datasets and test datasets.

[0007] Based on the training dataset, wavelet transform is performed on the basis used for convolution operations obtained in the equivariant convolutional network based on partial differential operators to obtain wavelet transform coefficients containing low-frequency and high-frequency components.

[0008] Based on the wavelet transform coefficients, perform inverse wavelet transform to reconstruct the signal and obtain the basis after wavelet transform;

[0009] Based on the wavelet transform-transformed basis, the model is trained in the equivariant convolutional network to obtain an equivariant convolutional network model that simultaneously possesses translation equivariance, rotation equivariance, and scale equivariance.

[0010] Based on the test dataset, the preprocessed low-resolution digital elevation model image is input into the trained equivariant convolutional network model for super-resolution reconstruction to obtain a high-resolution digital elevation model image.

[0011] Preferably, the process of preprocessing the high-resolution digital elevation model data includes:

[0012] The elevation values ​​are normalized to the [0-255] range to generate grayscale images; the grayscale images are cut into image blocks that meet the network input requirements and divided into training datasets and test datasets; the high-resolution images in the training dataset are downsampled by bicubic interpolation to obtain low-resolution images.

[0013] Preferably, the process of normalizing the elevation values ​​to the [0-255] interval includes:

[0014] Get the maximum and minimum elevation values;

[0015] The elevation value of each pixel is normalized;

[0016] The normalized elevation values ​​are set as the pixel values ​​of the grayscale image to generate the grayscale image.

[0017] Preferably, the process of performing wavelet transform on the basis obtained for convolution operations in the isovariant convolutional network based on partial differential operators includes:

[0018] Determine the base in the isovariant convolutional network;

[0019] Discrete wavelet transform is performed on each channel of the basis using the Daubechies 2 wavelet;

[0020] The wavelet transform coefficients containing both low-frequency and high-frequency components are obtained.

[0021] Preferably, the process of performing the inverse wavelet transform includes:

[0022] The wavelet transform coefficients obtained by the db2 wavelet are subjected to inverse wavelet transform.

[0023] Assign the result of the inverse wavelet transform to wavelet_basis;

[0024] Convert wavelet_basis to a tensor to obtain the basis after wavelet transform.

[0025] Preferably, the process of training the model in the equivariant convolutional network based on the wavelet-transformed basis includes:

[0026] Based on the basis transformed by wavelet transform, construct equivariant convolutions for the input layer, intermediate layer, and output layer;

[0027] In the input layer, an isovariant convolution is constructed to process the input image data;

[0028] In the intermediate layer, an equivariant convolution is constructed to process the intermediate feature maps;

[0029] In the output layer, an isovariant convolution is constructed to process the output image data;

[0030] The equivariant convolutional network is trained using the training dataset to obtain an equivariant convolutional network model with translation equivariance, rotation equivariance, and scale equivariance.

[0031] Preferably, the step of constructing the equivariant convolution of the input layer, intermediate layer, and output layer specifically includes:

[0032] In the input layer, all differential operators in the polynomial are discretized to obtain the isovariant convolution of the input layer;

[0033] In the intermediate layer, all differential operators in the polynomial are discretized to obtain the equivariant convolution of the intermediate layer;

[0034] In the output layer, all differential operators in the polynomial are discretized to obtain the isovariant convolution of the output layer;

[0035] The coefficients in the polynomials at different levels vary depending on the data to be processed and the function of each level.

[0036] Preferably, the equivariant convolutional network model is implemented based on an enhanced deep residual network, and the specific steps include:

[0037] Construct parameterized equivariant convolutions in the input, intermediate, and output layers of an enhanced deep residual network;

[0038] By using the basis derived from wavelet transform, an equivariant convolutional network with translational equivariance, rotational equivariance, and scale equivariance is achieved.

[0039] The training process of the equivariant convolutional network model specifically includes:

[0040] Training was performed using the deep learning framework PyTorch.

[0041] Set the initial learning rate to 1.00e. -4 ;

[0042] Set the residual block size to 16;

[0043] When training with double upsampling, the patch size is set to 48; when training with quadruple upsampling, the patch size is set to 96.

[0044] During training, the model with the highest peak signal-to-noise ratio is selected as the optimal model.

[0045] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0046] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0047] Compared with the prior art, the present invention has the following advantages and technical effects:

[0048] This invention provides a super-resolution reconstruction method for digital elevation models (DEMs) based on self-similarity and parameterized convolution. It utilizes the self-similarity of DEM data, combines the equivariance of equivariant convolution, and uses wavelet transform to further design a scale-equivariant convolutional network model based on translational and rotational equivariance, making it more efficient for DEM super-resolution reconstruction.

[0049] This invention first preprocesses high-resolution DEM data to generate high- and low-resolution image datasets. Next, wavelet transform is applied to the convolutional basis in an equivariant convolutional network based on partial differential operators to extract low- and high-frequency information, and the basis is reconstructed through inverse wavelet transform. The reconstructed basis is then input into the equivariant convolutional network for training, constructing a model with translation, rotation, and scaling equivariance. Finally, a low-resolution DEM image is input into the trained model to achieve super-resolution reconstruction of the high-resolution DEM image. This method fully utilizes the self-similarity of DEM data and combines it with the characteristics of equivariant convolutional networks to effectively improve the accuracy and efficiency of DEM super-resolution reconstruction. Attached Figure Description

[0050] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1This is a flowchart illustrating the super-resolution reconstruction of a digital elevation model based on self-similarity and parameterized convolution, according to an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of the architecture of the equivariant convolutional network model according to an embodiment of the present invention. Detailed Implementation

[0053] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a super-resolution reconstruction method for digital elevation models based on self-similarity and parameterized convolution, including:

[0057] Data preprocessing is performed on high-resolution digital elevation model data to obtain high-resolution images and corresponding low-resolution images, which are divided into training datasets and test datasets.

[0058] Wavelet transform is performed on the basis used for convolution operations obtained in the isovariant convolutional network based on partial differential operators. The resulting wavelet transform coefficients contain both low-frequency and high-frequency components.

[0059] Perform inverse wavelet transform on the wavelet transform coefficients to reconstruct the signal and obtain the basis after wavelet transform;

[0060] The model is trained in the equivariant convolutional network with wavelet transform as the basis input, resulting in an equivariant convolutional network model that simultaneously possesses translation equivariance, rotation equivariance, and scale equivariance.

[0061] The low-resolution digital elevation model images from the preprocessed test set are input into the trained equivariant convolutional network model for super-resolution reconstruction, resulting in high-resolution digital elevation model images, thus achieving super-resolution image reconstruction.

[0062] Step 1: Perform data preprocessing on the high-resolution digital elevation model data to obtain high-resolution images and corresponding low-resolution images, which are divided into training datasets and test datasets.

[0063] In this example, the high-resolution DEM dataset obtained directly is in the Tag Image File Format (TIFF) format by default. The default TIFF format data contains the geographic coordinates and elevation values ​​of the DEM data. The elevation value can be understood as the altitude, and its data range can be as low as negative numbers or as high as over a thousand. If you want to obtain the corresponding PNG format images from the DEM for super-resolution reconstruction experiments, you need to first perform data preprocessing on the DEM data to obtain the dataset to be used for training and testing.

[0064] The specific data preprocessing operations in this embodiment are as follows:

[0065] The elevation value, similar to altitude, is scaled down to the range of [0-255]. Then, R=G=B=the scaled elevation value is set to convert the DEM data from TIFF format to PNG format, thus describing the DEM image data as a grayscale image.

[0066] The specific scaling formula is as follows:

[0067] Normalized elevation value = (original elevation value - minimum elevation value) / (maximum elevation value - minimum elevation value) × 255;

[0068] After the first step of processing, seven high-precision grayscale PNG images of different areas can be obtained.

[0069] The seven DEM grayscale images obtained in the first step are high-precision, uncut 7200×7200 original images from seven different regions of my country. The actual size of these images does not match the input required by the network model. Therefore, to further preprocess and obtain the dataset needed for training and testing, the grayscale images from the seven different regions are further cut into a total of 1372 high-resolution (HR) images of 512×512 pixels. From these 1372 high-resolution DEM images, 900 images are randomly selected for training and 150 images are selected for testing.

[0070] Rename the 900 files in the training dataset folder to 0001.png~0900.png. Then generate a complete list of filenames in the training dataset folder and record it in the celeba_test_list.txt file. Based on the filename list, use bicubic interpolation to downsample the 900 training images to half their original scale, resulting in 900 256×256 low-resolution (LR) DEM images. Rename these images to 0001x2.png~0900x2.png. Next, use bicubic interpolation to downsample the original HR images to one-quarter of their original scale, resulting in 900 128×128 LR DEM images. Rename all these images to 0001x4.png~0900x4.png.

[0071] The same operation was performed on 150 HR DEM images from the test dataset to obtain corresponding LR DEM images after 2x downsampling and 4x downsampling. For easier observation of the results, 100 images were placed in the B100 folder, and the remaining 50 images were placed in the Set5 folder.

[0072] In summary, this embodiment selects 1372 pairs of HRDEM data with an image size of 512×512 from seven regions, and performs 2x downsampling and 4x downsampling on them respectively using bicubic interpolation to obtain the corresponding LR DEM images. Among them, 900 pairs of DEM data are used as the training set and 150 pairs of DEM data are used as the test set.

[0073] like Figure 2 As shown, step 2: Perform wavelet transform on the basis obtained for convolution operation in the equivariant convolution network based on partial differential operators. The resulting wavelet transform coefficients include low-frequency and high-frequency components.

[0074] Figure 2 The symbols with tildes represent the discretized form of the corresponding operators. The image example only shows three transformations, Ψ. (A) Indicates continuous, It indicates discrete.

[0075] The bivariate polynomial representation used in equivariant convolutional networks based on partial differential operators is as follows:

[0076] H(x,y)=β1+β2x+β3y+β4x 2 +β5xy+β6y 2 +β7x 2 y+β8xy 2 +β9x 2 y 2 (1)

[0077] Here, H(x,y) represents a bivariate polynomial, and β1-β9 represent the coefficients of each term.

[0078] The equivariance group S represents a transformation group with translation, rotation, and reflection transformations, and a transformation group with rotation and translation transformations. In Equation 2, A is the orthogonal matrix in the equivariance group S, and the two differential operators depend on A. and Represented as Equation 2:

[0079]

[0080] The initial basis is represented as the set of bases in the polynomial of Equation 1:

[0081] Basis = {1, x, y, x 2 ,xy,y 2 ,x 2 y,xy 2 ,x 2 y 2} (3)

[0082] The equivariant convolutional network based on partial differential operators performs a rotation transformation on the Basis in Equation 3. That is, it uses the set of coefficients β={β1,β2,β3,β4,β5,β6,β7,β8,β9} to linearly combine these bases. The base BasisR obtained after rotation is the set of terms in the polynomial H(x,y).

[0083] Perform wavelet transform on BasisR and recombine the coefficients:

[0084]

[0085] Where W represents the wavelet transform matrix, which is an orthogonal matrix.

[0086] The reorganized bivariate polynomial is represented as:

[0087]

[0088] The specific method for performing wavelet transform is as follows: Obtain the shape of BasisR, and create an empty array wavelet_basis with the same shape as BasisR to store the wavelet-transformed basis; iterate through each channel of BasisR, and use Daubechies 2 wavelet (db2) to perform discrete wavelet transform decomposition (DWT) on each channel of the basis. The resulting wavelet transform coefficients coeffs contain low-frequency and high-frequency components.

[0089] The expression for obtaining the coefficients coeffs using db2 discrete wavelet transform decomposition is as follows:

[0090] coeffs=DWT(BasisR, 'db2') (6)

[0091] Step 3: Perform inverse wavelet transform on the wavelet transform coefficients to reconstruct the signal and obtain the wavelet-transformed basis.

[0092] The specific method for performing inverse wavelet transform on the wavelet transform coefficients is as follows: the obtained wavelet transform coefficients coeffs are subjected to inverse wavelet transform (IDWT) using the db2 wavelet, the result of the inverse wavelet transform is assigned to wavelet_basis, and wavelet_basis is converted into a tensor, which is the basis after wavelet transform.

[0093] Using db2 to perform an inverse wavelet transform on the wavelet transform coefficients coeffs, the resulting basis wavelet_basis is expressed as:

[0094] wavelet_basis=IDWT(coeffs[0], coeffs[1], 'db2') (7)

[0095] Wherein, coeffs[0] is the low-frequency part of the coefficients, and coeffs[1] is the high-frequency part of the coefficients.

[0096] Step 4: Train the model in the wavelet-transformed base input equivariant convolutional network to obtain an equivariant convolutional network model that simultaneously possesses translation equivariance, rotation equivariance, and scale equivariance.

[0097] The final structure of the equivariant convolutional network model is as follows:

[0098] In the input layer, an isovariant convolution is constructed to process the input image data:

[0099]

[0100] Where A is an orthogonal matrix in the isovariant group S, Ψ (A) It is a family of differential operators, discretizing all differential operators in the 8-polynomial, and the resulting new operator is the equivariant convolution of the input layer.

[0101] In the intermediate layers, equivariant convolutions are constructed to process the intermediate feature maps:

[0102]

[0103] in, Let be a polynomial with coefficient set β(B). It is a family of differential operators. Discretizing all the differential operators in Equation 9, the resulting new operator is the equivariant convolution of the intermediate layer.

[0104] In the output layer, construct an equivariant convolution to process the output:

[0105]

[0106] Among them, Υ (B) It is a family of differential operators. Discretize all the differential operators in Equation 10, and the resulting new operator is the equivariant convolution of the output layer.

[0107] The polynomials differ depending on the data to be processed and the function of each layer. coefficients in They are different.

[0108] The method for training the model in the wavelet-transformed base-input equivariant convolutional network is as follows: Based on the image super-resolution deep learning model, equivariant convolutions based on wavelet transform are designed for the input layer, intermediate layer, and output layer using the above method, such as... Figure 2 As shown, the filters of the input layer, intermediate layer and output layer are subjected to corresponding scale transformations during rotation transformation, resulting in an equivariant convolutional network model that simultaneously possesses translation equivariance, rotation equivariance and scale equivariance, and can achieve image super-resolution.

[0109] This example uses an enhanced deep residual network (EDSR) model as the basis for a super-resolution network model. Parameterized equivariant convolutions are designed in the input, intermediate, and output layers of the EDSR to achieve an equivariant convolutional network with translational equivariance, rotational equivariance, and scale equivariance, capable of image super-resolution reconstruction.

[0110] This example uses the deep learning framework PyTorch, with a training process that iterates through 6 rounds and an initial learning rate of 1.00e. -4 The residual block size was set to 16, and the patch size was set to 48 during training with double upsampling and 96 during training with quadruple upsampling. During training, the model with the highest peak signal-to-noise ratio was selected as the best model for testing.

[0111] Step 5: Input the low-resolution digital elevation model images from the preprocessed test set into the trained equivariant convolutional network model for super-resolution reconstruction to obtain high-resolution digital elevation model images.

[0112] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0113] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0114] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A super-resolution reconstruction method for digital elevation models based on self-similarity and parameterized convolution, characterized in that, include: Data preprocessing is performed on high-resolution digital elevation model data to obtain high-resolution images and corresponding low-resolution images, and the data is divided into training datasets and test datasets. Based on the training dataset, wavelet transform is performed on the basis used for convolution operations obtained in the equivariant convolutional network based on partial differential operators to obtain wavelet transform coefficients containing low-frequency and high-frequency components. Based on the wavelet transform coefficients, perform inverse wavelet transform to reconstruct the signal and obtain the basis after wavelet transform; Based on the wavelet transform-transformed basis, the model is trained in the equivariant convolutional network to obtain an equivariant convolutional network model that simultaneously possesses translation equivariance, rotation equivariance, and scale equivariance. Based on the test dataset, the preprocessed low-resolution digital elevation model image is input into the trained equivariant convolutional network model for super-resolution reconstruction to obtain a high-resolution digital elevation model image. The process of training the model in the equivariant convolutional network based on the wavelet transform basis includes: Based on the basis transformed by wavelet transform, construct equivariant convolutions for the input layer, intermediate layer, and output layer; In the input layer, an isovariant convolution is constructed to process the input image data; In the intermediate layer, an equivariant convolution is constructed to process the intermediate feature maps; In the output layer, an isovariant convolution is constructed to process the output image data; The equivariant convolutional network is trained using the training dataset to obtain an equivariant convolutional network model with translation equivariance, rotation equivariance, and scale equivariance. The steps for constructing the equivariant convolutional layer, including the input layer, intermediate layer, and output layer, specifically include: In the input layer, all differential operators in the polynomial are discretized to obtain the isovariant convolution of the input layer; In the intermediate layer, all differential operators in the polynomial are discretized to obtain the equivariant convolution of the intermediate layer; In the output layer, all differential operators in the polynomial are discretized to obtain the isovariant convolution of the output layer; The coefficients in the polynomials at different levels vary depending on the data to be processed and the function of each level. The final structure of the equivariant convolutional network model is as follows: In the input layer, an isovariant convolution is constructed to process the input image data: （8）； Where A is an orthogonal matrix in the equivariant group S. It is a family of differential operators. Discretize all the differential operators in the polynomial of formula (8) and the resulting new operator is the equivariant convolution of the input layer. In the intermediate layers, equivariant convolutions are constructed to process the intermediate feature maps: （9）； in, The set of coefficients is polynomial, It is a family of differential operators. Discretize all the differential operators in equation (9) and the resulting new operator is the equivariant convolution of the intermediate layer. In the output layer, construct an equivariant convolution to process the output: （10）； in, It is a family of differential operators. Discretize all the differential operators in equation (10) and the resulting new operator is the equivariant convolution of the output layer.

2. The method according to claim 1, characterized in that, The process of preprocessing the high-resolution digital elevation model data includes: The elevation values ​​are normalized to the [0-255] range to generate grayscale images; the grayscale images are cut into image blocks that meet the network input requirements and divided into training datasets and test datasets; the high-resolution images in the training dataset are downsampled by bicubic interpolation to obtain low-resolution images.

3. The method according to claim 2, characterized in that, The process of normalizing the elevation values ​​to the [0-255] interval includes: Get the maximum and minimum elevation values; The elevation value of each pixel is normalized; The normalized elevation values ​​are set as the pixel values ​​of the grayscale image to generate the grayscale image.

4. The method according to claim 1, characterized in that, The process of performing wavelet transform on the basis obtained for convolution operations in the isovariant convolutional network based on partial differential operators includes: Determine the base in an isovariant convolutional network; Discrete wavelet transform is performed on each channel of the basis using the db2 wavelet; The wavelet transform coefficients containing both low-frequency and high-frequency components are obtained.

5. The method according to claim 4, characterized in that, The process of performing the inverse wavelet transform includes: The obtained wavelet transform coefficients are subjected to inverse wavelet transform using the db2 wavelet; Assign the result of the inverse wavelet transform to wavelet_basis; Convert wavelet_basis to a tensor to obtain the basis after wavelet transform.

6. The method according to claim 1, characterized in that, The equivariant convolutional network model is implemented based on an enhanced deep residual network, and the specific steps include: Construct parameterized equivariant convolutions in the input, intermediate, and output layers of an enhanced deep residual network; By using the basis derived from wavelet transform, an equivariant convolutional network with translational equivariance, rotational equivariance, and scale equivariance is achieved. The training process of the equivariant convolutional network model specifically includes: Training was performed using the deep learning framework PyTorch. Set the initial learning rate to 1.00e. -4 ; Set the residual block size to 16; When training with double upsampling, the patch size is set to 48; when training with quadruple upsampling, the patch size is set to 96. During training, the model with the highest peak signal-to-noise ratio is selected as the optimal model.

7. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images