A method for reconstructing digital elevation model using images

Abstract

The present invention relates to the technical field of image processing and three-dimensional reconstruction, and in particular to a method for reconstructing a digital elevation model using an image. The method comprises: 1) pre-processing image data and its corresponding digital elevation model (DEM) data to produce a training data set for generating a high-resolution DEM. 2) constructing a deep learning network, using the data in the training data set to train the network, and obtaining an elevation reconstruction network after the training is completed; 3) pre-processing the image of the area to be reconstructed and the low-resolution DEM data of the area; 4) using the elevation reconstruction network to obtain an elevation reconstruction result consistent with the input image resolution; 5) post-processing the elevation reconstruction result output by the network to obtain a DEM. This method uses the grayscale information of a single image, takes the low-resolution DEM as a constraint, and realizes end-to-end high-resolution DEM reconstruction through a trained deep network model.

Description

Technical Field

[0001] The invention relates to the technical field of image processing and three-dimensional reconstruction, and in particular to a method for reconstructing a digital elevation model using images. Background Art

[0002] Digital elevation model (DEM) is a digital simulation of ground terrain using regular grids and their corresponding elevation values. At present, the main methods of producing ground DEM are aerospace stereo imaging, laser scanning and other technologies. However, due to the limitations of measurement costs and technical application scenarios, especially in restricted environments such as deep space exploration, obtaining high-resolution and high-precision DEM data still requires a high cost.

[0003] In restricted environments, some areas lack stereo images, and DEM cannot be obtained using traditional production methods, and laser scanning can only provide low-resolution terrain information. For a single image, since the illumination and terrain undulations are reflected as differences in light and dark on the image, the terrain information can be restored using the traditional Shape from Shading (SFS) method. However, the problem with this method is that the restrictions are harsh, the model is complex, and the model parameters need to be readjusted for data acquired at different times, resulting in unsatisfactory accuracy of the produced DEM.

[0004] With the rapid development of computer vision and artificial intelligence technology, deep learning-based methods can build a model between images and elevations to achieve end-to-end DEM production, which has the advantages of few restrictions, good portability, and low cost. Therefore, it is of great significance to study a method for reconstructing DEM from images based on deep learning. Summary of the invention

[0005] In order to solve the problems of the prior art, the present invention provides a method for reconstructing a digital elevation model using images. The method utilizes the grayscale information of a single image and takes a DEM with a resolution lower than the image resolution as a constraint. Through a trained deep network model, end-to-end DEM reconstruction consistent with the image resolution is achieved.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for reconstructing a digital elevation model using images comprises the following steps: 1) pre-processing image data and corresponding DEM data to prepare a training data set for generating a high-resolution DEM; 2) constructing a deep learning network, using data in the training data set to perform network training, and obtaining an elevation reconstruction network after the training is completed; 3) for each scene of image data for which DEM needs to be reconstructed, performing high-resolution DEM reconstruction using steps 3)-5): pre-processing the image to be reconstructed and low-resolution DEM data of the area; 4) using the elevation reconstruction network to obtain an elevation reconstruction result that is consistent with the input image resolution; and 5) post-processing the elevation reconstruction result output by the network to obtain a DEM.

[0008] Furthermore, the specific process of step 1) is as follows: 1.1) obtaining the required data, including: an image dataset X of a certain area on the surface and its homologous or heterologous DEM dataset Y; 1.2) normalizing all data; 1.3) geometrically registering each image in the dataset (denoted as x) and the corresponding DEM (denoted as y); 1.4) producing deep learning network training data and establishing a training dataset.

[0009] Furthermore, the specific process of step 1.4) is as follows: 1.4.1) the image data x and the DEM data y are cut into the same size according to a certain overlap, denoted as C_x i , C_y i (i=1,2,…,m), m represents the number of images or corresponding DEMs; 1.4.2) Use bilinear interpolation or nearest neighbor interpolation to convert C_y i Downsample by a multiple of D_y i ; 1.4.3) Then D_y i Upsample to U_D_y i , so that its grid size is the same as C_x i Consistent; 1.4.4) C_x i As band 1, U_D_y i As band 2, synthesize a two-channel image data file S_x i , as training data (S_x i , C_y i ); 1.4.5) Optional operation: The training data in the present invention is not limited to being produced using only x and y. Multi-source data can be processed according to steps 1.4.2) and 1.4.3) and synthesized as a new band channel into S_x i If multiple source image bands are added, the subsequent image data to be converted also needs to be changed accordingly to ensure that the number of bands and band types of the image to be processed are consistent with S_x i Consistent.

[0010] Furthermore, the specific process of step 2) is as follows: 2.1) using the adversarial generative network model mechanism, respectively establishing a generative network model G and a discriminative network model D; 2.2) adding training data, and training the established network by forward operation and error back propagation to obtain the elevation reconstruction network.

[0011] Furthermore, the specific process of step 2.1) is as follows: 2.1.1) Establish a generative network, using a U-shaped convolutional network structure, according to S_x i The image size is used to construct a multi-layer convolutional network, n = log2 (x i ), the loss function is the L1 regularized loss between the discriminant network output and the real image; 2.1.2) Establish a discriminant network, using a network structure with multiple convolutional layers. Except for the first and last layers, activation functions and normalization operations need to be added after the convolutional layers. The discriminant network is used to generate the output image G(S_x i ) and the truth value C_y i The discrimination result is taken as the loss.

[0012] Furthermore, the specific process of establishing the generation network in step 2.1.1) is as follows: the U-shaped convolutional network is a symmetrical structure, which is divided into two parts: the first half is a downsampling convolutional layer, which inputs the synthetic image data S_x i While performing downsampling convolution, the number of feature channels is increased. It needs to be normalized and activated before being passed to the next layer, and finally the features are converted into a one-dimensional feature sequence. The second half is the upsampling convolution. Each layer of convolution operation in this part combines the output of the previous layer and the symmetric layer based on the center of the network as input. After the convolution is completed, it needs to be normalized and activated before being passed out. The activation function is a function.

[0013] Furthermore, in step 2.1.2, the discriminant network sets multiple layers of convolution for feature encoding, and the feature image size after encoding is consistent with the original image size. After encoding, the image can be divided into N*N blocks for discriminant calculation, and the average loss is used as the final loss. The number of blocks is adjusted by the receptive field of the convolutional network, which can be expanded from a single pixel to the entire image data.

[0014] Furthermore, the specific process of the step 3) pretreatment is as follows:

[0015] 3.1) Obtain the image data x' of the area to be reconstructed, and the low-resolution DEM data y' corresponding to the area; 3.2) Perform geometric calibration and normalization on x' and y', and cut them into an image C_x of the same size as the training data image according to a certain overlap i '、C_y i '; 3.3) C_y i 'Upsample to U_C_yi ', so that its grid size is the same as C_x i 'Consistent; 3.4) C_x i 'As band 1, U_C_y i 'As band 2, combine the two data into one image data file as the image data to be converted S_x i '.

[0016] Furthermore, the specific process of step 5) is as follows: 5.1) performing inverse normalization processing on the DEM generation network operation result to restore the absolute elevation scale it carries; 5.2) performing image mosaicking and merging into a scene elevation data; 5.3) adding projection and other information to the merged data, and outputting the result, which is high-resolution DEM data.

[0017] The beneficial effects brought by the technical solution provided by the present invention are:

[0018] 1. The present invention can construct a model based on the information shown in the image, and recover the elevation information of the corresponding position of each pixel point, so as to complete the generation of high-precision DEM. It can be applied to the situation where there is only an image but no high-precision DEM in a restricted environment, and a high-precision DEM with the same resolution as the image can be obtained by using the image and the low-resolution DEM.

[0019] 2. The present invention can take advantage of the deep learning network and construct a network to accurately represent the mapping relationship between the image and the DEM by learning the training data. At the same time, it can flexibly select multiple types of data as input to reconstruct the DEM according to the characteristics of different regions, and select training data according to regional characteristics. It has the characteristics of high precision, strong stability and high adaptability, and can be widely used in the field of generating high-resolution DEM. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flow chart of the present invention.

[0022] Figure 2 It is a schematic diagram of the U-shaped network structure in the present invention.

[0023] Figure 3 This is a schematic diagram of the U network convolution kernel setting in the present invention.

[0024] Figure 4It is a schematic diagram of the discriminator convolutional network in the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] Embodiment 1

[0027] like Figure 1 As shown, this embodiment provides a method for generating a digital elevation model using images based on a deep learning network, and the method comprises the following steps:

[0028] 1) Preprocess the image data and its corresponding DEM data to produce a training data set for generating high-resolution DEM;

[0029] 1.1) Obtain the required data, including: image data X of a part of the surface area and its corresponding generated or heterogeneous high-resolution DEM data Y;

[0030] 1.2) Normalize all data;

[0031] 1.3) Perform geometric registration on the data X and respectively, and cut them into the same size according to a certain overlap, denoted as C_x i , C_y i ;

[0032] 1.4) Create deep learning network training data. The specific process is as follows:

[0033] 1.4.1) Use bilinear interpolation or nearest neighbor interpolation to convert C_y i Downsample to D_y i ;

[0034] 1.4.2) D_y i Upsample to U_D_y i , so that its pixel resolution is the same as C_x i Consistency;

[0035] 1.4.3) C_x i As band 1, U_D_y i As band 2, synthesize a two-channel image data file S_x i , as training data (S_x i , C_y i );

[0036] 1.4.4) Optional operation: The training data in the present invention is not limited to being produced using only images x and y. Multi-source data can be processed according to steps 1.4.2) and 1.4.3) and synthesized as a new band channel into S_xi If multiple source image bands are added, the subsequent image data to be converted also needs to be changed accordingly to ensure that the number of bands and band types of the converted image are consistent with S_x i Consistent.

[0037] 2) Construct a deep learning network, which includes a generative network and a discriminative network. Use the training data set to train the network, and use different loss functions to optimize the parameters of the generative network and the discriminative network respectively. After the training is completed, the DEM reconstruction network is obtained. The specific process is as follows:

[0038] 2.1) Using the adversarial generative network model mechanism, the generative network model G and the discriminative network model D are established respectively. The specific process is as follows:

[0039] 2.1.1) Establish a generative network using a U-shaped convolutional network structure, such as Figure 2 As shown, according to x i The image size is constructed to construct an n-layer convolutional network, n = 2*log2(S_x i ), with (S_x i , C_y i ) is an example of an image with a size of 256*256*2. The U-shaped network has 16 layers and can be divided into two parts:

[0040] (1) Layers 1-8: Figure 3 As shown in the figure, it is a downsampling convolution layer, which includes downsampling convolution, normalization, and activation function. The convolution kernel step size is 2. The number of convolution kernels in the first four layers is 64, 64*2, 64*4, and 64*8. The size of the convolution kernel changes accordingly, which are: 4*4*2, 4*4*64, 4*4*128, 4*4*256, and 4*4*512. The size of the convolution kernel in the last four layers remains unchanged, all 4*4*512 (it can be adjusted according to actual needs, and is described here as an example). After the convolution is completed, it is processed by normalization and activation function and passed to the next layer. After 8 layers of downsampling convolution, the original 256*256*2 synthetic image data can be converted into 1*1*512 feature data. The activation function uses the LeakyRelu activation function, as shown below:

[0041]

[0042] (2) 9-16 layers: Figure 3As shown in the figure, it is an upsampling deconvolution layer, which includes upsampling deconvolution, normalization, and activation function. There is a skip link in this part. The input of the i-th layer is the output of the i-1st and ni-th layers as input, and the convolution kernel step size is 2. The number of convolution kernels in the 9th layer is 64*8, and the size is 1*1*512; the convolution kernels in the 10th-12th layers are 4*4*1024; the number of convolution kernels corresponding to the 13th-16th layers are 64*4, 64*2, 64, and 1, and the sizes are 4*4*1024, 4*4*512, 4*4*256, 4*4*128, and 4*4*64. After this part, the 1*1*512 feature data is reconstructed into data with relative elevation information. The activation function uses the Relu activation function.

[0043] The loss calculation of the generated network is shown in the following formula:

[0044] Loss G =w i [log(1-D(x i ,G(x i ))]+λw i ||y i -G(x i )||1

[0045] The above formula represents the L1 regularization loss between the discriminant network’s discriminant result and the real image. Each input synthetic image can be assigned a weight w according to its quality i , and the parameter λ needs to be balanced so that the two parts are in the same order of magnitude.

[0046] 2.1.2) Establish a discriminant network, taking the receptive field as a single pixel size as an example, Figure 4 As shown in the figure, a three-layer convolutional network structure is used. The original data and the generated data are combined as input. The activation function uses the LeakyRelu function. Normalization is required after the second convolutional layer before the activation function. The loss function is the discriminant network for the generated value G(x i ), the true value C_y i The results of the discrimination are weighted and summed according to the image quality, as follows:

[0047]

[0048] 2.2) Add training data (S_x) i , C_y i ) The established network is optimized by forward operation and back propagation to obtain the elevation reconstruction network structure.

[0049] 3) For each scene where DEM image data needs to be reconstructed, high-resolution DEM reconstruction is performed using steps 3)-5): the image to be reconstructed and the low-resolution DEM data of the area are preprocessed to obtain input data. The specific process is as follows:

[0050] 3.1) Obtain the image data x' of the area to be super-reconstructed and the low-resolution DEM data y' corresponding to the area;

[0051] 3.2) Perform geometric calibration and normalization on x' and y', and cut them into C_x of the same size as the training data according to a certain overlap i '、C_y i ';

[0052] 3.3) C_y i 'Upsample to U_C_y i ', so that its pixel resolution is the same as C_x i 'Consistency;

[0053] 3.4) C_x i 'As band 1, U_C_y i 'As band 2, combine the two data into one image data file as the image data to be converted S_x i '.

[0054] 4) Use the trained elevation reconstruction network to generate high-resolution elevation images, and the grid size of the output image is consistent with the input image;

[0055] 5) Post-process the output of the elevation reconstruction network to restore the DEM data information and obtain the final high-resolution DEM data. The specific process is as follows:

[0056] 5.1) The output of the elevation reconstruction network is inversely normalized to restore its absolute elevation scale;

[0057] 5.2) Mosaic and merge the images into a scene of elevation data;

[0058] 5.3) Add projection information and other information to the merged data and output the result, which is high-resolution DEM data.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for reconstructing a digital elevation model using images, comprising the following steps: A. Preprocess the image data and its corresponding digital elevation model data to produce a training data set for generating a digital elevation model with the same resolution as the input image; B. Construct a deep learning network, use the data in the training data set to perform network training, and obtain an elevation reconstruction network after the training is completed; The specific process of step B is as follows: B1. Use the adversarial generative network model mechanism to establish a generative network model and a discriminative network model respectively; B2, adding training data, training the established network by forward operation and error back propagation to obtain the elevation reconstruction network; The specific process of step B1 is as follows: B11. Establish a generative network, use the U-shaped convolutional network structure, and build an n-layer convolutional network according to the image size: n = log2 (x i ), the loss function is the L1 regularization loss between the discriminant network output and the real image; B12. Establish a discriminant network, using a network structure with multiple convolutional layers. Except for the first and last layers, activation functions and normalization operations need to be added after the convolutional layers. Use the discriminant network to generate the output image G(S_x i ) and the truth value C_y i The judgment result is taken as the loss; In the step B11, the specific process of generating the network is as follows: the U-shaped convolutional network is a symmetrical structure, which is divided into two parts: the first half is a downsampling convolution layer, which inputs the synthetic image data for downsampling convolution while increasing the number of feature channels, and needs to be normalized and activated before being passed to the next layer, and finally the features are converted into a one-dimensional feature sequence; The second half is the upsampling convolution. Each layer of convolution operation in this part combines the output of the previous layer and the symmetric layer based on the center of the network as input. After the convolution is completed, it needs to be normalized and processed by the activation function before being output; In the step B12, the discriminant network sets multi-layer convolution to perform feature encoding, and the size of the feature image after encoding is consistent with the size of the original image; after encoding, the image can be divided into N*N blocks for discriminant calculation respectively, and the average loss is taken as the final loss; The number of blocks is adjusted by the receptive field of the convolutional network, and the block size can be expanded from a single pixel to the entire image data; C. preprocessing the image of the area to be reconstructed and the digital elevation model data of the area with a resolution lower than that of the input image; D. Use the elevation reconstruction network to obtain an elevation reconstruction result that is consistent with the input image resolution; E. Post-process the elevation reconstruction results output by the network to obtain a digital elevation model.

2. The method for reconstructing a digital elevation model using images according to claim 1, characterized in that: The step A specifically comprises: A1. Obtain the required data, including: an image dataset X of a certain area on the surface and its homologous or heterologous digital elevation model dataset Y; A2. Normalize all data; A3. Geometrically align each image data in the data set with the corresponding digital elevation model data; A4. Create deep learning network training data and establish a training data set.

3. The method for reconstructing a digital elevation model using images according to claim 2, characterized in that: The step A4 specifically includes: A41. Crop the image data and digital elevation model data to the same size, denoted as C_x i , C_y i (i=1,2,…,m), m represents the number of images or corresponding DEMs, x represents image data, and y represents digital elevation model data; A42. Use bilinear interpolation or nearest neighbor interpolation to convert C_y i Downsample by a multiple of D_y i ; A43, then D_y i Upsample to U_D_y i , so that its grid size is the same as C_x i Consistency; A44, C_x i As the first band, U_D_y i As the second band, synthesize a two-channel image data file, denoted as S_x i , as training data, denoted as (S_x i , C_y i ).

4. The method for reconstructing a digital elevation model using images according to claim 3, characterized in that: The step A4 also includes step A45. The training data is not limited to being produced using only image data and digital elevation model data, that is, the multi-source data is processed according to steps A42 and A43 and synthesized as a new band channel into S_x i If multiple source image bands are added, the subsequent image data to be reconstructed also needs to be changed accordingly to ensure that the number of bands and band types of the image to be processed are consistent with S_x i Consistent.

5. The method for reconstructing a digital elevation model using images according to claim 1, characterized in that: The specific process of step C is as follows: C1. Obtain the image data of the area to be reconstructed, denoted as x', and the digital elevation model data of the area with a resolution lower than the image resolution, denoted as y'; C2, geometrically calibrate and normalize the x' and y', and cut them into multiple images C_x of the same size as the training data image i '、C_y i '; C3, the C_y i 'Upsample to U_C_y i ', so that its grid size is the same as C_x i 'Consistency; C4, the C_x i 'As the first band, U_C_y i 'As the second band, combine the two data into one image data file as the image data to be converted S_x i '.

6. The method for reconstructing a digital elevation model using images according to claim 1, characterized in that: The specific process of step E is as follows: E1. Perform inverse normalization on the network operation results generated by the digital elevation model to restore its absolute elevation scale; E2. Mosaic and merge the images into a scene of elevation data; E3. Add projection information to the merged data and output the result, which is high-resolution digital elevation model data.

Images