A Building Height Prediction Method Based on Wavelet Transform and Convolutional Neural Network

Abstract

The present invention discloses a method for predicting building height based on wavelet transform and convolutional neural network, belonging to the technical field of building measurement. First, the present invention constructs a data set using satellite optical images and synthetic aperture radar images; then combines wavelet transform and convolutional neural network to construct an initial building height prediction model; then uses the training set and labels to train the initial building height prediction model; finally uses the target building height prediction model to predict the height of the building. The present invention integrates wavelet transform into the convolutional neural network, captures four different frequency information of the image, makes full use of the high-frequency and low-frequency information of the image, enhances the expression of building features, and further improves the accuracy of building height prediction by using the building height recognition ability of the convolutional neural network.

Description

Technical Field

[0001] The invention relates to a building height prediction method, in particular to a building height prediction method based on wavelet transform and convolutional neural network, belonging to the technical field of building measurement. Background Art

[0002] With the acceleration of urbanization and the rise of the low-altitude economy, accurate measurement of building heights has become increasingly important. The low-altitude economy refers to the use of low-altitude airspace for a series of commercial activities, including drone aerial photography and low-altitude logistics. These activities not only rely on urban planning, air traffic management and environmental monitoring, but also put forward more accurate requirements for the measurement of urban building heights.

[0003] Traditional building height acquisition methods, such as manual measurement and formula-based calculation, have certain limitations. Manual measurement is usually time-consuming and labor-intensive, while building height estimation based on analytical formulas is performed using a shadow-based formula. In shadow-based building height estimation, the length of the building in the image and its shadow in the pixel are combined with the sun elevation to obtain the true building height, but this method usually requires other assumptions, resulting in inaccurate estimation results.

[0004] With the development of machine learning and deep learning, both have made significant progress in the estimation of building height in the field of remote sensing. Among machine learning methods, support vector machines are often used to predict building height. They mainly extract features such as texture and shape from images for classification or regression analysis, but they usually perform better when the amount of data is small and it is difficult to process large-scale remote sensing image data. Convolutional neural network is a deep learning technology that can extract image features through multi-layer convolution, and thus use the features of radar images and optical images to predict building heights. It has been widely used, especially synthetic aperture radar (Synthetic Aperture Radar, Sar) images. Convolutional neural networks can quickly process large-scale remote sensing image data, improve the computational efficiency of building height prediction, and reduce the consumption of human and material resources. However, when processing radar images, traditional convolutional neural networks ignore the frequency domain features of the image, which makes the network incomplete in utilizing the image information, resulting in a decrease in prediction accuracy. How to efficiently and accurately predict building heights has become a technical problem that needs to be solved in this field. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to propose a building height prediction method based on wavelet transform and convolutional neural network.

[0006] Technical solution: A building height prediction method based on wavelet transform and convolutional neural network of the present invention comprises the following steps:

[0007] Step 1: Obtain satellite optical images and synthetic aperture radar images, construct a data set using the satellite optical images and synthetic aperture radar images, and collect the height information of the corresponding buildings as labels. Preprocess the images in the data set and divide them into a training set and a test set according to a ratio;

[0008] Step 2: Combine wavelet transform and convolutional neural network to construct an initial building height prediction model;

[0009] Step 3: Use the training set and labels to train the initial building height prediction model to obtain a target building height prediction model;

[0010] Step 4: Use the target building height prediction model to predict the height of the building.

[0011] Furthermore, combining wavelet transform and convolutional neural network to construct an initial building height prediction model includes:

[0012] Construct a first encoder and a second encoder with the same structure. Each encoder includes a first convolutional layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolutional block layer, a first residual block layer, a second convolutional block layer, a second residual block layer, a third convolutional block layer, and a third residual block layer arranged in sequence;

[0013] Construct a decoder. The decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence. The structure of each decoding layer is the same, and each decoding layer includes a convolutional layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence;

[0014] Construct a second convolutional layer.

[0015] Furthermore, Step 3 includes:

[0016] Input the satellite optical images in the training set into the first encoder to extract four groups of feature map tensors, denoted as f 1 、f 2 、f 3 、f 4 ;

[0017] Input the synthetic aperture radar images in the training set into the second encoder to extract four groups of feature map tensors, denoted as ;

[0018] Add the feature map tensor f 4 and the feature map tensor and input the result into the first decoding layer. Add the feature map tensor f 3 and the feature map tensor After addition, it is concatenated with the tensor output by the first decoding layer in terms of dimensions and then input into the second decoding layer. The feature map tensor f 2 is added to the feature map tensor After addition, it is concatenated with the tensor output by the second decoding layer in terms of dimensions and then input into the third decoding layer. The feature map tensor f 1 is added to the feature map tensor After addition, it is concatenated with the tensor output by the third decoding layer in terms of dimensions and then input into the fourth decoding layer. The tensor output by the fourth decoding layer passes through the second convolutional layer to obtain the height of the building.

[0019] Furthermore, the satellite optical image input into the first encoder adopts a four-channel input, including the red, green, blue, and near-infrared bands.

[0020] Furthermore, the synthetic aperture radar image input into the second encoder adopts a four-channel input, including the ascending and descending orbits in the VV polarization mode and the ascending and descending orbits in the VH polarization mode.

[0021] Furthermore, the input image of the wavelet transform is represented as a two-dimensional matrix , where represents the pixel value of the i-th row and j-th column, , , and M and N respectively represent the total number of rows and the total number of columns;

[0022] In the wavelet transform, first perform a one-dimensional wavelet transform on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is:

[0023] ,

[0024] ,

[0025] In the formula, represents the low-frequency component calculated from two adjacent pixel points in the row direction, represents the high-frequency component calculated from two adjacent pixel points in the row direction;

[0026] Then perform a one-dimensional wavelet transform on the remaining other direction. The expression of the output image is:

[0027] ,

[0028] ,

[0029] ,

[0030] ,

[0031] In the formula, represents the low-frequency components in the horizontal and vertical directions, represents the low-frequency component in the horizontal direction and the high-frequency component in the vertical direction, represents the high-frequency component in the horizontal direction and the low-frequency component in the vertical direction, represents the high-frequency components in the horizontal and vertical directions.

[0032] Furthermore, the expression of the loss function of the initial building height prediction model is:

[0033] ,

[0034] In the formula, CS represents the cosine similarity loss, MSE represents the mean square error loss, is the weight coefficient.

[0035] Beneficial effects: Compared with the prior art, the remarkable advantages of the present invention are:

[0036] The present invention integrates wavelet transform into the convolutional neural network, and proposes a new method for predicting building height based on wavelet transform and convolutional neural network, and finally returns to a specific numerical value of the building height. By using wavelet transform to capture four different frequency information of the image, it makes full use of the high-frequency and low-frequency information of the image, and also avoids using deconvolution, thus avoiding problems such as artifacts and interpolation errors that may be caused by deconvolution, enhancing the expression of building features, and making full use of the ability of the convolutional neural network to identify building height, so that the accuracy of predicting building height is greatly improved. Description of the Drawings

[0037] Figure 1 is a flowchart of a method for predicting building height based on wavelet transform and convolutional neural network;

[0038] Figure 2 is a schematic structural diagram of the initial building height prediction model;

[0039] Figure 3 is a schematic diagram of the principle of wavelet transform;

[0040] Figure 4 is a schematic diagram of the visualization of the building height prediction result. Detailed Embodiment

[0041] In order to make the purpose, technical solution and advantages of the present application clearer, the following further describes the present application in detail with reference to the drawings and embodiments.

[0042] A method for predicting building height based on wavelet transform and convolutional neural network described in this embodiment, the flowchart is as Figure 1 shown, and the method includes the following steps:

[0043] Step 1: Obtain satellite optical images and synthetic aperture radar images, construct a dataset using the satellite optical images and synthetic aperture radar images, and collect the height information of corresponding buildings as labels. Preprocess the images in the dataset and divide them into a training set and a test set according to a certain proportion.

[0044] Step 2: Combine wavelet transform and convolutional neural network to construct an initial building height prediction model.

[0045] Step 3: Use the training set and labels to train the initial building height prediction model to obtain a target building height prediction model.

[0046] Step 4: Use the target building height prediction model to predict the height of buildings.

[0047] The present invention combines the multi-resolution characteristics of wavelet transform and the feature extraction ability of convolutional neural network to realize the prediction of building height. Specifically, through wavelet transform, the information of the image at four different frequencies is captured, making full use of the high-frequency and low-frequency information of the image, enhancing the useful features of the image, and making more complete use of the image information; using convolutional neural network avoids problems such as artifacts and interpolation errors that may be caused by deconvolution, enhances the expression of building features, and greatly improves the accuracy of building height prediction.

[0048] In one example, the dataset used includes synthetic aperture radar images, optical images, and the height information of corresponding buildings as reference data. The synthetic aperture radar images are from Sentinel-1 satellite, and the optical images are from Sentinel-2 satellite. The corresponding building height information comes from different datasets, and the corresponding height information can be collected from a certain map company as reference data, or obtained from Microsoft's building height training dataset, or the dataset Eubucco V0.1 can also be used as a reference.

[0049] In this example, the dataset contains 45,000 samples, mainly distributed in multiple urban areas in China, North America, and Europe, with 15,000 samples in each region. The spatial range of each sample is , the resolution is 10m, and the year of the dataset is 2020. In order to obtain data that meets the input standards of the deep learning model, the samples in the dataset are rasterized, and 80% of the samples are used as the training set, and 20% of the samples are used as the test set.

[0050] Furthermore, as Figure 2 shown, combining wavelet transform and convolutional neural network to construct an initial building height prediction model includes:

[0051] Construct a first encoder and a second encoder with the same structure. Each encoder includes a first convolutional layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolutional block layer, a first residual block layer, a second convolutional block layer, a second residual block layer, a third convolutional block layer, and a third residual block layer arranged in sequence;

[0052] Construct a decoder. The decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence. The structure of each decoding layer is the same, and each decoding layer includes a convolutional layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence;

[0053] Construct a second convolutional layer.

[0054] In this example, the initially constructed building height prediction model adopts a U-net model. The U-net model includes two encoders with the same structure and a decoder. It uses skip connections to directly transmit the features of the encoding part to the decoding part. The two encoders are the first encoder and the second encoder respectively. The first encoder is used to extract the features of satellite optical images, and the second encoder is used to extract the features of synthetic aperture radar images.

[0055] Further, step 3 includes:

[0056] Input the satellite optical images in the training set into the first encoder, and four groups of feature map tensors are extracted, denoted as ;

[0057] Input the synthetic aperture radar images in the training set into the second encoder, and four groups of feature map tensors are extracted, denoted as ;

[0058] Add the feature map tensor f 4 to the feature map tensor and then input the result into the first decoding layer. Add the feature map tensor f 3 to the feature map tensor , then concatenate the result with the tensor output by the first decoding layer in dimension and input the concatenated result into the second decoding layer. Add the feature map tensor f 2 to the feature map tensor , then concatenate the result with the tensor output by the second decoding layer in dimension and input the concatenated result into the third decoding layer. Add the feature map tensor f 1 to the feature map tensor , then concatenate the result with the tensor output by the third decoding layer in dimension and input the concatenated result into the fourth decoding layer. Pass the tensor output by the fourth decoding layer through the second convolutional layer to obtain the height of the building.

[0059] Further, the satellite optical images input into the first encoder adopt a four-channel input, including the red, green, blue, and near-infrared bands.

[0060] Further, the synthetic aperture radar images input into the second encoder adopt a four-channel input, including ascending and descending orbits in the vertical transmit - vertical receive (VV) polarization mode, and ascending and descending orbits in the vertical transmit - horizontal receive (VH) polarization mode.

[0061] In this example, since the pixels of the image can be regarded as numbers, the input of each channel of the image can be regarded as a two-dimensional matrix, and the input end of the encoder is a tensor, where represents the height of the tensor width, and 4 is the number of input channels.

[0062] Taking the input of satellite optical images into the first encoder as an example for illustration. After the two-dimensional matrix of the satellite optical image is input, it first passes through the first convolutional layer. The stride set for this first convolutional layer is 2, the convolutional kernel size is , the number of convolutional kernels is 64. After passing through the convolutional layer, the tensor size becomes , and this tensor is denoted as f 1 . Then it passes through the normalization layer and uses the activation function Relu to enhance the non-linear features of the model. As Figure 3 shown, then it enters the wavelet transform layer to capture the features of the four frequency components HH, HL, LH, and LL of the image, enhancing the useful features of the image. The wavelet transform can be performed separately on each channel. After passing through the wavelet transform layer, the number of channels of the input image becomes four times the original, that is, the tensor size becomes . Among them, LL represents the low-frequency components in the horizontal and vertical directions, LH represents the horizontal low-frequency and vertical high-frequency components, HL represents the horizontal high-frequency and vertical low-frequency components, and HH represents the high-frequency components in the horizontal and vertical directions. Then it sequentially enters the first convolutional block layer and the first residual block layer. The first convolutional block layer consists of three repeated convolutional layers, a normalization layer, and the activation function Relu. The convolutional kernel sizes of the convolutional layers are all , the stride is set to 2, and the numbers of convolutional kernels of the three convolutional layers are 64, 64, and 256 respectively. The first residual block layer does not change the number of input channels and the size, enabling the gradient to propagate more stably in the network and alleviating the problem of gradient disappearance. Therefore, after the first residual block layer, the output tensor size is , denoted as f 2 .

[0063] Next, it successively passes through the second convolutional block layer, the second residual block layer, the third convolutional block layer, and the third residual block layer. The structure of the second convolutional block layer is the same as that of the first convolutional block layer, and the kernel size and stride size are also the same, only the number of convolutional kernels is different. The numbers of convolutional kernels in the three convolutional layers of the second convolutional block layer are 128, 128, and 512 respectively. The second residual block layer does not change the number of input channels and the size. After the second residual block layer, the output tensor size is , denoted as f 3 . The numbers of convolutional kernels in the three convolutional layers of the third convolutional block layer are 256, 256, and 1024 respectively. The third residual block layer does not change the number of input channels and the size. After the third residual block layer, the output tensor size is , denoted as f 4 .

[0064] f 1 , f 2 , f 3 , f 4 are four groups of feature map tensors extracted from an encoder, with sizes of , , and respectively. The two encoders have a symmetric structure and are respectively used to extract the features of optical images and synthetic aperture radar images. Therefore, the other encoder also extracts four groups of feature map tensors, and the tensor sizes are the same as those of f 1 , f 2 , f 3 , f 4 , denoted as .

[0065] Wavelet transform can be extended to multi-dimensional signals, especially in the case of two-dimensional signals such as images. The wavelet transform used in this example is the Haar wavelet transform, which is the simplest wavelet transform.

[0066] Furthermore, the input image of the wavelet transform is represented as a two-dimensional matrix , where represents the pixel value of the i-th row and j-th column, , , M and N respectively represent the total number of rows and the total number of columns;

[0067] In the wavelet transform, first perform a one-dimensional wavelet transform on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is:

[0068] ,

[0069] ,

[0070] In the formula, represents the low-frequency component obtained by calculating two adjacent pixel points in the row direction, represents the high-frequency component obtained by calculating two adjacent pixel points in the row direction;

[0071] Then perform a one-dimensional wavelet transform on the remaining other direction, and the expression of the output image is:

[0072] ,

[0073] ,

[0074] ,

[0075] ,

[0076] In the formula, represents the low-frequency components in the horizontal and vertical directions, represents the low-frequency component in the horizontal direction and the high-frequency component in the vertical direction, represents the high-frequency component in the horizontal direction and the low-frequency component in the vertical direction, represents the high-frequency components in the horizontal and vertical directions.

[0077] The decoder structure is based on an upsampling network, which is used to recover the multi-scale features extracted from the encoder. Different from the traditional upsampling layer, in this example, an inverse wavelet transform operation is used for spatial upsampling, and through operations in the frequency domain, the detailed features are better recovered. The decoder consists of four decoding layers with the same structure, namely the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer. Each decoding layer includes a convolutional layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence. Using the self-attention mechanism in the channel space, the high-resolution details of the image are effectively recovered. The feature maps output by the encoder are concatenated and then input to each decoding layer respectively. Specifically, the feature map tensor f 4 is added to the feature map tensor and then input to the first decoding layer. The feature map tensor f 3 is added to the feature map tensor and then dimensionally concatenated with the tensor output by the first decoding layer and input to the second decoding layer. The feature map tensor f 2 is added to the feature map tensor and then dimensionally concatenated with the tensor output by the second decoding layer and input to the third decoding layer. The feature map tensor f 1 is added to the feature map tensor After addition, it is concatenated with the tensor output by the third decoding layer in terms of dimensions and then input into the fourth decoding layer. The tensor output by the fourth decoding layer finally passes through a second convolutional layer with a convolutional kernel of 1, and the number of channels is reduced to 1 to obtain the height of the building in the image.

[0078] Furthermore, the expression of the loss function of the initial building height prediction model is:

[0079] ,

[0080] In the formula, CS represents the cosine similarity loss, and MSE represents the mean squared error loss. is the weight coefficient.

[0081] To better train the initial building height prediction model, it is necessary to set the loss function. In this example, a weighted combination of the cosine similarity loss and the mean squared error loss is used as the training loss. The weight of 0.8 represents the cosine similarity loss, and the weight of 0.2 represents the mean squared error loss. The formula for the cosine similarity loss is as follows:

[0082] ,

[0083] In the formula, represents the true height of the building, represents the height of the building predicted by the initial building height prediction model, represents the L2 norm of, represents the L2 norm of, represents the dot product of vectors;

[0084] The formula for the mean squared error loss is as follows:

[0085] ,

[0086] Then the loss function of the initial building height prediction model is expressed as:

[0087] ,

[0088] When training the initial building height prediction model, the batch value is set to 16, the Adam optimizer is used, the initial learning rate is set to 0.0001, and a learning rate scheduler is used. If the validation loss no longer decreases, the learning rate is reduced to 10% of the current learning rate, and the minimum learning rate is set to 0.00001. The trained model is denoted as the target building height detection model, and the validation set and the target building height prediction model are used to predict the height of the buildings in the input optical image and the corresponding synthetic aperture radar image.

[0089] In this example, the root mean square error (RMSE) is used to evaluate the experimental results. RMSE represents the accuracy of the predicted height relative to the reference, and the formula is as follows:

[0090] ,

[0091] where n is the number of validation samples, represents the reference value of the building height, represents the predicted value of the building height obtained from the target building height prediction model.

[0092] The root mean square error calculated by the target building height prediction model constructed based on the present invention is 5.85 m, while the root mean square error when only using a convolutional neural network to predict the building height is 16.8 m, indicating that adding wavelet transform can significantly improve the accuracy of building height prediction. At the same time, other models are also compared. If deconvolution is used instead of wavelet transform and the same convolutional neural network is used for training, the root mean square error of the building height obtained is 6.706 m. If the U-Net structure in this example is changed to the SegNet structure, the root mean square error obtained is 7.21 m, proving that using the U-Net structure can significantly improve the accuracy of building height prediction.

[0093] In this example, the results are visualized, as Figure 4 shown. They are the visualization comparison charts of the predicted building heights in Hangzhou, Hefei, Shanghai, and Xi'an. The first column is the Sar image, the second column is the corresponding optical image, the third column is the reference data of the building height, the fourth column is the building height predicted by the convolutional neural network with wavelet transform added, and the fifth column is the building height predicted only using the convolutional neural network. It can Figure 4 be seen that the method of the present invention basically predicts the building height correctly, and the convolutional neural transform with wavelet transform added predicts the building height more carefully and has clearer edges.

Claims

1. A building height prediction method based on wavelet transform and convolutional neural network, characterized in that: The steps include: Step 1: Obtain satellite optical images and synthetic aperture radar images, use the satellite optical images and synthetic aperture radar images to construct a data set, collect the height information of the corresponding buildings as labels, pre-process the images in the data set, and divide them into a training set and a test set according to the ratio; Step 2, combining wavelet transform and convolutional neural network to build an initial building height prediction model; Step 3, using the training set and labels to train the initial building height prediction model to obtain the target building height prediction model; Step 4, predicting the height of the building using the target building height prediction model; Combining wavelet transform and convolutional neural network to build an initial building height prediction model includes: Constructing a first encoder and a second encoder with the same structure, each encoder comprising a first convolution layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolution block layer, a first residual block layer, a second convolution block layer, a second residual block layer, a third convolution block layer, and a third residual block layer, which are arranged in sequence; Constructing a decoder, the decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence, each decoding layer has the same structure, and each decoding layer includes a convolution layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence; Construct the second convolutional layer; Step 3 includes: The satellite optical images in the training set are input into the first encoder, and four sets of feature map tensors are extracted, which are recorded as f1, f2, f3, and f4; The synthetic aperture radar images in the training set are input into the second encoder, and four sets of feature map tensors are extracted, denoted as ; Combine the feature map tensor f4 with the feature map tensor After addition, it is input to the first decoding layer, and the feature map tensor f3 is combined with the feature map tensor After addition, the tensor output by the first decoding layer is concatenated in dimension and input into the second decoding layer, and the feature map tensor f2 is combined with the feature map tensor After addition, the tensor output by the second decoding layer is concatenated in dimension and input into the third decoding layer, and the feature map tensor f1 is combined with the feature map tensor After addition, the tensor output by the third decoding layer is concatenated in dimension and input into the fourth decoding layer. The tensor output by the fourth decoding layer passes through the second convolutional layer to obtain the height of the building.

2. A building height prediction method based on wavelet transform and convolutional neural network according to claim 1, characterized in that: The satellite optical image input into the first encoder uses four-channel input, including red, green, blue and near-infrared bands.

3. A building height prediction method based on wavelet transform and convolutional neural network according to claim 2, characterized in that: The synthetic aperture radar image input into the second encoder adopts a four-channel input, including an ascending orbit and a descending orbit in a VV polarization mode, and an ascending orbit and a descending orbit in a VH polarization mode.

4. The building height prediction method based on wavelet transform and convolutional neural network according to claim 3 is characterized in that: Represent the input image of the wavelet transform as a two-dimensional matrix ,in, represents the pixel value of the i-th row and j-th column, , , M and N represent the total number of rows and columns respectively; In the wavelet transform, a one-dimensional wavelet transform is first performed on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is: ， ， In the formula, Represents the low-frequency component calculated from two adjacent pixels in the row direction. Represents the high-frequency component calculated from two adjacent pixels in the row direction; Then perform one-dimensional wavelet transform on the remaining direction, and the expression of the output image is: ， ， ， ， In the formula, Represents the low-frequency components in the horizontal and vertical directions, Represents low-frequency components in the horizontal direction and high-frequency components in the vertical direction. Represents the high-frequency components in the horizontal direction and the low-frequency components in the vertical direction. Represents high-frequency components in horizontal and vertical directions.

5. A building height prediction method based on wavelet transform and convolutional neural network according to any one of claims 1 to 4, characterized in that: The loss function expression of the initial building height prediction model is: ， In the formula, CS represents cosine similarity loss, MSE represents mean square error loss, is the weight coefficient.

Images