Building height prediction method based on wavelet transform and convolutional neural network

Abstract

The invention discloses a building height prediction method based on wavelet transform and a convolutional neural network, and belongs to the technical field of building measurement. The method comprises the following steps: constructing a data set by using a satellite optical image and a synthetic aperture radar image; then combining wavelet transform and a convolutional neural network to construct an initial building height prediction model; training the initial building height prediction model by using the training set and the label; and finally predicting the height of the building by using the target building height prediction model. Wavelet transformation is fused into the convolutional neural network, information of four different frequencies of the image is captured, high-frequency and low-frequency information of the image is fully utilized, expression of building features is enhanced, the building height recognition capability of the convolutional neural network is utilized, and the building height prediction accuracy is further improved.

Description

Technical Field

[0001] The invention relates to a building height prediction method, in particular to a building height prediction method based on wavelet transform and convolutional neural network, belonging to the technical field of building measurement. Background Art

[0002] With the acceleration of urbanization and the rise of the low-altitude economy, accurate measurement of building heights has become increasingly important. The low-altitude economy refers to the use of low-altitude airspace for a series of commercial activities, including drone aerial photography and low-altitude logistics. These activities not only rely on urban planning, air traffic management and environmental monitoring, but also put forward more accurate requirements for the measurement of urban building heights.

[0003] Traditional building height acquisition methods, such as manual measurement and formula-based calculation, have certain limitations. Manual measurement is usually time-consuming and labor-intensive, while building height estimation based on analytical formulas is performed using a shadow-based formula. In shadow-based building height estimation, the length of the building in the image and its shadow in the pixel are combined with the sun elevation to obtain the true building height, but this method usually requires other assumptions, resulting in inaccurate estimation results.

[0004] With the development of machine learning and deep learning, both have made significant progress in the estimation of building height in the field of remote sensing. Among machine learning methods, support vector machines are often used to predict building height. They mainly extract features such as texture and shape from images for classification or regression analysis, but they usually perform better when the amount of data is small and it is difficult to process large-scale remote sensing image data. Convolutional neural network is a deep learning technology that can extract image features through multi-layer convolution, and thus use the features of radar images and optical images to predict building heights. It has been widely used, especially synthetic aperture radar (Synthetic Aperture Radar, Sar) images. Convolutional neural networks can quickly process large-scale remote sensing image data, improve the computational efficiency of building height prediction, and reduce the consumption of human and material resources. However, when processing radar images, traditional convolutional neural networks ignore the frequency domain features of the image, which makes the network incomplete in utilizing the image information, resulting in a decrease in prediction accuracy. How to efficiently and accurately predict building heights has become a technical problem that needs to be solved in this field. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to propose a building height prediction method based on wavelet transform and convolutional neural network.

[0006] Technical solution: A building height prediction method based on wavelet transform and convolutional neural network of the present invention comprises the following steps: Step 1: Obtain satellite optical images and synthetic aperture radar images, use the satellite optical images and synthetic aperture radar images to construct a data set, collect the height information of the corresponding buildings as labels, pre-process the images in the data set, and divide them into a training set and a test set according to the ratio; Step 2, combining wavelet transform and convolutional neural network to build an initial building height prediction model; Step 3, using the training set and labels to train the initial building height prediction model to obtain the target building height prediction model; Step 4: Use the target building height prediction model to predict the building height.

[0007] Furthermore, the initial building height prediction model is constructed by combining wavelet transform and convolutional neural network, including: Constructing a first encoder and a second encoder with the same structure, each encoder comprising a first convolution layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolution block layer, a first residual block layer, a second convolution block layer, a second residual block layer, a third convolution block layer, and a third residual block layer, which are arranged in sequence; Constructing a decoder, the decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence, each decoding layer has the same structure, and each decoding layer includes a convolution layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence; Construct the second convolutional layer.

[0008] Further, step 3 includes: The satellite optical images in the training set are input into the first encoder, and four sets of feature map tensors are extracted, which are recorded as f1, f2, f3, and f4; The synthetic aperture radar images in the training set are input into the second encoder, and four sets of feature map tensors are extracted, denoted as ; Combine the feature map tensor f4 with the feature map tensor After addition, it is input to the first decoding layer, and the feature map tensor f3 is combined with the feature map tensor After addition, the tensor output by the first decoding layer is concatenated in dimension and input into the second decoding layer, and the feature map tensor f2 is combined with the feature map tensor After addition, the tensor output by the second decoding layer is concatenated in dimension and input into the third decoding layer, and the feature map tensor f1 is combined with the feature map tensor After addition, the tensor output by the third decoding layer is concatenated in dimension and input into the fourth decoding layer. The tensor output by the fourth decoding layer passes through the second convolutional layer to obtain the height of the building.

[0009] Furthermore, the satellite optical image input into the first encoder adopts four-channel input, including red, green, blue and near-infrared bands.

[0010] Furthermore, the synthetic aperture radar image input into the second encoder adopts a four-channel input, including an ascending orbit and a descending orbit of a VV polarization mode, and an ascending orbit and a descending orbit of a VH polarization mode.

[0011] Furthermore, the input image of the wavelet transform is represented as a two-dimensional matrix ,in, represents the pixel value of the i-th row and j-th column, , , M and N represent the total number of rows and columns respectively; In the wavelet transform, a one-dimensional wavelet transform is first performed on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is: , , In the formula, Represents the low-frequency component calculated from two adjacent pixels in the row direction. Represents the high-frequency component calculated from two adjacent pixels in the row direction; Then perform one-dimensional wavelet transform on the remaining direction, and the expression of the output image is: , , , , In the formula, Represents the low-frequency components in the horizontal and vertical directions, Represents low-frequency components in the horizontal direction and high-frequency components in the vertical direction. Represents the high-frequency components in the horizontal direction and the low-frequency components in the vertical direction. Represents high-frequency components in horizontal and vertical directions.

[0012] Furthermore, the loss function expression of the initial building height prediction model is: , In the formula, CS represents cosine similarity loss, MSE represents mean square error loss, is the weight coefficient.

[0013] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The present invention integrates wavelet transform into convolutional neural network, and proposes a new building height prediction method based on wavelet transform and convolutional neural network, which finally regresses to a specific numerical value of building height. The four different frequency information of the image are captured by wavelet transform, and the high-frequency and low-frequency information of the image are fully utilized. The use of deconvolution is also avoided, and the problems such as artifacts and interpolation errors that may be caused by deconvolution are avoided. The expression of building features is enhanced, and the ability of convolutional neural network to recognize building height is fully utilized, so that the accuracy of building height prediction is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of a building height prediction method based on wavelet transform and convolutional neural network; Figure 2 It is a structural diagram of the initial building height prediction model; Figure 3 This is the principle diagram of wavelet transform; Figure 4 A diagram showing the visualization of building height prediction results. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0016] The method for predicting building height based on wavelet transform and convolutional neural network described in this embodiment is as follows: Figure 1 As shown, the method comprises the following steps: Step 1: Obtain satellite optical images and synthetic aperture radar images, use the satellite optical images and synthetic aperture radar images to construct a data set, collect the height information of the corresponding buildings as labels, pre-process the images in the data set, and divide them into a training set and a test set according to the ratio; Step 2, combining wavelet transform and convolutional neural network to build an initial building height prediction model; Step 3, using the training set and labels to train the initial building height prediction model to obtain the target building height prediction model; Step 4: Use the target building height prediction model to predict the building height.

[0017] The present invention combines the multi-resolution characteristics of wavelet transform and the feature extraction ability of convolutional neural network to realize the prediction of building height. Specifically, the information of the image at four different frequencies is captured by wavelet transform, and the high-frequency information and low-frequency information of the image are fully utilized to enhance the useful features of the image and make more complete use of the image information. The convolutional neural network is used to avoid problems such as artifacts and interpolation errors that may be caused by deconvolution, enhance the expression of building features, and greatly improve the accuracy of building height prediction.

[0018] In one example, the dataset used includes synthetic aperture radar images and optical images, as well as height information of corresponding buildings as reference data. The synthetic aperture radar images come from the Sentinel-1 satellite, and the optical images come from the Sentinel-2 satellite. The corresponding building height information comes from different datasets, and the corresponding height information can be collected from a certain map company as reference data, or obtained from Microsoft's building height training dataset, or the dataset Eubucco V0.1 can also be used as a reference.

[0019] In this example, the dataset contains 45,000 samples, mainly distributed in multiple urban areas in China, North America, and Europe, with 15,000 samples in each area. The spatial range of each sample is , the resolution is 10m, and the year of the dataset is 2020. In order to obtain data that meets the input standards of the deep learning model, the samples in the dataset are rasterized, and 80% of the samples are used as training sets and 20% of the samples are used as test sets.

[0020] Furthermore, if Figure 2 As shown, the initial building height prediction model constructed by combining wavelet transform and convolutional neural network includes: Constructing a first encoder and a second encoder with the same structure, each encoder comprising a first convolution layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolution block layer, a first residual block layer, a second convolution block layer, a second residual block layer, a third convolution block layer, and a third residual block layer, which are arranged in sequence; Constructing a decoder, the decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence, each decoding layer has the same structure, and each decoding layer includes a convolution layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence; Construct the second convolutional layer.

[0021] The initial building height prediction model constructed in this example adopts the U-net model, which includes two encoders with the same structure and a decoder. The features of the encoding part are directly transmitted to the decoding part using skip links. The two encoders are the first encoder and the second encoder. The first encoder is used to extract the features of satellite optical images, and the second encoder is used to extract the features of synthetic aperture radar images.

[0022] Further, step 3 includes: The satellite optical images in the training set are input into the first encoder, and four sets of feature map tensors are extracted, denoted as ; The synthetic aperture radar images in the training set are input into the second encoder, and four sets of feature map tensors are extracted, denoted as ; Combine the feature map tensor f4 with the feature map tensor After addition, it is input to the first decoding layer, and the feature map tensor f3 is combined with the feature map tensor After addition, the tensor output by the first decoding layer is concatenated in dimension and input into the second decoding layer, and the feature map tensor f2 is combined with the feature map tensor After addition, the tensor output by the second decoding layer is concatenated in dimension and input into the third decoding layer, and the feature map tensor f1 is combined with the feature map tensor After addition, the tensor output by the third decoding layer is concatenated in dimension and input into the fourth decoding layer. The tensor output by the fourth decoding layer passes through the second convolutional layer to obtain the height of the building.

[0023] Furthermore, the satellite optical image input into the first encoder adopts four-channel input, including red, green, blue and near-infrared bands.

[0024] Furthermore, the synthetic aperture radar image input into the second encoder adopts a four-channel input, including an ascending orbit and a descending orbit of a vertical transmission-vertical reception (VV) polarization mode, and an ascending orbit and a descending orbit of a vertical transmission-horizontal reception (VH) polarization mode.

[0025] In this example, since the pixels of an image can be viewed as numbers, the input of each channel of the image can be viewed as a The input of the encoder is a two-dimensional matrix. A tensor of Represented as the height of the tensor Width, 4 is the number of input channels.

[0026] The satellite optical image is input to the first encoder as an example. After the two-dimensional matrix of the satellite optical image is input, it first passes through the first convolution layer. The step size of the first convolution layer is set to 2 and the convolution kernel size is , the number of convolution kernels is 64, and the tensor size becomes , and record the tensor as f1. Then pass through the normalization layer and use the activation function Relu to enhance the nonlinear characteristics of the model. Figure 3 As shown, it then enters the wavelet transform layer to capture the features of the four frequency components of the image, HH, HL, LH, and LL, and enhance the useful features of the image. The wavelet transform can be performed separately on each channel. After the wavelet transform layer, the number of channels of the input image becomes four times the original, that is, the tensor size becomes . LL represents the low-frequency components in the horizontal and vertical directions, LH represents the low-frequency components in the horizontal and vertical directions, HL represents the high-frequency components in the horizontal and vertical directions, and HH represents the high-frequency components in the horizontal and vertical directions. Then, we enter the first convolutional block layer and the first residual block layer in turn. The first convolutional block layer consists of three repeated convolutional layers, a normalization layer, and an activation function Relu. The convolution kernel size of the convolutional layer is , the step size is set to 2, and the number of convolution kernels of the three convolutional layers are 64, 64, and 256 respectively. The first residual block layer does not change the number and size of input channels, making the gradient propagation more stable in the network and alleviating the gradient vanishing problem. Therefore, the output tensor size after the first residual block layer is , denoted as f2.

[0027] Next, it passes through the second convolutional block layer, the second residual block layer, the third convolutional block layer, and the third residual block layer in sequence. The structure of the second convolutional block layer is the same as that of the first convolutional block layer, and the size of the convolution kernel and the step size are also the same. Only the number of convolution kernels is different. The number of convolution kernels of the three convolutional layers in the second convolutional block layer is 128, 128, and 512 respectively. The second residual block layer does not change the number and size of input channels. After the second residual block layer, the output tensor size is , denoted as f3. The number of convolution kernels in the three convolutional layers in the third convolutional block layer are 256, 256, and 1024 respectively. The third residual block layer does not change the number and size of input channels. The output tensor size after the third residual block layer is , denoted as f4.

[0028] f1, f2, f3, and f4 are four sets of feature map tensors extracted from an encoder, with sizes of , , as well as The two encoders are symmetrical structures, which are used to extract the features of optical images and synthetic aperture radar images respectively. Therefore, the other encoder also extracts four sets of feature map tensors, and the size of the tensors is the same as f1, f2, f3, and f4, which are recorded as .

[0029] Wavelet transform can be extended to multi-dimensional signals, especially in the case of two-dimensional signals such as images. The wavelet transform used in this example is Haar wavelet transform, which is the simplest wavelet transform.

[0030] Furthermore, the input image of the wavelet transform is represented as a two-dimensional matrix ,in, represents the pixel value of the i-th row and j-th column, , , M and N represent the total number of rows and columns respectively; In the wavelet transform, a one-dimensional wavelet transform is first performed on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is: , , In the formula, Represents the low-frequency component calculated from two adjacent pixels in the row direction. Represents the high-frequency component calculated from two adjacent pixels in the row direction; Then perform one-dimensional wavelet transform on the remaining direction, and the expression of the output image is: , , , , In the formula, Represents the low-frequency components in the horizontal and vertical directions, Represents low-frequency components in the horizontal direction and high-frequency components in the vertical direction. Represents the high-frequency components in the horizontal direction and the low-frequency components in the vertical direction. Represents high-frequency components in horizontal and vertical directions.

[0031] The decoder structure is based on an upsampling network, which is used to restore the multi-scale features extracted from the encoder. Unlike the traditional upsampling layer, this example uses an inverse wavelet transform operation to perform spatial upsampling. Through operations in the frequency domain, the detail features can be better restored. The decoder consists of four decoding layers with the same structure, namely the first decoding layer, the second decoding layer, the third decoding layer and the fourth decoding layer. Each decoding layer includes a convolution layer, a normalization layer, an inverse wavelet transform layer and an attention mechanism layer arranged in sequence. The self-attention mechanism in the channel space is used to effectively restore the high-resolution details of the image. The feature maps output by the encoder are spliced ​​and input into each decoding layer respectively. Specifically, the feature map tensor f4 is combined with the feature map tensor After addition, it is input to the first decoding layer, and the feature map tensor f3 is combined with the feature map tensor After addition, the tensor output by the first decoding layer is concatenated in dimension and input into the second decoding layer, and the feature map tensor f2 is combined with the feature map tensor After addition, the tensor output by the second decoding layer is concatenated in dimension and input into the third decoding layer, and the feature map tensor f1 is combined with the feature map tensor After addition, the tensor output by the third decoding layer is concatenated in dimension and input into the fourth decoding layer. The tensor output by the fourth decoding layer finally passes through a second convolution layer with a convolution kernel of 1, and the number of channels is reduced to 1 to obtain the height of the building in the image.

[0032] Furthermore, the loss function expression of the initial building height prediction model is: , In the formula, CS represents cosine similarity loss, MSE represents mean square error loss, is the weight coefficient.

[0033] In order to better train the initial building height prediction model, it is necessary to set the loss function. This example uses a weighted combination of cosine similarity loss and mean square error loss as the training loss, with a weight of 0.8 for cosine similarity loss and a weight of 0.2 for mean square error loss. The formula for cosine similarity loss is as follows: , In the formula, Indicates the true height of the building. represents the building height predicted by the initial building height prediction model, express The L2 norm of express The L2 norm of Represents the dot product of vectors; The formula for the mean square error loss is as follows: , Then the loss function of the initial building height prediction model is expressed as: , When training the initial building height prediction model, the batch value is set to 16, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the learning rate scheduler is used. If the validation loss no longer decreases, the learning rate is reduced to 10% of the current learning rate, and the minimum learning rate is set to 0.00001. The trained model is recorded as the target building height detection model, and the validation set and the target building height prediction model are used to predict the height of the buildings in the input optical image and the corresponding synthetic aperture radar image.

[0034] In this example, the root mean square error (RMSE) is used to evaluate the experimental results. RMSE represents the accuracy of the predicted height relative to the reference. The formula is as follows: , In the formula, n is the number of validation samples, Indicates the reference value of the building height, Represents the predicted value of the building height obtained by the target building height prediction model.

[0035] The root mean square error calculated by the target building height prediction model constructed based on the present invention is 5.85m, while the root mean square error when only the convolutional neural network is used to predict the building height is 16.8m, which shows that the addition of wavelet transform can significantly improve the accuracy of building height prediction. At the same time, other models are also compared. If deconvolution is used instead of wavelet transform, the root mean square error of the building height obtained when the convolutional neural network is used for training is 6.706m. If the U-Net structure in this example is changed to the SegNet structure, the root mean square error obtained is 7.21m, which proves that the use of the U-Net structure can significantly improve the accuracy of building height prediction.

[0036] In this example, the results are visualized as Figure 4 As shown in the figure, the visual comparison of building height prediction results in Hangzhou, Hefei, Shanghai, and Xi'an are shown. The first column is the Sar image, the second column is the corresponding optical image, the third column is the reference data of the building height, the fourth column is the building height predicted by the convolutional neural network with wavelet transform, and the fifth column is the building height predicted by the convolutional neural network only. Figure 4 It can be seen that the method of the present invention is basically correct in predicting the building height, and the convolutional neural transform with wavelet transform makes the prediction of the building height more detailed and the edge clearer.

Claims

1. A building height prediction method based on wavelet transform and convolutional neural network, characterized in that: The steps include: Step 1: Obtain satellite optical images and synthetic aperture radar images, use the satellite optical images and synthetic aperture radar images to construct a data set, collect the height information of the corresponding buildings as labels, pre-process the images in the data set, and divide them into a training set and a test set according to the ratio; Step 2, combining wavelet transform and convolutional neural network to build an initial building height prediction model; Step 3, using the training set and labels to train the initial building height prediction model to obtain the target building height prediction model; Step 4: Use the target building height prediction model to predict the building height.

2. A building height prediction method based on wavelet transform and convolutional neural network according to claim 1, characterized in that: Combining wavelet transform and convolutional neural network to build an initial building height prediction model includes: Constructing a first encoder and a second encoder with the same structure, each encoder comprising a first convolution layer, a normalization layer, an activation function layer, a wavelet transform layer, a first convolution block layer, a first residual block layer, a second convolution block layer, a second residual block layer, a third convolution block layer, and a third residual block layer, which are arranged in sequence; Constructing a decoder, the decoder includes a first decoding layer, a second decoding layer, a third decoding layer, and a fourth decoding layer arranged in sequence, each decoding layer has the same structure, and each decoding layer includes a convolution layer, a normalization layer, an inverse wavelet transform layer, and an attention mechanism layer arranged in sequence; Construct the second convolutional layer.

3. A building height prediction method based on wavelet transform and convolutional neural network according to claim 2, characterized in that: Step 3 includes: The satellite optical images in the training set are input into the first encoder, and four sets of feature map tensors are extracted, which are recorded as f1, f2, f3, and f4; The synthetic aperture radar images in the training set are input into the second encoder, and four sets of feature map tensors are extracted, denoted as ; Combine the feature map tensor f4 with the feature map tensor After addition, it is input to the first decoding layer, and the feature map tensor f3 is combined with the feature map tensor After addition, the tensor output by the first decoding layer is concatenated in dimension and input into the second decoding layer, and the feature map tensor f2 is combined with the feature map tensor After addition, the tensor output by the second decoding layer is concatenated in dimension and input into the third decoding layer, and the feature map tensor f1 is combined with the feature map tensor After addition, the tensor output by the third decoding layer is concatenated in dimension and input into the fourth decoding layer. The tensor output by the fourth decoding layer passes through the second convolutional layer to obtain the height of the building.

4. The building height prediction method based on wavelet transform and convolutional neural network according to claim 3 is characterized in that: The satellite optical image input into the first encoder uses four-channel input, including red, green, blue and near-infrared bands.

5. The method for predicting building height based on wavelet transform and convolutional neural network according to claim 4, characterized in that: The synthetic aperture radar image input into the second encoder adopts a four-channel input, including an ascending orbit and a descending orbit in a VV polarization mode, and an ascending orbit and a descending orbit in a VH polarization mode.

6. A building height prediction method based on wavelet transform and convolutional neural network according to claim 5, characterized in that: Represent the input image of the wavelet transform as a two-dimensional matrix ,in, represents the pixel value of the i-th row and j-th column, , , M and N represent the total number of rows and columns respectively; In the wavelet transform, a one-dimensional wavelet transform is first performed on the two-dimensional matrix in either the row direction or the column direction. The expression of the output image is: ， ， In the formula, Represents the low-frequency component calculated from two adjacent pixels in the row direction. Represents the high-frequency component calculated from two adjacent pixels in the row direction; Then perform one-dimensional wavelet transform on the remaining direction, and the expression of the output image is: ， ， ， ， In the formula, Represents the low-frequency components in the horizontal and vertical directions, Represents low-frequency components in the horizontal direction and high-frequency components in the vertical direction. Represents the high-frequency components in the horizontal direction and the low-frequency components in the vertical direction. Represents high-frequency components in horizontal and vertical directions.

7. A building height prediction method based on wavelet transform and convolutional neural network according to any one of claims 1 to 6, characterized in that: The loss function expression of the initial building height prediction model is: ， In the formula, CS represents cosine similarity loss, MSE represents mean square error loss, is the weight coefficient.

Images