Convolutional neural network-based lightweight building three-dimensional model extraction method and system

Abstract

The invention relates to a lightweight building three-dimensional model extraction method and system based on a convolutional neural network, and the method comprises the steps: constructing a sample library through employing two kinds of remote sensing image data, i.e., a digital orthographic map (DOM) and a digital surface model (DSM), and corresponding building contour and building real height values; and training the convolutional neural network for extracting the three-dimensional model of the lightweight building, and learning two-dimensional and three-dimensional features of the building. The trained network model is used for predicting new DOM and DSM data, and a prismatic lightweight building three-dimensional model can be directly extracted. The method has the advantages that building contour extraction and height estimation are carried out without depending on artificial experience design rules, and the automation degree is high; the efficient reasoning speed of the convolutional neural network can ensure that a lightweight building three-dimensional model in a large-range region can be quickly constructed, and the extraction efficiency is high.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to a method for extracting a lightweight building three-dimensional model based on a convolutional neural network, which can realize automatic extraction of a prismatic lightweight building three-dimensional model from two types of remote sensing image data, namely a digital orthophoto map (DOM) and a digital surface model (DSM). The method can be used in urban management and planning, population density analysis, autonomous driving and other fields. Background Art

[0002] Buildings are the main places for human activities. The automatic extraction of building three-dimensional models using remote sensing image data has long been an important task in the field of photogrammetry and computer vision. According to the definition of urban geographic markup language, the LoD-1 level building three-dimensional model is a lightweight prismatic model that can be directly constructed from the building's outline and vertical height. Traditional methods use empirical rules to extract building outlines, such as methods combining region growing and edge detection, methods based on level set segmentation, and methods based on Markov random fields; the height is calculated by generating a normalized digital surface model (nDSM), using building neighborhood elevation change trend analysis, or using shadow analysis. Traditional methods are limited by the setting of empirical parameters in terms of building outline extraction and height calculation, and have low efficiency and poor versatility. In recent years, convolutional neural networks, which have shown excellent performance, have been widely used in many tasks in computer vision, photogrammetry, and remote sensing. The designed convolutional neural network is trained using manually annotated image data. Based on a large number of learnable parameters in the network, the mapping relationship between the input image and the desired output can be automatically established. Because there is no need to manually set empirical parameters, it has good versatility. There have been many studies on building contour extraction based on convolutional neural networks, but the related research on using convolutional neural networks to extract lightweight building three-dimensional models is still relatively small and incomplete, and is not yet mature. Summary of the invention

[0003] In view of the current situation that existing convolutional neural networks are rarely used and have poor effects in the task of extracting lightweight building three-dimensional models, the present invention uses two kinds of remote sensing image data, digital orthophoto map (DOM) and digital surface model (DSM), as input sources, carries out structural design and model building of convolutional neural networks, and proposes a lightweight building three-dimensional model extraction method based on convolutional neural networks. The method can efficiently and reliably extract lightweight prismatic building three-dimensional models from remote sensing image data, which has important practical significance and practical value for realizing fully automatic extraction of lightweight building three-dimensional models.

[0004] The technical solution used to achieve the purpose of the present invention is: a method for extracting a lightweight building three-dimensional model using a convolutional neural network, comprising the following steps: Step 1: Construct a convolutional neural network for lightweight building 3D model extraction. The network is used to learn the 2D and 3D features of buildings from two types of remote sensing image data: digital orthophoto map (DOM) and digital surface model (DSM), and establish a mapping relationship between the learned features and the target output (building outline, height); The convolutional neural network for extracting lightweight building 3D models described in step 1 includes a feature extraction subnetwork, a 3D candidate bounding box generation subnetwork, a 3D bounding box regression and a mask image generation subnetwork, a total of 3 parts; Step 2: Combine the two remote sensing image data of Digital Orthophoto Map (DOM) and Digital Surface Model (DSM) with the corresponding building outlines and real building height values ​​to build a sample library, and use the sample library to train the constructed convolutional neural network model in the Pytorch deep learning framework. By loading the trained network model and predicting the new digital orthophoto map (DOM) and digital surface model (DSM) data, the building outline and height information of the target area are obtained, and then the prismatic lightweight building 3D model is directly constructed by combining the outline and height.

[0005] Furthermore, the feature extraction sub-network can directly use the currently popular residual structure network (ResidualNetwork, ResNet), visual transformer network (Visual Transformer, ViT), etc. to perform feature encoding on the input digital orthophoto map (Digital Orthophoto map, DOM) and digital surface model (Digital Surface Model, DSM) data, and generate feature maps for use in subsequent parts of the network.

[0006] Furthermore, the three-dimensional candidate border generation subnetwork includes four parallel predictors: a border center point predictor, a border length predictor, a border width predictor and a border height predictor, and the four predictors all use the feature map output by the feature extraction subnetwork as input. The center point predictor consists of a 3×3 convolution layer, a rectified linear unit (ReLU), a 1×1 convolution layer and a Sigmoid activation function. The center point predictor is responsible for finding the center point position of the three-dimensional border of the building in the input image data to identify different individual buildings. The border length predictor, the border width predictor and the border height predictor have the same structure, all consisting of a 3×3 convolution layer, a rectified linear unit (ReLU), and a 1×1 convolution layer, which are responsible for predicting the absolute length, width and height values ​​of the building in three-dimensional space respectively. The local peak point on the activation feature map output by the center point predictor is used as the center position of the building, and the length, width and height values ​​of the building's three-dimensional border output by the border length predictor, border width predictor and border height predictor at the corresponding position are taken out synchronously to obtain the three-dimensional border of the building in space. The position of the three-dimensional border on the plane and the feature map output by the feature extraction subnetwork are used as input, and the region of interest alignment pooling (RoIAlign) operation is used to sample at two resolutions of 7×7 and 14×14 respectively, to obtain a 7×7 low-resolution Region of Interest (RoI) feature and a 14×14 high-resolution RoI feature, which are used in the subsequent parts of the network.

[0007] Furthermore, the three-dimensional bounding box regression and mask map generation subnetwork includes three branches: a bounding box correction branch, a height correction branch, and a building mask map prediction branch. The bounding box correction branch takes the flattened low-resolution RoI feature as input, and is responsible for regressing and correcting the two-dimensional bounding box of the building on the plane dimension, that is, correcting the length and width of the three-dimensional bounding box of the building. The bounding box correction branch includes three fully connected layers, wherein the first two fully connected layers are followed by a rectified linear unit (ReLU), and the last fully connected layer directly outputs the regression parameters of the two-dimensional bounding box of the building. The height correction branch shares the first two fully connected layers with the bounding box correction branch, and finally uses a separate fully connected layer to predict the building height regression parameters. The building mask map prediction branch takes the high-resolution RoI feature as input, and is composed of three groups of sequentially connected convolution layers (convolution layer) + rectified linear units (RectifiedLinear Unit, ReLU) and a group of convolution layers (convolution layer) + Sigmoid activation functions, which is responsible for predicting the coverage area of ​​a single building, and uses a raster-to-vector operation to convert the prediction results into building outline polygons in vector format.

[0008] Furthermore, the specific implementation of step 2 includes the following sub-steps: Step 2.1: Annotate the outlines of all buildings that can be visually interpreted in the digital orthophoto map (DOM) in polygonal form, and combine with the digital surface model (DSM) data to manually measure the height of each building. Record the outline polygon and height value of each building as a line of a txt file to generate an annotation file. Then, the original large-size DOM and DSM data are cropped to the standard size supported by the network model, that is, 512×512 pixel tiles, and the three-channel DOM and the single-channel DSM are connected in series along the channel direction to obtain a four-channel synthetic tile. According to the tile cropping range, the building outline polygons in the annotation file are cropped and the coordinates of the polygons are converted to the coordinates in the tile, and the txt annotation file corresponding to each tile is regenerated.

[0009] Step 2.2: Train the convolutional neural network. Use the synthetic tiles generated in step 2.1 and the corresponding annotation files in txt format as training data, and iteratively train the convolutional neural network in the Pytorch deep learning framework until convergence.

[0010] Step 2.3, given the new data to be extracted from the building 3D model, the size of the image cropping is the same as that in the process of making training data, the original large-size DOM and DSM image data are cropped and input into the trained convolutional neural network model in series to obtain the prediction results on each tile. After all tiles are predicted, all predicted building outline coordinates and height information are mapped back to the original large-size DOM and DSM images to obtain all building outlines and corresponding heights in the original coordinate system. Finally, the maximum value on the DSM image within the building outline range is taken as the building roof elevation, the roof elevation minus the predicted height to obtain the building base elevation, the building outline is stretched from the roof elevation to the base elevation to obtain the building side, and then together with the building outline surface at the roof elevation and the building outline surface at the base position, a closed prismatic lightweight building 3D model is formed.

[0011] Furthermore, in step 2.2, a specific loss function is used to supervise the output of each part of the convolutional neural network during the training process. The output of the center point predictor of the 3D candidate bounding box generation subnetwork is supervised by Focal Loss, and the outputs of the length predictor, width predictor, and height predictor are all supervised by Smooth L1 Loss. The outputs of the bounding box correction branch and the height correction branch in the 3D bounding box regression and mask map generation subnetwork are supervised by Smooth L1 Loss, and the output of the mask map prediction branch is supervised by Binary Cross Entropy Loss.

[0012] The present invention also provides a convolutional neural network lightweight building three-dimensional model extraction system, comprising: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the lightweight building three-dimensional model extraction method of the convolutional neural network as described in the above technical solution.

[0013] The invention has the following advantages: 1) It does not rely on artificial empirical design rules to extract building outlines and estimate heights. Through an end-to-end, fully automatic learning convolutional neural network model, it can automatically extract a prismatic lightweight building three-dimensional model from two remote sensing image data, namely, a digital orthophoto map (DOM) and a digital surface model (DSM). 2) It has strong reusability, and the model can be trained on additional data, and can be used for continuous online learning, and can be continuously iterated and optimized. 3) It has good scalability, and the trained convolutional neural network model can be adjusted and applied to other purposes, such as three-dimensional change detection of buildings based on remote sensing image data. 4) It has strong robustness, and the two remote sensing image data, namely, a digital orthophoto map (DOM) and a digital surface model (DSM), are used to extract lightweight building three-dimensional models, which can complement each other and achieve good extraction effects in complex building areas. 5) High extraction efficiency: The efficient reasoning speed of the convolutional neural network can ensure the rapid construction of lightweight building three-dimensional models in a large area. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of the lightweight building three-dimensional model extraction method based on convolutional neural network proposed in the present invention.

[0015] Figure 2 It is a structural schematic diagram of a border correction branch and a height correction branch that share two fully connected layers in the method of the present invention.

[0016] Figure 3 The three-dimensional model result of a lightweight building extracted by the method of the present invention on a certain data set.

[0017] Figure 4 The three-dimensional model of the lightweight building extracted by the method of the present invention is projected onto the original aerial image to achieve the effect of fitting the model with the real building in the image. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0019] In the convolutional neural network for extracting lightweight building three-dimensional models constructed by the method of the present invention, the three-dimensional candidate frame generation subnetwork is first used to obtain the three-dimensional frame of the building in space, and then the three-dimensional frame regression and mask image generation subnetwork are used to regress and correct the plane frame and vertical height of the building, and predict the contour range of a single building. By taking the maximum value on the DSM image within the building contour range as the building roof elevation, the roof elevation minus the predicted height to obtain the building base elevation, the building contour is stretched from the roof elevation to the base elevation to obtain the building side, and then combined with the building contour surface at the roof elevation and the building contour surface at the base position, a closed lightweight building three-dimensional model is directly extracted.

[0020] like Figure 1 As shown, the present invention proposes a method for extracting a lightweight building three-dimensional model based on a convolutional neural network, comprising the following steps: Step 1: Construct a convolutional neural network for lightweight building 3D model extraction. The network is used to learn the 2D and 3D features of buildings from two types of remote sensing image data: digital orthophoto map (DOM) and digital surface model (DSM), and establish a mapping relationship between the learned features and the target output (building outline, height).

[0021] The convolutional neural network used for lightweight building 3D model extraction includes three parts: feature extraction subnetwork, 3D candidate bounding box generation subnetwork, 3D bounding box regression and mask map generation subnetwork. The feature extraction subnetwork can directly use the currently popular residual structure network (Residual Network, ResNet), visual transformer network (Visual Transformer, ViT), etc. to perform feature encoding on the input digital orthophoto map (Digital Orthophoto map, DOM) and digital surface model (Digital Surface Model, DSM) data, and generate feature maps for use in subsequent parts of the network.

[0022] The 3D candidate bounding box generation subnetwork includes four parallel predictors: bounding box center point predictor, bounding box length predictor, bounding box width predictor and bounding box height predictor. All four predictors take the feature map output by the feature extraction subnetwork as input. The center point predictor consists of a 3×3 convolution layer, a rectified linear unit (ReLU), a 1×1 convolution layer and a sigmoid activation function. The center point predictor is responsible for finding the center point position of the 3D bounding box of the building in the input image data to identify different individual buildings. The bounding box length predictor, bounding box width predictor and bounding box height predictor have the same structure, which consists of a 3×3 convolution layer, a rectified linear unit (ReLU), and a 1×1 convolution layer, which are respectively responsible for predicting the absolute length, width and height values ​​of the building in 3D space. The local peak point on the activation feature map output by the center point predictor is used as the center position of the building, and the length, width and height values ​​of the building's three-dimensional border output by the border length predictor, border width predictor and border height predictor at the corresponding position are taken out synchronously to obtain the three-dimensional border of the building in space. The position of the three-dimensional border on the plane and the feature map output by the feature extraction subnetwork are used as input, and the RoIAlign operation is used to sample at two resolutions of 7×7 and 14×14 respectively to obtain a 7×7 low-resolution Region of Interest (RoI) feature and a 14×14 high-resolution RoI feature, which are used in the subsequent parts of the network.

[0023] The 3D bounding box regression and mask generation subnetwork consists of three branches: bounding box correction branch, height correction branch, and building mask prediction branch. Figure 2 As shown in the figure, the border correction branch takes the flattened low-resolution RoI features as input and is responsible for regressing and correcting the 2D border of the building on the plane dimension, that is, correcting the length and width of the 3D border of the building. The border correction branch includes three fully connected layers, of which the first two fully connected layers are followed by a Rectified Linear Unit (ReLU), and the last fully connected layer directly outputs the regression parameters of the 2D border of the building, including the coordinate correction number in the border. t’ x , t’ y , and border width t’w , altitude correction t’ h The height correction branch and the border correction branch share the first two fully connected layers (including the corrected linear units after the fully connected layer), and finally use a separate fully connected layer to predict the building height regression parameter, that is, the height regression factor t p The building mask prediction branch takes high-resolution RoI features as input and consists of three groups of sequentially connected convolution layers + rectified linear units (ReLU) and a group of convolution layers + Sigmoid activation functions, that is, a convolution layer + rectified linear unit is taken as a group, three groups are set in sequence, and finally a convolution layer + Sigmoid activation function is taken as a group. These four groups are connected in sequence and are responsible for predicting the coverage area of ​​a single building. The prediction results are converted into building outline polygons in vector format using raster-to-vector operations.

[0024] Step 2: Use two types of remote sensing image data, Digital Orthophoto map (DOM) and Digital Surface Model (DSM), and the corresponding building outlines and true building height values ​​to build a sample library, and iteratively train the convolutional neural network model in the Pytorch deep learning framework. By loading the trained model weights to predict the new DOM and DSM, the end-to-end extraction of prismatic lightweight building 3D models can be achieved.

[0025] Step 2.1: Annotate the outlines of all buildings that can be visually interpreted in the digital orthophoto map (DOM) in polygonal form, and combine with the digital surface model (DSM) data to manually measure the height of each building. Record the outline polygon and height value of each building as a line of a txt file to generate an annotation file. Then, the original large-size DOM and DSM data are cropped to the standard size supported by the network model, that is, 512×512 pixel tiles, and the three-channel DOM and the single-channel DSM are connected in series along the channel direction to obtain a four-channel synthetic tile. According to the tile cropping range, the building outline polygons in the annotation file are cropped and the coordinates of the polygons are converted to the coordinates in the tile, and the txt annotation file corresponding to each tile is regenerated.

[0026] Step 2.2, train the convolutional neural network. Use the synthetic tiles generated in step 2.1 and the corresponding annotation files in txt format as training data, and iteratively train the convolutional neural network in the Pytorch deep learning framework until convergence. During the training process, a specific loss function is used to supervise the output of each part of the convolutional neural network. The output of the center point predictor of the 3D candidate bounding box generation subnetwork is supervised by focal loss, and the output of the length predictor, width predictor, and height predictor are all supervised by smooth L1 loss. The output of the bounding box correction branch and the height correction branch in the 3D bounding box regression and mask map generation subnetwork is supervised by smooth L1 loss, and the output of the mask map prediction branch is supervised by binary cross entropy loss.

[0027] Step 2.3, given the new data to be extracted from the building 3D model, the size of the image cropping is the same as that in the process of making training data, the original large-size DOM and DSM image data are cropped and input into the trained convolutional neural network model in series to obtain the prediction results on each tile. After all tiles are predicted, all predicted building outline coordinates and height information are mapped back to the original large-size DOM and DSM images to obtain all building outlines and corresponding heights in the original coordinate system. Finally, the maximum value on the DSM image within the building outline range is taken as the building roof elevation, the roof elevation minus the predicted height to obtain the building base elevation, the building outline is stretched from the roof elevation to the base elevation to obtain the building side, and then together with the building outline surface at the roof elevation and the building outline surface at the base position, a closed prismatic lightweight building 3D model is formed.

[0028] First, a convolutional neural network for extracting lightweight building three-dimensional models is constructed according to the description of the present invention. Then, training sample data is prepared, and the sample data is used to train the constructed convolutional neural network model in the Pytorch deep learning framework. The sample data used in the embodiment is data of a certain area, including two remote sensing image data, digital orthophotomap (DOM) and digital surface model (DSM), and the true value of the LoD-1 level prismatic lightweight building three-dimensional model constructed by the corresponding building outline and the true height value of the building. The data is reasonably divided into a training area and a test area, wherein the test area contains a total of 243 three-dimensional building models. We cut the DOM and DSM data corresponding to the training area into tiles of 512×512 pixels, connect them in series and input them into the convolutional neural network model for iterative training until the model converges to obtain the optimal weight file. After the model training is completed, the DOM and DSM data of the test area to be extracted for 3D buildings are also cut into 512×512 pixel tiles and input into the convolutional neural network in series. After the model prediction, the tile prediction results are mapped back to the original large-size DOM and DSM. Finally, the maximum value within the building outline range on the DSM image is taken as the building roof elevation, and the building base elevation is obtained by subtracting the predicted height from the roof elevation. Then, the building outline is stretched from the roof elevation to the base elevation to obtain the side of the building, which together with the building outline surface at the roof elevation and the building outline surface at the base position constitute a closed prismatic lightweight building 3D model.

[0029] In order to verify the effectiveness and advancement of the proposed method, we compared the proposed method with other advanced deep learning building 3D model extraction methods, including two methods based on pixel-by-pixel building classification and height prediction proposed by Srivastava and Carvalho, for details, please refer to the following references: [1] Srivastava S, Volpi M, Tuia D. Joint height estimation and semantic labeling of monocular aerial images with CNNs; proceedings of the 2017 IEEE International Geoscience and RemoteSensing Symposium (IGARSS), F, 2017 [C]. IEEE, and [2] Carvalho M, Le Saux B, Trouv -Peloux P, et al. Multitask learning of height and semantics from aerial images [J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(8):1391-5; Chen and Ji proposed a method based on Mask R-CNN to simultaneously predict building instance masks and heights, for details, please refer to the following references: [3] Chen S, Mou L, Li Q, et al. Mask-height R-CNN: An end-to-endnetwork for 3D building reconstruction from monocular remote sensing imagery; proceedings of the 2021 IEEE International Geoscience and Remote SensingSymposium IGARSS, F, 2021 [C]. IEEE, and [4] Ji C, Tang H. Number of buildingstories estimation from monocular satellite image using a modified mask R-CNN[J]. Remote Sensing, 2020, 12(22): 3833.All methods are trained on the same hardware environment (a desktop computer equipped with NVIDIA GeForce GTX 3090 Ti GPU, Intel Core i9-12900KF CPU and Windows operating system) using the same training data. The prediction results of all methods are quantitatively evaluated using completeness, correctness and quality, and are recorded in Table 1 below. Quality combines completeness and correctness, and is used as the main indicator to reflect the quality of the prediction results. It can be seen from the values ​​of the three indicators in Table 1 that the effects achieved by the method of the present invention are significantly better than those of several other deep learning methods. In terms of the main indicator (quality), the method of the present invention has a very obvious advantage over other existing methods, which directly proves the advanced nature of the method of the present invention. Compared with these existing methods, the excellent performance of the method of the present invention is proved, and it can reliably and automatically extract lightweight building three-dimensional models from remote sensing image data. In summary, the method of the present invention has good engineering practical value.

[0030] Table 1 Comparison of the effects of the method of the present invention and other methods in extracting lightweight building 3D models

[0031] In addition, some lightweight building three-dimensional model extraction results obtained by the method of the present invention on two kinds of remote sensing image data, namely, real digital orthophoto map (DOM) and digital surface model (DSM). Figure 3 The results of the lightweight building 3D model extracted by the method of the present invention on a certain data set are shown. Figure 4 The three-dimensional model of the lightweight building extracted by the method of the present invention is projected onto the original aerial image to achieve the effect of fitting the model with the real building in the image. Figure 3 and Figure 4 Medium blue represents the sides of buildings, and different colors are used to represent the roofs of different buildings. Figure 4 It can be seen that the lightweight building three-dimensional model extracted by the method of the present invention can closely fit the real building on the original aerial image and reliably reflect the actual situation of the building in the three-dimensional space. This proves the innovation and effectiveness of the present invention and its good engineering practical value.

[0032] On the other hand, an embodiment of the present invention also provides a lightweight building three-dimensional model extraction system using a convolutional neural network, including: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the lightweight building three-dimensional model extraction method of the convolutional neural network as described in the above technical solution.

[0033] The specific implementations described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for extracting lightweight building three-dimensional models using a convolutional neural network, characterized in that: The steps include: Step 1: Construct a convolutional neural network for lightweight building 3D model extraction, learn the 2D and 3D features of buildings from two types of remote sensing image data: digital orthophoto DOM and digital surface model DSM, and establish a mapping relationship between the learned features and the building outline and height; The convolutional neural network includes a feature extraction subnetwork for encoding input features, a 3D candidate bounding box generation subnetwork for generating a 3D bounding box of a building, and a 3D bounding box regression and mask map generation subnetwork for obtaining a building outline polygon, a total of 3 parts; Step 2: Use two types of remote sensing image data, digital orthophoto DOM and digital surface model DSM, and the corresponding building contours and true building height values ​​to build a sample library, iteratively train the convolutional neural network, and predict the new DOM and DSM data by loading the trained model weights to obtain the building contours and height information of the target area, and then directly construct a prismatic lightweight building three-dimensional model based on the building contours and heights.

2. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 1, characterized in that: The feature extraction subnetwork is a residual structure network ResNet or a visual transformer network ViT, which performs feature encoding on the input digital orthophoto image and digital surface model data to generate a feature map.

3. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 1, characterized in that: The 3D candidate bounding box generation subnetwork includes four parallel predictors: bounding box center predictor, bounding box length predictor, bounding box width predictor, and bounding box height predictor. All four predictors take the feature map output by the feature extraction subnetwork as input. The center point predictor consists of a 3×3 convolution layer, a rectified linear unit, a 1×1 convolution layer, and a sigmoid activation function; the center point predictor is responsible for finding the center point position of the 3D bounding box of the building in the input image data to identify different individual buildings; The border length predictor, border width predictor and border height predictor have the same structure, which consists of a 3×3 convolutional layer, a rectified linear unit, and a 1×1 convolutional layer, which are responsible for predicting the absolute length, width and height values ​​of the building in three-dimensional space, respectively.

4. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 3, characterized in that: The local peak point on the activation feature map output by the center point predictor is taken as the center position of the building, and then the length, width and height values ​​of the three-dimensional bounding box of the building output by the border length predictor, border width predictor and border height predictor at the corresponding position are synchronously taken out to obtain the three-dimensional bounding box of the building in space; the position of the three-dimensional bounding box on the plane and the feature map output by the feature extraction subnetwork are taken as input, and the region of interest alignment pooling operation RoIAlign is used to sample at two resolutions of 7×7 and 14×14 respectively to obtain a 7×7 low-resolution RoI feature and a 14×14 high-resolution RoI feature.

5. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 4, characterized in that: The 3D bounding box regression and mask map generation subnetwork consists of three branches: a bounding box correction branch, a height correction branch, and a building mask map prediction branch; the bounding box correction branch takes the flattened low-resolution RoI features as input and is responsible for regressing and correcting the 2D bounding box of the building in the plane dimension, that is, correcting the length and width of the 3D bounding box of the building; the bounding box correction branch consists of three fully connected layers, of which the first two fully connected layers are followed by a corrected linear unit, and the last fully connected layer directly outputs the regression parameters of the 2D bounding box of the building.

6. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 5, characterized in that: The height correction branch shares the first two fully connected layers with the bounding box correction branch, and finally uses a separate fully connected layer to predict the building height regression parameters.

7. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 5, characterized in that: The building mask prediction branch takes high-resolution RoI features as input and consists of three sets of sequentially connected convolutional layers + rectified linear units and one set of convolutional layers + sigmoid activation functions. It is responsible for predicting the coverage area of ​​a single building and converting the prediction results into building outline polygons in vector format using raster-to-vector operations.

8. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 1, characterized in that: The specific implementation of step 2 includes the following sub-steps: Step 2.1, annotate the outlines of all buildings that can be visually interpreted in the digital orthophoto map in the form of polygons, and at the same time manually measure the height value of each building in combination with the digital surface model data, and record the outline polygon and height value of each building as a line of a txt file to generate an annotation file; Then, the original large-size DOM and DSM data are cropped to the standard supported by the network model, tiles of a certain size are generated, and the three-channel DOM and the single-channel DSM are connected in series along the channel direction to obtain four-channel synthetic tiles; the building outline polygons in the annotation file are cropped according to the tile cropping range and the coordinates of the polygons are converted into coordinates in the synthetic tile, and the txt annotation file corresponding to each tile is regenerated; Step 2.2, training the convolutional neural network: using the synthetic tiles and the annotation files in txt format generated in step 2.1 as training data, iteratively train the convolutional neural network in the deep learning framework until convergence; Step 2.3, given new data to be used for building 3D model extraction, the size of the original large-size DOM and DSM image data is the same as the size of the image cropping performed in the process of making training data, and the original large-size DOM and DSM image data are cropped and input in series into the trained convolutional neural network to obtain the prediction results on each tile; after all tiles are predicted, all predicted building outline coordinates and height information are mapped back to the original large-size DOM and DSM images to obtain all building outlines and corresponding heights in the original coordinate system; finally, the maximum value on the DSM image within the building outline range is taken as the building roof elevation, the roof elevation minus the predicted height to obtain the building base elevation, the building outline is stretched from the roof elevation to the base elevation to obtain the building side, and then together with the building outline surface at the roof elevation and the building outline surface at the base position, a closed prismatic lightweight building 3D model is formed.

9. The method for extracting lightweight building 3D models using a convolutional neural network according to claim 5, characterized in that: During the training process, a loss function is used to supervise the output of each part of the convolutional neural network used for lightweight building 3D model extraction: the output of the center point predictor of the 3D candidate bounding box generation subnetwork is supervised by focal loss, and the outputs of the bounding box length predictor, bounding box width predictor, and bounding box height predictor are all supervised by smooth L1 loss; the outputs of the bounding box correction branch and the height correction branch in the 3D bounding box regression and mask map generation subnetwork are supervised by smooth L1 loss, and the output of the mask map prediction branch is supervised by binary cross entropy loss.

10. A convolutional neural network lightweight building three-dimensional model extraction system, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the lightweight building three-dimensional model extraction method using a convolutional neural network as described in any one of claims 1 to 9.

Images