Building extraction and roof identification method and device based on RGB satellite picture

Abstract

The invention relates to a building extraction and roof identification method and device based on RGB satellite pictures, and the method comprises the steps: carrying out the preprocessing of the RGB satellite pictures, obtaining a picture data set, marking the picture data set, and dividing the marked picture data set into a training set, a test set and a verification set; an improved MASK-RCNN model is built; a training set and a verification set are adopted to train and verify the improved MASK-RCNN model, and an optimized improved MASK-RCNN model is obtained; and adopting the optimized improved MASK-RCNN model to predict the test set, and extracting the category of the building and the corresponding roof classification. According to the method, high-precision building identification and roof classification can be realized, and different types of buildings can be accurately distinguished in a complex satellite image; satellite images can be automatically processed, manual intervention is reduced, and processing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of building recognition and processing, and in particular relates to a method and a device for building extraction and roof recognition based on RGB satellite images. Background Art

[0002] In the construction of modern smart cities, the extraction of buildings and the accurate identification and classification of roofs are important foundations for urban management and planning. Traditional building roof survey methods often rely on manual on-site surveys, which use manual marking of building outlines and record of roof types. The principle of manual marking of building outlines and recording of roof types is to accurately depict the outline of the building by manual means and record the type of roof in detail. This technology includes multiple steps: first, the labeler needs to collect relevant image data of the building and perform preprocessing to improve the accuracy of the labeling; second, use professional labeling tools to carefully depict the outline of the building on the image data; finally, identify the type of roof based on the design drawings or on-site photos, and record the relevant information in a table or database. Through this technology, the precise shape and location information of the building outline, as well as the detailed classification of roof types, can be obtained.

[0003] The currently widely used deep learning methods include semantic segmentation, target inspection, and point cloud image recognition and analysis. The common building extraction and roof recognition and classification methods in the existing market are carried out separately: for building extraction, semantic segmentation technology is used, which is a pixel-level image classification technology that assigns a category label to each pixel in the image, thereby achieving fine segmentation of the image. When extracting buildings from satellite images, semantic segmentation technology enables the model to learn the feature representation of buildings by training the deep learning model, and divides the input satellite image into different areas during the test phase, where the building area is assigned a specific category label, thereby achieving building extraction. For roof recognition and classification, the current common technology is target detection, which can effectively locate the roof position in the image. However, target detection can usually only provide the approximate location and category information of the roof, and cannot achieve fine pixel-level classification. The second is to identify and classify the roof through the three-dimensional point cloud obtained by the laser radar. The laser radar calculates the three-dimensional coordinates of the target surface by emitting a laser beam and receiving its reflected signal, thereby generating three-dimensional point cloud data. These point cloud data contain rich spatial information and geometric features, providing a data basis for roof recognition and classification. By using the coordinates of the points in the point cloud data, the reflection intensity and other information, combined with algorithm analysis, accurate identification and classification of roof types can be achieved. The current semantic segmentation and target detection technologies each have significant limitations. Semantic segmentation technology can only extract the mask information of the building (mask information: the outline of the building), which is presented in the form of a binary grayscale image and lacks sufficient details to reveal the specific characteristics and information of each building roof. In other words, it cannot provide detailed data on the shape of the building roof. In contrast, although target detection technology can extract and display the roof information of each building, this technology is also accompanied by an obvious disadvantage: it can usually only generate the bounding box of the building. This bounding box is a simplified rectangular box used to roughly identify the location of the building, but there is a large gap compared to the actual outline information of the building. Therefore, target detection technology is unable to accurately depict the shape and outline of the building. In point cloud image recognition and analysis, three-dimensional point clouds obtained by lidar are used to identify and classify roofs. This technology can provide detailed location and category information of roofs, but this method requires flying equipment such as drones to scan and shoot, which greatly increases the manpower, material resources and time costs required. Another reason is that the no-fly zone makes this technology unable to meet actual needs.

[0004] In summary, whether it is semantic segmentation technology, target detection technology or point cloud recognition and analysis technology, they all have obvious shortcomings in current applications. Semantic segmentation technology cannot provide enough roof detail information, while target detection technology is insufficient in accurately depicting the outline of buildings. Point cloud recognition and analysis technology is time-consuming, labor-intensive and costly, and is not suitable for extracting buildings over a large area and identifying roof types. These limitations limit the effectiveness, timeliness and economy of these technologies in application scenarios that require accurate building information and outline details. Summary of the invention

[0005] In order to overcome the problems existing in the prior art, the present invention provides a method and device for building extraction and roof recognition based on RGB satellite images, which are used to overcome the current defects.

[0006] A method for building extraction and roof recognition based on RGB satellite images, the method comprising the steps of: S1. Preprocessing the RGB satellite images to obtain an image dataset, and then annotating the image dataset and dividing it into a training set, a test set, and a validation set; S2. Build an improved MASK-RCNN model; S3. Using the training set to train the improved MASK-RCNN model to obtain a trained improved MASK-RCNN model; S4. Using the validation set to validate the trained improved MASK-RCNN model to obtain an optimized improved MASK-RCNN model; S5. Use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

[0007] According to the aspects and any possible implementation methods described above, an implementation method is further provided, wherein the improved MASK-RCNN model includes a core Backbone part, a region proposal network RPN part, and a candidate region head ROI Head part, which are connected in sequence.

[0008] According to the above aspects and any possible implementation, an implementation is further provided, wherein the core Backbone part comprises the following steps: S211. The input picture is segmented into overlapping pacth pictures by using an encoder; S212. The overlapping pacth images are propagated using the cmViT model to obtain an intermediate feature map; S213. The feature pyramid network FPN is used to fuse the intermediate feature maps to obtain feature images of different levels.

[0009] According to the aspects and any possible implementation methods described above, an implementation method is further provided, wherein the cmViT model includes sixteen cViT modules connected in series, wherein the first to fourth cViT modules constitute a first submodule, the fifth to eighth cViT modules constitute a second submodule, the ninth to fourteenth cViT modules constitute a third submodule, and the fifteenth to sixteenth cViT modules constitute a fourth submodule; a first patch fusion module connected to the first submodule is arranged between the second submodule and the third submodule, a second patch fusion module connected to the second submodule is arranged between the second submodule and the third submodule; a third patch fusion module connected to the third submodule is arranged between the third submodule and the fourth submodule, the first submodule is also connected to a 4-fold upsampling module, the second submodule is also connected to a 2-fold upsampling module, and the fourth submodule is connected to a 2-fold downsampling module.

[0010] According to the above aspects and any possible implementation, an implementation is further provided, wherein the intermediate feature maps include at least four images; S212 specifically includes: S212 outputs a first intermediate feature image after the overlapping patch images are propagated by the first submodule and processed by the upsampling 4x module; the image output by the first submodule is fused by the first patch fusion module, propagated by the second submodule, and sampled by the upsampling 2x module to output a second intermediate feature image; The image output by the second submodule is fused by the second patch fusion module, propagated by the third submodule, and then outputs the third intermediate feature image; The image output by the third submodule is fused by the third patch fusion module, propagated by the fourth submodule and sampled by the downsampling 2x module to output the fourth intermediate feature image; wherein the sizes of the first to fourth intermediate feature images are all different.

[0011] According to the above aspects and any possible implementation, an implementation is further provided, wherein S213 specifically includes: a feature pyramid network FPN is processed in a top-down and lateral connection manner, specifically including: performing a 1x1 convolution on the fourth intermediate feature image C4 to generate a fifth image P5; The fifth image P5 is upsampled by 2 times to generate a feature map with the same resolution as the third intermediate feature image C3; the upsampled fifth image P5 is added to the feature map after 1x1 convolution of the third intermediate feature image C3, that is, horizontally connected, to generate a fourth image P4; The fourth image P4 is upsampled by 2 times to generate a feature map with the same resolution as the second intermediate feature image C2, and the upsampled fourth image P4 is added to the feature map after 1x1 convolution of the second intermediate feature image C2, that is, horizontally connected, to generate a third image P3; The third image P3 is upsampled by 2 times to generate a feature map with the same resolution as the first intermediate feature image C1, and the upsampled third image P3 is added to the feature map of the first intermediate feature image C1 after 1x1 convolution, that is, horizontally connected, to generate the second image P2; The fifth image P5 is directly downsampled by 2 times to generate the sixth image P6.

[0012] As described above, an implementation method is further provided, and the operation of the region proposal network RPN part specifically includes: the RPN performs feature extraction and conversion, classification and regression processing on the feature images of different levels, and screens the results obtained by the classification and regression processing to obtain candidate regions.

[0013] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the operation of the candidate region head ROI Head part specifically includes: S221. Perform normalized weighted averaging on the feature images P2-P6 at different levels to obtain feature images at different levels; S222. Determine the specific position of the candidate region in the feature images of different levels, obtain local feature maps of the candidate region of different sizes, resize and average pool the maps, and obtain the final features of the candidate region; S223. The final features are processed by a fully connected layer to obtain the category probability, bounding box coordinates and mask of each feature.

[0014] According to the aspects described above and any possible implementation, an implementation is further provided, wherein each cViT module includes a multi-head attention operator, a normalization layer, and a convolutional multilayer perceptron connected in sequence.

[0015] The present invention also provides a device for building extraction and roof recognition based on RGB satellite images, and the device is used to implement the method, including the following modules: A processing module is used to pre-process the RGB satellite images to obtain an image dataset, and divide the image dataset into a training set, a test set, and a validation set after annotating the image dataset; Building modules for building improved MASK-RCNN models; A training module, used to train the improved MASK-RCNN model using the training set to obtain a trained improved MASK-RCNN model; A verification module, used to verify the trained improved MASK-RCNN model using the verification set to obtain an optimized improved MASK-RCNN model; The prediction module is used to use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

[0016] Beneficial Effects of the Invention The method of the present invention comprises: preprocessing RGB satellite images to obtain an image data set, and marking the image data set and dividing it into a training set, a test set and a verification set; An improved MASK-RCNN model is constructed; the improved MASK-RCNN model is trained using the training set to obtain a trained improved MASK-RCNN model; the trained improved MASK-RCNN model is verified using the verification set to obtain an optimized improved MASK-RCNN model; the test set is predicted using the optimized improved MASK-RCNN model to extract the building category and the corresponding roof category.

[0017] The beneficial effects of the present invention include: the present invention applies the CMVIT module and the improved NLRoI Align module to the deep learning model of instance segmentation, which can simultaneously realize the identification of building extraction and roof recognition and classification. The present invention can achieve (1) high-precision recognition and classification: Mask R-CNN combines the advantages of target detection (identifying the location of buildings) and semantic segmentation (pixel-level classification of building roofs), and can achieve high-precision building recognition and roof classification. This helps to accurately distinguish different types of buildings in complex satellite images. (2) Improve the level of automation: Automatically process satellite images, reduce manual intervention, and improve processing efficiency and accuracy. In the fields of urban planning, disaster assessment, etc., this technology can quickly provide detailed information on buildings and their roofs to support decision-making.

[0018] (3) Strong adaptability: The improved Mask R-CNN model has strong generalization ability and can achieve good recognition results on satellite images of different regions. (4) Multi-task processing: The improved Mask R-CNN can simultaneously complete the two tasks of target detection and semantic segmentation in the same model, reducing the complexity of model training and computing resource consumption. (5) Significant cost reduction: Compared with point cloud data, satellite images have lower acquisition and processing costs, and have lower requirements for hardware and algorithms. The Mask R-CNN model can achieve high-precision building recognition and roof classification on two-dimensional images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention; Figure 2 A schematic diagram for marking the outline of a building in the present invention; Figure 3 This is a schematic diagram of the roof type of the present invention; Figure 4 Schematic diagram of the structure of the improved Mask R-CNN model of the present invention; Figure 5 This is a schematic diagram of the patch encoding module structure; Figure 6 This is a schematic diagram of the patch fusion module structure; Figure 7 This is a schematic diagram of the cViT module structure of the present invention; Figure 8 It is a schematic diagram of the anchor point frame generation process of the present invention; Fig. 9 This is a schematic diagram of the NLRoI Align model structure; Fig.10 This is a schematic diagram of the roof effect before identification; Fig.11 Schematic diagram of the identified roof effect. DETAILED DESCRIPTION

[0020] In order to better understand the technical solution of the present invention, the content of the present invention includes but is not limited to the specific implementation methods described below, and similar technologies and methods should be considered to be within the scope of protection of the present invention. In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0021] It should be clear that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the present invention provides a building extraction and roof recognition method based on RGB satellite images, the method comprising the steps of: S1. Preprocessing the RGB satellite images to obtain an image dataset, and then annotating the image dataset and dividing it into a training set, a test set, and a validation set; S2. Build an improved MASK-RCNN model; S3. Using the training set to train the improved MASK-RCNN model to obtain a trained improved MASK-RCNN model; S4. Using the validation set to validate the trained improved MASK-RCNN model to obtain an optimized improved MASK-RCNN model; S5. Use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

[0023] Furthermore, the improved MASK-RCNN model includes a core Backbone part, a region proposal network RPN part and a candidate region head ROI Head part which are connected in sequence.

[0024] Furthermore, the core Backbone part includes the following steps: S211. Using an encoder to segment the input image into overlapping pacth images; S212. The overlapping pacth images are propagated using the cmViT model to obtain an intermediate feature map; S213. The feature pyramid network FPN is used to fuse the intermediate feature maps to obtain feature images of different levels.

[0025] Furthermore, the cmViT model includes sixteen cViT modules connected in series, wherein the first to fourth cViT modules constitute a first submodule, the fifth to eighth cViT modules constitute a second submodule, the ninth to fourteenth cViT modules constitute a third submodule, and the fifteenth to sixteenth cViT modules constitute a fourth submodule; a first patch fusion module connected is arranged between the first submodule and the second submodule, a second patch fusion module connected is arranged between the second submodule and the third submodule; a third patch fusion module connected is arranged between the third submodule and the fourth submodule, the first submodule is also connected to a 4-fold upsampling module, the second submodule is also connected to a 2-fold upsampling module, and the fourth submodule is connected to a 2-fold downsampling module.

[0026] Furthermore, the intermediate feature map includes at least four images; S212 specifically includes: S212 outputs the first intermediate feature image C1 after the overlapping patch images are propagated by the first submodule and processed by the upsampling 4 times module; the image output by the first submodule is fused by the first patch fusion module, propagated by the second submodule, and sampled by the upsampling 2 times module to output the second intermediate feature image C2; The image output by the second submodule is fused by the second patch fusion module and then propagated by the third submodule to output the third intermediate feature image C3; The image output by the third submodule is fused by the third patch fusion module, propagated by the fourth submodule and sampled by the downsampling 2x module to output the fourth intermediate feature image C4; wherein the sizes of the first to fourth intermediate feature images are all different.

[0027] Further, the S213 specifically includes: the feature pyramid network FPN adopts a top-down and lateral connection method for processing, specifically including: performing a 1x1 convolution on the fourth intermediate feature image C4 to generate an initial fifth image P5; The fifth image P5 is upsampled by 2 times to generate a feature map with the same resolution as the third intermediate feature image C3; the upsampled fifth image P5 is added to the feature map after 1x1 convolution of the third intermediate feature image C3, that is, horizontally connected, to generate a fourth image P4; The fourth image P4 is upsampled by 2 times to generate a feature map with the same resolution as the second intermediate feature image C2, and the upsampled fourth image P4 is added to the feature map after 1x1 convolution of the second intermediate feature image C2, that is, horizontally connected, to generate a third image P3; The third image P3 is upsampled by 2 times to generate a feature map with the same resolution as the first intermediate feature image C1, and the upsampled third image P3 is added to the feature map of the first intermediate feature image C1 after 1x1 convolution, that is, horizontally connected, to generate the second image P2; The fifth image P5 is directly downsampled by 2 times to generate the sixth image P6.

[0028] Furthermore, the operation of the RPN part of the region proposal network specifically includes: the RPN performs feature extraction and conversion, classification and regression processing on the feature images of different levels, and screens the results obtained by the classification and regression processing to obtain candidate regions.

[0029] Furthermore, the operations of the candidate region head ROI Head part specifically include: S221. Perform normalized weighted averaging on the feature images P2-P6 at different levels to obtain feature images at different levels; S222. Determine the specific position of the candidate region in the feature images of different levels, obtain local feature maps of the candidate region of different sizes, resize and average pool the maps, and obtain the final features of the candidate region; S223. The final features are processed by a fully connected layer to obtain the category probability, bounding box coordinates and mask of each feature.

[0030] Furthermore, each cViT module includes a multi-head attention operator, a normalization layer, and a convolutional multilayer perceptron connected in sequence.

[0031] The purpose of this invention is to solve the problems of slow extraction speed, high cost, no roof information and inaccurate contours in the prior art in the future, and propose a building extraction and roof recognition and classification method based on RGB satellite images. This method draws on the idea of ​​the visual Transformer module (mViT) and the idea of ​​non-local building blocks to improve the deep learning model MASK-RCNN. The model combines the advantages of semantic segmentation and target detection, overcomes the defects of high cost and slow speed of point cloud recognition and classification technology, realizes the fast and accurate extraction of buildings in RGB satellite images, and can accurately identify and classify roof information, significantly improving the extraction speed and contour accuracy.

[0032] Specifically, the process of the present invention is as follows: Step 1. Obtain original satellite images and data preprocessing The satellite images used in the present invention are acquired through Sentinel satellites, and remote sensing images of a 30km×30km area with a resolution of 50cm around a city or region are downloaded. The downloaded original high-resolution remote sensing images are subjected to data preprocessing (atmospheric correction for remote sensing effects, band synthesis, truncation and stretching, data slicing, etc.), and finally 9409 pictures in TIFF format are obtained to form sliced ​​image data.

[0033] Step 2. Dataset creation. This dataset is used for deep learning model training. The open source software labelme is used to label satellite image samples. The labels include building outlines and roof types. The results are as follows: Figure 2 As shown in the figure, since the best height and width of the deep learning model for processing images are between 400 pixels and 600 pixels, the 1024×1024 pixel image is cut into four 512×512 pixel images, and finally 37,636 satellite images are obtained. The satellite image data is divided into training set, validation set and test set in a ratio of 2:1:7, and a json file that conforms to the coco dataset format is generated using a python script. The building roof outline is marked as follows Figure 2 As shown in the figure, the roofs of the buildings are marked with different colors. In order to indicate that the 1024×1024 pixel image is cut into four 512×512 pixel images, the four colors represent the four 512×512 pixel images. The colors are only used to distinguish the images of different blocks and identify the type (part) of the roof. Figure 3 As shown, the roof includes a flat roof, a double-slope roof with the roof inclined on both sides, a mansard roof composed of multiple sloped surfaces, a concave mansard roof, etc., and the present invention is not limited thereto.

[0034] Step 3. Build an improved Mask R-CNN model The overall structure of the improved Mask R-CNN can be divided into the following main parts: Backbone part cmViT+FPN, region proposal network RPN part and region of interest head ROI Head, head ROI Head includes non-local region of interest alignment NLRoI Align, classification head, regression head and mask head. The overall structure of the improved Mask R-CNN model is as follows Figure 4 As shown, Figure 4 The input in is Fig.10 The pictures shown, Fig.10 is a picture in the test set of the dataset.

[0035] Step 3.1. The Backbone part is composed of cmViT+FPN, in which the cmViT technology connects the multi-scale feature hierarchy of satellite images with the principle of attention mechanism, and then uses a multi-scale feature pyramid structure to gradually expand the channel capacity and reduce the spatial resolution, thereby extracting richer multi-scale features and extracting visual information at different scales. Since mViT in the prior art directly divides the input image into non-overlapping image patches, this method has limitations in modeling the internal structural information of the image. In order to solve this problem, the present invention uses a convolutional layer instead of a direct segmentation method to divide the input image into overlapping small image patches; in addition, the ViT small module in the mViT module is improved to a new cmViT with a convolutional multi-layer perceptron, which is described below using cmViT and cViT.

[0036] The specific method of cmViT to process satellite images is to first divide the input image into overlapping small image patches through the patch encoding module, such as Figure 5 As shown in Figure 1, the specific approach of the patch encoding module is to use two standard 3×3 convolutional layers to expand the R, G, and B channel dimensions of the input image to 96 and reduce the resolution to 1 / 4. In the first standard 3×3 convolutional layer, batch normalization is performed after the convolution operation and ReLU function is used for activation. Next, the next standard 3×3 depth convolution is performed and residual connections are used to enhance the features of the patch. Figure 5 The process of obtaining X0, wherein the convolution, normalization operation, relu activation, deep convolution residual connection, etc. are all existing technologies and will not be repeated here.

[0037] In order to make the feature map output by the cmViT backbone model contain multi-scale feature levels, the feature vector x0 obtained in the previous step is propagated to the high-dimensional cmViT model, which is mainly used to extract the global context information and multi-scale features of the image. Its core functions include: Global feature modeling: Capturing long-range dependencies in the image through the self-attention mechanism Self-Attention. Multi-scale feature extraction: Gradually extracting multi-scale features from local to global through hierarchical design. High-dimensional feature mapping: Mapping the low-dimensional feature vector x0 to a high-dimensional space to enhance the expressiveness of the feature. The cmViT model includes sixteen cViT modules connected in series, of which the first to fourth cViT modules constitute the first submodule, which is used to extract low-level features of the image (such as edges, textures, etc.), which are the basis for subsequent high-level feature extraction. The extraction of low-level features requires more levels to be gradually refined. Four cViT modules as the first submodule can fully capture the local details of the image without excessively increasing the computational complexity. For most image tasks (such as classification and detection), four cViT modules connected in series are an empirical balance point. The fifth to eighth cViT modules constitute the second submodule, which is used to extract intermediate features of the image, such as local shape, partial structure, etc. The extraction of intermediate features requires a certain level to integrate local information. The four cViT modules can gradually enhance the semantic information of the features while maintaining computational efficiency. The second submodule is symmetrical with the first submodule, which is also convenient for the structural design of the model. The ninth to fourteenth cViT modules constitute the third submodule, which is used to extract high-level features of the image, such as global structure, semantic information, etc. The extraction of high-level features requires more levels to model global context information. The six cViT modules can more fully capture the long-distance dependencies of the image. For complex image tasks (such as segmentation and generation), more serial modules can help improve the performance of the model. The fifteenth to sixteenth cViT modules constitute the fourth submodule, which is used for feature refinement and output, integrating multi-scale features into the final feature map. The feature refinement stage does not require too many levels. Two cViT modules are sufficient to complete feature fusion and optimization. Reducing the number of modules can reduce the amount of calculation and avoid overfitting. A first patch fusion module connected to each other is arranged between the first submodule and the second submodule, a second patch fusion module connected to each other is arranged between the second submodule and the third submodule; a third patch fusion module connected to each other is arranged between the third submodule and the fourth submodule, the first submodule is also connected to a 4-fold upsampling module, the second submodule is also connected to a 2-fold upsampling module, the third submodule does not need sampling and directly outputs, and the fourth submodule is connected to a 2-fold downsampling module.

[0038] The role of the cmViT model is to gradually extract multi-scale features from local to global. The size of buildings in satellite images varies significantly, and multi-scale feature extraction can capture the features of small buildings and small buildings at the same time. The upsampling 4x module, the upsampling 2x module, and the downsampling 2x module upsample or downsample the output of the intermediate cViT module feature map to obtain the feature map required by FPN, namely the first intermediate feature image C1, the second intermediate feature image C2, the third intermediate feature image C3, and the fourth intermediate feature image C4. The cmViT model inserts a sampling module and a patch fusion module every time a fixed number of cViT modules are passed. This model is set to perform sampling and fusion after 4, 4, 6, and 2 cViT modules, and the last 2 cViT modules do not require fusion operations, such as Figure 4 As shown in Figure 1, because it is the last part of the cViT module, its output is not connected to other cViT modules. The sampling module is to make the feature vector output size of the network meet the requirements of the subsequent FPN network. , , and The size of the patch fusion module is to hierarchize the feature vector. A total of three patch fusion modules are used. The structure of each module is the same. The module structure is as follows: Figure 6 As shown in the figure, the feature vector, i.e., the output of the cViT module, is first normalized, then downsampled to 1 / 2 using a 2×2 convolution layer, and its channel dimension is expanded to 2 times. Finally, the patch encoding module structure is borrowed to enhance the features of the patch through a standard 3×3 deep convolution and residual connection. In deep neural networks, residual connections can enhance the features of images, and deep convolution and residual are commonly used operations. The summation is to add the values ​​corresponding to the two feature images, that is, to add the corresponding elements of two matrices of the same size.

[0039] Each cViT module consists of a multi-head attention operator MHPA, a convolutional multi-layer perceptron cMLP, and a normalization layer LN. The output X n The calculation formula is as follows: X=MHPA(X n-1 )+Pool(X n-1 )(1) X n =cMLP(LN(X1))+X(2) Where Pool is the pooling operation, X is the output of the multi-head attention operator, n is [1,16]; when n is 1, x0 is the feature vector x obtained previously. 0。

[0040] like Figure 7As shown, each cViT module converts the feature vector X obtained in the previous step n-1 After normalization, the input sequence of the multi-head attention operator MHPA is obtained, and the feature vector X n-1 After inputting the multi-head attention operator MHPA, it first undergoes three linear operations (here the three weight matrices are respectively n-1 Multiply) to get the query vector Q, key vector K and value vector V respectively; then use the dot product method to calculate the attention score between the pooled query vector Q and its corresponding pooled key vector K, and the score is: Score=QK T (3), Then the attention score Score is normalized to obtain the attention weight coefficient in the multi-head attention operator MHPA. The attention weight coefficient is multiplied by its corresponding value vector V and then multiplied by the directly pooled feature vector X n-1 The output X of the multi-head attention operator in equation (3) can be obtained by weighted summation. The calculation formula is as follows: (4) Among them, d k is the dimension of the key vector K. In order to avoid the Score value being too large, a scaling factor equal to the square root of the K dimension is used.

[0041] The output of the multi-head attention operator is processed by normalization LN and then input into the multi-layer perceptron MLP. The weighted sum of the output of MLP and the output of the multi-head attention operator is the input X of the nth cViT module. n .

[0042] Here, the convolutional multilayer perceptron cMLP is composed of a convolutional neural network, where Figure 7 As shown in the figure, from top to bottom, there are three modules: normalized convolution, ReLU activated depth convolution, and convolution ReLU activated normalized convolution.

[0043] The hierarchical structure of cmViT is divided into 4 stages to generate multi-scale feature maps and integrate them into the feature pyramid network FPN for target detection tasks, such as Figure 4 As shown, in these four stages, the output of each cViT submodule needs to be upsampled by 4 times, upsampled by 2 times, not sampled, and downsampled by 2 times, and finally Figure 4 The first to fourth intermediate feature images or pictures C1, C2, C3 and C4 in . These intermediate feature images contain information of different scales. The size of the first intermediate feature image C1 is , the size of the second intermediate feature image C2 is , the size of the third intermediate feature image C3 is , the size of the fourth intermediate feature image C4 is , the first intermediate feature image C1 is of high resolution and suitable for small target detection. The second intermediate feature image C2 is of medium resolution and suitable for medium-sized target detection. The third and fourth intermediate feature images C3 and C4 are of low resolution and suitable for large target detection. Where H and W represent the height and width pixels of the satellite image, which are 512 and 512. The feature pyramid network FPN processes C1-C4 to obtain images P2-P6 by top-down and lateral connection, wherein the top-down and lateral connection methods specifically include: firstly, performing 1x1 convolution on the fourth intermediate feature image C4 to generate the fifth image P5; then upsampling the fifth image P5 by 2 times to generate a feature map with the same resolution as the third intermediate feature image C3. The upsampled fifth image P5, i.e., the feature map with the same resolution as the third intermediate feature image C3, is added to the feature map of the third intermediate feature image C3 after 1x1 convolution, i.e., horizontal connection, to generate the fourth image P4. The fourth image P4 is upsampled by 2 times to generate a feature map with the same resolution as the second intermediate feature image C2. The upsampled fourth image P4, i.e., the feature map with the same resolution as the second intermediate feature image C2, is added to the feature map of the second intermediate feature image C2 after 1x1 convolution, i.e., horizontally connected, to generate the third image P3. The third image P3 is upsampled by 2 times to generate a feature map with the same resolution as the first intermediate feature image C1. The upsampled third image P3 is added to the feature map of the first intermediate feature image C1 after 1x1 convolution, i.e., horizontally connected, to generate the second image P2. The fifth image P5 is directly downsampled by 2 times to generate the sixth image P6. From top to bottom, the fifth image P5 is realized to the second image P2, which is obtained by upsampling the fifth image P5. The sixth image P6 is obtained by downsampling the fifth image P5. The above process obtains high-level feature images P2, P3, P4, P5 and P6 at different levels.

[0044] Step 3.2. The Rpn network generates candidate regions by processing the high-level feature images P2, P3, P4, P5 and P6 at different levels output from the Backbone network. These feature images have different scales and semantic information and are suitable for detecting buildings of different sizes. This process can be further refined into the following steps: Generate anchor boxes: The Rpn network generates 9 anchor boxes through sliding windows on the five high-level feature images P2-P6. Anchor Boxes are predefined rectangular boxes with different scales and aspect ratios, covering areas of different positions and sizes on the high-level feature map. Specifically, 9 Anchor Boxes are extracted for each pixel in each high-level feature image P2-P6, and each pixel includes 18 scores and 36 coordinates after this transformation. The anchor box selection part corresponding to each pixel in the high-level feature images P2, P3, P4, P5 and P6 is formed by extracting features and converting them using convolution kernels to obtain a 256-dimensional vector, and then two 1x1 convolutions are applied to the 256-dimensional feature vectors to obtain scores and coordinates. For example, a 3×3 convolution kernel is used to extract and transform the second-level feature map P2, and 9 anchor boxes are generated for each pixel on each feature map. These anchor boxes have different scales and aspect ratios to cover buildings of different sizes and shapes. The sizes and shapes of these anchor boxes are determined according to the preset scales and aspect ratios, such as Figure 8 As shown, classification and regression: The RPN network contains two main branches: classification branch and regression branch. The classification branch classifies each Anchor Box through a 1×1 convolution layer to determine whether it contains the target (foreground) or does not contain the target (background), and finally obtains 18 scores (each pixel has nine anchor boxes, each anchor box has a foreground score and a background score) and 4 coordinates (the classification branch only has scores but no coordinates). The regression branch also regresses each Anchor Box through a 1×1 convolution layer and outputs its coordinate offset, and finally obtains 36 coordinates (each anchor box uses two coordinates and two offsets) to adjust the position and size of the Anchor Box to more accurately match the target, as shown in Figure 8 As shown. After the above process, the scores and coordinates of all anchor boxes corresponding to all pixel points in the high-level feature maps P2 to P6 are obtained. The classification branch obtains the scores and the regression branch obtains the coordinates.

[0045] Candidate box proposal generation: According to the classification and regression results of multiple anchor boxes generated by generating anchor boxes, the RPN network screens out high-quality candidate region proposals. Specifically, the candidate region proposals are screened here to obtain feature vectors with a foreground score greater than 0.7. Then, the non-maximum suppression method NMS is used to remove overlaps of the screened feature vectors again to remove candidate boxes with high overlap, that is, to remove redundant bounding boxes and retain the most representative candidate region proposals. For example, if the first anchor box contains half of the building, and the second anchor box contains the entire building (the same building), the second anchor box is selected because it includes the entire building. Finally, the RPN network outputs a series of candidate regions, which will be used for target classification, positioning, and mask extraction tasks of the subsequent NLRoI Align network.

[0046] Step 3.3.ROI Head part, including NLRoI Align, classification head (roof category), regression head (bounding box coordinates) and mask head (mask).

[0047] NLRoI Align layer: Accurately extract the features of a series of candidate regions, namely the RoI region, from the feature map, perform finer divisions inside the RoI region through bilinear interpolation, and perform pooling operations on each divided sub-region to obtain a fixed-size feature map. Since the RoI extractor of the MASK-RCNN model has the limitation that it can only select one (optimal) layer from FPN, but the non-optimal layer (the adjacent layer of the optimal layer) also has useful information, a non-local building block is used to improve the RoI extractor of the MASK-RCNN model, which is described here as NLRoI Align.

[0048] Classification head (roof category): The classification head is responsible for classifying the objects in the RoI area and outputting the probability of each category. Regression head (bounding box coordinates): The bounding box regression head is responsible for fine-tuning the bounding box of the RoI area and outputting more accurate bounding box coordinates. Mask header (mask): The mask header is a unique part of Mask R-CNN and is responsible for generating the corresponding segmentation mask for each RoI area.

[0049] The specific steps are as follows: NLRoI Align receives the feature images Feature Maps output in 3.1 and the candidate region proposals output in 3.2 for processing. First, NLRoI Align processes the Feature Maps output in 3.1. Maps, the six high-level feature maps P2-P6 of different sizes are normalized and weighted averaged five times, specifically: the first time is to downsample the third high-level feature map P3 by 2 times, add it to the second high-level feature map P2, and divide it by 2 to obtain the weighted feature N1; the second time is to downsample the fourth high-level feature map P4 by 2 times, upsample the second high-level feature map P2 by 2 times, add it to the third high-level feature map P3, and divide it by 3 to obtain the weighted feature N2; the third time is to downsample the fifth high-level feature map P5 by 2 times, upsample the third high-level feature map P3 by 2 times, add it to the fourth high-level feature map P4, and divide it by 3 to obtain the weighted feature N3; the fourth time is to downsample the sixth high-level feature map P6 by 2 times, upsample the fourth high-level feature map P4 by 2 times, add it to the fifth high-level feature map P5, and divide it by 3 to obtain the weighted feature N4; the second time is to upsample the fifth high-level feature map P5 by 2 times, add it to the sixth high-level feature map P6, and divide it by 2 to obtain the weighted feature N5. Since the RoI extractor of the existing MASK-RCNN model has the limitation that it can only select one (optimal) layer from FPN, but the non-optimal layers (the adjacent layers of the optimal layer) also contain useful information, the present invention performs five weighted calculations on the high-level feature maps P2-P6 to obtain weighted feature images N1-N5, which helps to improve the building extraction effect.

[0050] NLRoI Align then analyzes the series of candidate regions generated in the RPN network and combines the foreground and background scores with the anchor frame coordinates to accurately locate the specific positions of the series of candidate regions in the weighted feature images N1, N2, N3, N4, and N5 at different levels output by the Backbone network. The specific number of layers can be obtained through the following formula (5). The output of NLRoI Align is the local feature map of the candidate region. The specific positioning formula is: (5) Where 512 represents the size of the input image, , variables w and h are the width and height in the coordinate offset. If the width or height is less than 512, the corresponding area is extracted from the feature map below N5, such as N4, N3, N2 or N1. This is because large-scale candidate regions are extracted from low-resolution feature maps, which is conducive to detecting large targets, and small-scale candidate regions are cut from high-resolution feature maps, which is conducive to detecting small targets.

[0051] Finally, since the NLRoI Align part outputs a series of candidate regions, namely the local feature maps of the RoI region, they need to be input into the fully connected network for the next step. The sizes of these local feature maps are different, so it is necessary to use the bilinear interpolation method to uniformly adjust these local feature maps of different sizes to a fixed size of 7×7 feature map F, and then aggregate the extracted features through average pooling to obtain the final feature representation of the candidate region. Fig. 9 shown.

[0052] After that, the feature map F is input into the fully connected layer fc layers, which includes three independent fully connected layers fc layers, that is, the feature map F is input into three independent fully connected layers fc layers respectively, Figure 4 The first fully connected layer fc layer in the image performs bounding box regression on the feature map F through the bbox reg function, thereby fine-tuning the bounding box of the RoI area corresponding to the feature map F, obtaining more accurate bounding box coordinates, and obtaining the offset at the same time. The bounding box coordinates consist of two coordinate points and two offsets, so the two accurate coordinate points and two offsets are obtained, and the offset of the bounding box of the RoI area corresponding to each feature map F is output. The second fully connected layer fc layer in the middle performs multi-classification on the feature map F output by the ROI Align layer through the softmax function, and outputs the probability distribution of the corresponding category of the RoI area corresponding to each feature map F. The third fully connected layer fc layer uses the fully convolutional network Fully Convolutional Network (FCN) to take the feature map F output by the NLROI Align layer as input and output a segmentation mask of the same size as the RoI area corresponding to the feature map F. The segmentation mask is a binary image of the same size as the RoI area corresponding to the feature map F, where white (or non-zero value) pixels represent buildings and black (or zero value) pixels represent non-buildings. This mask can accurately describe the shape and contour of the object, thus achieving instance segmentation.

[0053] The first fully connected layer and the second fully connected layer are used for classification and bounding box regression, respectively, to obtain the final classification and regression results. The mask terminal is a fully convolutional network (FCN), which receives the feature map F extracted by the ROIAlign layer as input and outputs a segmentation mask of the same size as the RoI region corresponding to the feature map F. The segmentation mask of each category is independent, so the number of output channels of the mask terminal is the same as the number of categories. At this point, this step obtains the improved Mask R-CNN model.

[0054] Step 4. Train the improved Mask R-CNN model to obtain an optimized improved Mask R-CNN model.

[0055] The training sample set prepared in step 2 is used as the input data for training the improved Mask R-CNN model in step 3, and finally the bounding box coordinates of the buildings in the satellite image, the roof type and the mask of the building outline are obtained. When the data in the training set is processed by the improved Mask R-CNN model neural network model established in step 3, the buildings, roofs and outlines in the satellite images in the training set pass through the CMVIT module, FPN module, and module NLRoI Align, and then pass through three independent fully connected layers fc layers and a fully convolutional layer to obtain the bounding box coordinates of the buildings in the satellite image, the roof type and the mask of the building outline. The number of epochs is set to 25 during training, and the initial learning rate of the optimizer is set to 0.01 using the Adam optimizer. The learning rate remains unchanged in the first 10 epochs, and then the learning rate is reduced by 0.001 in each epoch until it is reduced to 0.002. The batch size is set to 8 during training. After each cycle, the loss function value of the current training cycle is calculated using the validation set, and the model with the smallest loss function value in the current training cycle is selected as the optimization model to update the model parameters; the training ends when the maximum number of training epochs is reached.

[0056] Step 5. Use the improved Mask R-CNN model to make predictions on unlabeled satellite images.

[0057] First, the satellite image to be predicted, that is, the test set in step 2, is input into the trained Mask R-CNN model; then, the model extracts the features of the image through the feature extraction network; then, the region proposal network RPN generates candidate areas that may contain buildings based on these features in the aforementioned processing process; then, each candidate area is classified to determine whether it belongs to a building and the type of roof of the building, and bounding box regression and mask generation are performed simultaneously; finally, redundant bounding boxes are removed by non-maximum suppression NMS, and the category, bounding box position and corresponding roof classification mask of each detected building are output. The hardware equipment used in the experimental process of the present invention is equipped with Intel 13400 version of the CPU and NVIDIA 4060ti 16G version of the GPU. The time required to predict satellite images of a 30km×30km area in a certain area is 1h, or 0.25 / s, the roof recognition and classification effect is as follows Fig.11As shown in the figure, flat|0.99 represents a flat roof and a confidence level of 0.99, which indicates the percentage of the recognized building category that is the true situation; gable|0.99 represents a gable and a confidence level of 0.99; complex|0.85 represents a complex roof and a confidence level of 0.85. The corresponding RGB satellite image, i.e. the original image, is shown in the figure below. Fig.10 As shown, the results show that the proposed method can effectively identify buildings and their corresponding roofs.

[0058] The present invention can automatically identify and classify different types of building roofs, such as flat roofs, sloping roofs, and arched roofs, by processing and analyzing high-resolution RGB satellite images. The method can be combined with the urban geographic information system (GIS) to achieve visual management and analysis of building roof data. This helps city managers monitor the status of urban buildings in real time and promptly discover and deal with potential safety hazards. At the same time, the technology can also provide basic data support for applications such as energy management and environmental monitoring in smart cities.

[0059] In summary, the building roof recognition and classification technology based on RGB satellite images has broad application prospects in the construction of smart cities. It not only improves the efficiency and accuracy of urban planning and management, but also provides strong support for the sustainable development and intelligent construction of cities.

[0060] As an embodiment disclosed by the present invention, the present invention also provides a device for building extraction and roof recognition based on RGB satellite images, and the device is used to implement the method, including the following modules: A processing module is used to pre-process the RGB satellite images to obtain an image dataset, and divide the image dataset into a training set, a test set, and a validation set after annotating the image dataset; Building modules for building improved MASK-RCNN models; A training module, used to train the improved MASK-RCNN model using the training set to obtain a trained improved MASK-RCNN model; A verification module, used to verify the trained improved MASK-RCNN model using the verification set to obtain an optimized improved MASK-RCNN model; The prediction module is used to use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

[0061] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0062] The above description shows and describes several preferred embodiments of the present invention, but as mentioned above, it should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the application concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A building extraction and roof recognition method based on RGB satellite images, characterized in that: The method comprises the steps of: S1. Preprocessing the RGB satellite images to obtain an image dataset, and then annotating the image dataset and dividing it into a training set, a test set, and a validation set; S2. Build an improved MASK-RCNN model; S3. Using the training set to train the improved MASK-RCNN model to obtain a trained improved MASK-RCNN model; S4. Using the validation set to validate the trained improved MASK-RCNN model to obtain an optimized improved MASK-RCNN model; S5. Use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

2. The method according to claim 1, characterized in that: The improved MASK-RCNN model includes a core Backbone part, a region proposal network RPN part and a candidate region head ROI Head part which are connected in sequence.

3. The method according to claim 2, characterized in that The core Backbone part includes the following steps: S211. Using an encoder to segment the input image into overlapping image pacths; S212. The overlapping picture pacth is propagated using the cmViT model to obtain an intermediate feature map; S213. The feature pyramid network FPN is used to fuse the intermediate feature maps to obtain feature images of different levels.

4. The method according to claim 3, characterized in that The cmViT model includes sixteen cViT modules connected in series, wherein the first to fourth cViT modules constitute a first submodule, the fifth to eighth cViT modules constitute a second submodule, the ninth to fourteenth cViT modules constitute a third submodule, and the fifteenth to sixteenth cViT modules constitute a fourth submodule; a first patch fusion module connected to the first submodule is arranged between the second submodule and the third submodule, a second patch fusion module connected to the second submodule is arranged between the second submodule and the third submodule; a third patch fusion module connected to the third submodule is arranged between the third submodule and the fourth submodule, the first submodule is also connected to a 4-fold upsampling module, the second submodule is also connected to a 2-fold upsampling module, and the fourth submodule is connected to a 2-fold downsampling module.

5. The method according to claim 4, characterized in that The intermediate feature images include at least four images, and S212 specifically includes: the overlapping image patches are propagated by the first submodule and processed by the upsampling 4-fold module to output the first intermediate feature image C1; The image output by the first submodule is fused by the first patch fusion module, propagated by the second submodule, and sampled by the upsampling 2x module to output the second intermediate feature image C2; The image output by the second submodule is fused by the second patch fusion module and then propagated by the third submodule to output the third intermediate feature image C3; The image output by the third submodule is fused by the third patch fusion module, propagated by the fourth submodule and sampled by the downsampling 2x module to output the fourth intermediate feature image C4; wherein the sizes of the first to fourth intermediate feature images are all different.

6. The method according to claim 5, characterized in that The S213 specifically includes: processing the feature pyramid network FPN in a top-down and lateral connection manner, specifically including: performing a 1x1 convolution on the fourth intermediate feature image C4 to generate a fifth image P5; The fifth image P5 is upsampled by 2 times to generate a feature map with the same resolution as the third intermediate feature image C3; the upsampled fifth image P5 is added to the feature map after 1x1 convolution of the third intermediate feature image C3, that is, horizontally connected, to generate a fourth image P4; The fourth image P4 is upsampled by 2 times to generate a feature map with the same resolution as the second intermediate feature image C2, and the upsampled fourth image P4 is added to the feature map after 1x1 convolution of the second intermediate feature image C2, that is, horizontally connected, to generate a third image P3; The third image P3 is upsampled by 2 times to generate a feature map with the same resolution as the first intermediate feature image C1, and the upsampled third image P3 is added to the feature map of the first intermediate feature image C1 after 1x1 convolution, that is, horizontally connected, to generate the second image P2; The fifth image P5 is directly downsampled by a factor of 2 to generate a sixth image P6.

7. The method according to claim 3, characterized in that The operation of the region proposal network RPN part specifically includes: the RPN performs feature extraction and conversion, classification and regression processing on the images P2-P6 at different levels, and screens the results obtained by the classification and regression processing to obtain candidate regions.

8. The method according to claim 7, characterized in that The operations of the candidate region head ROI Head part specifically include: S221. Performing normalized weighted average processing on the feature images at different levels to obtain feature images at different levels; S222. Determine the specific position of the candidate region in the feature images of different levels, obtain local feature maps of the candidate region of different sizes, resize and average pool the maps, and obtain the final features of the candidate region; S223. The final features are processed by a fully connected layer to obtain the category probability, bounding box coordinates and mask of each feature.

9. The method according to claim 3, characterized in that: Each cViT module consists of a multi-head attention operator, a normalization layer, and a convolutional multilayer perceptron connected in sequence.

10. A building extraction and roof recognition method based on RGB satellite images, characterized in that: The device is used to implement the method according to any one of claims 1 to 9, and comprises the following modules: A processing module is used to pre-process the RGB satellite images to obtain an image dataset, and divide the image dataset into a training set, a test set, and a validation set after annotating the image dataset; Building a module for building an improved MASK-RCNN model; A training module, used to train the improved MASK-RCNN model using the training set to obtain a trained improved MASK-RCNN model; A verification module, used to verify the trained improved MASK-RCNN model using the verification set to obtain an optimized improved MASK-RCNN model; The prediction module is used to use the optimized improved MASK-RCNN model to predict the test set and extract the building category and the corresponding roof classification.

Images