Method, device, terminal and storage medium for constructing three-dimensional model of building

Abstract

The present invention provides a method, device, terminal and storage medium for constructing a three-dimensional model of a building. The method for constructing a three-dimensional model of a building includes: performing segmentation processing on an acquired remote sensing image to obtain multiple regional images; the remote sensing image includes a building; performing target detection on the multiple regional images respectively to obtain roof information of the building; performing height and direction detection on the multiple regional images respectively to obtain height information and direction information of the building; and constructing a three-dimensional model of the building based on the roof information, height information and direction information of the building. The present application performs target detection, height and direction detection on the regional images respectively, and then predicts the roof information, height information and direction information of the building, and constructs a three-dimensional model of the building according to the predicted roof information, height information and direction information of the building, thereby improving the accuracy of constructing a three-dimensional model of the building in the remote sensing image.

Description

Technical Field

[0001] The present invention relates to the technical field of model construction, and in particular to a method, device, terminal and storage medium for constructing a three-dimensional model of a building. Background Art

[0002] 3D reconstruction of buildings is of great significance in urban construction and planning, disaster loss assessment, urban land environment assessment, military surveillance, etc. Remote sensing images have the advantages of large data volume, rich information, wide coverage, and high resolution. Therefore, using remote sensing images for 3D reconstruction of buildings has become a hot topic in the current application research of remote sensing.

[0003] In the existing technology, the most common method is to use binocular stereo matching technology for reconstruction. This method can achieve higher reconstruction accuracy. However, this technology often requires two or more remote sensing images with a certain inclination, a certain intersection angle, a base height ratio, etc., which are satellite-borne or airborne, as input data. It has higher requirements on the data source, is time-consuming, and has high costs. In addition, the data source is relatively limited, and there are certain limitations on the detection range. Secondly, using a single remote sensing image for reconstruction, this type of method often uses shadows in the image, additional elevation data information, such as digital surface models (DSM), etc. However, the method using additional elevation data information is severely limited by the influence of terrain undulations and complex input information. Although the algorithm using shadows is simple and easy to use, there are large differences in the shadow change range in different images, resulting in poor universality and robustness of the method, and the green space, trees, water bodies and other objects are similar to shadows, resulting in a high degree of false detection in the algorithm, and the complex information of remote sensing image objects leads to serious speckle noise in the shadow area. Summary of the invention

[0004] The main technical problem solved by the present invention is to provide a method, device, terminal and storage medium for constructing a three-dimensional model of a building, so as to solve the problem of low accuracy of reconstruction of three-dimensional models of buildings in the prior art.

[0005] To solve the above technical problems, the first technical solution adopted by the present invention is: to provide a method for constructing a three-dimensional model of a building, the method for constructing a three-dimensional model of a building comprising: segmenting an acquired remote sensing image to obtain a plurality of regional images; the remote sensing image includes a building; performing target detection on the plurality of regional images respectively to obtain roof information of the building; performing height and direction detection on the plurality of regional images respectively to obtain height information and direction information of the building; and constructing a three-dimensional model of the building based on the roof information, height information and direction information of the building.

[0006] The acquired remote sensing image is segmented to obtain a plurality of regional images, including: cropping the acquired remote sensing image by an overlapping cropping method to obtain a plurality of regional images.

[0007] Among them, target detection is performed on multiple regional images respectively to obtain roof information of the building, including: feature extraction of the regional image to obtain target features; detection of the target features to obtain a sub-regional image containing the roof; image masking of the sub-regional image based on the target features corresponding to the regional image to obtain a roof mask image; based on the correlation between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information corresponding to the remote sensing image.

[0008] Among them, feature extraction is performed on the regional image to obtain target features, which also includes: feature extraction on the regional image to obtain regional image features; upsampling or downsampling the regional image features to obtain a preprocessed regional image; feature extraction is performed on the preprocessed regional image to obtain corresponding target features; target features are detected to obtain a sub-regional image containing a roof, including: roof detection is performed on the target features corresponding to the preprocessed regional image respectively, to obtain a sub-regional image containing the roof in the regional image.

[0009] Among them, the roof information includes the detection frame of the building and the roof outline of the building; based on the association between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information corresponding to the remote sensing image, including: based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image is mapped to the remote sensing image to obtain the detection frames of all buildings contained in the remote sensing image and the roof outlines of the buildings contained in the detection frames.

[0010] Among them, based on the association between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information of the remote sensing image, and also include: in response to the overlap of multiple regional images, the overlap rate and confidence of the roof mask image corresponding to the regional image are calculated; retaining the roof mask image with an overlap rate less than a preset overlap rate and a confidence level greater than a preset confidence level.

[0011] Among them, based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image is mapped onto the remote sensing image to obtain the detection frames of all buildings contained in the remote sensing image and the roof outlines of the buildings contained in the detection frames, and then it also includes: judging whether the angles between the adjacent edges that make up the outline are less than the preset angle; if the angle between two adjacent edges is less than the preset angle, deleting the common points of the two edges.

[0012] Among them, target detection is performed on multiple regional images respectively to obtain the roof information of the building, including: performing target detection on the regional images through an instance segmentation network to obtain the roof information of the building; wherein the instance segmentation network includes a first feature extraction module, a target detection module and a semantic segmentation mask module.

[0013] Among them, the step of training to obtain an instance segmentation network includes: obtaining a first training sample set; the first training sample set includes multiple first sample images, and the first sample image is annotated with a real box of the target building, a real category of the semantic information in the real box, a real center position of the roof, and a real mask map of the roof; the first sample image is detected by the initial target detection network to obtain a predicted box of the target building, a predicted category of the semantic information in the predicted box, a predicted center position of the roof, and a predicted mask map of the roof; a first loss function is constructed based on the real box and predicted box of the target building, the real category of the semantic information in the real box and the predicted category of the semantic information in the predicted box, the real center position of the roof and the predicted center position of the roof, the real mask map of the roof, and the predicted mask map of the roof; and the initial target detection network is iteratively trained using the first loss function to obtain an instance segmentation network.

[0014] Among them, height and direction detection are performed on multiple regional images respectively to obtain height information and direction information of the building, including: feature extraction of the regional image through a height and direction detection network to obtain a grayscale image corresponding to the regional image; based on the correlation between each regional image and the remote sensing image, the grayscale images corresponding to the multiple regional images are spliced ​​to obtain an overall grayscale image corresponding to the remote sensing image; based on the overall grayscale image corresponding to the remote sensing image, the direction information and height information of the building are determined.

[0015] Among them, the height and direction detection network includes a second feature extraction module; the second feature extraction module includes a feature conversion layer, a first feature extraction layer and a second feature extraction layer cascaded in sequence; the feature of the regional image is extracted through the height and direction detection network to obtain a grayscale image corresponding to the regional image, including: the feature conversion layer performs feature conversion on the regional image to obtain a feature map of three-dimensional data; the first feature extraction layer performs feature extraction on the feature map of three-dimensional data to obtain a regional feature map; the second feature extraction layer performs feature extraction on the regional feature map to obtain a first feature map.

[0016] Among them, the height and direction detection network also includes a height information extraction module, the height information extraction module includes a plurality of height information extraction units cascaded in sequence, the height information extraction unit includes a first linear layer and a first activation function layer; the height information extraction module is connected to the second feature extraction module; the feature extraction of the regional image is performed through the height and direction detection network to obtain a grayscale image corresponding to the regional image, and also includes: the first linear layer associates the result output by the previous height information extraction unit with the result output by the corresponding second feature extraction module and then performs feature extraction to obtain a corresponding height feature map; the first activation function layer decodes the height feature map to obtain the height data of each pixel in the regional image.

[0017] Among them, the height and direction detection network also includes a direction information extraction module, the direction information extraction module includes a plurality of direction information extraction units cascaded in sequence, the direction information extraction unit includes a second linear layer and a second activation function layer; the direction information extraction module is connected to the second feature extraction module; the feature extraction of the regional image is performed through the height and direction detection network to obtain a grayscale image corresponding to the regional image, and also includes: the second linear layer associates the result output by the previous direction information extraction unit with the result output by the corresponding second feature extraction module and then performs feature extraction to obtain a corresponding direction feature map; the second activation function layer decodes the direction feature map to obtain the direction data of each pixel in the regional image.

[0018] Wherein, based on the overall grayscale image corresponding to the remote sensing image, determining the direction information and height information of the building includes: determining the direction information and height information of the building according to the direction data and height data of each pixel in the overall grayscale image.

[0019] Among them, based on the correlation between each regional image and the remote sensing image, the grayscale images corresponding to the multiple regional images are spliced ​​to obtain the height information and direction information of the building in the remote sensing image, and also include: judging whether there are overlapping areas between the regional images; if there are overlapping areas between at least two regional images, at least two grayscale images corresponding to the overlapping areas are merged to obtain the grayscale image corresponding to the overlapping areas.

[0020] The at least two grayscale images corresponding to the overlapping area are merged to obtain the grayscale image corresponding to the overlapping area, including: processing the at least two grayscale images corresponding to the overlapping area based on the following splicing formula; the splicing formula is as follows:

[0021] H i =α*B i +(1-α)*P i (Formula 1)

[0022]

[0023] Where: H i is the height value of the overlapping area; B i is the predicted value of the height of the overlapping area in the first grayscale image, P i is the height prediction value of the second grayscale image, β is a parameter, d represents the distance from pixel i to the overlapping edge; α is the weight, the default value is 0.5.

[0024] Among them, the steps of training to obtain a height and direction detection network include: obtaining a second training sample set; the second training sample set includes multiple second sample images, and the real height and real direction of the target building are marked in the second sample images; the second sample images are detected by the initial feature detection network to obtain the predicted height and predicted direction of the target building; a second loss function is constructed based on the real height and predicted height of the target building; a third loss function is constructed based on the real direction and predicted direction of the target building; and the initial feature detection network is iteratively trained using the second loss function and the third loss function to obtain the height and direction detection network.

[0025] Among them, based on the roof information, height information and direction information of the building, a three-dimensional model of the building is constructed, including: determining the foundation position of the building according to the roof information, height information and direction information; and constructing the three-dimensional model of the building according to the foundation position, height information and roof information.

[0026] To solve the above technical problems, the second technical solution adopted by the present invention is: to provide a three-dimensional model construction device for a building, the three-dimensional model construction device for a building includes: a segmentation module, used to segment the acquired remote sensing image to obtain multiple regional images; the remote sensing image includes a building; a target detection module, used to perform target detection on the multiple regional images respectively, to obtain the roof information of the building; a height and direction detection module, used to perform height and direction detection on the multiple regional images respectively, to obtain the height information and direction information of the building; a construction module, used to construct a three-dimensional model of the building based on the roof information, height information and direction information of the building.

[0027] In order to solve the above technical problems, the third technical solution adopted by the present invention is: to provide a terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to implement the steps in the above-mentioned method for constructing a three-dimensional model of a building.

[0028] In order to solve the above technical problems, the fourth technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method for constructing a three-dimensional model of a building are implemented.

[0029] The beneficial effects of the present invention are as follows: different from the prior art, the provided method, device, terminal and storage medium for constructing a three-dimensional model of a building, the method for constructing a three-dimensional model of a building comprising: segmenting the acquired remote sensing image to obtain a plurality of regional images; the remote sensing image includes a building; performing target detection on the plurality of regional images respectively to obtain the roof information of the building; performing height and direction detection on the plurality of regional images respectively to obtain the height information and direction information of the building; constructing a three-dimensional model of the building based on the roof information, height information and direction information of the building. The present application performs segmentation processing on the acquired remote sensing image, thereby effectively avoiding the roof, podium or densely connected buildings, which are adjacent to each other with different heights due to adhesion, from being identified as the same building; performing target detection, height and direction detection on the regional image respectively, thereby predicting the roof information, height information and direction information of the building, constructing a three-dimensional model of the building based on the predicted roof information, height information and direction information of the building, thereby improving the accuracy of constructing a three-dimensional model of the building in the remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 It is a schematic flow chart of a method for constructing a three-dimensional model of a building provided by the present invention;

[0032] Figure 2 It is a flow chart of an embodiment of a method for constructing a three-dimensional model of a building provided by the present invention;

[0033] Figure 3 It is a flow chart of a specific embodiment of a method for constructing a three-dimensional model of a building provided by the present invention;

[0034] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step S201 in the method for constructing a three-dimensional model of a building is provided;

[0035] Figure 5 yes Figure 3 A flowchart of a specific embodiment of step S202 in the method for constructing a three-dimensional model of a building is provided;

[0036] Figure 6 is a schematic diagram of overlapping cropping of remote sensing images with an overlap rate of 50% provided in this embodiment;

[0037] Figure 7 yes Figure 3 A flowchart of a specific embodiment of step S205 in the method for constructing a three-dimensional model of a building is provided;

[0038] Figure 8 It is a structural diagram of a specific embodiment of an instance segmentation network provided by the present invention;

[0039] Fig. 9 yes Figure 7 A roof information map obtained by the three-dimensional model construction method of the provided building;

[0040] Fig.10 yes Figure 3 A flowchart of a specific embodiment of step S206 in the method for constructing a three-dimensional model of a building is provided;

[0041] Fig.11 It is a structural schematic diagram of a specific embodiment of a height and direction feature extraction network provided by the present invention;

[0042] Fig.12 yes Fig.10 An overall grayscale image obtained from the provided method for constructing a three-dimensional model of a building;

[0043] Fig.13 (a) is an overall grayscale image obtained by splicing grayscale images using the soft transition splicing method in this application;

[0044] Fig.13 (b) is the overall grayscale image obtained by splicing the grayscale images using the traditional method;

[0045] Fig.14 (a) is a schematic diagram of an embodiment of original remote sensing image segmentation provided by the present application;

[0046] Fig.14 (b) is a schematic diagram of another embodiment of original remote sensing image segmentation provided by the present application;

[0047] Fig.14 (c) is a curve diagram between the weight parameter α and the distance d provided by the present application;

[0048] Fig.15 (a) is an original remote sensing image in an embodiment provided by the present application;

[0049] Fig.15 (b) is a roof information map corresponding to the original remote sensing image in an embodiment provided by the present application;

[0050] Fig.15(c) is an overall grayscale image corresponding to the original remote sensing image in an embodiment provided by the present application;

[0051] Fig.15 (d) is a three-dimensional model of a building corresponding to the original remote sensing image in an embodiment provided by the present application;

[0052] Fig.16 It is a schematic block diagram of a three-dimensional model construction device of a building provided by the present invention;

[0053] Fig.17 yes Fig.16 A schematic block diagram of a specific embodiment of a target detection module in a three-dimensional model of a building is provided;

[0054] Fig.18 yes Fig.16 A schematic block diagram of a specific embodiment of a height and direction detection module in a three-dimensional model of a building is provided;

[0055] Fig.19 is a schematic block diagram of an implementation manner of a terminal provided by the present invention;

[0056] Fig. 20 It is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0057] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0058] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0059] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0060] In order to enable those skilled in the art to better understand the technical solution of the present invention, a method for constructing a three-dimensional model of a building provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0061] See also Figure 1 , Figure 1 1 is a flow chart of a method for constructing a three-dimensional model of a building provided by the present invention. In this embodiment, a method for constructing a three-dimensional model of a building is provided, and the method for constructing a three-dimensional model of a building comprises the following steps.

[0062] S11: Segment the acquired remote sensing image to obtain multiple regional images.

[0063] Specifically, a remote sensing image is acquired, wherein the remote sensing image includes buildings, and the remote sensing image is cropped by an overlapping cropping method to obtain a plurality of regional images.

[0064] S12: Perform target detection on multiple area images respectively to obtain roof information of the building.

[0065] Specifically, feature extraction is performed on the regional image to obtain target features; the target features are detected to obtain a sub-regional image containing the roof; image mask processing is performed on the sub-regional image based on the target features corresponding to the regional image to obtain a roof mask image; based on the association between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information corresponding to the remote sensing image. The roof information includes the detection frame of the building and the roof outline of the building; based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image is mapped to the remote sensing image to obtain the detection frames of all buildings contained in the remote sensing image and the roof outlines of the buildings contained in the detection frames.

[0066] In a specific embodiment, feature extraction is performed on the regional image to obtain regional image features; upsampling or downsampling is performed on the regional image features to obtain a preprocessed regional image; feature extraction is performed on the preprocessed regional image to obtain corresponding target features; roof detection is performed on the target features corresponding to the preprocessed regional image to obtain a sub-region image containing the roof in the regional image.

[0067] In a specific embodiment, in response to the overlap of multiple area images, the overlap rate and confidence of the roof mask image corresponding to the area image are calculated; and the roof mask image with an overlap rate less than a preset overlap rate and a confidence greater than a preset confidence is retained.

[0068] In another specific embodiment, it is determined whether the angle between adjacent edges constituting the contour is smaller than a preset angle; if the angle between two adjacent edges is smaller than the preset angle, the common point of the two edges is deleted.

[0069] In another specific embodiment, target detection is performed on the regional image through an instance segmentation network to obtain roof information of the building; wherein the instance segmentation network includes a first feature extraction module, a target detection module and a semantic segmentation mask module.

[0070] S13: Perform height and direction detection on the multiple area images respectively to obtain height information and direction information of the building.

[0071] Specifically, the features of the regional image are extracted through the height and direction detection network to obtain the grayscale image corresponding to the regional image; based on the correlation between each regional image and the remote sensing image, the grayscale images corresponding to multiple regional images are spliced ​​to obtain the overall grayscale image corresponding to the remote sensing image; based on the overall grayscale image corresponding to the remote sensing image, the direction information and height information of the building are determined.

[0072] In a specific embodiment, the height and direction detection network includes a second feature extraction module; the second feature extraction module includes a feature conversion layer, a first feature extraction layer and a second feature extraction layer cascaded in sequence; the feature conversion layer performs feature conversion on the regional image to obtain a feature map of the three-dimensional data; the first feature extraction layer performs feature extraction on the feature map of the three-dimensional data to obtain a regional feature map; the second feature extraction layer performs feature extraction on the regional feature map to obtain a first feature map.

[0073] The height and direction detection network also includes a height information extraction module, which includes a plurality of height information extraction units cascaded in sequence, and the height information extraction unit includes a first linear layer and a first activation function layer; the height information extraction module is connected to the second feature extraction module. The first linear layer associates the result output by the previous height information extraction unit with the result output by the corresponding second feature extraction module, and then performs feature extraction to obtain a corresponding height feature map; the first activation function layer decodes the height feature map to obtain the height data of each pixel in the regional image.

[0074] The height and direction detection network also includes a direction information extraction module, which includes a plurality of direction information extraction units cascaded in sequence, and the direction information extraction unit includes a second linear layer and a second activation function layer; the direction information extraction module is connected to the second feature extraction module; the second linear layer associates the result output by the previous direction information extraction unit with the result output by the corresponding second feature extraction module and then performs feature extraction to obtain a corresponding direction feature map; the second activation function layer decodes the direction feature map to obtain the direction data of each pixel in the regional image.

[0075] The direction information and height information of the building are determined according to the direction data and height data of each pixel in the overall grayscale image.

[0076] In an optional embodiment, it is determined whether there is an overlapping area between the regional images; if there is an overlapping area between at least two regional images, at least two grayscale images corresponding to the overlapping area are merged to obtain a grayscale image corresponding to the overlapping area.

[0077] In a specific embodiment, the overlapping portion of at least two grayscale images is processed based on the following splicing formula; the splicing formula is as follows:

[0078] Hi =α*B i +(1-α)*P i (Formula 1)

[0079]

[0080] Where: H i is the height value of the overlapping area; B i is the predicted value of the height of the overlapping area in the first grayscale image, P i is the height prediction value of the second grayscale image, β is a parameter, d represents the distance from pixel i to the overlapping edge; α is the weight, the default value is 0.5.

[0081] S14: Based on the roof information, height information and direction information of the building, a three-dimensional model of the building is constructed.

[0082] Specifically, the foundation position of the building is determined according to the roof information, height information and direction information; and a three-dimensional model of the building is constructed according to the foundation position, height information and roof information.

[0083] See also Figure 2 , Figure 2 The present invention is a flow chart of an embodiment of a method for constructing a three-dimensional model of a building provided by the present invention. In a specific embodiment, a high-resolution remote sensing image is obtained, and the original remote sensing image obtained is subjected to sliding window cropping by an overlapping cropping method to obtain a remote sensing image set consisting of a plurality of low-resolution regional images. The roof of the building is detected for each regional image in the remote sensing image set by an instance segmentation network, and the detection result corresponding to each regional image is obtained. The detection result includes a detection frame of the building and a roof outline of the building. According to the association between each regional image and the remote sensing image, the detection frame of the building corresponding to each regional image and the roof outline of the building are spliced ​​to obtain the roof information of each building in the remote sensing image. The height and direction of each regional image in the remote sensing image set are detected by a height and direction feature extraction network to obtain the height result and direction result corresponding to each regional image. The height result includes the height data of each pixel in the regional image, and the direction result includes the direction data of each pixel in the regional image. The direction result and the height result corresponding to each regional image are spliced ​​to obtain the height information and direction information of each building in the remote sensing image. The three-dimensional model of the building is constructed according to the roof information, direction information and height information of the building.

[0084] The method for constructing a three-dimensional model of a building provided in this embodiment includes: performing segmentation processing on the acquired remote sensing image to obtain multiple regional images; the remote sensing image includes a building; performing target detection on the multiple regional images respectively to obtain the roof information of the building; performing height and direction detection on the multiple regional images respectively to obtain the height information and direction information of the building; and constructing a three-dimensional model of the building based on the roof information, height information and direction information of the building. The present application performs segmentation processing on the acquired remote sensing image, thereby effectively avoiding the roof, podium or densely connected buildings, which are adjacent to each other with different heights due to adhesion, from being identified as the same building; performing target detection, height and direction detection on the regional image respectively, thereby predicting the roof information, height information and direction information of the building, and constructing a three-dimensional model of the building based on the predicted roof information, height information and direction information of the building, thereby improving the accuracy of constructing a three-dimensional model of the building in the remote sensing image.

[0085] See also Figure 3 , Figure 3 The figure is a flow chart of a specific embodiment of a method for constructing a three-dimensional model of a building provided by the present invention. In this embodiment, a method for constructing a three-dimensional model of a building is provided, and the method for constructing a three-dimensional model of a building comprises the following steps.

[0086] S201: Train to obtain an instance segmentation network.

[0087] Specifically, the initial object detection network includes an initial first feature extraction module, an initial object detection module, and an initial semantic segmentation mask module. The parameters in each module in the initial object detection network are corrected through the following steps to obtain an instance segmentation network. Among them, the instance segmentation network includes a first feature extraction module, an object detection module, and a semantic segmentation mask module.

[0088] In a specific embodiment, the initial target detection network uses centermask as the network framework. Among them, the first feature extraction module uses VoVNetV2+FPN (Feature Pyramid Networks). In the first feature extraction module, each time a pooling layer is passed, the feature map is reduced by one scale, and finally a feature map with the same number of pooling layers is obtained. In this embodiment, the number of pooling layers is five, so five feature maps of different sizes can be obtained. The target detection network uses FCOS (Fully Convolutional One-Stage Object Detection) network. The semantic segmentation mask module uses SAG-Mask (Spatial Attention-Guided Mask) network. The SAG-Mask network can predict a mask map of size 28*28, and finally scale it to the corresponding size of the prediction box.

[0089] See also Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step S201 in a method for constructing a three-dimensional model of a building is provided.

[0090] The specific steps for training the primary object detection network to obtain the instance segmentation network are as follows.

[0091] S2011: Obtain a first training sample set.

[0092] Specifically, the first training sample set includes a plurality of first sample images, and the first sample images are images annotated with real information of the roof of the target building. In a specific embodiment, the first sample image is annotated with a real frame of the target building, a real category of semantic information in the real frame, a real center position of the roof, and a real mask map of the roof. The first sample image can be a regional image obtained by segmentation of a high-resolution remote sensing image, or can be an image containing the target building captured by a camera.

[0093] S2012: Detect the first sample image through the initial target detection network to obtain a predicted box of the target building, a predicted category of the semantic information in the predicted box, a predicted center position of the roof, and a predicted mask map of the roof.

[0094] Specifically, the initial target detection network performs building detection on the first sample image to obtain the roof prediction information of the target building contained in the first sample image. In a specific embodiment, the initial target detection network performs building roof detection on the first sample image object to obtain the prediction box of the target building, the prediction category of the semantic information in the prediction box, the prediction center position of the roof, and the prediction mask map of the roof.

[0095] S2013: Construct a first loss function based on the real box and predicted box of the target building, the real category of the semantic information in the real box and the predicted category of the semantic information in the predicted box, the real center position of the roof and the predicted center position of the roof, the real mask map of the roof and the predicted mask map of the roof.

[0096] Specifically, the first loss function is constructed by the real box and predicted box of the target building, the real category of the semantic information in the real box and the predicted category of the semantic information in the predicted box, the real center position of the roof and the predicted center position of the roof, the real mask map of the roof and the predicted mask map of the roof. Among them, the first loss function includes MSE (Mean Squared Error) Loss and BCE (Binary Cross-Entropy) Loss.

[0097] S2014: Using the first loss function, iteratively train the initial target detection network to obtain an instance segmentation network.

[0098] Specifically, the initial target detection network is iteratively trained through the error values ​​between the true box and the predicted box of the target building, the true category of the semantic information in the true box and the predicted category of the semantic information in the predicted box, the true center position of the roof and the predicted center position of the roof, the true mask map of the roof and the predicted mask map of the roof to obtain the instance segmentation network.

[0099] In an optional embodiment, the result of the initial target detection network is back-propagated, and the weight of the initial target detection network is modified according to the loss value fed back by the first loss function. In an optional embodiment, the parameters of the initial target detection network can also be modified to achieve training of the initial target detection network.

[0100] The remote sensing image containing the roof of the building is input into the initial target detection network, and the initial target detection network predicts the roof, roof category, roof center position and roof real mask map of the target building. When the error value between the real frame and the predicted frame of the target building, the real category of the semantic information in the real frame and the predicted category of the semantic information in the predicted frame, the real center position of the roof and the predicted center position of the roof, the real mask map of the roof and the predicted mask map of the roof is less than the preset threshold, which can be set by yourself, such as 1%, 5%, etc., the training of the initial target detection network is stopped and the instance segmentation network is obtained.

[0101] S202: Train to obtain a height and direction detection network.

[0102] Specifically, the initial feature detection network is trained to obtain a height and direction detection network, wherein the height and direction detection network is similar to the U-Net network.

[0103] See also Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step S202 in a method for constructing a three-dimensional model of a building is provided.

[0104] S2021: Obtain a second training sample set.

[0105] Specifically, the second training sample set includes a plurality of second sample images, in which the real height and real direction of the target building contained in each pixel in the image are annotated. The second sample image can be a regional image obtained by segmenting a high-resolution remote sensing image, or can be an image containing the target building collected by a camera.

[0106] S2022: Detect the second sample image through the initial feature detection network to obtain the predicted height and predicted direction of the target building.

[0107] Specifically, the target building is detected by the initial second feature extraction module in the initial feature detection network to obtain a target building feature map, and the initial height information extraction module extracts the height information of the target building in the target building feature map to obtain the predicted height of the target building in each pixel. The initial direction information extraction module predicts the predicted direction of the target building from the roof to the foundation according to the target building feature map.

[0108] S2023: Construct a second loss function based on the actual height and predicted height of the target building; construct a third loss function based on the actual direction and predicted direction of the target building.

[0109] Specifically, a second loss function is constructed based on the real height and predicted height of the target building contained in each pixel in the second sample image. The second loss function is a weighted sum of the structural similarity index (SSIM) loss function and the MSE loss function. The SSIM loss function can calculate the brightness, contrast and structural indicators of each individual pixel in the image, and pays more attention to detail differences than the MSE loss function. The SSIM loss function and the MSE loss function take into account both the accuracy of each individual pixel and the details of the entire image.

[0110] The third loss function is constructed based on the real direction and predicted direction of the target building contained in each pixel in the second sample image. The third loss function is the weighted sum of the MSE loss function and the cosine similarity loss function. The training accuracy of the initial direction information extraction module can be improved by the MSE loss function and the cosine similarity loss function.

[0111] S2024: Use the second loss function and the third loss function to iteratively train the initial feature detection network to obtain a height and direction detection network.

[0112] Specifically, the initial feature detection network is iteratively trained through the actual height and predicted height of the target building and the error value between the actual direction and predicted direction of the target building to obtain the height and direction detection network.

[0113] In an optional embodiment, the result of the initial feature detection network is back-propagated, and the weights of the initial feature detection network are modified according to the loss values ​​fed back by the second loss function and the third loss function. In an optional embodiment, the parameters of the initial feature detection network can also be modified to achieve training of the initial feature detection network.

[0114] The remote sensing image containing the building is input into the initial feature detection network, and the initial feature detection network predicts the height and direction of the target building in each pixel in the image. When the error between the predicted height and the actual height, and the predicted direction and the actual direction of the target building in each pixel in the image are less than the preset threshold, which can be set by yourself, such as 1%, 5%, etc., the training of the initial feature detection network is stopped and the height and direction detection network is obtained.

[0115] The height and direction detection network obtained through training can accurately detect the height information and direction information of the building contained in the remote sensing image. The direction information is the direction from the roof of the building to the foundation of the building.

[0116] S203: Acquire remote sensing images.

[0117] Specifically, the remote sensing image can be a remote sensing image obtained by scanning the earth's surface using a satellite as a remote sensing platform through an earth observation remote sensing instrument mounted on the satellite. The remote sensing image in this embodiment has the characteristics of high resolution, large mapping area, rich information and wide shooting area. The remote sensing image can be a color image or a black and white image. The image remote sensing image includes at least one building.

[0118] S204: cropping the remote sensing image by an overlapping cropping method to obtain multiple regional images.

[0119] Specifically, since high-resolution remote sensing images require a large amount of storage space, in order to reduce the amount of calculation in the detection process and improve the detection accuracy and precision, the original high-resolution remote sensing images can be cropped to obtain multiple small-resolution color images or black-and-white images.

[0120] In the subsequent steps, the height information, direction information, and roof information detected in each small-resolution color image or black-and-white image need to be stitched back to the size of the original remote sensing image. In order to avoid the problem of stitching gaps when the detection results are stitched, this embodiment uses an overlapping cropping method to crop the remote sensing image, so that when the detection results of each small-resolution color image or black-and-white image are stitched back to the size of the original remote sensing image, they can be continuous and without gaps. In a specific embodiment, the overlap rate of two adjacent small-resolution color images or black-and-white images is 25%, and the pixel size of the cropped small-resolution color image or black-and-white image is 512*512. Please refer to Figure 6 , Figure 6 Schematic diagram of overlapping cropping of remote sensing images with an overlap rate of 50% provided in this embodiment. In other optional embodiments, the overlap rate of two adjacent low-resolution color images or black-and-white images can also be 50%. Specifically, the cropping overlap rate can be set according to demand. Among them, the low-resolution color image or black-and-white image is used as the regional image.

[0121] In an optional embodiment, other methods may be used to crop the original remote sensing image according to different detection results.

[0122] S205: Perform target detection on the regional image through an instance segmentation network to obtain roof information of the building.

[0123] See also Figures 7 to 9 , Figure 7 yes Figure 3 A flowchart of a specific embodiment of step S205 in the method for constructing a three-dimensional model of a building is provided; Figure 8 It is a structural diagram of a specific embodiment of an instance segmentation network provided by the present invention; Fig. 9 yes Figure 7 A roof information diagram obtained by the method for constructing a three-dimensional model of a building is provided.

[0124] Specifically, the instance segmentation network includes a first feature extraction module, a target detection module, and a semantic segmentation mask module, such as Figure 8 In one embodiment, the regional image may be preprocessed in advance. For example, converting data types.

[0125] S2051: Extract features from the regional image to obtain target features.

[0126] Specifically, the first feature extraction module includes a plurality of roof feature extraction units cascaded in sequence. The regional image is upsampled or downsampled to obtain a plurality of preprocessed regional images of different scales; and the preprocessed image is feature extracted to obtain corresponding target features.

[0127] In this embodiment, the first feature extraction module includes five roof feature extraction units. The acquired regional images are respectively input into the instance segmentation network. The first roof feature extraction unit in the first feature extraction module extracts features from the regional image and performs size adjustment to obtain the first target feature; the second roof feature extraction unit continues to extract features from the first target feature and performs size adjustment to obtain the second target feature; the third roof feature extraction unit continues to extract features from the second target feature and performs size adjustment to obtain the third target feature; the fourth roof feature extraction unit continues to extract features from the third target feature and performs size adjustment to obtain the fourth target feature; the fifth roof feature extraction unit continues to extract features from the fourth target feature and performs size adjustment to obtain the fifth target feature. Among them, the sizes and information richness of the first target feature, the second target feature, the third target feature, the fourth target feature and the fifth target feature are all different, but the corresponding target buildings are the same. The first target feature, the second target feature, the third target feature, the fourth target feature and the fifth target feature are all feature maps of different sizes corresponding to the regional image.

[0128] S2052: Detect target features to obtain location information of the roof in the regional image.

[0129] Specifically, the target features corresponding to the multiple pre-processed regional images are detected by the target detection module in the instance segmentation network to obtain the position information of the roof in the regional image.

[0130] In a specific embodiment, the target detection module detects the roof of a building on the first target feature, the second target feature, the third target feature, the fourth target feature and the fifth target feature obtained by the first feature extraction module to obtain the position information of the roof in the regional image. That is, the detection frame of the roof of the building is obtained according to the first target feature, the second target feature, the third target feature, the fourth target feature and the fifth target feature.

[0131] S2053: Perform image mask processing based on the target features corresponding to the regional image and the position information of the roof in the regional image to obtain a roof mask image.

[0132] Specifically, masking an image means recalculating the value of each pixel in the image through a mask kernel operator. The mask kernel describes the degree of influence of neighborhood pixels on the new pixel value. At the same time, the original pixels are weighted averaged according to the weight factor in the mask operator to determine the roof contour in the regional image and detect the roof area in the regional image.

[0133] In a specific embodiment, the position information of the roof in the regional image and the acquired first target feature, second target feature, third target feature, fourth target feature and fifth target feature are input into a semantic segmentation mask module. The semantic segmentation mask module binarizes the regional image based on the feature information of the corresponding position of the position information of the roof in the first target feature, second target feature, third target feature, fourth target feature and fifth target feature. The pixel value of the roof area is 255, and the pixel value of the part of the regional image that does not belong to the roof area is 0, thereby obtaining a roof mask image.

[0134] Through the above steps, the roof mask image corresponding to each regional image can be obtained. In a specific embodiment, through the above steps, the roof detection frame in each regional image and the roof contour contained in the roof detection frame can be obtained.

[0135] S2054: Based on the association between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information of the remote sensing image.

[0136] Specifically, based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image corresponding to each regional image is mapped onto the remote sensing image, and the roof information is formed by the roof mask image corresponding to each regional image. In other words, the roof mask images corresponding to each regional image are spliced ​​to obtain the roof information map corresponding to the remote sensing image, such as Fig. 9 .

[0137] Since the remote sensing image is cropped by overlapping cropping in the above steps, there are overlapping parts in the adjacent multiple regional images, and overlap will also occur during the splicing process. Therefore, it is necessary to determine whether the two adjacent regional images have overlapping areas. If the two adjacent regional images do not have overlapping areas, the roof mask image corresponding to the regional image is directly spliced ​​at the corresponding position of the remote sensing image. That is, the detection frame of the roof in the regional image can be directly used and matched to the corresponding position of the remote sensing image. If the two adjacent regional images have overlapping areas, the roof mask image corresponding to the regional image is mapped to the high-resolution remote sensing image, and the roof mask image of the overlapping area is processed based on the non-maximum suppression of the mask, and the roof mask image with an overlap rate not less than the preset overlap rate and / or a confidence not greater than the preset confidence is deleted, and the roof mask image with an overlap rate less than the preset overlap rate and a confidence greater than the preset confidence is retained. That is to say, the detection frames of all roofs are mapped back to the high-resolution remote sensing image, and the detection frames of all roofs are suppressed based on the non-maximum of the mask, thereby retaining the roof mask images with an overlap rate less than a preset overlap rate and a confidence level greater than a preset confidence level.

[0138] S2055: Extract the roof outline based on the roof mask image.

[0139] Specifically, since the pixel value of the roof area in the roof mask image is 255 and the pixel value of the non-roof area is 0, when the pixel values ​​of two adjacent pixels in the roof mask image are different, the pixel with a pixel value of 255 is a boundary point of the roof area, and multiple boundary points are connected to form the outline of the roof.

[0140] S2056: Compare the angles between adjacent edges constituting the contour with a preset angle.

[0141] Specifically, in order to further prevent the inconsistently sized holes in the roof mask image from affecting the roof detection result and to improve the speed of locating the pixel position of the roof of the target building, the roof mask image needs to be preprocessed.

[0142] In this embodiment, the preset shape of the roof is a rectangle or a circle, so there should be no holes within the outline of the roof. The area within the outline is formed into a connected domain for extraction, and the area of ​​each connected domain is calculated. The connected domain with an area smaller than the preset area is taken as the roof.

[0143] Furthermore, in order to improve the detection accuracy of the roof contour, in the conventional definition, the roof contour does not have a bulge with an angle less than a preset angle. Therefore, the obtained roof contour is extracted to determine whether the angle between two adjacent edges constituting the roof contour is less than the preset angle, and then determine whether there is an edge constituting a bulge between the two adjacent edges constituting the roof contour.

[0144] S2057: In response to the angle between two adjacent edges of the contour being smaller than a preset angle, deleting a common point between the two edges.

[0145] Specifically, if the angle between two adjacent edges is smaller than a preset angle, the common point of the two edges is deleted, and the endpoints of the two edges far from the common point are connected to each other.

[0146] In this embodiment, the roof of the building is segmented into individual parts through the instance segmentation network, which can effectively prevent the roof, podium or densely connected building mask images from being identified as the same building due to adhesion.

[0147] S206: Extract features of the regional image through a height and direction detection network to obtain a grayscale image corresponding to the regional image.

[0148] See also Figures 10 to 12 , Fig.10 yes Figure 3 A flowchart of a specific embodiment of step S206 in the method for constructing a three-dimensional model of a building is provided; Fig.11 It is a structural schematic diagram of a specific embodiment of a height and direction feature extraction network provided by the present invention; Fig.12 yes Fig.10 The overall grayscale image obtained by the method for constructing a three-dimensional model of a building is provided.

[0149] Specifically, the height and direction detection network includes a second feature extraction module, a height information extraction module and a direction information extraction module. The second feature extraction module is connected to the height information extraction module and the direction information extraction module respectively. The second feature extraction module is used as an encoding part, and the height information extraction module and the direction information extraction module are used as decoding parts. Fig.11 .

[0150] S2061: The second feature extraction module extracts features from the regional image to obtain a corresponding feature map.

[0151] The second feature extraction module includes a plurality of sub-feature extraction units, and the sub-feature extraction units include a feature conversion layer, a first feature extraction layer, and a second feature extraction layer cascaded in sequence. In this embodiment, there are four sub-feature extraction units, and the four sub-feature extraction units are cascaded in sequence. Among them, the second feature extraction module adopts a segformer network structure as the network framework, and encodes layer by layer through the cascaded sub-feature extraction units. The feature conversion layer corresponds to an OverlapPatch Embeddings (Overlap Patch Embeddings), and Overlap Patch Embeddings is used to convert four-dimensional data into three-dimensional data. The first feature extraction layer corresponds to an Attn (Attention), and Attn serves as an attention module. The second feature extraction layer corresponds to an MLP (Multilayer Perceptron), and the second feature extraction layer includes a linear layer, a convolutional layer, and an activation layer. The second feature extraction layer is used to continue feature extraction of the feature map extracted by the first feature extraction layer.

[0152] In a specific embodiment, the structures of the four sub-feature extraction units are the same. The regional image is input into the second feature extraction module, and the feature conversion layer in the first sub-feature extraction unit in the second feature extraction module converts the four-bit data in the regional image into a feature map of three-dimensional data; the first feature extraction layer performs feature extraction on the feature map of three-dimensional data to obtain a regional feature map; the second feature extraction layer then performs feature extraction on the regional feature map to obtain a first feature map, the second sub-feature extraction unit performs feature extraction on the first feature map to obtain a second feature map, the third sub-feature extraction unit performs feature extraction on the second feature map to obtain a third feature map, and the fourth sub-feature extraction unit performs feature extraction on the third feature map to obtain a fourth feature map. Among them, the sizes of the first feature map, the second feature map, the third feature map, and the fourth feature map are different, and the richness of the feature data contained in each feature map is also different.

[0153] S2062: Detect the feature map corresponding to the regional feature through the height information extraction module to obtain the height data of each pixel in the regional feature.

[0154] The height information extraction module includes a plurality of height information extraction units cascaded in sequence. In the present embodiment, there are four height information extraction units. The height information extraction unit includes a first linear layer and a first activation function layer connected to the first linear layer. The feature map extracted by the last sub-feature extraction unit in the second feature extraction module is input to the first linear layer of the first height information extraction unit in the height information extraction module.

[0155] In a specific embodiment, the fourth feature map is input into the first linear layer of the first height information extraction unit in the height information extraction module, and the first linear layer extracts features from the fourth feature map to obtain a corresponding height feature map; and the height feature map is activated through the first activation function layer to obtain a fifth feature map. The fifth feature map and the third feature map are input into the second height information extraction unit for feature extraction to obtain a sixth feature map; the sixth feature map and the second feature map are input into the third height information extraction unit for feature extraction to obtain a seventh feature map; the seventh feature map and the first feature map are input into the fourth height information extraction unit for feature extraction to obtain an eighth feature map. The height of the target building in each pixel in the regional image is detected through the eighth feature map to obtain the height data of each pixel.

[0156] Through the above steps, the height data corresponding to each pixel in each area image can be obtained.

[0157] S2063: Detect the feature map corresponding to the regional image through the direction information extraction module to obtain direction data of each pixel in the regional image.

[0158] The directional information extraction module includes a plurality of directional information extraction units cascaded in sequence. In this embodiment, there are four directional information extraction units. The directional information extraction unit includes a second linear layer and a second activation function layer connected to the second linear layer. The feature map extracted by the last sub-feature extraction unit in the second feature extraction module is input to the second linear layer of the first directional information extraction unit in the directional information extraction module.

[0159] In a specific embodiment, the fourth feature map is input into the second linear layer of the first directional information extraction unit in the directional information extraction module, and the second linear layer extracts features from the fourth feature map to obtain a directional feature map; and the directional feature map is activated through the second activation function layer to obtain a ninth feature map. The ninth feature map and the third feature map are input into the second directional information extraction unit for feature extraction to obtain a tenth feature map; the tenth feature map and the second feature map are input into the third directional information extraction unit for feature extraction to obtain an eleventh feature map; the eleventh feature map and the first feature map are input into the fourth directional information extraction unit for feature extraction to obtain a twelfth feature map. The direction of the target building in each pixel in the regional image is detected through the twelfth feature map to obtain the direction data of each pixel.

[0160] Through the above steps, the direction data corresponding to each pixel in each regional image can be obtained.

[0161] S207: Based on the positional relationship and size relationship between the regional image and the remote sensing image, the grayscale images corresponding to the regional images are spliced ​​to obtain an overall grayscale image corresponding to the remote sensing image.

[0162] Specifically, based on the association between each regional image and the remote sensing image, the grayscale image corresponding to the regional image is mapped to the remote sensing image, and multiple grayscale images constitute the overall grayscale image of the remote sensing image, such as Fig.12 Among them, the grayscale value in the overall grayscale image represents the height of the pixel, and the pixel value ranges from 1 to 1000.

[0163] That is to say, the overall grayscale image of the remote sensing image is a map of the height and direction data corresponding to the remote sensing image obtained by splicing the height data and direction data corresponding to each regional image.

[0164] Since there is an overlapping area between two adjacent regions or multiple region images, in order to avoid the grayscale images corresponding to the overlapping areas being mapped into the overall grayscale image, resulting in information duplication, it is necessary to process the overlapping parts of at least two corresponding grayscale images based on the overlapping parts corresponding to the two region images.

[0165] In response to the grayscale images respectively corresponding to at least two regional images overlapping, the overlapping portion is processed in combination with the height information and direction information of the overlapping portion of the at least two grayscale images.

[0166] See also Fig.13 (a) and Fig.13 (b) Fig.13 (a) is an overall grayscale image obtained by splicing grayscale images using the soft transition splicing method in this application; Fig.13 (b) is the overall grayscale image obtained by splicing grayscale images using traditional methods.

[0167] If the traditional method is used to retain a set of height information and direction information corresponding to the overlapping parts and then stitch the height information and direction information, there will be stitching seams between the grayscale images corresponding to the images of each region, such as Fig.13 (b) As shown in the present embodiment, a soft transition stitching method is used for stitching, and the grayscale value of the overlapping area of ​​two or more adjacent area images can be obtained by the corresponding stitching formula, thereby obtaining a seamless overall grayscale image, as shown in FIG. Fig.13 (a) shown.

[0168] Specifically, the overlapping parts of at least two grayscale images are processed based on the following splicing formula.

[0169] The concatenation formula is as follows:

[0170] H i =α*B i +(1-α)*P i (Formula 1)

[0171]

[0172] Where: H i is the height value of the overlapping area; B i is the predicted value of the height of the overlapping area in the first grayscale image, P i is the height prediction value of the second grayscale image, β is a parameter, d represents the distance from pixel i to the overlapping edge; α is the weight, the default value is 0.5.

[0173] See also Fig.14 (a) to Fig.14 (c) Fig.14 (a) is a schematic diagram of an embodiment of original remote sensing image segmentation provided by the present application; Fig.14 (b) is a schematic diagram of another embodiment of original remote sensing image segmentation provided by the present application; Fig.14 (c) is a curve diagram between the weight parameter α and the distance d provided by the present application.

[0174] The original remote sensing image is cropped according to the set overlap ratio to determine the overlapping areas of different regional images, such as Fig.14 (a) and Fig.14 (b) As shown. The weight value of the same pixel in the first grayscale image and the second grayscale image is determined according to the distance between different pixels in the overlapping area and the overlapping edge. Among them, the value range of the weight value α is 0.5 by default, such as Fig.14 In this embodiment, β is set to 0.08.

[0175] The soft transition stitching method proposed in this embodiment can effectively eliminate the gaps that appear at the stitching boundary when the grayscale images corresponding to the regional images are stitched. This method is applicable to stitching two grayscale images adjacent to each other on the left and right, and also to stitching two grayscale images above and below.

[0176] S208: Determine direction information and height information of the building based on the overall grayscale image corresponding to the remote sensing image.

[0177] Specifically, according to step S206 and step S207, the height information and direction information corresponding to the building to which each roof belongs can be obtained. According to the height information and direction information of the building, the average height, average visual direction and position information of the roof can be quickly obtained, and then the relevant information of the roofs of all buildings contained in the remote sensing image can be obtained.

[0178] S209: Determine the foundation position of the building according to the roof information, height information and direction information.

[0179] Specifically, the direction information of the building to which the roof belongs is decomposed into (d x , d y), according to the average height H of the corresponding building roof obtained in step S208 and the direction information of the building, the distance from the building roof to the building foundation is (H*d x , H*d y ), and then the foundation position of the building is obtained according to the position information of the roof, the distance between the roof and the foundation, and the direction information between the roof and the foundation.

[0180] In one embodiment, the rooftop pixel R i Along (d x , d y ) direction by H pixels to obtain the foundation position of the building, and then determine the height of the foundation.

[0181] S210: Construct a three-dimensional model of the building according to the foundation location, height information, and roof information.

[0182] Specifically, a three-dimensional model of a building corresponding to the building is constructed through the foundation position of the building, the direction information of the building, and the roof position.

[0183] See also Fig.15 (a) to Fig.15 (d) Fig.15 (a) is an original remote sensing image in an embodiment provided by the present application; Fig.15 (b) is a roof information map corresponding to the original remote sensing image in an embodiment provided by the present application; Fig.15 (c) is an overall grayscale image corresponding to the original remote sensing image in an embodiment provided by the present application; Fig.15 (d) is a three-dimensional model of a building corresponding to the original remote sensing image in an embodiment provided by the present application.

[0184] In a specific embodiment, an image to be processed is obtained, and the image to be processed contains buildings, such as Fig.15 (a). The image to be processed is detected by the instance segmentation network to obtain the roof information map of the building contained in the image to be processed, such as Fig.15 (b). The image to be processed is detected by the height and direction detection network to obtain the overall grayscale image corresponding to the image to be processed, such as Fig.15 (c). The overall grayscale image contains the direction information and height information of the building. The foundation position of the building is determined based on the roof information, the direction information and the height information of the building. The three-dimensional building model corresponding to the building is constructed based on the foundation position, the height information and the roof information of the building. Fig.15 (d).

[0185] The present embodiment provides a method for constructing a three-dimensional model of a building, by segmenting the acquired remote sensing image to obtain multiple regional images; the remote sensing image includes a building; target detection is performed on the multiple regional images respectively to obtain the roof information of the building; height detection is performed on the multiple regional images respectively to obtain the height information and direction information of the building; based on the roof information, height information and direction information of the building, a three-dimensional model of the building is constructed. The present application performs segmentation processing on the acquired remote sensing image, thereby effectively avoiding the adjacent buildings of different heights being identified as the same building due to adhesion of the roof, podium or densely connected buildings, and is convenient for identifying the attic on the roof; target detection is performed on the regional images respectively through an instance segmentation network, thereby improving the detection accuracy of the roof information; height and direction detection are performed on the regional images respectively through a height and direction detection network, thereby improving the detection accuracy of the height information and direction information of the building, and constructing a three-dimensional model of the building according to the detected roof information, height information and direction information of the building, thereby improving the accuracy of the construction of the three-dimensional model of the building and the roof attic in the remote sensing image.

[0186] See also Fig.16 , Fig.16 1 is a schematic block diagram of a 3D model construction device for a building provided by the present invention. In this embodiment, a 3D model construction device 100 for a building is provided, and the 3D model construction device 100 for a building comprises a segmentation module 1, a target detection module 2, a height and direction detection module 3 and a construction module 4.

[0187] The segmentation module 1 is used to perform segmentation processing on the acquired remote sensing image to obtain multiple regional images; the remote sensing image includes buildings; the target detection module 2 is used to perform target detection on the multiple regional images respectively to obtain the roof information of the building; the height and direction detection module 3 is used to perform height and direction detection on the multiple regional images respectively to obtain the height information and direction information of the building; the construction module 4 is used to construct a three-dimensional model of the building based on the roof information, height information and direction information of the building.

[0188] The segmentation module 1 crops the acquired remote sensing image by an overlapping cropping method to obtain a plurality of regional images.

[0189] See also Fig.17 , Fig.17 yes Fig.16 A schematic block diagram of a specific embodiment of a target detection module in a three-dimensional model of a building is provided.

[0190] In a specific embodiment, the object detection module 2 includes a first feature extraction module 21 , a roof detection module 22 and a roof mask module 23 .

[0191] The first feature extraction module 21 is used to perform feature extraction on the regional image to obtain regional image features; perform upsampling or downsampling on the regional image features to obtain a preprocessed regional image; perform feature extraction on the preprocessed regional image to obtain corresponding target features.

[0192] The roof detection module 22 is used to extract features from the regional image to obtain target features. Specifically, the roof detection module 22 is used to perform roof detection on the target features corresponding to the pre-processed regional images to obtain a sub-region image containing the roof in the regional image.

[0193] The roof mask module 23 is used to perform image mask processing on the sub-region image according to the target features corresponding to the region image to obtain a roof mask image.

[0194] The target detection module 2 is also used to splice the roof mask images corresponding to the multiple regional images based on the association relationship between the regional image and the remote sensing image, and obtain the roof information corresponding to the remote sensing image. Specifically, the roof information includes the detection frame of the building and the roof outline of the building. The target detection module 2 maps the roof mask image to the remote sensing image based on the positional relationship and size relationship between the regional image and the remote sensing image, and obtains the detection frames of all buildings contained in the remote sensing image and the roof outline of the building contained in the detection frame. And the target detection module 2 is also used to calculate the overlap rate and confidence of the roof mask image corresponding to the regional image in response to the overlap of multiple regional images; retain the roof mask image with low overlap rate and high confidence; and judge whether the angle between each adjacent edge constituting the outline is less than the preset angle; if the angle between two adjacent edges is less than the preset angle, delete the common point of the two edges.

[0195] See also Fig.18 , Fig.18 yes Fig.16 A schematic block diagram of a specific embodiment of a height and direction detection module in a three-dimensional model of a building is provided.

[0196] In a specific embodiment, the height and direction detection module 3 includes a second feature extraction module 31 , a height information extraction module 32 and a direction information extraction module 33 .

[0197] Specifically, the second feature extraction module 31 includes a plurality of second feature extraction units 311 cascaded in sequence, and the second feature extraction unit 311 includes a feature conversion layer 3111, a first feature extraction layer 3112, and a second feature extraction layer 3113 cascaded in sequence. The feature conversion layer 3111 is used to perform feature conversion on the regional image to obtain a feature map of three-dimensional data; the first feature extraction layer 3112 is used to perform feature extraction on the feature map of three-dimensional data to obtain a regional feature map; the second feature extraction layer 3113 is used to perform feature extraction on the regional feature map to obtain a first feature map.

[0198] Specifically, the height information extraction module 32 includes a plurality of height information extraction units 321 cascaded in sequence, and the height information extraction unit 321 includes a first linear layer 3211 and a first activation function layer 3212; the height information extraction module 32 is connected to the second feature extraction module 31. The first linear layer 3211 is used to associate the result output by the previous height information extraction unit 321 with the result output by the corresponding second feature extraction module 31, and then perform feature extraction to obtain a corresponding height feature map; the first activation function layer 3212 is used to decode the height feature map to obtain the height data of each pixel in the regional image.

[0199] Specifically, the direction information extraction module 33 includes a plurality of direction information extraction units 331 cascaded in sequence, and the direction information extraction unit 331 includes a second linear layer 3311 and a second activation function layer 3312; the direction information extraction module 33 is connected to the second feature extraction module 31. The second linear layer 3311 is used to associate the result output by the previous direction information extraction unit 331 with the result output by the corresponding second feature extraction module 31, and then perform feature extraction to obtain a corresponding direction feature map; the second activation function layer 3312 is used to decode the direction feature map to obtain the direction data of each pixel in the regional image.

[0200] The height and direction detection module 3 is also used to determine the direction information and height information of the building based on the direction data and height data of each pixel in the overall grayscale image; it is also used to determine whether there is an overlapping area between the regional images; in response to the overlapping area between at least two regional images, it is used to merge at least two grayscale images corresponding to the overlapping area to obtain a grayscale image corresponding to the overlapping area.

[0201] In a specific embodiment, the height and direction detection module 3 is used to process at least two grayscale images corresponding to the overlapping area according to the following splicing formula.

[0202] The concatenation formula is as follows:

[0203] H i =α*B i +(1-α)*P i (Formula 1)

[0204]

[0205] Where: H i is the height value of the overlapping area; B i is the predicted value of the height of the overlapping area in the first grayscale image, P i is the height prediction value of the second grayscale image, β is a parameter, d represents the distance from pixel i to the overlapping edge; α is the weight, the default value is 0.5.

[0206] Specifically, the construction module 4 is used to determine the foundation position of the building according to the roof information, height information and direction information; and to construct a three-dimensional model of the building according to the foundation position, height information and roof information.

[0207] The three-dimensional model construction device of the building provided in this embodiment includes a segmentation module, a target detection module, a height and direction detection module and a construction module. The segmentation module is used to segment the acquired remote sensing image to obtain multiple regional images; the target detection module is used to perform target detection on the multiple regional images respectively to obtain the roof information of the building; the height and direction detection module is used to perform height and direction detection on the multiple regional images respectively to obtain the height information and direction information of the building; the construction module is used to construct the three-dimensional model of the building based on the roof information, height information and direction information of the building. By segmenting the acquired remote sensing image, it is effectively avoided that the roof, podium or densely connected buildings with different heights are recognized as the same building due to adhesion; by performing target detection and height detection on the regional images respectively, the roof information, height information and direction information of the building are predicted, and the three-dimensional model of the building is constructed according to the predicted roof information, height information and direction information of the building, thereby improving the accuracy of the three-dimensional model construction of the building in the remote sensing image.

[0208] See also Fig.19 , Fig.19 1 is a schematic block diagram of an embodiment of a terminal provided by the present invention. The terminal 70 in this embodiment includes: a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, the method for constructing a three-dimensional model of the building described above is implemented. To avoid repetition, it is not described one by one here.

[0209] See also Fig. 20 , Fig. 20 The embodiment of the present application also provides a computer-readable storage medium 90, which stores a computer program 901. The computer program 901 includes program instructions. The processor executes the program instructions to implement the method for constructing a three-dimensional model of a building provided in the embodiment of the present application.

[0210] The computer-readable storage medium 90 may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium 90 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0211] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for constructing a three-dimensional model of a building, characterized in that: The three-dimensional model construction method of the building comprises: Segmenting the acquired remote sensing image to obtain a plurality of regional images; the remote sensing image includes buildings; Performing target detection on the multiple regional images respectively to obtain roof information of the building; Performing height and direction detection on the multiple area images respectively to obtain height information and direction information of the building; Constructing a three-dimensional model of the building based on the roof information, the height information and the direction information of the building; The step of performing target detection on the plurality of regional images respectively to obtain roof information of the building includes: Extracting features from the regional image to obtain target features; Detecting the target features to obtain a sub-region image including the roof; performing image mask processing on the sub-region image based on the target feature corresponding to the region image to obtain a roof mask image; Based on the association relationship between the regional image and the remote sensing image, the roof mask images corresponding to a plurality of the regional images are spliced ​​to obtain the roof information corresponding to the remote sensing image; The step of performing height and direction detection on the plurality of regional images respectively to obtain the height information and direction information of the building includes: Extract features of the regional image through a height and direction detection network to obtain a grayscale image corresponding to the regional image; Based on the association relationship between each of the regional images and the remote sensing image, the grayscale images corresponding to the multiple regional images are spliced ​​to obtain an overall grayscale image corresponding to the remote sensing image; Based on the overall grayscale image corresponding to the remote sensing image, the direction information and height information of the building are determined.

2. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The acquired remote sensing image is segmented to obtain multiple regional images, including: The acquired remote sensing image is cropped by overlapping cropping method to obtain a plurality of regional images.

3. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The extracting features of the regional image to obtain target features also includes: Extracting features from the regional image to obtain regional image features; Performing upsampling or downsampling processing on the regional image features to obtain a preprocessed regional image; Extracting features from the preprocessed region image to obtain the corresponding target features; The detecting of the target feature to obtain a sub-region image including the roof comprises: Roof detection is performed on the target features respectively corresponding to the preprocessed regional images to obtain a sub-region image containing the roof in the regional image.

4. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The roof information includes a detection frame of a building and a roof outline of the building; The step of splicing the roof mask images respectively corresponding to a plurality of the regional images based on the association relationship between the regional image and the remote sensing image to obtain the roof information corresponding to the remote sensing image includes: Based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image is mapped onto the remote sensing image to obtain detection frames of all the buildings contained in the remote sensing image and roof outlines of the buildings contained in the detection frames.

5. The method for constructing a three-dimensional model of a building according to claim 4, characterized in that: Based on the association between the regional image and the remote sensing image, the roof mask images corresponding to the multiple regional images are spliced ​​to obtain the roof information of the remote sensing image, and further includes: In response to the overlap of a plurality of the area images, calculating the overlap rate and confidence of the roof mask image corresponding to the area image; The roof mask image whose overlap rate is less than a preset overlap rate and whose confidence level is greater than a preset confidence level is retained.

6. The method for constructing a three-dimensional model of a building according to claim 4, characterized in that: Based on the positional relationship and size relationship between the regional image and the remote sensing image, the roof mask image is mapped onto the remote sensing image to obtain the detection frames of all the buildings contained in the remote sensing image and the roof outlines of the buildings contained in the detection frames, and then further comprising: Determining whether the angle between adjacent edges constituting the contour is less than a preset angle; If the angle between two adjacent edges is smaller than the preset angle, the common point of the two edges is deleted.

7. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The performing target detection on the multiple regional images respectively to obtain the roof information of the building includes: The target is detected on the regional image through an instance segmentation network to obtain the roof information of the building; wherein the instance segmentation network includes a first feature extraction module, a target detection module and a semantic segmentation mask module.

8. The method for constructing a three-dimensional model of a building according to claim 7, characterized in that: The step of training the instance segmentation network comprises: Acquire a first training sample set; the first training sample set includes a plurality of first sample images, the first sample images are annotated with a real frame of the target building, a real category of the semantic information in the real frame, a real center position of the roof, and a real mask image of the roof; Detecting the first sample image through an initial target detection network to obtain a predicted frame of the target building, a predicted category of semantic information in the predicted frame, a predicted center position of the roof, and a predicted mask map of the roof; Constructing a first loss function based on the real box and the predicted box of the target building, the real category of the semantic information in the real box and the predicted category of the semantic information in the predicted box, the real center position of the roof and the predicted center position of the roof, the real mask map of the roof and the predicted mask map of the roof; The initial object detection network is iteratively trained using the first loss function to obtain the instance segmentation network.

9. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The height and direction detection network includes a second feature extraction module; the second feature extraction module includes a feature conversion layer, a first feature extraction layer and a second feature extraction layer cascaded in sequence; The step of extracting features from the regional image through a height and direction detection network to obtain a grayscale image corresponding to the regional image includes: The feature conversion layer performs feature conversion on the regional image to obtain a feature map of three-dimensional data; The first feature extraction layer extracts features from the feature map of the three-dimensional data to obtain a regional feature map; The second feature extraction layer performs feature extraction on the regional feature map to obtain a first feature map.

10. The method for constructing a three-dimensional model of a building according to claim 9, characterized in that: The height and direction detection network also includes a height information extraction module, the height information extraction module includes a plurality of height information extraction units cascaded in sequence, the height information extraction unit includes a first linear layer and a first activation function layer; the height information extraction module is connected to the second feature extraction module; The feature extraction of the regional image by the height and direction detection network to obtain a grayscale image corresponding to the regional image also includes: The first linear layer associates the result output by the previous height information extraction unit with the result output by the corresponding second feature extraction module and then performs feature extraction to obtain a corresponding height feature map; The first activation function layer decodes the height feature map to obtain height data of each pixel in the regional image.

11. The method for constructing a three-dimensional model of a building according to claim 10, characterized in that: The height and direction detection network also includes a direction information extraction module, the direction information extraction module includes a plurality of direction information extraction units cascaded in sequence, the direction information extraction unit includes a second linear layer and a second activation function layer; the direction information extraction module is connected to the second feature extraction module; The feature extraction of the regional image by the height and direction detection network to obtain a grayscale image corresponding to the regional image also includes: The second linear layer associates the result output by the previous direction information extraction unit with the result output by the corresponding second feature extraction module and then performs feature extraction to obtain a corresponding direction feature map; The second activation function layer decodes the directional feature map to obtain directional data of each pixel in the regional image.

12. The method for constructing a three-dimensional model of a building according to claim 11, characterized in that: The determining the direction information and height information of the building based on the overall grayscale image corresponding to the remote sensing image includes: The direction information and the height information of the building are determined according to the direction data and the height data of each pixel in the overall grayscale image.

13. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: Based on the association between each of the regional images and the remote sensing image, the grayscale images corresponding to the multiple regional images are spliced ​​to obtain an overall grayscale image corresponding to the remote sensing image, and further includes: Determining whether there is an overlapping area between the regional images; If there is an overlapping area between at least two of the regional images, at least two grayscale images corresponding to the overlapping area are merged to obtain a grayscale image corresponding to the overlapping area.

14. The method for constructing a three-dimensional model of a building according to claim 13, characterized in that: The merging of at least two grayscale images corresponding to the overlapping area to obtain a grayscale image corresponding to the overlapping area includes: At least two grayscale images corresponding to the overlapping area are processed based on the following splicing formula; the splicing formula is as follows: H i = α * B i + (1 - α) * P i (Equation 1) Where: H i is the height value of the overlapping area; B i is the predicted value of the height of the overlapping area in the first grayscale image, P i is the height prediction value of the second grayscale image, β is a parameter, d represents the distance from pixel i to the overlapping edge; α is the weight, the default value is 0.

5.

15. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The steps of training the height and direction detection network include: Acquire a second training sample set; the second training sample set includes a plurality of second sample images, and the real height and real direction of the target building are marked in the second sample images; Detecting the second sample image through an initial feature detection network to obtain a predicted height and a predicted direction of the target building; Constructing a second loss function based on the real height and the predicted height of the target building; constructing a third loss function based on the real direction and the predicted direction of the target building; The initial feature detection network is iteratively trained using the second loss function and the third loss function to obtain the height and direction detection network.

16. The method for constructing a three-dimensional model of a building according to claim 1, characterized in that: The step of constructing a three-dimensional building model based on the roof information, the height information and the direction information of the building comprises: Determining the foundation position of the building according to the roof information, the height information and the direction information; The three-dimensional model of the building is constructed according to the foundation position, the height information, and the roof information.

17. A device for constructing a three-dimensional model of a building, characterized in that: The three-dimensional model building device of the building comprises: A segmentation module is used to segment the acquired remote sensing image to obtain multiple regional images; the remote sensing image includes buildings; The target detection module is used to perform target detection on the multiple regional images respectively to obtain the roof information of the building; to perform feature extraction on the regional image to obtain target features; to detect the target features to obtain a sub-region image containing the roof; to perform image mask processing on the sub-region image based on the target features corresponding to the regional image to obtain a roof mask image; based on the association relationship between the regional image and the remote sensing image, to splice the roof mask images corresponding to the multiple regional images respectively to obtain the roof information corresponding to the remote sensing image; A height and direction detection module is used to perform height and direction detection on the multiple regional images respectively to obtain the height information and direction information of the building; to extract features of the regional images through a height and direction detection network to obtain a grayscale image corresponding to the regional image; based on the association relationship between each of the regional images and the remote sensing image, the grayscale images corresponding to the multiple regional images are spliced ​​to obtain an overall grayscale image corresponding to the remote sensing image; based on the overall grayscale image corresponding to the remote sensing image, the direction information and height information of the building are determined; A construction module is used to construct a three-dimensional model of the building based on the roof information, the height information and the direction information of the building.

18. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor is used to execute program data to implement the steps in the method for constructing a three-dimensional model of a building as claimed in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method for constructing a three-dimensional model of a building as claimed in any one of claims 1 to 16 are implemented.

Images