Method and device for 3D reconstruction of buildings based on single-view non-orthogonal satellite remote sensing images

Abstract

The present invention provides a method and device for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images, including: generating a first building roof outline and a roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model; performing height inversion on the building to obtain height information according to the roof-to-foundation offset; matching the second building roof outline with the first building roof outline to generate a building foundation outline and a building white mold corresponding to the building; visualizing and analyzing the side elevation formed by the second building roof outline and the building foundation outline to extract a texture map corresponding to the building; and performing 3D reconstruction of the building according to the building white mold and the texture map. The present invention can effectively improve the existing 3D reconstruction methods of buildings, such as small application scope, high data quality requirements, large manpower consumption, and long model production cycle.

Description

Technical Field

[0001] The present invention relates to the technical field of building three-dimensional reconstruction, and in particular to a method and device for building three-dimensional reconstruction based on single-view non-orthophoto satellite remote sensing images. Background Art

[0002] With the continuous advancement of urbanization, 3D reconstruction of urban scenes has become a key technology in the fields of geographic information systems, urban planning, virtual reality, etc. The 3D model of the city combines the characteristics of intuitive and vivid information expression of image products with the 3D measurability of topographic maps, which can help researchers better understand the urban spatial structure. As one of the most important ground targets and the most spatially characteristic targets in urban scenes, buildings are the focus of 3D reconstruction of urban scenes.

[0003] At present, the three-dimensional reconstruction of buildings is mainly carried out through technical means such as laser radar, synthetic aperture radar, and optical remote sensing images. The three-dimensional reconstruction based on laser radar is mainly used in vehicle-mounted and airborne sensors, and is more suitable for small-scale, refined, and high-precision three-dimensional reconstruction. The three-dimensional reconstruction method based on synthetic aperture radar can quickly simulate the terrain, but for areas such as buildings that are prone to geometric distortion such as overlap and shadow, higher resolution, more phases, or more perspectives of image data are required for simulation. The reconstruction technology based on optical remote sensing is currently mostly used in aerial photogrammetry, taking photos according to a pre-set route and obtaining a three-dimensional model of the surface through a feature matching algorithm. Most of the above mainstream three-dimensional reconstruction methods have high requirements for the amount of data, imaging angle, and data quality. The limitations of the airborne or vehicle-mounted platforms also make it difficult for these methods to achieve large-scale reconstruction. The complex process requires a lot of manpower, which greatly prolongs the time cycle for generating three-dimensional models.

[0004] Satellite remote sensing images have the characteristics of large data coverage and large historical image reserves. They are one of the most important data sources in remote sensing interpretation. Many traditional methods and machine learning methods have been used to conduct automated extraction research on various types of objects in satellite remote sensing images. Satellite sensors have an incident angle with the ground when imaging, so most satellite remote sensing images are not orthographic. Especially for high-rise buildings, the roof of the building and its actual foundation contour often do not coincide, and on the same image, the taller the building, the greater the offset from the roof to the foundation. This feature makes it possible to use single-view non-orthogonal satellite remote sensing images to perform three-dimensional reconstruction of buildings.

[0005] At present, the interpretation of buildings in satellite remote sensing images is mostly focused on the extraction of two-dimensional contours. A small number of studies use deep learning methods to directly regress the height of buildings in single-view remote sensing images, but since such methods implicitly learn regression clues such as building textures and shadows, they usually require a large amount of data to ensure the generalization ability of the model in different regions. Some researchers have also used stereo mapping satellites or multi-view satellite remote sensing images to conduct three-dimensional reconstruction research, and have achieved good results, but there are still high requirements for image data and corresponding imaging parameters, which limits the large-scale application of these methods. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a method and device for three-dimensional reconstruction of buildings based on single-view non-orthogonal satellite remote sensing images, which can effectively improve the problems of existing three-dimensional reconstruction methods of buildings, such as small application scope, high data quality requirements, high manpower consumption, and long model production cycle.

[0007] In a first aspect, an embodiment of the present invention provides a method for 3D reconstruction of a building based on a single-view non-orthogonal satellite remote sensing image, comprising:

[0008] Obtain a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline;

[0009] Generate the roof outline of the first building and the offset from the roof to the foundation corresponding to the building contained in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model;

[0010] According to the offset from the roof to the foundation, the height of the building is inverted to obtain the corresponding height information of the building;

[0011] Matching the roof outline of the second building with the roof outline of the first building, and assigning the roof-to-foundation offset and height information to the roof outline of the second building based on the matching relationship, so as to generate a building foundation outline and a building white mold corresponding to the building; wherein the accuracy of the roof outline of the second building is higher than the accuracy of the roof outline of the first building;

[0012] Performing a visual analysis on the side elevation formed by the roof outline and the foundation outline of the second building to obtain visibility features corresponding to the side elevation, which are used to extract a texture map corresponding to the building;

[0013] The building is reconstructed in three dimensions based on the building white model and texture map.

[0014] In one embodiment, before generating the first building roof outline and the roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through the instance-level multi-task learning network model, the method further includes:

[0015] Acquire a labeled satellite remote sensing image, wherein the labeled satellite remote sensing image includes a pair of building roof outline annotations and building foundation outline annotations, and determine a roof-to-foundation offset between the building roof outline annotation and the building foundation outline annotation;

[0016] Perform sliding window cropping on the labeled satellite remote sensing images to obtain multiple labeled satellite remote sensing image slices;

[0017] According to the roof-to-foundation offset corresponding to the building roof outline annotation contained in the labeled satellite remote sensing image slice, the building roof outline annotation is moved to the building foundation outline annotation;

[0018] During the moving process, if the top outline of the building foundation exceeds the range of the marked satellite remote sensing image slice, the new roof-to-foundation offset corresponding to the building roof outline is determined, and the building roof outline continues to be moved according to the new roof-to-foundation offset;

[0019] Generate a full building bounding box based on the moved building roof outline annotation and building foundation outline annotation;

[0020] The labeled satellite remote sensing image slices are used as model input, and the building roof outline annotations and their corresponding roof-to-foundation offsets and the entire building bounding box are used as training labels to train the instance-level multi-task learning network model.

[0021] In one embodiment, determining a new roof-to-foundation offset corresponding to a building roof outline annotation includes:

[0022] Based on the relative position relationship between the building roof outline annotation and the building foundation outline annotation, and the relative position relationship between the building roof outline annotation and the annotated satellite remote sensing image slice, a target offset calculation function is determined;

[0023] Through the target offset calculation function, based on the size of the labeled satellite remote sensing image slice, the vertex coordinates of the building roof outline, and the offset from the roof to the foundation, the offset adjustment coefficient is output;

[0024] If the number of target offset calculation functions is 1, the product of the offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset; if the number of target offset calculation functions is multiple, the product of the smallest offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset.

[0025] In one embodiment, generating a first building roof outline and a roof-to-foundation offset corresponding to a building included in a non-orthophoto satellite remote sensing image by using an instance-level multi-task learning network model includes:

[0026] Perform sliding window cropping on non-orthophoto satellite remote sensing images to obtain multiple non-orthophoto satellite remote sensing image slices;

[0027] The non-orthophoto satellite remote sensing image slices are rotated and enhanced to obtain the non-orthophoto satellite remote sensing image slices at multiple rotation angles;

[0028] Generate the roof outline of the first building and the offset from the roof to the foundation corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at each rotation angle through an instance-level multi-task learning network model;

[0029] Performing angular restoration on the first building's roof profile and the roof-to-foundation offset;

[0030] The mean value of the restored roof outline of the first building is used as the roof outline of the first building corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle; and the maximum value of the restored roof-to-foundation offset is used as the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle.

[0031] In one embodiment, matching the second building roof profile with the first building roof profile includes:

[0032] Performing rough matching of the outer bounding box of the second building roof outline and the first building roof outline to select the roughly matched first building roof outline;

[0033] Performing an IoF fine matching of the second building roof profile and the roughly matched first building roof profile to screen out a target first building roof profile that matches the second building roof profile, or to screen out a finely matched first building roof profile;

[0034] The second building roof outline and the precisely matched first building roof outline are nestedly matched based on edge coincidence ratio, so as to replace the second building roof outline with the precisely matched first building roof outline.

[0035] In one embodiment, performing outer bounding box coarse matching of the second building roof outline and the first building roof outline to select a coarse matching first building roof outline includes:

[0036] If the outer bounding box of the second building roof outline intersects with the outer bounding box of the first building roof outline, the first building roof outline is taken as the coarse matching first building roof outline.

[0037] In one embodiment, performing an IoF fine matching of the second building roof profile and the roughly matched first building roof profile to select a target first building roof profile that matches the second building roof profile, or to select a finely matched first building roof profile, includes:

[0038] determining an IoF value between a second building roof profile and a roughly matched first building roof profile;

[0039] If there is a coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, the coarse matching first building roof outline corresponding to the maximum IoF value is used as the target first building roof outline that matches the second building roof outline;

[0040] If there is no coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, determining whether the number of coarse matching first building roof outlines whose IoF values ​​are greater than the second IoF threshold and less than the first IoF threshold is 1;

[0041] If the number is 1, the coarsely matched first building roof outline is used as the target first building roof outline that matches the second building roof outline;

[0042] If the number is greater than 1, the internal IoF filtering is performed on the coarsely matched first building roof outline;

[0043] Determine whether the roof-to-foundation offsets corresponding to the filtered coarsely matched first building roof outlines are all no higher than the offset threshold; if so, use the coarsely matched first building roof outline corresponding to the largest roof-to-foundation offset as the target first building roof outline that matches the second building roof outline; if not, use the filtered coarsely matched first building roof outline as the precise matched first building roof outline.

[0044] In one embodiment, performing edge overlap ratio nested matching on the second building roof outline and the precisely matched first building roof outline to replace the second building roof outline with the precisely matched first building roof outline includes:

[0045] determining an edge overlap ratio between a second building roof profile and an exactly matching first building roof profile;

[0046] Adding the precisely matched first building roof outline corresponding to the highest edge overlap rate to the second building roof outline, and determining the difference between the precisely matched first building roof outline corresponding to the highest edge overlap rate and the second building roof outline;

[0047] The edge overlap rate between the difference set and the remaining precisely matched first building roof outlines is re-determined until all precisely matched first building roof outlines are cycled through, so as to replace the second building roof outline with the precisely matched first building roof outline.

[0048] In one embodiment, a side elevation formed by a roof outline of a second building and a foundation outline of the building is visualized and analyzed to obtain visibility features corresponding to the side elevation, so as to extract a texture map corresponding to the building, including:

[0049] Select two points from the roof outline of the second building, the distance between the two points is greater than a distance threshold, and select two points from the foundation outline of the building to form a side elevation;

[0050] According to the height information corresponding to the roof outline of the second building, linear interpolation is performed from the foundation edge to the roof edge of the side facade to obtain the visibility characteristics corresponding to the side facade;

[0051] Visible side facades are screened out from the side facades based on visibility features, so as to extract texture maps corresponding to the visible side facades from non-orthophoto satellite remote sensing images.

[0052] In a second aspect, an embodiment of the present invention further provides a device for 3D reconstruction of buildings based on single-view non-orthogonal satellite remote sensing images, comprising:

[0053] An acquisition module, used for acquiring a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline;

[0054] A roof outline and offset determination module is used to generate a first building roof outline and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model;

[0055] The height inversion module is used to invert the height of the building according to the offset from the roof to the foundation to obtain the corresponding height information of the building;

[0056] A foundation outline and white mold generation module is used to match the second building roof outline with the first building roof outline, and assign roof-to-foundation offset and height information to the second building roof outline based on the matching relationship, so as to generate a building foundation outline and a building white mold corresponding to the building;

[0057] A texture extraction module is used to perform a visual analysis on the side elevation formed by the roof outline of the second building and the building foundation outline, and obtain the visibility features corresponding to the side elevation to extract the texture map corresponding to the building; wherein the accuracy of the roof outline of the second building is higher than the accuracy of the roof outline of the first building;

[0058] The 3D reconstruction module is used to perform 3D reconstruction of the building based on the building white model and texture map.

[0059] The embodiment of the present invention provides a method and device for three-dimensional reconstruction of a building based on a single-view non-orthophoto satellite remote sensing image. First, a single-view non-orthophoto satellite remote sensing image and a corresponding second building roof outline are obtained, and a first building roof outline and a roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image are generated through an instance-level multi-task learning network model; then, according to the roof-to-foundation offset, the building is height-inverted to obtain height information corresponding to the building; the second building roof outline is matched with the first building roof outline, and the roof-to-foundation offset and height information are assigned to the second building roof outline based on the matching relationship to generate a building foundation outline and a building white mold corresponding to the building; finally, a side elevation formed by the second building roof outline and the building foundation outline is visualized and analyzed to obtain visibility features corresponding to the side elevation, so as to extract a texture map corresponding to the building, thereby performing three-dimensional reconstruction of the building according to the building white mold and the texture map. The above method can process single-view, non-orthophoto satellite remote sensing images, and directly use the instance-level multi-task learning network model to generate a low-precision first building roof outline and a roof-to-foundation offset. By matching the low-precision first building roof outline with the high-precision second building roof outline, the building foundation outline is confirmed and the building white mold is generated. The side elevation formed by the second building roof outline and the building foundation outline is further visualized and analyzed to obtain a texture map for the building white mold, thereby realizing three-dimensional reconstruction of the building. The embodiments of the present invention can effectively improve the existing building three-dimensional reconstruction methods, such as small application scope, high data quality requirements, high manpower consumption, and long model production cycle.

[0060] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A schematic diagram of a flow chart of a method for three-dimensional reconstruction of a building based on a single-view non-orthophoto satellite remote sensing image provided by an embodiment of the present invention;

[0064] Figure 2 A technical flow chart of a method for three-dimensional reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images provided by an embodiment of the present invention;

[0065] Figure 3 An overall flow chart of a method for three-dimensional reconstruction of a building based on a single-view non-orthophoto satellite remote sensing image provided by an embodiment of the present invention;

[0066] Figure 4 A flowchart of training data collection provided by an embodiment of the present invention;

[0067] Figure 5 A multi-level and multi-mode matching strategy for high-precision roof profiles and low-precision roof profiles of buildings provided in an embodiment of the present invention;

[0068] Figure 6 A schematic diagram of an outer bounding box intersection condition provided by an embodiment of the present invention;

[0069] Figure 7 A flow chart of extracting high-visibility side elevations of a building provided by an embodiment of the present invention;

[0070] Figure 8 A schematic diagram of the structure of a device for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images provided by an embodiment of the present invention;

[0071] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] At present, the 3D reconstruction method of buildings based on deep learning requires a large amount of data to ensure the generalization ability of the model in different regions; the 3D reconstruction method of buildings based on stereo mapping satellites or multi-view satellite remote sensing images still has high requirements for image data and corresponding imaging parameters, which limits the large-scale application of these methods. Based on this, the present invention provides a method and device for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images, which can effectively improve the existing 3D reconstruction methods of buildings, such as small application range, high data quality requirements, high manpower consumption, and long model production cycle.

[0074] To facilitate understanding of this embodiment, firstly, a method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images disclosed in an embodiment of the present invention is described in detail. Figure 1 The flowchart of a method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images is shown, and the method mainly includes the following steps S102 to S112:

[0075] Step S102, obtaining a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline.

[0076] In one example, the roof outline of the second building may come from public or manually annotated roof outline data, or may be a roof outline extraction result obtained through methods such as machine learning.

[0077] Step S104, generating a first building roof outline and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model.

[0078] Among them, the input of the instance-level multi-task learning network model is the non-orthogonal satellite remote sensing image, and the output is the full building bounding box, the low-precision roof outline of the first building (referred to as the low-precision building roof outline) and the roof to foundation offset.

[0079] In one example, a non-orthophoto satellite remote sensing image can be subjected to sliding window cropping to obtain multiple non-orthophoto satellite remote sensing image slices; the non-orthophoto satellite remote sensing image slices at the original angle are rotationally enhanced to obtain non-orthophoto satellite remote sensing image slices at multiple rotation angles; the non-orthophoto satellite remote sensing image slices at multiple rotation angles are used as input to an instance-level multi-task learning network model to obtain the corresponding first building roof outline and roof-to-foundation offset; each rotation angle is restored to the original angle, and finally, based on the restored first building roof outline and roof-to-foundation offset, the first building roof outline and roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at the original angle are determined.

[0080] Step S106, performing height inversion on the building according to the offset from the roof to the foundation to obtain the height information corresponding to the building.

[0081] In one example, the height information of all buildings can be quickly inverted based on the geometric relationship between the roof-to-foundation offset and the incident angle, resolution, and building height information of the non-orthophoto satellite remote sensing image, in combination with the known image incident angle or the known height information of a certain building.

[0082] Step S108, matching the second building roof outline with the first building roof outline, and assigning roof-to-foundation offset and height information to the second building roof outline based on the matching relationship, so as to generate a building foundation outline and a building white model corresponding to the building.

[0083] Among them, the accuracy of the second building roof outline is higher than the accuracy of the first building roof outline. The high-precision second building roof outline is referred to as the high-precision building roof outline, and the building foundation outline can also be referred to as the high-precision building foundation outline.

[0084] In one example, by performing multi-level and multi-mode matching on the high-precision roof outline of the second building and the low-precision roof outline of the first building, the roof outline of the first building that matches the second building roof outline can be determined, so that the roof-to-foundation offset and height information corresponding to the building to which the first building roof outline belongs are assigned to the second building roof outline that matches the first building roof outline, and the building foundation outline can be obtained by moving the second building roof outline according to the roof-to-foundation offset, and the building white model can be constructed according to the building foundation outline, the second building roof outline and the height information.

[0085] Step S110 , performing a visual analysis on the side elevation formed by the roof outline and the foundation outline of the second building to obtain visibility features corresponding to the side elevation, which are used to extract a texture map corresponding to the building.

[0086] In one example, according to the height information, linear interpolation is performed from the foundation edge to the roof edge of the side facade to obtain the visibility feature of the side facade. The visibility feature is used to characterize the visibility of the side facade, that is, the higher the visibility feature, the higher the visibility of the side facade. Based on the visibility feature, the visible side facade is determined from the side facade, and the texture map of the visible side facade is extracted.

[0087] Step S112, performing three-dimensional reconstruction of the building based on the building white model and texture map.

[0088] The method for 3D reconstruction of a building based on a single-view non-orthophoto satellite remote sensing image provided by an embodiment of the present invention can process single-view non-orthophoto satellite remote sensing images, directly use an instance-level multi-task learning network model to generate a low-precision first building roof outline and a roof-to-foundation offset, and complete the confirmation of the building foundation outline and the generation of a building white mold by matching the low-precision first building roof outline with the high-precision second building roof outline. Further, a visual analysis is performed on the side elevation formed by the second building roof outline and the building foundation outline to obtain a texture map for the building white mold, thereby realizing 3D reconstruction of the building. The embodiment of the present invention can effectively improve the problems of existing building 3D reconstruction methods such as a small application range, high data quality requirements, high manpower consumption, and a long model production cycle.

[0089] For easier understanding, see Figure 2 The technical flow chart of a method for three-dimensional reconstruction of a building based on a single-view non-orthophoto satellite remote sensing image is shown, comprising: inputting the non-orthophoto satellite remote sensing image into an instance-level multi-task learning network model to obtain a roof-to-foundation offset and a low-precision building roof outline; calculating the building height by combining the roof-to-foundation offset, the image incidence angle or known building height information; combining the low-precision building roof outline and the high-precision building roof outline to perform multi-level and multi-mode matching to obtain a high-precision building foundation outline; generating a building white model by combining the height information obtained by the building height calculation and the high-precision building foundation outline; combining the non-orthophoto satellite remote sensing image, the high-precision building roof outline, and the high-precision building foundation outline to perform visibility-based texture extraction to obtain the building texture; and realizing three-dimensional reconstruction of the building by combining the building white model and the building texture.

[0090] Furthermore, the embodiment of the present invention also provides Figure 3 The overall flow chart of a method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images shown in FIG. 1 includes the following steps S302 to S314:

[0091] Step S302 , collecting training data required for the instance-level multi-task learning network model, including labeled satellite remote sensing images, labeled building roof outlines, full building bounding boxes, and offsets from building roofs to foundations.

[0092] Training the instance-level multi-task learning network model requires cropped satellite remote sensing images and labels for building roof outlines, full building bounding boxes, and roof-to-foundation offsets. In order to reduce the difficulty of labeling, paired building roof outline labels and building foundation outline labels are used as pre-data for making full building bounding boxes and roof-to-foundation offset labels. Since image cropping may cause the loss of important building side elevation information and building foundation outline information in the image, the labels are corrected during the data generation process, see Figure 4 The flowchart of a training data collection shown in the figure includes: performing image window cropping on the labeled satellite remote sensing image to obtain the cropped image, that is, the labeled satellite remote sensing image slice; converting the building roof outline annotation to a raster, whose pixel value is consistent with the key value ID, and further performing roof outline grid window cropping to obtain the cropped roof outline grid; combining the building roof outline annotation and the building foundation outline annotation to calculate the roof to foundation offset, combining the cropped roof outline grid, and finding the corresponding roof to foundation offset according to the ID; moving the cropped roof outline grid to the building foundation outline annotation, and judging whether the offset building foundation outline annotation exceeds the image boundary; if so, calculating the new roof to foundation offset, and calculating the whole building bounding box of the roof and foundation as a whole, if not, directly calculating the whole building bounding box of the roof and foundation as a whole; finally obtaining the labels: building roof outline, roof to foundation offset, whole building bounding box.

[0093] For ease of understanding, please refer to steps 1.1 to 1.7 below:

[0094] Step 1.1, obtaining labeled satellite remote sensing images, wherein the labeled satellite remote sensing images include paired building roof outline annotations and building foundation outline annotations.

[0095] In one example, pairs of building roof outline annotations and building foundation outline annotations are obtained. Generally, public data usually gives one of the roof or foundation outlines, and manual annotations usually mark the roof outline. Assuming that there are already building roof outline annotations, based on the assumption that most building roof outlines and foundation outlines are similar on satellite remote sensing images, the building roof outline annotations are moved to the foundation position through partially visible building foundation outlines or side elevation information, and the paired building roof outline annotations and building foundation outline annotations are stored in a vector file. At the same time, a key value field ID is established (ID cannot be 0), and the roof outline records and foundation outline records with the same ID are corresponding.

[0096] Step 1.2, determine the roof-to-foundation offset between the building roof outline marking and the building foundation outline marking.

[0097] In one example, find each pair of building roof outline annotations and building foundation outline annotations based on the key field ID, and calculate the roof to foundation offset in the geographic coordinate system. , , and recorded in a vector file.

[0098] Step 1.3, convert the vector annotation of the roof outline into a raster format. The label of each raster patch is consistent with the key value ID in the vector annotation, which facilitates the query of the offset from the roof to the foundation.

[0099] Step 1.4, performing sliding window cropping processing on the labeled satellite remote sensing image to obtain multiple labeled satellite remote sensing image slices.

[0100] In one example, the same clipping strategy is used to clip labeled satellite remote sensing images and raster-based building roof outline labels into fixed-size slices.

[0101] Step 1.5, read the cropped building roof outline annotation, find the roof to foundation offset corresponding to the same ID in the roof vector file, and convert the roof to foundation offset to the offset along the x-axis and y-axis in the pixel coordinate system according to the image's affine matrix and .when When , the foundation outline is on the left side of the roof outline, otherwise, the foundation outline is on the right side of the roof outline. When the foundation is above the roof outline, the foundation outline is above the roof outline, otherwise, the foundation outline is below the roof outline.

[0102] Step 1.6, move the building roof outline annotation to the building foundation outline annotation according to the roof-to-foundation offset corresponding to the building roof outline annotation contained in the labeled satellite remote sensing image slice; during the movement, if the building foundation outline annotation exceeds the range of the labeled satellite remote sensing image slice, determine the new roof-to-foundation offset corresponding to the building roof outline annotation, and continue to move the building roof outline annotation according to the new roof-to-foundation offset.

[0103] In one example, the building roof outline annotation is moved to the building foundation outline annotation according to the original roof-to-foundation offset. If the building foundation outline annotation exceeds the range of the cropped image, it is moved in the direction of the original roof-to-foundation offset until it first touches the image boundary, and the new roof-to-foundation offset is recorded. The process of determining the new roof-to-foundation offset is as follows:

[0104] (1) Based on the relative position relationship between the building roof outline annotation and the building foundation outline annotation, and the relative position relationship between the building roof outline annotation and the annotated satellite remote sensing image slice, determine the target offset calculation function. For details, please refer to the following cases 1 to 4:

[0105] Case 1: If , that is, the foundation is on the left side of the roof and exceeds the left boundary of the image:

[0106] ;

[0107] Case 2: If , that is, the foundation is on the right side of the roof and exceeds the right boundary of the image:

[0108] ;

[0109] Case 3: If , that is, the foundation is above the roof and exceeds the upper boundary of the image:

[0110] ;

[0111] Case 4: If , that is, the foundation is below the roof and beyond the lower boundary of the image:

[0112] ;

[0113] in, is the offset adjustment factor, which can be understood as the ratio of the new roof-to-foundation offset to the original roof-to-foundation offset. is the size of the labeled satellite remote sensing image slice, that is, the cropped image size. , The vertex coordinates of the building roof outline are marked. , is the original roof to foundation offset.

[0114] (2) Through the target offset calculation function, the offset adjustment coefficient is output based on the size of the labeled satellite remote sensing image slice, the vertex coordinates of the building roof outline, and the offset from the roof to the foundation.

[0115] (3) If the number of target offset calculation functions is one, the product of the offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset; if the number of target offset calculation functions is multiple, the product of the smallest offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset.

[0116] For example, the foundation is on the left side of the roof. In this case, the function shown in Case 1 is used as the target offset calculation function, and the product of the offset adjustment coefficient output by the function and the roof-to-foundation offset is directly used as the new roof-to-foundation offset. For another example, the foundation is on the upper left side of the roof, which satisfies Case 1 and Case 3. In this case, the functions shown in Case 1 and Case 3 are both target offset calculation functions, and two offset adjustment coefficients are obtained. In this case, the product of the smaller offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset.

[0117] The new roof to foundation offset calculation formula is as follows:

[0118] ;

[0119] Typically, the new roof-to-foundation offset is less than the original roof-to-foundation offset.

[0120] Step 1.7, generating a full building bounding box based on the moved building roof outline annotation and building foundation outline annotation.

[0121] In one example, the roof outline is moved to the foundation according to the new offset, and the full building bounding box is calculated based on the roof outline range and the foundation outline range. From then on, the cropped satellite remote sensing image required for model training and the building roof outline, full building bounding box, and offset labels from the building roof to the foundation are obtained.

[0122] Step S304, training the instance-level multi-task learning network model so that it can achieve the building roof extraction task and the roof-to-foundation offset estimation task. In one embodiment, the labeled satellite remote sensing image slices are used as model input, the building roof outline annotations and their corresponding roof-to-foundation offsets and the full building bounding box are used as training labels to train the instance-level multi-task learning network model.

[0123] Specifically, the labeled satellite remote sensing image slices and corresponding labels are fed into a two-stage instance-level multi-task learning network model. The first stage quickly identifies areas that may contain buildings and regresses their precise locations, providing candidate area ranges for the second stage; the second stage further analyzes the feature maps within the candidate area range provided in the first stage, regresses the bounding box, category, contour, and offset respectively; calculates the loss with the true value, and performs gradient backpropagation to optimize the parameters in the model. Iterate the optimization until the model converges.

[0124] Step S306, sliding the non-orthogonal satellite remote sensing images block by block into the instance-level multi-task learning network model to obtain the low-precision building roof outline and the roof-to-foundation offset.

[0125] Since the estimation task of the roof-to-foundation offset depends largely on the visible building side elevation information and part of the foundation contour information on the image, it is inevitable to lose some or even all of the side elevation information when performing sliding window reasoning on the image, which may lead to an underestimate or even wrong estimation of the offset. During the reasoning process, follow the steps 3.1 to 3.3 below:

[0126] Step 3.1, perform sliding window cropping on the non-orthophoto satellite remote sensing image to obtain multiple non-orthophoto satellite remote sensing image slices.

[0127] In one example, the sliding windows of non-orthophoto satellite remote sensing images that originally did not overlap were padded so that there were overlapping areas between different windows. The scale of the pad should be larger than the maximum unidirectional offset of the building to ensure that at least one of the two adjacent windows can cover the entire building. In practice, each window is padded by 128.

[0128] Step 3.2, through the instance-level multi-task learning network model, generate the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at the original angle. The details are as follows:

[0129] Firstly, the non-orthophoto satellite remote sensing image slices are rotated and enhanced to obtain the non-orthophoto satellite remote sensing image slices at multiple rotation angles;

[0130] Then, the first building roof outline and the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at each rotation angle are generated through an instance-level multi-task learning network model;

[0131] Finally, the angle of the roof outline of the first building and the offset from the roof to the foundation are restored; the average value of the restored roof outline of the first building is used as the roof outline of the first building corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle; and the maximum value of the restored roof-to-foundation offset is used as the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle.

[0132] In practical applications, considering that the direction of the offset vector of the training data may be relatively single, the model has a biased estimate of the offset in a certain direction. Therefore, before the sliding window image enters the model, the non-orthophoto satellite remote sensing image slice is rotationally enhanced, and the rotation-enhanced non-orthophoto satellite remote sensing image slice is input into the instance-level multi-task learning network model to obtain the roof outline of the first building and the roof-to-foundation offset at multiple rotation angles. Then, it is restored to the original angle according to the rotation matrix, and the mean of the roof outline of the first building and the maximum value of the roof-to-foundation offset are obtained as output. The output is the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle. The roof-to-foundation offset is calculated by and Indicates.

[0133] Step 3.3: Within a certain range, buildings in satellite remote sensing images tend to maintain the same offset direction. The roof-to-foundation offset estimates within a certain orientation are statistically analyzed to obtain the mean and standard deviation of the roof-to-foundation offset. Outliers that are inconsistent with the mean direction and exceed 3 times the standard deviation are considered to be estimated incorrectly, and these results are discarded.

[0134] Step S308, combining the known image incident angle or the known height of a certain building, and according to the geometric relationship between the roof-to-foundation offset and the image incident angle, resolution, and building height, the height information of all buildings is quickly inverted.

[0135] The following geometric relationship exists between building height, image incidence angle, resolution, and roof-to-foundation offset:

[0136] ;

[0137] in, is the building height, is the image incident angle, is the image resolution, usually in meters / pixel, is the angle between the offset vector and the pixel coordinate x-axis. Usually the image resolution s is known, and the offset vector can be estimated in step S103. As long as the image incident angle is known, the height of the building can be calculated, specifically according to the following steps 4.1 to 4.2:

[0138] Step 4.1, obtain the incident angle of the image. If the incident angle information is available, it can be used directly. If there is no incident angle information, the incident angle can be estimated by the known height of a certain building:

[0139] ;

[0140] When there are multiple known building heights, the average can be taken to reduce the estimation error of the image incidence angle.

[0141] Step 4.2: Based on the relationship between the roof-to-foundation offset and the building height, image incidence angle, and resolution, other building heights can be quickly estimated:

[0142] .

[0143] Step S310, in combination with the high-precision building roof outline, a multi-level and multi-mode matching strategy is adopted to achieve matching with the low-precision building roof outline in the instance-level multi-task learning model, and the high-precision building roof outline is moved according to the roof-to-foundation offset to obtain the high-precision building foundation outline, and the height information is given to construct a white model of the building.

[0144] In the instance-level multi-task learning network model, since the feature map of the candidate area will be scaled at a fixed scale in the second stage, the building roof outline obtained by the instance-level multi-task learning network model has low accuracy and rough edge contours, which is not conducive to the subsequent construction of the three-dimensional model. Therefore, consider replacing the low-precision building roof outline output by the instance-level multi-task learning network model with a higher-precision building roof outline. The specific steps 5.1 to 5.3 are as follows:

[0145] Step 5.1, obtaining a high-precision building roof outline, which can be from public or manually annotated roof outline data, or can be a roof outline extraction result obtained by machine learning and other methods. In this example, a semantic-level high-precision building roof outline is obtained by a deep learning method;

[0146] Step 5.2, using multi-level and multi-method matching of the high-precision building roof outline and the low-precision building roof outline output by the instance-level multi-task learning network model, and assigning the roof-to-foundation offset and height information to the high-precision building roof outline;

[0147] Step 5.3, move the high-precision building roof outline to the foundation position according to the matched roof-to-foundation offset to obtain the high-precision building foundation outline, and construct a building white model by combining the high-precision building roof outline, high-precision building foundation outline, and height information.

[0148] In step 5.2, since there are differences between the high-precision building roof outline and the low-precision building roof outline output by the instance-level multi-task learning network model, it is impossible to achieve a one-to-one match. Therefore, a multi-level and multi-faceted approach is used to achieve outline matching and attribute assignment, such as Figure 5 The multi-level and multi-mode matching strategy of high-precision roof outline and low-precision roof outline of a building is shown, including three parts: rough matching of outer bounding box, fine matching of outline IoF, and nested matching of edge overlap rate. The specific steps are as follows:

[0149] (1) The outer bounding box of the second building roof outline is roughly matched with the outer bounding box of the first building roof outline to select the roughly matched first building roof outline. In one example, if the outer bounding box of the second building roof outline intersects with the outer bounding box of the first building roof outline, the first building roof outline is used as the roughly matched first building roof outline, that is, the low-precision building roof outline after the outer bounding box is roughly matched.

[0150] Specifically, an outer bounding box is used for coarse matching to determine whether each high-precision building roof outline intersects with the outer bounding box of the low-precision building roof outline. This step can greatly reduce the number of subsequent matching times and speed up the matching process.

[0151] In the embodiment of the present invention, whether two outer bounding boxes intersect is determined by the relationship between the outer bounding box coordinates, such as Figure 6Schematic diagram of the intersection condition of the outer bounding box. If two outer bounding boxes intersect, they should simultaneously meet the following conditions: Condition 1, the upper border (A) of the lower one of the two boxes should be above the lower border (B) of the upper one of the two boxes, that is, A < B; Condition 2, the left border (C) of the right one of the two boxes should be to the left of the right border (D) of the left one of the two boxes, that is, C < D. After rough matching of the outer bounding boxes, the low-precision building roof contours that may match each high-precision building roof contour are obtained. If there is no intersection with the outer bounding box of any low-precision building roof contour, a 0 offset is assigned to it.

[0152] (2) Perform fine matching of the contour IoF (Intersection over Frame) between the second building roof contour and the roughly matched first building roof contour to screen out the target first building roof contour that matches the second building roof contour, or screen out the finely matched first building roof contour. Specifically:

[0153] (2.1) Determine the IoF value between the second building roof contour and the roughly matched first building roof contour. In one example, fine matching is performed using the contour IoF. Based on the rough matching, calculate the ratio of the intersection of each building roof contour and each possible matching building roof contour (i.e., the roughly matched first building roof contour) to its own area, which is called IoF here.

[0154] (2.2) If there is a roughly matched first building roof contour with an IoF value greater than the first IoF threshold, then take the roughly matched first building roof contour corresponding to the maximum IoF value as the target first building roof contour that matches the second building roof contour.

[0155] In one example, when the IoF is greater than the threshold of 0.75, it is considered that the matching degree is good, and the building roof contour with the largest offset is selected as the matching object, and this matching object is the target first building roof contour that matches the second building roof contour.

[0156] (2.3) If there is no roughly matched first building roof contour with an IoF value greater than the first IoF threshold, then judge whether the number of roughly matched first building roof contours with an IoF value greater than the second IoF threshold and less than the first IoF threshold is 1.

[0157] In one example, when the IoF is less than the threshold of 0.75 and not lower than the limit of 0.2, judge whether the number of low-precision building roof contours matched is 1.

[0158] (2.4) If the number is 1, then take the roughly matched first building roof contour as the target first building roof contour that matches the second building roof contour.

[0159] In one example, if the number of matched low-precision building roof outlines is 1, it is taken as a matching object.

[0160] (2.5) If the number is greater than 1, the coarse matching first building roof outline is subjected to internal IoF filtering; it is determined whether the roof-to-foundation offsets corresponding to the filtered coarse matching first building roof outline are not higher than the offset threshold; if so, the coarse matching first building roof outline corresponding to the largest roof-to-foundation offset is used as the target first building roof outline to be matched with the second building roof outline; if not, the filtered coarse matching first building roof outline is used as the precise matching first building roof outline, that is, the low-precision building roof outline after the precise matching of the outline IoF.

[0161] In one example, if the number of matched low-precision building roof outlines is not 1, that is, multiple low-precision building roof outlines are matched, these low-precision building roof outlines are first internally filtered by IoF to reduce repeated outlines and broken outlines caused by pad clipping, and then determine whether the roof-to-foundation offsets corresponding to several low-precision building roof outlines are consistent. For example, if the roof-to-foundation offset is not higher than the threshold value 5, it is considered that the roof-to-foundation offsets corresponding to the low-precision building roof outlines are consistent; otherwise, they are inconsistent. Based on this, when the maximum roof-to-foundation offset difference is not higher than the threshold value 5, the low-precision building roof outline with the largest roof-to-foundation offset is matched to the high-precision building roof outline as the matching object; otherwise, edge overlap rate nested matching is performed.

[0162] (3) Performing edge overlap ratio nested matching on the roof outline of the second building and the roof outline of the precisely matched first building, so as to replace the roof outline of the second building with the roof outline of the precisely matched first building.

[0163] The edge overlap ratio is used for nested matching. The patches that have not been successfully matched after precise matching can be interpreted as containing multiple low-precision building roof outlines, and these outlines do not belong to the same building because the difference in their roof-to-foundation offset is large. Specifically:

[0164] (3.1) Determine the edge overlap ratio between the second building roof profile and the exactly matched first building roof profile.

[0165] In one example, multiple low-precision building roof outlines are calculated and compared with the current high-precision building roof outline. The edge overlap rate is defined as the ratio of the points within the low-precision building roof outline to the points within the high-precision building roof outline whose closest distance is less than the threshold to all the points contained in the low-precision building roof outline. The formula is as follows:

[0166] ;

[0167] is located at the low-resolution building roof outline The point on It is located at the high-precision building roof outline The point on is the distance threshold, Refers to low-precision building roof outlines The number of all contour points on .

[0168] (3.2) Add the precisely matched first building roof outline corresponding to the highest edge overlap rate to the second building roof outline, and determine the difference between the precisely matched first building roof outline corresponding to the highest edge overlap rate and the second building roof outline.

[0169] (3.3) Re-determine the edge overlap rate between the difference set and the remaining precisely matched first building roof outlines until all precisely matched first building roof outlines are cycled through, so as to replace the second building roof outline with the precisely matched first building roof outline.

[0170] In one example, the low-resolution building roof outline with the highest overlap rate is selected , add the low-precision building roof outline to the high-precision building roof outline, and assign the corresponding roof-to-foundation offset of the low-precision building roof outline. Calculate the high-precision building roof outline With low-precision building roof outlines The largest part of the difference and the remaining low-precision building roof outlines are selected as the input of the nested matching, and the execution is performed until all the low-precision building roof outlines are cycled; and the original high-precision building roof outlines will be replaced.

[0171] Step S312, visually analyzing the side facades formed by the high-precision building roof outline and the high-precision building foundation outline, determining the visible range of each side facade, and selecting facade textures and roof textures with high visibility for texture mapping.

[0172] The side facades of buildings are not all visible in non-orthophoto satellite remote sensing images. In order to apply the correct texture map to the white model of the building, it is necessary to ensure that the extracted texture map only contains the texture of the current side facade, and does not contain the texture of the building roof and other side facades. Figure 7 A flowchart of extracting high-visibility side facades of a building is shown. The visibility index of the side facade is constructed to extract the complete side facade texture. The specific steps 6.1 to 6.3 are as follows:

[0173] Step 6.1, select two points from the roof outline of the second building, the distance between the two points is greater than the distance threshold, and select two points from the foundation outline of the building to form a side facade.

[0174] In one example, to construct a side elevation quadrilateral of a building, two adjacent points are selected on the high-precision building roof outline. Note that the distance between the two points should be greater than a threshold. Two corresponding points are selected on the high-precision building foundation outline. The four points constitute a side elevation quadrilateral, which is converted to pixel coordinates for subsequent pixel-level feature calculations.

[0175] Step 6.2: According to the height information corresponding to the roof outline of the second building, linear interpolation is performed from the foundation edge to the roof edge of the side facade to obtain the visibility characteristics corresponding to the side facade.

[0176] In one example, the side elevation visibility feature is constructed, and according to the height information of the building, linear interpolation is performed from 0 to the height value from the foundation edge of the side elevation quadrilateral to the roof edge to obtain the visibility feature of the elevation.

[0177] Step 6.3, screen out visible side facades from the side facades based on visibility features, so as to extract texture maps corresponding to the visible side facades from the non-orthophoto satellite remote sensing images.

[0178] First, the visibility features of all side facades are stitched together. The higher the visibility feature value, the higher the visibility of the facade. The area with the highest visibility feature value is the area where the facade can be directly seen in the image. The number of pixels containing the maximum visibility feature value in each side facade is counted, and the ratio of it to the number of pixels of the side facade is calculated. If the ratio exceeds the threshold, the facade is considered to be a visible facade. If two visible facades are adjacent, the two side facade quadrilaterals are merged.

[0179] Then, the texture covered by the visible facade quadrilateral in the non-orthophoto satellite remote sensing image is taken out and transformed into a rectangular texture through affine transformation, which is used for texture mapping of the building white mold.

[0180] Step S314, performing three-dimensional reconstruction of the building based on the building white model and texture map.

[0181] To summarize, the embodiments of the present invention provide a method for three-dimensional reconstruction of buildings based on single-view non-orthogonal satellite remote sensing images to address technical and practical application problems such as the small application scope, high data quality requirements, high manpower consumption, and long model production cycle of the three-dimensional reconstruction method described in the background technology. The embodiment of the present invention is based on the assumption that buildings are basically affine transformations on satellite remote sensing images, and provides a set of rapid three-dimensional reconstruction process methods for buildings based on single-view non-orthophoto satellite remote sensing images, including: offset estimation, using a two-stage instance-level multi-task learning framework, combining full-building bounding box regression and building roof contour extraction, establishing explicit clues for the offset estimation task from the building roof to the foundation, and improving the accuracy of the model for offset estimation; height estimation, combining the image incidence angle or the known height of a certain building, according to the geometric relationship between the offset and the building height, the height information of all buildings can be quickly inverted; model construction, combining a high-precision building roof segmentation model, using a multi-level and multi-mode matching strategy, achieving matching with a low-precision building roof segmentation model in an instance-level multi-task learning model, giving the building contour three-dimensional height information, and constructing an accurate building white model; texture mapping, performing overlapping analysis on the side facade formed by the roof contour and the foundation contour, and selecting the facade for texture mapping based on visibility. The process method described in the embodiment of the present invention can achieve large-scale, rapid, and automated three-dimensional reconstruction of buildings.

[0182] Based on the above embodiments, the present invention provides a device for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images. Figure 8 The schematic diagram of the structure of a building 3D reconstruction device based on single-view non-orthophoto satellite remote sensing images is shown in FIG. The device mainly includes the following parts:

[0183] An acquisition module 802 is used to acquire a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline;

[0184] The roof profile and offset determination module 804 is used to generate a first building roof profile and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model;

[0185] The height inversion module 806 is used to perform height inversion on the building according to the offset from the roof to the foundation to obtain the height information corresponding to the building;

[0186] The foundation outline and white mold generation module 808 is used to match the second building roof outline with the first building roof outline, and assign the roof-to-foundation offset and height information to the second building roof outline based on the matching relationship, so as to generate the building foundation outline and the building white mold corresponding to the building; wherein the accuracy of the second building roof outline is higher than the accuracy of the first building roof outline;

[0187] A texture extraction module 810 is used to perform a visual analysis on the side elevation formed by the roof outline and the foundation outline of the second building to obtain visibility features corresponding to the side elevation, so as to extract a texture map corresponding to the building;

[0188] The three-dimensional reconstruction module 812 is used to perform three-dimensional reconstruction of the building based on the building white model and texture map.

[0189] The device for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images provided by the embodiment of the present invention can process single-view non-orthophoto satellite remote sensing images, directly use the instance-level multi-task learning network model to generate a low-precision first building roof outline and a roof-to-foundation offset, and complete the confirmation of the building foundation outline and the generation of the building white mold by matching the low-precision first building roof outline with the high-precision second building roof outline. The side elevation formed by the second building roof outline and the building foundation outline is further visualized and analyzed to obtain a texture map for the building white mold, thereby realizing 3D reconstruction of the building. The embodiment of the present invention can effectively improve the problems of the existing building 3D reconstruction method, such as small application scope, high data quality requirements, large manpower consumption, and long model production cycle.

[0190] In one embodiment, a model training module is further included, which is used to:

[0191] Acquire a labeled satellite remote sensing image, wherein the labeled satellite remote sensing image includes a pair of building roof outline annotations and building foundation outline annotations, and determine a roof-to-foundation offset between the building roof outline annotation and the building foundation outline annotation;

[0192] Perform sliding window cropping on the labeled satellite remote sensing images to obtain multiple labeled satellite remote sensing image slices;

[0193] According to the roof-to-foundation offset corresponding to the building roof outline annotation contained in the labeled satellite remote sensing image slice, the building roof outline annotation is moved to the building foundation outline annotation;

[0194] During the moving process, if the building foundation outline exceeds the range of the marked satellite remote sensing image slice, the new roof-to-foundation offset corresponding to the building roof outline is determined, and the building roof outline continues to be moved according to the new roof-to-foundation offset;

[0195] Generate a full building bounding box based on the moved building roof outline annotation and building foundation outline annotation;

[0196] The labeled satellite remote sensing image slices are used as model input, and the building roof outline annotations and their corresponding roof-to-foundation offsets and the entire building bounding box are used as training labels to train the instance-level multi-task learning network model.

[0197] In one embodiment, the model training module is specifically used to:

[0198] Based on the relative position relationship between the building roof outline annotation and the building foundation outline annotation, and the relative position relationship between the building roof outline annotation and the annotated satellite remote sensing image slice, a target offset calculation function is determined;

[0199] Through the target offset calculation function, based on the size of the labeled satellite remote sensing image slice, the vertex coordinates of the building roof outline, and the offset from the roof to the foundation, the offset adjustment coefficient is output;

[0200] If the number of target offset calculation functions is 1, the product of the offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset; if the number of target offset calculation functions is multiple, the product of the smallest offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset.

[0201] In one implementation, the roof profile and offset determination module 804 is specifically configured to:

[0202] Perform sliding window cropping on non-orthophoto satellite remote sensing images to obtain multiple non-orthophoto satellite remote sensing image slices;

[0203] The non-orthophoto satellite remote sensing image slices are rotated and enhanced to obtain the non-orthophoto satellite remote sensing image slices at multiple rotation angles;

[0204] Generate the roof outline of the first building and the offset from the roof to the foundation corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at each rotation angle through an instance-level multi-task learning network model;

[0205] Performing angular restoration on the first building's roof profile and the roof-to-foundation offset;

[0206] The mean value of the restored roof outline of the first building is used as the roof outline of the first building corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle; and the maximum value of the restored roof-to-foundation offset is used as the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice of the original angle.

[0207] In one embodiment, the foundation contour and white model generation module 808 is specifically used to:

[0208] Performing rough matching of the outer bounding box of the second building roof outline and the first building roof outline to select the roughly matched first building roof outline;

[0209] Performing an IoF fine matching of the second building roof profile and the roughly matched first building roof profile to screen out a target first building roof profile that matches the second building roof profile, or to screen out a finely matched first building roof profile;

[0210] The second building roof outline and the precisely matched first building roof outline are nestedly matched based on edge coincidence ratio, so as to replace the second building roof outline with the precisely matched first building roof outline.

[0211] In one embodiment, the foundation contour and white model generation module 808 is specifically used to:

[0212] If the outer bounding box of the second building roof outline intersects with the outer bounding box of the first building roof outline, the first building roof outline is taken as the coarse matching first building roof outline.

[0213] In one embodiment, the foundation contour and white model generation module 808 is specifically used to:

[0214] determining an IoF value between a second building roof profile and a roughly matched first building roof profile;

[0215] If there is a coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, the coarse matching first building roof outline corresponding to the maximum IoF value is used as the target first building roof outline that matches the second building roof outline;

[0216] If there is no coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, determining whether the number of coarse matching first building roof outlines whose IoF values ​​are greater than the second IoF threshold and less than the first IoF threshold is 1;

[0217] If the number is 1, the coarsely matched first building roof outline is used as the target first building roof outline that matches the second building roof outline;

[0218] If the number is greater than 1, the internal IoF filtering is performed on the coarsely matched first building roof outline;

[0219] Determine whether the roof-to-foundation offsets corresponding to the filtered coarsely matched first building roof outlines are all no higher than the offset threshold; if so, use the coarsely matched first building roof outline corresponding to the largest roof-to-foundation offset as the target first building roof outline that matches the second building roof outline; if not, use the filtered coarsely matched first building roof outline as the precise matched first building roof outline.

[0220] In one embodiment, the foundation contour and white model generation module 808 is specifically used to:

[0221] determining an edge overlap ratio between a second building roof profile and an exactly matching first building roof profile;

[0222] Adding the precisely matched first building roof outline corresponding to the highest edge overlap rate to the second building roof outline, and determining the difference between the precisely matched first building roof outline corresponding to the highest edge overlap rate and the second building roof outline;

[0223] The edge overlap rate between the difference set and the remaining precisely matched first building roof outlines is re-determined until all precisely matched first building roof outlines are cycled through, so as to replace the second building roof outline with the precisely matched first building roof outline.

[0224] In one implementation, the texture extraction module 810 is specifically used to:

[0225] Select two points from the roof outline of the second building, the distance between the two points is greater than a distance threshold, and select two points from the foundation outline of the building to form a side elevation;

[0226] According to the height information corresponding to the roof outline of the second building, linear interpolation is performed from the foundation edge to the roof edge of the side facade to obtain the visibility characteristics corresponding to the side facade;

[0227] Visible side facades are screened out from the side facades based on visibility features, so as to extract texture maps corresponding to the visible side facades from non-orthophoto satellite remote sensing images.

[0228] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0229] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0230] Fig. 9A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 90, a memory 91, a bus 92 and a communication interface 93, wherein the processor 90, the communication interface 93 and the memory 91 are connected via the bus 92; the processor 90 is used to execute an executable module stored in the memory 91, such as a computer program.

[0231] The memory 91 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 93 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0232] The bus 92 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0233] Among them, the memory 91 is used to store programs, and the processor 90 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiments of the present invention can be applied to the processor 90 or implemented by the processor 90.

[0234] The processor 90 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 90. The above processor 90 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 91, and the processor 90 reads the information in the memory 91 and completes the steps of the above method in combination with its hardware.

[0235] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0236] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0237] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images, characterized in that: include: Obtain a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline; Generate a first building roof outline and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model; According to the offset from the roof to the foundation, performing height inversion on the building to obtain height information corresponding to the building; Matching the second building roof outline with the first building roof outline, and assigning the roof-to-foundation offset and the height information to the second building roof outline based on the matching relationship, so as to generate a building foundation outline and a building white mold corresponding to the building; wherein the accuracy of the second building roof outline is higher than that of the first building roof outline; Performing a visual analysis on the side elevation formed by the roof outline of the second building and the foundation outline of the building to obtain visibility features corresponding to the side elevation, so as to extract a texture map corresponding to the building; Reconstructing the building in three dimensions according to the building white model and the texture map; Matching the second building roof outline with the first building roof outline comprises: Performing outer bounding box rough matching of the second building roof outline and the first building roof outline to screen out a roughly matched first building roof outline; Performing an IoF fine matching on the second building roof profile and the roughly matched first building roof profile, so as to screen out a target first building roof profile that matches the second building roof profile, or screen out a finely matched first building roof profile; Performing edge overlap ratio nested matching on the second building roof outline and the precisely matched first building roof outline, so as to replace the second building roof outline with the precisely matched first building roof outline; Performing an IoF fine matching on the second building roof profile and the roughly matched first building roof profile to screen out a target first building roof profile that matches the second building roof profile, or to screen out a finely matched first building roof profile, comprising: Determine an IoF value between the second building roof outline and the roughly matched first building roof outline; wherein the ratio of the intersection of the second building roof outline and the roughly matched first building roof outline to its own area is called IoF; If there is a coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, taking the coarse matching first building roof outline corresponding to the maximum IoF value as the target first building roof outline that matches the second building roof outline; If there is no coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, determining whether the number of coarse matching first building roof outlines whose IoF value is greater than the second IoF threshold and less than the first IoF threshold is 1; If the number is 1, the coarsely matched first building roof outline is used as the target first building roof outline that matches the second building roof outline; If the number is greater than 1, internal IoF filtering is performed on the coarsely matched first building roof outline; Determine whether the roof-to-foundation offsets corresponding to the filtered coarsely matched first building roof outlines are all not higher than the offset threshold; if so, use the coarsely matched first building roof outline corresponding to the largest roof-to-foundation offset as the target first building roof outline that matches the second building roof outline; if not, use the filtered coarsely matched first building roof outline as the precise match first building roof outline.

2. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 1, characterized in that: Before generating the first building roof outline and the roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through the instance-level multi-task learning network model, the method further includes: Acquire a labeled satellite remote sensing image, wherein the labeled satellite remote sensing image includes a pair of building roof outline annotations and building foundation outline annotations, and determine a roof-to-foundation offset between the building roof outline annotation and the building foundation outline annotation; Performing sliding window cropping processing on the labeled satellite remote sensing image to obtain a plurality of labeled satellite remote sensing image slices; According to the roof-to-foundation offset corresponding to the building roof outline annotation contained in the labeled satellite remote sensing image slice, the building roof outline annotation is moved to the building foundation outline annotation; During the moving process, if the building foundation outline annotation exceeds the range of the annotated satellite remote sensing image slice, a new roof-to-foundation offset corresponding to the building roof outline annotation is determined, and the building roof outline annotation is continued to be moved according to the new roof-to-foundation offset; Generate a full building bounding box according to the moved building roof outline marking and building foundation outline marking; The labeled satellite remote sensing image slice is used as a model input, and the building roof outline annotation and its corresponding roof-to-foundation offset and the full building bounding box are used as training labels to train the instance-level multi-task learning network model.

3. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 2, characterized in that: Determining a new roof-to-foundation offset corresponding to the building roof outline annotation includes: Determine a target offset calculation function based on the relative position relationship between the building roof outline annotation and the building foundation outline annotation, and the relative position relationship between the building roof outline annotation and the annotated satellite remote sensing image slice; Outputting an offset adjustment coefficient through the target offset calculation function based on the size of the labeled satellite remote sensing image slice, the vertex coordinates of the building roof outline annotation, and the roof-to-foundation offset; If the number of the target offset calculation functions is one, the product of the offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset; if the number of the target offset calculation functions is multiple, the product of the smallest offset adjustment coefficient and the roof-to-foundation offset is used as the new roof-to-foundation offset.

4. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 1, characterized in that: Generate a first building roof outline and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model, including: Performing sliding window cropping processing on the non-orthophoto satellite remote sensing image to obtain a plurality of non-orthophoto satellite remote sensing image slices; Rotating and enhancing the non-orthophoto satellite remote sensing image slices to obtain non-orthophoto satellite remote sensing image slices at multiple rotation angles; Generate a first building roof outline and a roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at each rotation angle through an instance-level multi-task learning network model; Performing angular restoration on the first building roof profile and the roof-to-foundation offset; The mean value of the restored roof outline of the first building is used as the roof outline of the first building corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at the original angle; and the maximum value of the restored roof-to-foundation offset is used as the roof-to-foundation offset corresponding to the building contained in the non-orthophoto satellite remote sensing image slice at the original angle.

5. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 1, characterized in that: The step of roughly matching the second building roof outline with the first building roof outline in terms of an outer bounding box to select a roughly matched first building roof outline includes: If the outer bounding box of the second building roof outline intersects with the outer bounding box of the first building roof outline, the first building roof outline is taken as a coarse matching first building roof outline.

6. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 1, characterized in that: Performing edge overlap ratio nested matching on the second building roof outline and the precisely matched first building roof outline to replace the second building roof outline with the precisely matched first building roof outline, comprising: determining an edge overlap ratio between the second building roof profile and the precisely matched first building roof profile; Adding the precisely matched first building roof outline corresponding to the highest edge overlap rate to the second building roof outline, and determining a difference set between the precisely matched first building roof outline corresponding to the highest edge overlap rate and the second building roof outline; The edge overlap rate between the difference set and the remaining precisely matched first building roof outlines is re-determined until all the precisely matched first building roof outlines are cycled through, so as to replace the second building roof outline with the precisely matched first building roof outline.

7. The method for 3D reconstruction of buildings based on single-view non-orthophoto satellite remote sensing images according to claim 1, characterized in that: Performing a visual analysis on the side elevation formed by the roof outline of the second building and the foundation outline of the building to obtain visibility features corresponding to the side elevation for extracting a texture map corresponding to the building, including: Selecting two points from the second building roof outline, the distance between the two points being greater than a distance threshold, and selecting two points from the building foundation outline to form a side elevation; According to the height information corresponding to the roof outline of the second building, linear interpolation is performed from the foundation edge to the roof edge of the side facade to obtain the visibility feature corresponding to the side facade; A visible side facade is screened out from the side facades based on the visibility feature, so as to extract a texture map corresponding to the visible side facade from the non-orthophoto satellite remote sensing image.

8. A building 3D reconstruction device based on single-view non-orthophoto satellite remote sensing images, characterized in that: include: An acquisition module, used for acquiring a single-view non-orthophoto satellite remote sensing image and its corresponding second building roof outline; A roof profile and offset determination module, used to generate a first building roof profile and a roof-to-foundation offset corresponding to the building included in the non-orthophoto satellite remote sensing image through an instance-level multi-task learning network model; A height inversion module, used to perform height inversion on the building according to the offset from the roof to the foundation, to obtain height information corresponding to the building; A foundation outline and white mold generation module, used for matching the second building roof outline with the first building roof outline, and allocating the roof-to-foundation offset and the height information to the second building roof outline based on the matching relationship, so as to generate a building foundation outline and a building white mold corresponding to the building; wherein the accuracy of the second building roof outline is higher than that of the first building roof outline; A texture extraction module, used for visually analyzing the side elevation formed by the roof outline of the second building and the foundation outline of the building, obtaining visibility features corresponding to the side elevation, and extracting a texture map corresponding to the building; A three-dimensional reconstruction module, used for performing three-dimensional reconstruction of the building according to the building white model and the texture map; The foundation contour and white mold generation module is specifically used for: Performing outer bounding box rough matching of the second building roof outline and the first building roof outline to screen out a roughly matched first building roof outline; Performing an IoF fine matching on the second building roof profile and the roughly matched first building roof profile, so as to screen out a target first building roof profile that matches the second building roof profile, or screen out a finely matched first building roof profile; Performing edge overlap ratio nested matching on the second building roof outline and the precisely matched first building roof outline, so as to replace the second building roof outline with the precisely matched first building roof outline; The foundation contour and white mold generation module is specifically used for: Determine an IoF value between the second building roof outline and the roughly matched first building roof outline; wherein the ratio of the intersection of the second building roof outline and the roughly matched first building roof outline to its own area is called IoF; If there is a coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, taking the coarse matching first building roof outline corresponding to the maximum IoF value as the target first building roof outline that matches the second building roof outline; If there is no coarse matching first building roof outline whose IoF value is greater than the first IoF threshold, determining whether the number of coarse matching first building roof outlines whose IoF value is greater than the second IoF threshold and less than the first IoF threshold is 1; If the number is 1, the coarsely matched first building roof outline is used as the target first building roof outline that matches the second building roof outline; If the number is greater than 1, internal IoF filtering is performed on the coarsely matched first building roof outline; Determine whether the roof-to-foundation offsets corresponding to the filtered coarsely matched first building roof outlines are all not higher than the offset threshold; if so, use the coarsely matched first building roof outline corresponding to the largest roof-to-foundation offset as the target first building roof outline that matches the second building roof outline; if not, use the filtered coarsely matched first building roof outline as the precise match first building roof outline.

Images