3D modeling method based on semantic segmentation model and building shadow

Abstract

The present invention relates to the technical field of three-dimensional modeling, specifically a three-dimensional modeling method based on a semantic segmentation model and building shadows in remote sensing images. The method trains a semantic segmentation model by constructing a building and shadow data set, extracts building and shadow information in a single remote sensing image, and proposes a calculation method for judging the actual shadow length of a building based on the azimuth of the sun and a sensor, and the extracted building geometry, and then calculates the building height using the sun altitude angle; finally, the three-dimensional model of the building is constructed based on the building height, a texture library, image data, etc.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional modeling method based on a semantic segmentation model and building shadows. Background Art

[0002] With the continuous advancement of the construction of real 3D China, the 3D modeling of urban buildings is an important part of the construction of real 3D China. Efficient processing of 3D modeling of urban buildings has become a more urgent task. At present, there are many research results on 3D modeling of buildings, mainly including photogrammetry, laser ranging and extraction based on single images. Photogrammetry mainly calculates the 3D information of buildings through stereo image pairs, and laser ranging mainly uses technologies such as LiDAR and InSAR to obtain building elevation information. These two methods require specific types of sensors to obtain data and building heights, and the process is complicated and costly. Facing the vast land area, it is difficult to achieve large-scale promotion and use. The extraction based on a single image mainly uses the shadow of the building to measure the height and obtain DEM information. With the development of remote sensing technology, high temporal and spatial resolution images have covered the entire country, providing important data guarantee for the implementation of this method.

[0003] The core of the method of realizing 3D modeling based on building shadows is the calculation of the actual shadow length. The current method of calculating the shadow length mainly starts from the longitudinal section perspective of the building, determines whether the sun and the sensor are on the same side, and calculates the actual shadow length using trigonometric functions. This method does not take the polygonal structure of the building into consideration, resulting in the calculation results being greatly affected by the relative position of the sensor and the sun. Therefore, it is necessary to design a more efficient and accurate 3D building modeling method that can realize 3D modeling of urban buildings in batches and accurately to meet the needs of real-life 3D construction. Summary of the invention

[0004] The purpose of the present invention is to propose a method for quickly completing three-dimensional modeling based on a semantic segmentation model and building shadows, so as to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0005] The method constructs a building and shadow dataset to train a semantic segmentation model, extracts building and shadow information from a single remote sensing image, and proposes a calculation method for determining the actual shadow length of a building based on the sun and sensor azimuths and the extracted building geometry, and then calculates the building height using the sun altitude angle; finally, the three-dimensional model of the building is constructed based on the building height, texture library, remote sensing image data, etc.

[0006] Specifically, it includes: constructing a multi-source remote sensing data set of different building types and shadows based on prior knowledge. To ensure the accuracy of extracting buildings and shadows, the remote sensing data set contains different types of buildings in remote sensing images of different resolutions, and the shadows of buildings are of the same type; then using a semantic segmentation model to extract different buildings and building shadows, and obtain building and shadow patches; by obtaining information in the remote sensing image metadata file, calculating the solar azimuth and altitude angle and the sensor azimuth angle, obtaining the spatial position of the sun and the sensor relative to the building, and calculating the shadow length in the direction of the solar azimuth angle according to the building shadow vector patches, and calculating the building height based on the solar altitude angle and the shadow length; using the building vector patches and height data to construct a three-dimensional model, by constructing a building texture library, and automatically extracting the corresponding texture information according to the building type and fitting it to the side of the three-dimensional model, the constructed three-dimensional model is made more realistic, and then using the segmented building remote sensing image to fit the top layer of the three-dimensional model, batch construction of building three-dimensional models is realized.

[0007] Preferably, the building type classification is to extract buildings and shadows in the remote sensing image through semantic segmentation classification to obtain spots of different types of buildings and corresponding shadows.

[0008] The extraction of different types of buildings and shadows based on the semantic segmentation model is specifically as follows: classifying building types, such as oil tanks, greenhouses, houses, etc.; collecting multi-source remote sensing data, performing image preprocessing on the remote sensing data, and improving the accuracy of the image; cropping the remote sensing image, interpreting the image information using expert knowledge, and annotating the building and shadow information using labelme software to construct a data set containing multiple types of buildings and shadows; using the data set to train a semantic segmentation model; the trained model can automatically extract regions of interest of different types of buildings and shadows; assigning coordinate information to the extracted regions of interest and converting them into vector data containing multiple types of buildings and shadows.

[0009] The building type classification is specifically to classify the types of land features that need to construct three-dimensional models, such as oil tanks, greenhouses, buildings, etc., and clarify the specific types of each type of building. For example, buildings include self-built houses, commercial houses, etc., to provide a data basis for subsequent remote sensing interpretation and data set construction.

[0010] Remote sensing image preprocessing is specifically the preprocessing of the collected multi-source remote sensing data. Since the sensor has systematic errors and the remote sensing images are affected by atmospheric absorption, scattering, emission, etc., the color of the image is distorted and the ground object information is blurred. Therefore, it is necessary to perform radiometric calibration and atmospheric correction on the remote sensing images. Radiometric calibration can eliminate the systematic errors of the sensor itself, making the image clearer and brighter; atmospheric correction can correct the radiation value of the remote sensing image, restore the true color and brightness information of the ground object, and thus better extract the ground object information. The above image preprocessing can be operated using ENVI software.

[0011] Construct a data set of various types of buildings and shadows, specifically using python to crop remote sensing images, and the cropping size can be adaptively set according to the model type. Then use labelme software to annotate the buildings in the remote sensing images according to the classification type, and annotate the shadows in the pictures at the same time, so that the deep learning model can learn the shadow information of the buildings. In the annotation process, the images without buildings to be annotated need to be deleted. The annotation process requires rich experience in remote sensing image interpretation and high requirements for image resolution. Image resolution and data set annotation quality have a great impact on the quality of model training. In order to further increase the number of pictures in the data set and improve the generalization ability of the model, it is necessary to perform data enhancement processing on the data set. First, a data_augmentation function is defined to perform data enhancement operations, including random rotation, horizontal flipping, and random scaling. Then, traverse the image files in the original data set, generate multiple enhanced samples for each image, and save them in the specified folder.

[0012] The training semantic segmentation model is specifically divided into a training set, a validation set, and a test set. The training is performed using the prepared training set and the deep learning model. During the training process, the learning rate, optimizer and other parameters are continuously adjusted as needed, and the performance of the model is evaluated using the validation set or the test set, including indicators such as accuracy, recall rate, and F1 score. The prediction results of the model are visualized, the generalization ability and performance of the model are checked, and the optimal training model is selected for the classification and extraction of buildings and shadows.

[0013] The automatic extraction of regions of interest of different types of buildings and shadows is specifically to input the image to be detected into the trained semantic segmentation model, which can automatically segment different buildings and building shadow areas in the image and mark them in the image. The marked different types of buildings and building shadows have different colors, and the range marked with different colors is the region of interest.

[0014] The extracted region of interest is assigned coordinate information and converted into vector data. Since the semantic segmentation model processes image data, the output image loses coordinate information, and the above-identified buildings and shadow areas cannot be directly converted into vector data. The present invention proposes to establish a one-to-one mapping relationship according to the input image name and the input result image name, and then obtain multiple pixel coordinates of each segmented building and shadow by traversing the extracted result image, create a vector geometric object, and mark the type of building or shadow represented by the geometric object according to the color of the region of interest; then obtain the geographic reference information of the input remote sensing image according to the geographic database GDAL, including the geographic coordinate system, pixel resolution, etc., and use the mapping relationship between the input and output images to convert the multiple pixel coordinates of the building and shadow in the result image into geographic coordinates; use the Shapely module in the python library to convert the polygon surrounded by multiple geographic coordinates into a vector format, and assign category information to the vector spot, and finally save the conversion result as a vector data file with geographic coordinates. In order to further improve the accuracy of the spot, the present invention adjusts the converted layer by designing an editing tool so that the spot is more in line with the actual image. The layer editing tools include: adding features, deleting features, moving features, editing feature attributes, cutting features, merging features, splitting features, modifying feature shapes, rotating features, transforming features, etc.

[0015] Preferably, the building height is calculated by obtaining remote sensing image metadata information, calculating the solar azimuth and altitude angles and the sensor azimuth angle, obtaining the spatial position of the sun and the sensor relative to the building, and calculating the shadow length in the direction of the solar azimuth angle based on the building shadow vector pattern, and then using the shadow length and the solar altitude angle to calculate the building height information.

[0016] To obtain remote sensing image metadata information, the GetMetadata() function in the GDAL library is used to read the remote sensing information in the metadata, including shooting time, sensor parameters and other information, to provide parameters for building height calculation.

[0017] The solar azimuth and altitude angles and the sensor azimuth angle are calculated to obtain the spatial position of the sun and the sensor relative to the building, and the shadow length on the solar azimuth is calculated according to the building shadow vector spot, and then the height information of the building is calculated using the shadow length and the solar altitude angle. Specifically:

[0018] (1) By assigning coordinate information to the extracted buildings and shadows and converting the data format, the vector map of the buildings and shadows is obtained. This step uses the astronomical calculation method to calculate the position of the sun, including the solar azimuth, based on the shooting time and image location obtained from the metadata, starting from the true north direction. and the sun's altitude ;

[0019] (2) Obtain the location information of the satellite sensor through information such as satellite orbit parameters, take the building as the origin and the north direction as the starting point, and calculate the azimuth of the satellite sensor .

[0020] (3) Using the ray of the solar azimuth angle and its reverse extension as the dividing line, the building patch is divided into two left and right patches S1 and S2. The satellite sensor azimuth angle Determine whether the sensor is on the left side S1 or the right side S2 of the ray. hour, or , the sensor is on the left side of the ray S1, otherwise it is on the right side S2. hour, , the sensor is on the left side of the ray S1, otherwise it is on the right side S2.

[0021] (4) When the satellite sensor is on the left side of the ray S1, draw a perpendicular line from the left edge of the pattern to the ray, determine the coordinate point a1 corresponding to the longest perpendicular line, take a1 as the base point, and draw an extension line parallel to the ray direction. The coordinate point b1 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point, and the distance L1 is the actual shadow length of the building. Building height When the satellite sensor is on the right side of the ray S2, draw a perpendicular line from the right side of the pattern boundary to the ray, determine the coordinate point a2 corresponding to the longest perpendicular line, take a2 as the base point, and draw an extension line parallel to the ray direction. The coordinate point b2 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point, and its distance L2 is the actual shadow length of the building. Building height .

[0022] In step S300, the three-dimensional model of the building is constructed by stretching the two-dimensional plane pattern into a three-dimensional model according to the building pattern and the building height; and constructing a building texture library, by calling the building texture in the texture library and pasting it to the side of the three-dimensional model; and pasting the building in the remote sensing image to the top layer of the three-dimensional model to complete the construction of the three-dimensional model of the building;

[0023] To build a three-dimensional model, all vertex coordinates in each patch are stored in an array and the corresponding height values ​​are obtained. ;

[0024] Call the bpy library of blender software to stretch the two-dimensional contour map into 3D voxels according to the height information;

[0025] The construction of the building texture library. According to the above-classified building types, the corresponding building texture images are collected, and the image library is constructed and stored in a folder structure;

[0026] At the same time, metadata is built for each image, including the image name, source, resolution, width, height and license of the actual object represented;

[0027] The metadata is stored as a metadata file with the same name and a .json extension. The metadata files together constitute a metadata database, and the image library and the metadata database together constitute a texture library;

[0028] The texture library call, when the constructed three-dimensional building model is mapped sideways, retrieves a texture image of a corresponding type according to the building type of the pattern element, and obtains the actual building size width and height represented by the image by reading the image metadata file;

[0029] According to the coordinates and height information in the above array, the width and height of each facade of the building are calculated, and the number of pictures that need to be fitted to each facade of the building is obtained by using the width and height represented by the picture and the width and height of each facade of the building;

[0030] The calculation of the number of maps needs to be classified and calculated according to the building type. When the building type is a columnar building, the number of pictures required to be pasted needs to be calculated according to the width and height of the side of the building when it is unfolded, and the actual width and height represented by the picture; when the building is a house, the length and width of the building are first determined, and the side with the length and height of the building is used as the side length, and the corresponding pictures with side-by-side windows are called for pasting, and the number of pictures required to be pasted at this height is calculated according to the size of the picture. For the side with the width and height of the building as the side length, the corresponding wall picture is called for pasting, and the number of pictures required to be pasted at this height is calculated according to the size of the picture;

[0031] By traversing the 3D model with the vector patch as the base of the 3D model, the library can be called in batches to build 3D models for various types of buildings;

[0032] Use remote sensing images to build the top-level information of the three-dimensional model, read vector data, filter the target building according to the vector data field, query and select the target building vector patch, segment the remote sensing image according to the building patch, and obtain the corresponding remote sensing image. Use the height data to map the two-dimensional coordinates of the vector data to the vertices of the three-dimensional model. The top surface of the building can accurately reflect its corresponding area in the remote sensing image, so that the remote sensing image can be correctly fitted on the surface of the geometric body. Finally, by calling Blender's rendering system, a 3D model image with top and side textures is generated.

[0033] The present invention extracts building and shadow information from remote sensing images through a deep learning model, and converts the extraction results into editable vector data. This method can realize batch acquisition of building spatial information by utilizing the wide coverage of remote sensing images on the basis of meeting the accuracy requirements; at the same time, by developing vector data editing tools, the accuracy of building and shadow data is further improved. According to the metadata information of the remote sensing image and the relative position of the sun and the sensor, the actual shadow length of the building is calculated, and the actual height of the building is obtained through the altitude angle. This method solves the problems of complex process and low efficiency when obtaining elevation information through LiDAR and InSAR. By constructing and applying a texture library, segmenting the buildings in the remote sensing image, and calling Blender's API to realize batch mapping of the three-dimensional model, the three-dimensional modeling of the building is not only efficient and accurate, but also has a high degree of similarity between the model and the actual ground objects. It can further promote the construction of real-life three-dimensional China. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features of the present invention will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present invention represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0035] Figure 1 Flowchart for extracting building and shadow information based on deep learning;

[0036] Figure 2 Flowchart for calculating building height based on shadow length;

[0037] Figure 3 Schematic diagram to determine the spatial position of the sun and the sensor relative to the building;

[0038] Figure 4 Construct a flow chart for the 3D model of the building; DETAILED DESCRIPTION

[0039] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0040] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0041] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand the advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0042] like Figure 1 As shown, S110 represents collecting building information in remote sensing images of multiple resolutions, and based on expert prior knowledge, analyzing the apparent characteristics of different building types in remote sensing images, classifying the buildings in remote sensing images according to building characteristics and actual needs, such as oil tanks, greenhouses, houses, etc., and clarifying the specific types of each type of building, such as houses including self-built houses, commercial houses, etc., to provide a data basis for subsequent remote sensing interpretation and data set construction.

[0043] S120, construct a remote sensing data set according to the building types divided in S110 and the collected multi-source resolution remote sensing images. To ensure the accuracy of extracting buildings and shadows, the remote sensing data set contains different types of buildings in remote sensing images of different resolutions, and the shadows of buildings are of the same type. The data set is enhanced and divided, and finally a semantic segmentation model is trained based on the divided data set. In this embodiment, the Mask RCNN model is selected as an example model.

[0044] The remote sensing data set construction is specifically the remote sensing image preprocessing. In order to reduce the influence of sensor system errors and atmospheric absorption, scattering, reflection and other factors to improve the accuracy of remote sensing images, multi-source remote sensing images need to be preprocessed. The preprocessing includes radiation calibration and atmospheric correction, wherein radiation calibration can eliminate the system error of the sensor itself, making the image clearer and brighter; atmospheric correction can correct the radiation value of the remote sensing image, restore the true color and brightness information of the ground object, so as to better extract the ground object information. The image preprocessing can be operated using ENVI software. According to the remote sensing image after the above preprocessing, the remote sensing image is cropped using python, and the cropping size can be customized according to the remote sensing image resolution size and model selection. Then, the labelme software is used to mark the buildings in the remote sensing image according to the classification type, and the shadows in the picture are marked so that the Mask RCNN model can learn the shadow information of the building. In the cropped images, pictures without buildings to be labeled are deleted. This labeling process requires rich experience in remote sensing image interpretation and has high requirements on image resolution. Image resolution and data set labeling quality have a great impact on the quality of model training.

[0045] The remote sensing dataset is enhanced. Specifically, after the buildings and shadows in the remote sensing images are marked, in order to further increase the number of images in the dataset and improve the generalization ability of the model, the dataset needs to be enhanced. First, a data_augmentation function is defined to perform data enhancement operations, including random rotation, horizontal flipping, and random scaling. Then, the image files in the original dataset are traversed, multiple enhanced samples are generated for each image, and saved in the specified folder.

[0046] The data set division and model training and testing are to divide the data set after data enhancement into training set, validation set and test set according to certain standards, and train the MaskRCNN model with the training set according to the usage method of the Mask RCNN model, use the validation set to evaluate the performance of the model during the model training process, and use the test set to test the final effect of the model, evaluate the accuracy and error of the model, etc.

[0047] S130 inputs the image to be detected into the model based on the trained Mask RCNN model. The model can automatically identify the building area, category information and building shadow area in the image, and mark them in the image. For different categories of buildings and shadows, the colors of the marked areas of interest are different.

[0048] S140 Usually, the region of interest in the Mask RCNN output image is pixel coordinates, that is, the vertex in the upper left corner of the image is the origin, and the vertex coordinates of the region of interest are the relative position information in the image, which loses the geographic coordinate information of the remote sensing image, and cannot directly convert the above-identified buildings and shadow areas into vector data. The present invention proposes to establish a one-to-one mapping relationship according to the input image name and the input result image name, and then obtain multiple pixel coordinates of each segmented building and shadow by traversing the extracted result image, create a vector geometric object, and mark the type of building or shadow represented by the geometric object according to the color of the region of interest; then obtain the geographic reference information of the input remote sensing image according to the geographic database GDAL, including the geographic coordinate system, pixel resolution, etc., and use the mapping relationship between the input and output images to convert the multiple pixel coordinates of the building and shadow in the result image into geographic coordinates; use the Shapely module in the python library to convert the polygon surrounded by multiple geographic coordinates into a vector format, and assign category information to the vector map, and finally save the conversion result as a vector data file with geographic coordinates. In order to further improve the accuracy of the vector image spots, the present invention adjusts the converted layers by designing editing tools so that the spots are more closely aligned with the actual image. The layer editing tools include:

[0049] Add features: Add new features to the layer by manually drawing, copying and pasting, etc.

[0050] Delete Features: Delete features from the layer by selecting them.

[0051] Move element: Move the element by dragging or specifying the position.

[0052] Edit feature attributes: Edit the attribute information of a feature to make changes to incorrectly identified building types.

[0053] Split Feature: Split a feature into multiple parts for further editing.

[0054] Merge Features: Merge multiple adjacent features into a single feature.

[0055] Split Feature: Splits a feature into multiple parts for further editing.

[0056] Modify feature shape: Modify the shape of features, including moving nodes, adding nodes, deleting nodes, and other operations to fine-tune the vector data of buildings and shadows.

[0057] Rotate feature: Rotate the selected feature to change its direction or angle.

[0058] Transform elements: Perform transformation operations such as scaling, translating, and rotating on selected elements to change their size and shape.

[0059] The above application of deep learning in remote sensing extraction obtains the building and shadow vector data in the remote sensing image, which provides a data basis for the subsequent building height calculation and the construction of a three-dimensional model. Based on the data, the present invention provides a method for calculating the building height, such as Figure 2 Said:

[0060] S210 reads the remote sensing information in the metadata through the GetMetadata() function in the GDAL library, including shooting time, sensor parameters and other information, to provide parameters for building height calculation.

[0061] S220 Since the positions of the sun and the sensor are different, the methods for calculating the length of the building's shadow are different. Therefore, it is necessary to calculate the solar azimuth and solar altitude angle, as well as the azimuth of the sensor, through the metadata information obtained above. By assigning coordinate information to the extracted buildings and shadows and converting the data format, the vector image of the buildings and shadows is obtained. This step uses the astronomical calculation method to calculate the position information of the sun, including the solar azimuth, based on the shooting time obtained from the metadata and the location of the image, starting from the due north direction. and the sun's altitude ; Obtain the location information of the satellite sensor through the satellite orbit parameters, take the building as the origin and the north direction as the starting point, and calculate the azimuth of the satellite sensor .

[0062] S230 determines the spatial position of the sun and the sensor relative to the building. The building pattern is divided into two left and right patterns S1 and S2 using the rays of the sun's azimuth angle and the reverse extension line as the dividing line. Determine whether the sensor is on the left side S1 or the right side S2 of the ray. hour, or , the sensor is on the left side of the ray S1, otherwise it is on the right side S2. hour, , the sensor is on the left side of the ray S1, otherwise it is on the right side S2. Figure 3 shown.

[0063] S240 When the satellite sensor is on the left side of the ray S1, draw a perpendicular line from the left edge of the pattern to the ray, determine the coordinate point a1 corresponding to the longest perpendicular line, take a1 as the base point, and draw an extension line parallel to the ray direction. The coordinate point b1 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point, and the distance L1 is the actual shadow length of the building. Building height When the satellite sensor is on the right side of the ray S2, a perpendicular line is drawn from the right side of the pattern edge to the ray, and the coordinate point a2 corresponding to the longest perpendicular line is determined. With a2 as the base point, an extension line is drawn parallel to the direction of the ray. The coordinate point b2 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point. The distance L2 is the actual shadow length of the building. The building height .

[0064] Through the above calculation, the building height in the remote sensing image can be accurately obtained. Combined with the building vector data obtained in step S140, the present invention provides a method for constructing a three-dimensional building model, that is, through the above-obtained building spots and building heights, the two-dimensional plane spots are stretched into a three-dimensional stereo model; and a building texture library is constructed, and the building texture in the texture library is called and pasted to the side of the three-dimensional model; and the building image in the remote sensing image is segmented using the vector spots, and pasted to the top layer of the three-dimensional model to complete the construction of the three-dimensional building model. Specifically, Figure 4 As shown:

[0065] S310 Through the above calculation, each building in the remote sensing image has a corresponding vector map spot. The present invention obtains the building map spot data, stores all vertex coordinates of each map spot in an array, and obtains the height data of the corresponding building, such as:

[0066] polygons = [

[0067] {

[0068] vertices: [[x1, y1], [x2, y2], ..., [x n , y n ]],

[0069] height: z1

[0070] };

[0071] It stores all vertex coordinates of a building vector map and the height information of the building.

[0072] S320 uses the 2D plane coordinates of the building and the corresponding height data to call the bpy library of the blender software, and stretches the 2D contour map into 3D voxels according to the height information.

[0073] S330: Buildings of the same type have similar appearance features. Therefore, in order to efficiently complete the three-dimensional modeling of the building, the present invention proposes to build a building texture library, and realize the side texture of the three-dimensional model by calling the building texture library. Specifically, according to the building type divided by step S110, the corresponding building texture pictures are collected, and the picture library is built and stored in a folder structure, such as:

[0074] textures /

[0075] |-- Residential /

[0076] | |-- house1.jpg

[0077] | |-- house2.png

[0078] |-- Commercial /

[0079] | |-- office1.jpg

[0080] | |-- mall.png

[0081] | ... ;

[0082] At the same time, a metadata file with the same name and extension of .json is constructed for each image, which includes the image name, source, resolution, width and height of the actual object represented, license, etc. The metadata files together constitute the metadata database, and the image library and metadata database together constitute the texture library. The metadata structure example is as follows:

[0083] metadata = {

[0084] "name": filename,

[0085] "source": "https: / / example.com / textures",

[0086] "resolution": "1920x1080",

[0087] "width":8,

[0088] "height":3,

[0089] "license": "CC0" . . . . . .

[0090] }

[0091] S340 The 3D voxels constructed in step S320 do not have the appearance features of the building, so the present invention needs to use the appearance features of the building in the texture library and the remote sensing image to perform mapping processing on the side and top of the 3D model. This example takes the construction of a 3D model of a building as an example, specifically:

[0092] When it is necessary to perform side mapping on the three-dimensional model of a building, the present invention can retrieve a texture image of the corresponding type according to the building type of the pattern element, and obtain the actual building size width and height represented by the image by reading the image metadata file; calculate the width and height of each facade of the building according to the coordinates and height information in the above array, and use the width and height represented by the image and the width and height of each facade of the building to obtain the number of pictures that need to be fitted to each facade of the building.

[0093] The calculation of the number of maps needs to be classified according to the type of building. For example, when the building type is a cylindrical building such as an oil storage tank, the number of pictures required to be pasted needs to be calculated based on the width and height of the side of the building when it is unfolded, and the actual width and height represented by the picture; when the building is a residential building, the length and width of the building are first determined, and the side with the length and height of the building as the side length is called for the corresponding picture with side-by-side windows to be pasted, and the number of pictures required to be pasted at the height is calculated according to the size of the picture. For the side with the width and height of the building as the side length, the corresponding wall picture is called for pasting, and the number of pictures required to be pasted at the height is calculated according to the size of the picture.

[0094] When attaching the remote sensing image to the top layer of the 3D model, read the vector data, filter the target building according to the vector data field, query and select the building vector patch, segment the remote sensing image according to the selected building patch, and obtain the segmented building image. Then use the height data to map the two-dimensional coordinates of the vector data to the vertices of the 3D model. The top surface of the building can accurately reflect its corresponding area in the remote sensing image, so that the remote sensing image can be correctly attached to the surface of the geometric body. Finally, the 3D model image with top and side textures is generated by calling Blender's rendering system. By traversing the 3D voxels based on the vector patch, the texture library can be called in batches, and the remote sensing image can be segmented to build 3D models for various types of buildings.

[0095] Through the three-dimensional building modeling method in the above embodiment, the present invention can extract building and shadow information through a semantic segmentation model, and realize batch extraction of data in a single remote sensing image; and calculate the spatial position of the sun and the sensor relative to the building, and find and locate the base point of the building and the shadow position of the base point according to the different spatial positions, and calculate the length between the base point shadows as the actual shadow length of the building, and calculate the actual height of the building based on the solar altitude angle. This method solves the problem of building shadows and improves the calculation accuracy of building shadows and building heights. The present invention also proposes to build a texture library based on building types and apply it to the side of 3D voxels; and use the building information extracted by semantic segmentation to segment the real scene of the building in the remote sensing image, and map the image to the top layer of the 3D voxel through the height data, so as to build a three-dimensional model of the building. This method improves the construction efficiency of the three-dimensional model and ensures that the three-dimensional model is highly similar to the actual ground object. In this way, the construction of a three-dimensional model of a real scene of a building is realized efficiently and in batches based on a single remote sensing image, providing an effective method for the construction of a real scene three-dimensional China.

Claims

1. A 3D modeling method based on a semantic segmentation model and building shadows, characterized in that: The method comprises the following steps: S100: Classify and extract buildings; S200: Calculate building height; S300: Depicting a two-dimensional plane pattern according to building classification and building height, stretching the two-dimensional plane pattern into a three-dimensional model, and constructing a building texture library; S400: Complete the construction of the 3D model of the building based on the building height, building classification, and texture library; In step S200, the building height is calculated by obtaining metadata of the image file, combining the spatial position of the building and the relative position of the sun and the sensor, calculating the building shadow length information, and then using the shadow length to calculate the building height information; To obtain remote sensing image metadata, the GetMetadata() function in the GDAL library is used to read the remote sensing information in the metadata. The remote sensing information includes the shooting time and sensor parameters. The method combines the spatial position of the building and the relative positions of the sun and the sensor to the building to calculate the length information of the building's shadow, and uses the shadow length to calculate the height information of the building. The specific method includes the following steps: By assigning coordinate information to the extracted buildings and shadows and converting the data format, the vector map of the buildings and shadows was obtained; according to the shooting time obtained from the metadata and the location of the image, the north direction was defined as the starting point, and the position information of the sun was calculated using the astronomical calculation method. The position information includes the solar azimuth and the sun's altitude ; The location information of the satellite sensor is obtained through the satellite orbit parameters. The azimuth of the satellite sensor is calculated with the building as the origin and the north direction as the starting point. ; The building patch is divided into two patches S1 and S2 on the left and right, using the rays of the sun’s azimuth angle and its reverse extension as the dividing line. Determine whether the sensor is on the left side S1 or the right side S2 of the ray. hour, or , the sensor is on the left side of the ray S1, otherwise it is on the right side S2. hour, , the sensor is on the left side of the ray S1, otherwise it is on the right side S2; When the satellite sensor is on the left side of the ray S1, draw a perpendicular line from the left edge of the pattern to the ray, determine the coordinate point a1 corresponding to the longest perpendicular line, take a1 as the base point, and draw an extension line parallel to the ray direction. The coordinate point b1 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point. The distance L1 is the actual shadow length of the building, and the building height ,in is the solar altitude angle; when the satellite sensor is on the right side of the ray S2, a perpendicular line is drawn from the right side of the pattern boundary to the ray to determine the coordinate point a2 corresponding to the longest perpendicular line. With a2 as the base point, an extension line is drawn parallel to the direction of the ray. The coordinate point b2 where the extension line intersects with the shadow pattern and is farthest from the base point is the shadow corresponding to the base point. The distance L2 is the actual shadow length of the building. The building height .

2. The three-dimensional modeling method based on the semantic segmentation model and building shadows according to claim 1, characterized in that: In step S100, building classification is to extract buildings and shadows in remote sensing images through semantic segmentation model classification, and obtain different building classification spots and building shadows; The method of extracting different types of buildings and shadows based on the semantic segmentation model specifically comprises: dividing the building types, collecting multi-source remote sensing data, performing image preprocessing on the remote sensing data, cropping the remote sensing image, interpreting the image information using expert knowledge to label the building and shadow information, constructing a data set containing multiple types of buildings and shadows, using the data set to train the semantic segmentation model, and the trained model automatically extracts the regions of interest of different types of buildings and shadows, assigning coordinate information to the extracted regions of interest, and converting the coordinate information of the regions of interest into vector data containing multiple types of buildings and building shadows; Classification of building types, specifically classifying the types of objects that need to be built into a 3D model, and clarifying the specific types of buildings included in each type; Remote sensing image preprocessing, specifically preprocessing the collected multi-source remote sensing data, including radiometric calibration and atmospheric correction of remote sensing images in the multi-source remote sensing data; Constructing a dataset of various types of buildings and shadows involves cropping remote sensing images, annotating the buildings in the remote sensing images according to their types, and annotating the shadows in the remote sensing images so that the deep learning model can learn the shadow information of the buildings. During the labeling process, images without buildings to be labeled need to be deleted; The training semantic segmentation model specifically divides the data set into a training set, a validation set, and a test set, and uses the prepared training set and the deep learning model for training. During the training process, the learning rate and the parameters of the optimizer are continuously adjusted, and the performance of the model is evaluated using the validation set or the test set, including the accuracy, recall rate, and F1 score. The prediction results of the model are visualized, the generalization ability and performance of the model are checked, and the optimal training model is selected for the classification and extraction of buildings and shadows.

3. The three-dimensional modeling method based on the semantic segmentation model and building shadows according to claim 1, characterized in that: In step S300, the three-dimensional model of the building is constructed by stretching the two-dimensional plane pattern into a three-dimensional model according to the building pattern and the building height; and constructing a building texture library, by calling the building texture in the texture library and pasting it to the side of the three-dimensional model; and pasting the building in the remote sensing image to the top layer of the three-dimensional model to complete the construction of the three-dimensional model of the building; To build a three-dimensional model, all vertex coordinates in each patch are stored in an array and the corresponding height values ​​are obtained. ; Call the bpy library of blender software to stretch the two-dimensional contour map into 3D voxels according to the height information; The building texture library is constructed by collecting corresponding building texture pictures according to the above-mentioned building types, constructing a picture library, and storing them in a folder structure; At the same time, metadata is built for each image, including the image name, source, resolution, width, height and license of the actual object represented; The metadata is stored as a metadata file with the same name and a .json extension. The metadata files together constitute a metadata database, and the image library and the metadata database together constitute a texture library; The texture library call, when the constructed three-dimensional building model is mapped sideways, retrieves a texture image of a corresponding type according to the building type of the pattern element, and obtains the actual building size width and height represented by the image by reading the image metadata file; According to the coordinates and height information in the above array, the width and height of each facade of the building are calculated, and the number of pictures that need to be fitted to each facade of the building is obtained by using the width and height represented by the picture and the width and height of each facade of the building; The number of images to be pasted needs to be calculated based on the building type. When the building type is a columnar building, the number of images to be pasted needs to be calculated based on the width and height of the building side when it is unfolded, as well as the actual width and height represented by the image. When the building is a residential building, first determine the length and width of the building. For the side with the length and height of the building as the side length, call the corresponding pictures with side-by-side windows for pasting, and calculate the number of images to be pasted at the height of the building based on the size of the pictures. For the side with the width and height of the building as the side length, call the corresponding wall pictures for pasting, and calculate the number of images to be pasted at the height of the building based on the size of the pictures. By traversing the 3D model with the vector patch as the base of the 3D model, the library is called in batches to build 3D models for various types of buildings; Use remote sensing images to build the top-level information of the three-dimensional model, read vector data, filter the target building according to the vector data field, query and select the target building vector map, segment the remote sensing image according to the building map, and obtain the corresponding remote sensing image. Use the height data to map the two-dimensional coordinates of the vector data to the vertices of the three-dimensional model. The top surface of the building accurately reflects its corresponding area in the remote sensing image, so that the remote sensing image can be correctly fitted on the surface of the geometric body. Finally, by calling Blender's rendering system, a 3D model image with top and side textures is generated.

Images