Method for generating 3D objects using aerial imagery

Abstract

This invention provides a method for generating 3D objects from images of buildings taken from the air using an aerial videography device. [Solution] The method includes a calculation unit that generates an initial spatial object which is a 3D object corresponding to a building based on multiple aerial images, and a mapping candidate selection step, defining an object surface corresponding to the exterior surface of the building and a building surface normal vector perpendicular to the exterior surface of the building on the building surface. At least one of the multiple aerial images includes an initial texture corresponding to the exterior surface of the building, and a shooting direction vector from the shooting device toward the exterior surface of the building is defined for each of the multiple aerial images. The method also includes an intermediate texture selection step in which the calculation unit selects an aerial image for which the dot product of the shooting direction vector and the building surface normal vector is negative as a mapping candidate aerial image, and selects the initial texture with the largest area from among multiple initial textures corresponding to the multiple mapping candidate aerial images as an intermediate texture.

Description

Technical Field

[0001] The present invention relates to a method for generating a three-dimensional object corresponding to a building. Specifically, the present invention relates to a method for generating a three-dimensional object using two-dimensional aerial imagery.

Background Art

[0002] Digital Twin means a technology that utilizes a digital replica of an actual object (such as a building, facility, terrain, etc.) to perform monitoring and simulation on the object.

[0003] In order to embody a digital twin, a technology for modeling an object to generate a three-dimensional object is required. Conventionally, a technology for generating a three-dimensional object based on images collected in the air by an aircraft or a drone has been proposed, but there is still a problem that the process is carried out manually, resulting in low efficiency.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention is for solving the above-described problems and relates to a method for generating a three-dimensional object that improves the efficiency of the process of generating a three-dimensional object by utilizing various AI models.

Means for Solving the Problems

[0005] A method for generating a 3D object using aerial images according to one embodiment of the present invention may include an initial spatial object generation step, a mapping candidate selection step, and an intermediate texture selection step. The aerial images may be images of a building taken from the air using a camera. In the initial spatial object generation step, the computing unit may generate initial spatial objects, which are 3D objects corresponding to a building, based on a plurality of aerial images. The exterior surface of the building may be defined on the surface of the building. A building surface normal vector perpendicular to the exterior surface of the building may be defined. At least one of the plurality of aerial images may include an initial texture, which is an image corresponding to the exterior surface of the building. A shooting direction vector from the camera toward the exterior surface of the building may be defined for each of the plurality of aerial images. In the mapping candidate selection step, the computing unit may select an aerial image from among the plurality of aerial images that includes the initial texture and whose dot product of the shooting direction vector and the building surface normal vector is negative as a mapping candidate aerial image. The initial spatial object may include an object surface corresponding to the exterior surface of the building. In the intermediate texture selection step, the computing unit may select the initial texture with the largest area from among multiple initial textures corresponding to multiple mapping candidate aerial images as the intermediate texture.

[0006] In one embodiment of the present invention, the first and second aerial images among a plurality of aerial images may be aerial images of a building taken from different directions. The initial spatial object generation step may include a roof polygon generation step, a feature point matching step, a spatial coordinate calculation step, a building height calculation step, and a modeling step. In the roof polygon generation step, the computing unit may extract roof polygons corresponding to the roof of the building from the first aerial image. In the feature point matching step, the computing unit may select at least one first feature point having predetermined features from a plurality of polygonal pixels corresponding to the roof polygon from the first aerial image, and select at least one second feature point corresponding to the first feature point from the second aerial image. In the spatial coordinate calculation step, the computing unit may calculate feature point spatial coordinate information using a collinearity-based forward intersection method based on the first and second feature points. The feature point spatial coordinate information may be either the spatial coordinates of the first feature point or the spatial coordinates of the second feature point. In the building height calculation step, the computing unit may calculate the height of the building using the feature point spatial coordinate information. In the modeling step, the computing unit can generate initial spatial objects based on the roof polygon and building height.

[0007] In one embodiment of the present invention, at least one of the first AI model and the second AI model can be stored in the computing device.

[0008] The first AI model can extract at least one first polygon as a vector image from aerial imagery using polygon mapping. The backbones of both the first and second AI models may be neural network models. The second AI model can extract at least one second polygon as a raster image from aerial imagery using object segmentation. In the roof polygon generation step, the computing unit can extract roof polygons from the first aerial imagery using at least one of the first and second AI models.

[0009] In one embodiment of the present invention, the second AI model can convert the second polygon into a vector image.

[0010] In the feature point matching step according to one embodiment of the present invention, the computing device can use at least one of KAZE, ORB, and SIFT to calculate at least one first feature point and at least one second feature point.

[0011] In the feature point matching step according to one embodiment of the present invention, the computing device can use at least one of BFMatcher and FLANN to calculate at least one second feature point.

[0012] In the building height calculation step according to one embodiment of the present invention, the calculation device can further utilize a pre-stored digital elevation model (DEM) to calculate the building height.

[0013] In a modeling step according to one embodiment of the present invention, the computing device can perform tessellation on the roof polygon to divide it into multiple triangles, and generate an initial spatial object based on the multiple triangles and the height of the building.

[0014] In a modeling step according to one embodiment of the present invention, the computing device may perform tessellation using a Sweeping Line Algorithm and an Ear Clipping Algorithm.

[0015] In the mapping candidate selection step according to one embodiment of the present invention, the computing device may select aerial images of mapping candidates using a Backface Culling Algorithm.

[0016] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include a final texture generation step. In the final texture generation step, the computing unit may generate the final texture by performing an affine transformation on the intermediate texture and then interpolating it.

[0017] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include an atlas generation step. Multiple exterior surfaces of a building may be provided. In the atlas generation step, the computing unit may repeatedly perform a mapping candidate selection step, an intermediate texture selection step, and a final texture generation step for each of the multiple exterior surfaces of a building to generate multiple final textures. The computing unit may also generate an atlas using a Binary Space Partitioning Tree based on the multiple final textures.

[0018] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include a final spatial object generation step. In the final spatial object generation step, the computing unit can generate a final spatial object by mapping a plurality of final textures to the initial spatial object based on an atlas.

[0019] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include an inpainting step. In the inpainting step, the computing unit may perform inpainting on the final spatial object using a third AI model. The backbone of the third AI model may be a neural network model.

[0020] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include a super-resolution step. In the super-resolution step, the computing device may increase the resolution of the final spatial object using a super-resolution model. The backbone of the super-resolution model may be a neural network model. In one embodiment of the present invention, the neural network model that is the backbone of the super-resolution model may be a Deep Convolutional Network. The super-resolution model may be trained using multiple images having different resolutions. The super-resolution model may change the resolution of multiple images using an attention mechanism.

[0021] A method for generating a 3D object using aerial imagery according to one embodiment of the present invention may further include a texture editing step. In the texture editing step, one of the mapped final textures may be a first final texture and the other a second final texture. The computing unit may edit the second final texture using at least a portion of predetermined textures stored in advance or at least a portion of the first texture.

[0022] An initial spatial object generation step according to one embodiment of the present invention may further include a preprocessing step. Each of the multiple aerial images may include a plurality of pixels arranged on a plane extended in the x-axis and y-axis directions. The position of each of the plurality of pixels may be determined by the x-coordinate and y-coordinate. The first aerial image may include a plurality of polygonal pixels corresponding to a roof polygon. The smallest value among the plurality of x-coordinates corresponding to the plurality of polygonal pixels may be the first left-hand value, and the largest value may be the first right-hand value. The smallest value among the plurality of y-coordinates corresponding to the plurality of polygonal pixels may be the first bottom-hand value, and the largest value may be the first top-hand value. In the first aerial image, the region in which the first aerial image overlaps with the second aerial image may be the overlapping region, and the region in which they do not overlap may be the non-overlapping region. The overlapping direction may be the direction from the centroid of the non-overlapping region toward the centroid of the overlapping region. The non-overlapping length may be the length of the non-overlapping region measured in the overlapping direction. In the preprocessing step, if the overlapping directions are aligned along the x-axis, the calculation unit may remove the cropping area from the second aerial image using Equation 1 or Equation 2. If the overlapping directions are aligned along the y-axis, the calculation unit may remove the cropping area from the second aerial image using Equation 3 or Equation 4. The cropping area may be the area outside the region enclosed by the line with the second left-hand value x-coordinate, the line with the second bottom-hand value y-coordinate, the line with the second right-hand value x-coordinate, and the line with the second top-hand value y-coordinate in the second aerial image.

[0023] [Mathematics 1] Xmin2=Xmin1+Vmin, Xmax2=Xmax1+Vmax Ymin2 = Ymin1, Ymax2 = Ymax1 [Math 2] Xmin2=Xmin1-Vmax, Xmax1=Xmax1-Vmin Ymin2 = Ymin1, Ymax1 = Ymax1 [Math 3] Xmin2 = Xmin1, Xmax2 = Xmax1 Ymin2=Ymin1+Vmin, Ymax2=Ymax1+Vmax [Math 4] Xmin2 = Xmin1, Xmax2 = Xmax1 Ymin2 = Ymin1 - Vmin, Ymax2 = Ymax1 - Vmax

[0024] In each of Formulas 1 to 4, Xmin2 is the second left-side value, Ymin2 is the second bottom-side value, Xmax2 is the second right-side value, Ymax2 is the second top-side value, Xmin1 is the first left-side value, Xmax1 is the first right-side value, Ymin1 is the first bottom-side value, Ymax1 is the first top-side value, Vmin can be the minimum value of the non-overlapping length, and Vmax can be the maximum value of the non-overlapping length. In the feature point matching step, the second aerial image can be the second aerial image from which the crop target area has been removed.

[0025] In the preprocessing step according to an embodiment of the present invention, when the overlapping direction is opposite to the x-axis direction, the arithmetic unit can use Formula 1 to remove the crop target area from the second aerial image. When the overlapping direction is the same as the x-axis direction, the arithmetic unit can use Formula 2 to remove the crop target area from the second aerial image. When the overlapping direction is opposite to the y-axis direction, the arithmetic unit can use Formula 3 to remove the crop target area from the second aerial image. When the overlapping direction is the same as the y-axis direction, the arithmetic unit can use Formula 4 to remove the crop target area from the second aerial image.

Advantages of the Invention

[0026] According to an embodiment of the present invention, a method for generating a three-dimensional object with improved efficiency in the process of generating a three-dimensional object can be provided in order to utilize various AI models.

Brief Description of the Drawings

[0027] [Figure 1] A diagram exemplarily showing a state in which a plurality of three-dimensional objects are generated by the method for generating a three-dimensional object using the aerial image of the present invention. [Figure 2] A diagram exemplarily showing a sequence diagram of the method for generating a three-dimensional object according to an embodiment of the present invention. [Figure 3]This figure illustrates a detailed sequence diagram of the initial spatial object generation step according to one embodiment of the present invention. [Figure 4a] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 4b] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 4c] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 4d] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 4e] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 4f] This diagram illustrates the detailed steps of the initial space object generation process. [Figure 5a] This figure illustrates the process by which a mapping candidate aerial image is selected in the mapping candidate selection step according to one embodiment of the present invention. [Figure 5b] This figure illustrates the process by which a mapping candidate aerial image is selected in the mapping candidate selection step according to one embodiment of the present invention. [Figure 6] This figure illustrates an intermediate texture selection step according to one embodiment of the present invention. [Figure 7] This figure illustrates the final texture generation step according to one embodiment of the present invention. [Figure 8] This figure illustrates the atlas generation step according to one embodiment of the present invention. [Figure 9] This figure illustrates the final spatial object generation step according to one embodiment of the present invention. [Figure 10a] This figure illustrates a portion of the final spatial object to illustrate an inpainting step according to one embodiment of the present invention. [Figure 10b] This figure illustrates a portion of the final spatial object to illustrate an inpainting step according to one embodiment of the present invention. [Figure 11a] This figure illustrates the super-resolution step according to one embodiment of the present invention. [Figure 11b] This figure illustrates the super-resolution step according to one embodiment of the present invention. [Figure 12a] This figure illustrates a texture editing step according to one embodiment of the present invention. [Figure 12b] This figure illustrates a texture editing step according to one embodiment of the present invention. [Figure 12c] This figure illustrates a texture editing step according to one embodiment of the present invention. [Figure 12d] This figure illustrates a texture editing step according to one embodiment of the present invention. [Figure 13a] This figure illustrates a pretreatment step according to one embodiment of the present invention. [Figure 13b] This figure illustrates a pretreatment step according to one embodiment of the present invention. [Figure 13c] This figure illustrates a pretreatment step according to one embodiment of the present invention. [Figure 13d] This figure illustrates a pretreatment step according to one embodiment of the present invention. [Figure 13e] This figure illustrates a pretreatment step according to one embodiment of the present invention. [Modes for carrying out the invention]

[0028] Preferred embodiments of the present invention will be described in more detail below with reference to the attached drawings. In the drawings, the proportions and dimensions of the components may be exaggerated for the sake of effective explanation of the technical content.

[0029] Terms such as "includes" should be understood to indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, without prejudice to the presence or possibility of adding one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0030] Furthermore, when the term "above" is used for a component, it means either above or below that component, and does not necessarily mean that it is located on the upper side relative to the direction of gravity.

[0031] Furthermore, when it is stated that one component is “connected” or “joined” with another component, this may include not only cases where the component is directly connected or joined to another component, but also cases where the component is indirectly connected or joined through another component.

[0032] Furthermore, while terms such as "first," "second," etc., may be used to describe a particular component, such terms are used to distinguish that component from other components, and are not intended to limit the essence, order, or sequence of that component.

[0033] Preferred embodiments of the present invention will be described in more detail below with reference to the attached drawings. In the drawings, the proportions and dimensions of the components may be exaggerated for the sake of effective explanation of the technical content.

[0034] Figure 1 is an illustrative diagram showing how multiple 3D objects MD are generated by the OBM method for generating 3D objects using aerial images according to the present invention.

[0035] Referring to Figure 1, the aerial image PH may be an image of a building BD taken from the air using a camera CM. For example, the aerial image PH may be collected via a camera CM coupled to an aircraft VH. The aircraft VH may be at least one of an airplane and / or a drone. However, the aircraft VH of the present invention is not limited to these, and any device capable of flight is sufficient. Using the present invention, multiple 3D objects MD for multiple buildings BD can be generated based on multiple aerial images PH1 to PHn, each of which is a 2D image.

[0036] One embodiment of the present invention, a method for generating 3D objects using aerial imagery (OBM), can utilize a computing device PR. The computing device PR can receive aerial imagery PH from an external source.

[0037] Figure 2 is a diagram illustrating the sequence of steps for a 3D object generation method (OBM) according to one embodiment of the present invention. Referring to Figure 2, the 3D object generation method (OBM) using aerial imagery according to one embodiment of the present invention may include an initial spatial object generation step S10, a mapping candidate selection step S20, an intermediate texture selection step S30, a final texture generation step S40, an atlas generation step S50, a final spatial object generation step S60, an inpainting step S70, and a super-resolution step S80.

[0038] Figure 3 is a diagram illustrating the detailed sequence of the initial spatial object generation step S10 according to one embodiment of the present invention.

[0039] Referring to Figure 3, in one embodiment, the initial spatial object generation step S10 may include a roof polygon generation step S11, a feature point matching step S12, a spatial coordinate calculation step S13, a building height calculation step S14, and a modeling step S15.

[0040] Figures 4a to 4f are illustrative diagrams illustrating the S10 detail step of the initial space object generation step, respectively.

[0041] Referring to Figures 4a to 4f, in the initial spatial object generation step S10, the computing unit PR may generate an initial spatial object OB1 based on multiple aerial images PH1 to PHn. The initial spatial object OB1 may be a three-dimensional object corresponding to a building BD.

[0042] Figures 4a and 4b are illustrative diagrams illustrating the roof polygon generation step S11 according to one embodiment of the present invention. Multiple building exteriors can be defined as the surface of building BD. Figures 4a and 4b are enlarged views showing the portion of the first aerial image PH1 corresponding to the exterior surface BF1 of the first building. The exterior surface BF1 of the first building may be the exterior surface of a building corresponding to the roof among the multiple building exteriors.

[0043] Referring to Figures 4a and 4b, in the roof polygon generation step S11 according to one embodiment of the present invention, the computing device PR can extract a roof polygon PG corresponding to the exterior surface BF1 of the first building from the first aerial image PH1. The roof polygon PG can later be used to generate the initial spatial object OB1.

[0044] At least one of the first AI model and the second AI model can be stored in the computing device PR. In the roof polygon generation step S11 according to one embodiment of the present invention, the computing device PR can extract the roof polygon PG from the first aerial image PH1 using at least one of the first AI model and the second AI model.

[0045] The first AI model, based on HRNetV2 and multiple convolutional neural networks, can extract at least one first polygon from aerial imagery through polygon mapping. The first polygon can be a vector image. In other words, the first polygon extracted by the first AI model is vector data and can be stored in SHP, GeoJSON, and KML formats.

[0046] The second AI model is based on at least one of U-Net, Deeplab, DeeplabV3+, and HRNet and can extract at least one second polygon as a raster image from aerial imagery by object segmentation. In one embodiment of the present invention, the second AI model can convert the second polygon into a vector image using a skeletonization algorithm and a loss function. That is, the second polygon extracted by the second AI model using a skeletonization algorithm and a loss function is vector data and can be stored in SHP, GeoJSON, and KML formats.

[0047] However, the first and second AI models of the present invention are not limited to these. The first AI model of the present invention is sufficient if it is capable of extracting polygons from video as vector images by polygon mapping, and the second AI model is sufficient if it is capable of extracting polygons from video as raster images by object segmentation.

[0048] Figures 4c and 4d are illustrative diagrams illustrating the feature point matching step S12 according to one embodiment of the present invention.

[0049] Referring to Figures 4c and 4d, among the multiple aerial images PH1 to PHn, the first aerial image PH1 and the second aerial image PH2 may be aerial images of the same building BD taken from different directions.

[0050] In the feature point matching step S12 according to one embodiment of the present invention, the computing unit PR may select at least one first feature point P1 having predetermined features from among a plurality of polygonal pixels corresponding to the roof polygon PG from the first aerial image PH1, and select at least one second feature point P2 corresponding to the first feature point P1 from the second aerial image PH2. The second feature point P2 may also be a pixel having predetermined features in the second aerial image PH2.

[0051] In one embodiment of the present invention, the first feature point P1 and the second feature point P2 may each be provided in multiple quantities. A feature point pair means a set of any two feature points that share similar features, and the computing unit PR may select multiple feature point pairs from the first aerial image PH1 and the second aerial image PH2 in the feature point matching step S12.

[0052] In the feature point matching step S12 according to one embodiment of the present invention, the computing unit PR can calculate at least one first feature point P1 using at least one of KAZE, ORB (Oriented FAST and Rotated BRIEF), and SIFT (Scale-Invariant Feature Transform).

[0053] KAZE is used to extract image edges and textures and has the advantage of being highly resistant to noise. ORB combines a FAST keypoint detector and a BRIEF descriptor and has the advantage of low computational load. SIFT extracts feature points using Gaussian blurring and differences and has the advantage of effectively extracting complex features.

[0054] In the feature point matching step S12 according to one embodiment of the present invention, the computing unit PR can calculate at least one or more second feature points P2 using at least one of BFMatcher (Brute-Force Matcher) and FLANN (Fast Library for Approximate Nearest Neighbors).

[0055] BFMatcher refers to a method that calculates the distance between all pairs of feature points and searches for the closest feature point. BFMatcher is easy to implement and has the advantage of accurately matching feature points because it compares all pairs. FLANN is a library for approximate nearest neighbor search. Unlike BFMatcher, FLANN uses an approximation method without comparing all pairs, and therefore has the advantage of matching feature points faster than BFMatcher. For example, the approximation method used in FLANN may include at least one of the following: KD trees, hash-based methods, and other fast search algorithms.

[0056] Figure 4e is an illustrative diagram illustrating the spatial coordinate calculation step S13 according to one embodiment of the present invention.

[0057] In this specification, the collinearity condition means that the shooting point S on which the lens of the imaging device is positioned, the feature point P placed in the image, and the ground point T located on the ground corresponding to the feature point P should lie on a single straight line. Furthermore, the forward resection method means a method for calculating the actual position of an object in three-dimensional space using two or more photographs taken at different positions.

[0058] In the spatial coordinate calculation step S13 according to one embodiment of the present invention, the computing unit PR can calculate feature point spatial coordinate information using a collinearity-based forward intersection method based on the first feature point P1 and the second feature point P2. In one embodiment of the present invention, the feature point spatial coordinate information may be either the spatial coordinate of the ground point corresponding to the first feature point P1 or the spatial coordinate of the ground point corresponding to the second feature point P2.

[0059] Referring to Figure 4e, ground point T corresponds to the first feature point P1 and the second feature point P2, and may be a point located on the ground. The first aerial image PH1 is an aerial image of ground point T taken at the first shooting point S1, and the second aerial image PH2 may be an aerial image of ground point T taken at a second shooting point S2, which is different from the first shooting point S1.

[0060] More specifically, in the spatial coordinate calculation step S13, the computing unit PR can calculate the spatial coordinates of the ground point T using any one of the following: the first vector V1 from the first feature point P1 to the first shooting point S1, the second vector V2 from the second feature point P2 to the second shooting point S2, the third vector V3 from the first shooting point S1 to the second shooting point S2, the fourth vector V4 from the first feature point P1 to the ground, the fifth vector V5 from the second feature point P2 to the ground, the sixth vector (not shown) from the first shooting point S1 to the ground point T, and the seventh vector from the second shooting point S2 to the ground point T.

[0061] In step S14 of the building height calculation according to one embodiment of the present invention, the computing device PR may calculate the height of the building BD using feature point spatial coordinate information. In step S14 of the building height calculation according to one embodiment of the present invention, the computing device PR may further calculate the height of the building BD using a pre-stored digital elevation model (DEM). In this specification, a digital elevation model means a model that represents ground surface height information in a digital form.

[0062] Figure 4f is an illustrative diagram illustrating modeling step S15 according to one embodiment of the present invention.

[0063] Referring to Figure 4f, in modeling step S15 according to one embodiment of the present invention, the computing unit PR may generate an initial spatial object OB1 based on the height of the roof polygon PG and the building BD. The initial spatial object OB1 may include a first object surface MF1 corresponding to the outer surface BF1 of the first building in Figures 4a and 4b.

[0064] Tessellation generally refers to the process of dividing the faces of a 3D object into a number of triangles or quadrilaterals. Tessellation allows for a more detailed representation of 3D objects and more efficient rendering.

[0065] In modeling step S15 according to one embodiment of the present invention, the computing unit PR can perform tessellation on the roof polygon PG, dividing the roof polygon PG into multiple triangles. The computing unit PR can generate an initial spatial object OB1 based on the multiple triangles and the height of the building BD.

[0066] In modeling step S15 according to one embodiment of the present invention, the computing unit may perform tessellation using a Sweeping Line Algorithm and an Ear Clipping Algorithm. The Sweeping Line Algorithm and the Ear Clipping Algorithm are algorithms for performing tessellation, respectively.

[0067] Figures 5a and 5b are illustrative diagrams illustrating the process by which a mapping candidate aerial image AA is selected in the mapping candidate selection step S20 according to one embodiment of the present invention.

[0068] In the mapping candidate selection step S20 according to one embodiment of the present invention, the computing device PR may select a mapping candidate aerial image AA from among a plurality of aerial images PH1 to PHn.

[0069] Referring to Figure 5a, the mapping candidate aerial image AA may be an aerial image containing an initial texture IT, which is an image corresponding to the exterior surface BF1 of the first building among a plurality of aerial images PH1 to PHn. In one embodiment of the present invention, a plurality of mapping candidate aerial images AA may be provided. That is, each of the plurality of mapping candidate aerial images AA1 to AAn may be an aerial image PH containing a plurality of initial textures IT1 to ITn.

[0070] Referring to Figure 5, the shooting direction vector CV can be defined as a vector from the shooting device CM toward the center C1 of the outer surface of the first building. In other words, the shooting direction vector CV can represent the direction in which the mapping candidate aerial image AA is captured. The first building surface normal vector NV1 can be defined as a vector perpendicular to the outer surface BF1 of the first building.

[0071] A shooting direction vector CV is defined for each of the multiple aerial images PH1 to PHn, and the mapping candidate aerial image AA may be an aerial image PH from among the multiple aerial images PH1 to PHn in which the dot product of the shooting direction vector CV and the first building surface normal vector NV1 is negative. In other words, the mapping candidate aerial image AA may be an aerial image PH from among the multiple aerial images PH1 to PHn in which the outer surface BF1 of the first building and the shooting device CM are facing each other. On the other hand, aerial image PH from among the multiple aerial images PH1 to PHn in which the dot product of the shooting direction vector CV and the first building surface normal vector NV1 is positive may be an aerial image PH in which the outer surface BF1 of the first building and the shooting device CM are not facing each other. In the mapping candidate selection step S20 according to one embodiment of the present invention, the computing device PR may select the mapping candidate aerial image AA using the Backface Culling Algorithm. The Backface Culling Algorithm may be an algorithm used in the process of distinguishing between surfaces visible from the shooting device CM and surfaces not visible from the shooting device CM in order to improve rendering performance. The processing unit PR can use the Backface Culling Algorithm to determine whether a particular surface is not visible from the imaging device CM. This allows for improved computational efficiency, as calculations are not performed on surfaces not visible from the imaging device CM.

[0072] Figure 6 is an illustrative diagram illustrating the intermediate texture selection step S30 according to one embodiment of the present invention.

[0073] Referring to Figure 6, in the intermediate texture selection step S30 according to one embodiment of the present invention, the computing unit PR may select the initial texture with the largest area from among a plurality of initial textures IT1 to ITn corresponding to a plurality of mapping candidate aerial images AA1 to AAn as the intermediate texture MT.

[0074] Figure 7 is an illustrative diagram illustrating the final texture selection step S40 according to one embodiment of the present invention.

[0075] Referring to Figure 7, in the final texture generation step S40 according to one embodiment of the present invention, the computing unit PR may generate the final texture FT by performing an affine transformation on the intermediate texture MT and then interpolating it. The affine transformation can be a method for mapping the texture to a 3D object while minimizing distortion.

[0076] Figure 8 is an illustrative diagram illustrating the atlas generation step S50 according to one embodiment of the present invention.

[0077] Referring to Figure 8, in the atlas generation step S50 according to one embodiment of the present invention, the computing unit PR can generate final textures FT1 to FTn by repeatedly performing the mapping candidate selection step S20, the intermediate texture selection step S30, and the final texture generation step S40 for each of the multiple exterior surfaces of buildings. Each of the multiple final textures FT1 to FTn may correspond to one of the multiple exterior surfaces of buildings.

[0078] In one embodiment of the present invention, the computing unit PR may generate an atlas AT using a Binary Space Partitioning Tree based on a plurality of final textures FT1 to FTn. The Binary Space Partitioning Tree may be a method for efficiently arranging textures during the generation of the atlas AT, preventing overlap between textures, and improving rendering performance.

[0079] Figure 9 is an illustrative diagram illustrating the final spatial object generation step S60 according to one embodiment of the present invention.

[0080] Referring to Figure 9, in the final spatial object generation step S60 according to one embodiment of the present invention, the computing unit PR can generate the final spatial object OB2 by mapping a plurality of final textures FT1 to FTn to the initial spatial object OB1 based on the atlas AT. In this specification, the final spatial object OB2 means the three-dimensional object MD.

[0081] Figures 10a and 10b are illustrative diagrams showing a portion of the final spatial object OB2-1 to illustrate the inpainting step S70 according to one embodiment of the present invention. For convenience of explanation, Figures 10a and 10b illustratively show a final spatial object OB2-1 that is different from the final spatial object OB2 in Figure 9.

[0082] Inpainting refers to a technique for restoring defects or missing parts of a 3D object. Referring to Figures 10a and 10b, the computing unit PR can remove unnecessary parts BB from the final spatial object OB2-1 and naturally embed other images CC through inpainting. In the inpainting step S70 according to one embodiment of the present invention, the computing unit PR can perform inpainting on the final spatial object OB2-1 using a third AI model. The backbone of the third AI model may be a neural network model. The third AI model may utilize at least one of Mobile Inpainting GAN and LaMa (Large Mask inpainting model). Mobile Inpainting GAN is a GAN (Generative Adversarial Network) based inpainting model designed to operate efficiently on mobile devices, and has the advantage of low computational load and smooth operation on mobile devices. LaMa is a large mask inpainting model, and has the advantage of being able to effectively process large masks.

[0083] Figures 11a and 11b are diagrams illustrating a super-resolution step S80 according to one embodiment of the present invention. Figure 11a illustrates the final spatial object OB2-2 before the super-resolution step S80 is performed, and Figure 11b illustrates the final spatial object OB2-2 after the super-resolution step S80 is performed.

[0084] The computing unit PR can increase the resolution of the final spatial object OB2-2 by the super-resolution step S80. Referring to Figures 11a and 11b, in the super-resolution step S80 according to one embodiment of the present invention, the computing unit PR can increase the resolution of the final spatial object by utilizing a super-resolution model. The backbone of the super-resolution model may be a neural network model. In one embodiment of the present invention, the neural network model that is the backbone of the super-resolution model may be a Deep Convolutional Network. The super-resolution model can be trained on multiple images having different resolutions. The super-resolution model can change the resolution of multiple images by utilizing an Attention Mechanism.

[0085] In the super-resolution step S80 according to one embodiment of the present invention, the computing unit PR can increase the resolution of the final spatial object OB2-2 by utilizing a Hybrid Attention Transformer. The Hybrid Attention Transformer may be a transformer model that utilizes a hybrid attention mechanism.

[0086] Figures 12a to 12d are diagrams illustrating a texture editing step (not shown) according to one embodiment of the present invention.

[0087] Figure 12a is an illustrative diagram showing the final spatial object OB2-3 before the texture editing step is performed. Figure 12b is an illustrative diagram showing a predetermined texture ST that has been saved in advance.

[0088] Referring to Figure 12a, of the multiple mapped final textures FT1-1 to FTn-1, one may be the first final texture FT1-1, and the other may be the second final texture FT2-1. Referring to Figure 12b, in one embodiment of the present invention, a predetermined texture ST may be stored in advance in the computing unit PR.

[0089] Figures 12c and 12d are illustrative diagrams showing the final spatial object OB2-3 after the texture editing step has been performed.

[0090] Referring to Figures 12a and 12c, in a texture editing step according to one embodiment of the present invention, the computing unit PR can edit the second final texture FT2-1 using at least a portion PT of the first final texture FT1-1. In the texture editing step, the second final texture FT2-1 in Figure 12a can be edited into the second final texture FT2-2 in Figure 12b.

[0091] Referring to Figures 12a, 12b, and 12d, in a texture editing step according to one embodiment of the present invention, the computing unit PR can edit the first final texture FT1-1 using at least a portion of a predetermined texture ST that has been stored in advance. In the texture editing step, the first final texture FT1-1 in Figure 12a can be edited into the first final texture FT1-2 in Figure 12d.

[0092] In other words, by utilizing the present invention, a user can edit the second final texture FT2-1 using the first final texture FT1-1 or a predetermined texture ST.

[0093] The initial spatial object generation step S10 according to one embodiment of the present invention may further include a preprocessing step (not shown). The preprocessing step will be described in detail below with reference to Figures 13a to 13e.

[0094] Figure 13a is an illustrative diagram showing the first aerial image PH1-1.

[0095] Referring to Figure 13a, each of the multiple aerial images may contain multiple pixels positioned on a plane extended in the x-axis and y-axis directions. The position of each of the multiple pixels in any one of the multiple aerial images can be determined by its x-coordinate and y-coordinate.

[0096] The first aerial image PH1-1 may contain multiple polygonal pixels corresponding to roof polygon PG-1. Among the multiple x-coordinates corresponding to the multiple polygonal pixels, the smallest value may be the first left-hand side value Xmin1, and the largest value may be the first right-hand side value Xmax1. Among the multiple y-coordinates corresponding to the multiple polygonal pixels, the smallest value may be the first bottom-hand side value Ymin1, and the largest value may be the first top-hand side value Ymax1.

[0097] Figure 13b is a diagram illustrating the superimposed region AR1 and the non-superimposed region AR2 in the first aerial image PH1-1.

[0098] The first aerial image PH1-1 and the second aerial image PH2-1 can be sequentially acquired by a camera CM coupled to a moving aircraft VH. Therefore, a portion of the first aerial image PH1-1 may be superimposed on the second aerial image PH2-1.

[0099] Referring to Figure 13b, the superimposed region AR1 may be the region of the first aerial image PH1-1 that superimposes on the second aerial image PH2-1, and the non-superimposed region AR2 may be the region of the first aerial image PH1-1 that does not superimpose on the second aerial image PH2-1. The superimposed direction DR may be the direction from the centroid C2 of the non-superimposed region toward the centroid C3 of the superimposed region. The non-superimposed length DT may be the length of the non-superimposed region AR2 measured in the superimposed direction DR. In one embodiment of the present invention, the non-superimposed length DT may be provided as a range.

[0100] The roof polygon PG can be placed in the superimposed region AR1. Therefore, if the feature point matching step S12 is performed only on a portion of the second aerial image PH2-1 corresponding to the roof polygon PG, the amount of computation that the computing unit PR must perform may be less than if the feature point matching step S12 is performed on the entire second aerial image PH2-1.

[0101] Figures 13 and 13 are illustrative diagrams illustrating the process by which the computing unit PR determines the cropping area CR in the second aerial image PH2-1.

[0102] In this specification, the cropping region CR refers to an area that, even if removed from the aerial image PH, does not affect the computational unit PR's ability to perform the feature point matching step S12. Referring to Figure 13c, the preprocessing step of the present invention may be a process for removing the cropping region CR, which is an area that does not correspond to the roof polygon PG, from the second aerial image PH2-1, thereby reducing the amount of computation that the computational unit PR must perform.

[0103] Referring to Figures 13c and 13d, in the preprocessing step, the cropping area CR may be the area outside of the region PH2-2 enclosed by the line where the x-coordinate is the second left-hand value (x=Xmin2), the line where the y-coordinate is the second lower-hand value (y=Ymin2), the line where the x-coordinate is the second right-hand value (x=Xmax2), and the line where the y-coordinate is the second upper-hand value (y=Ymax2) in the second aerial image PH2-1.

[0104] In one embodiment of the present invention, when the superposition direction DR is aligned in the x-axis direction, the calculation unit PR can remove the crop target area CR from the second aerial image PH2-1 using formula 1 or formula 2. When the superposition direction DR is aligned in the y-axis direction, the calculation unit PR can remove the crop target area CR from the second aerial image PH2-1 using formula 3 or formula 4.

[0105] [Mathematics 1] Xmin2=Xmin1+Vmin, Xmax2=Xmax1+Vmax Ymin2 = Ymin1, Ymax2 = Ymax1 [Math 2] Xmin2=Xmin1-Vmax, Xmax1=Xmax1-Vmin Ymin2 = Ymin1, Ymax1 = Ymax1 [Math 3] Xmin2 = Xmin1, Xmax2 = Xmax1 Ymin2=Ymin1+Vmin, Ymax2=Ymax1+Vmax [Math 4] Xmin2 = Xmin1, Xmax2 = Xmax1 Ymin2=Ymin1-Vmin, Ymax2=Ymax1-Vmax

[0106] In each of equations 1 through 4, Xmin1 is the first left-hand side value, Xmax1 is the first right-hand side value, Ymin1 is the first lower-side value, Ymax1 is the first upper-side value, Vmin is the minimum value of the non-overlapping length DT, Vmax is the maximum value of the non-overlapping length DT, Xmin2 is the second left-hand side value, Ymin2 is the second lower-side value, Xmax2 is the second right-hand side value, and Ymax2 is the second upper-side value.

[0107] In the preprocessing step according to one embodiment of the present invention, if the superposition direction DR is in the opposite direction to the x-axis, the computing device PR can remove the crop target area CR from the second aerial image PH2-1 using formula 1.

[0108] In the preprocessing step according to one embodiment of the present invention, if the superposition direction DR is in the same direction as the x-axis direction, the calculation unit PR can use equation 2 to remove the crop target area CR from the second aerial image PH2-1. For example, in Figures 13b and 13c, since the superposition direction DR is in the same direction as the x-axis direction, the calculation unit PR can use equation 2 to determine and remove the crop target area CR from the second aerial image PH2-1.

[0109] In the preprocessing step according to one embodiment of the present invention, if the superposition direction DR is in the opposite direction to the y-axis direction, the calculation unit PR can remove the crop target area CR from the second aerial image PH2-1 using formula 3.

[0110] In the preprocessing step according to one embodiment of the present invention, if the superposition direction DR is in the same direction as the y-axis direction, the computing device PR can remove the crop target area CR from the second aerial image PH2-1 using formula 4.

[0111] Figure 13e is an illustrative diagram showing the second aerial image PH2-2 after the cropping target area CR has been removed from the second aerial image PH2-1 in Figure 13d. Referring to Figure 13e, the second aerial image used in the feature point matching step S12 may be the second aerial image PH2-2 after the cropping target area CR has been removed.

[0112] In one embodiment of the present invention, the pretreatment step may be performed between the roof polygon generation step S11 and the feature point matching step S21. In one embodiment of the present invention, the pretreatment step may be omitted.

[0113] In one embodiment of the present invention, any one of the following steps can be omitted: the final texture generation step S40, the atlas generation step S50, the final spatial object generation step S60, the inpainting step S70, the super-resolution step S80, and the text editing step. Furthermore, although Figure 2 shows the inpainting step S70 and the super-resolution step S80 being performed sequentially, this is merely illustrative, and the order of the inpainting step S70, the super-resolution step S80, and the text editing step is not limited to this in the present invention. In other embodiments of the present invention, the order of the inpainting step S70, the super-resolution step S80, and the text editing step can be freely changed.

[0114] As described with reference to the examples, a person skilled in the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and scope of the invention as described in the claims below. Furthermore, the disclosed examples of the present invention are not intended to limit the technical spirit of the invention, and all technical spirit within the claims below and equivalent scope should be interpreted as being included within the scope of the rights of the present invention. [Explanation of symbols]

[0115] SPM: A method for generating 3D objects using aerial imagery. BD: Building CM: Filming equipment PH:Aerial footage VH:Aircraft MD: Multiple 3D objects PR: Processing unit S10: Initial space object generation step S20: Mapping candidate selection step S30: Intermediate Texture Selection Step S40: Final Texture Generation Step S50: Atlas generation step S60: Final spatial object generation step S70: Inpainting Step S80: Super-resolution step

Claims

1. The aerial imagery is an image of a building taken from the air using a camera, and the process involves an initial space object generation step in which a computing device generates an initial space object, which is a three-dimensional object corresponding to the building, based on multiple aerial images, A mapping candidate selection step in which the calculation device selects an aerial image from among the multiple aerial images as a mapping candidate aerial image, wherein the outer surface of the building is defined on the surface of the building, a building surface normal vector perpendicular to the outer surface of the building is defined, at least one of the multiple aerial images includes an initial texture which is an image corresponding to the outer surface of the building, a shooting direction vector from the shooting device toward the outer surface of the building is defined, and the calculation device selects an aerial image from among the multiple aerial images which includes the initial texture and whose dot product of the shooting direction vector and the building surface normal vector is negative, A method for generating a three-dimensional object using aerial images, comprising: an intermediate texture selection step in which the computing device selects the initial texture with the largest area from among a plurality of initial textures corresponding to a plurality of mapping candidate aerial images as an intermediate texture.

2. Of the aforementioned multiple aerial images, the first aerial image and the second aerial image are aerial images of the building taken from different directions. The aforementioned initial spatial object generation step is: The computing device includes a roof polygon generation step of extracting a roof polygon corresponding to the roof of the building from the first aerial image, A feature point matching step in which the computing device selects at least one first feature point having predetermined characteristics from among a plurality of polygonal pixels corresponding to the roof polygon from the first aerial image, and selects at least one second feature point corresponding to the first feature point from the second aerial image, The calculation step involves the arithmetic unit calculating feature point spatial coordinate information using a collinearity-based forward intersection method based on the first and second feature points, wherein the feature point spatial coordinate information is one of the spatial coordinates of the first and second feature points, The calculation step involves the arithmetic unit using the feature point spatial coordinate information to calculate the height of the building, A method for generating a three-dimensional object using aerial images according to claim 1, comprising a modeling step in which the computing device generates the initial spatial object based on the roof polygon and the height of the building.

3. At least one of the first AI model and the second AI model is stored in the computing device, the first AI model extracts at least one first polygon from the aerial image as a vector image by polygon mapping, the second AI model extracts at least one second polygon from the aerial image as a raster image by object segmentation, and the Backbone of the first AI model and the second AI model are neural network models. The method for generating a three-dimensional object using aerial images according to claim 2, wherein in the step of generating the roof polygon, the computing device uses at least one of the first AI model and the second AI model to extract the roof polygon from the first aerial image.

4. The method for generating a three-dimensional object using aerial images according to claim 3, wherein the second AI model converts the second polygon into a vector image using a skeletonization algorithm and a loss function.

5. The method for generating a three-dimensional object using aerial images according to claim 2, wherein in the feature point matching step, the calculation device uses at least one of KAZE, ORB, and SIFT to calculate at least one of the first feature points and at least one of the second feature points.

6. The method for generating a three-dimensional object using aerial images according to claim 2, wherein in the feature point matching step, the computing device uses at least one of BFMatcher and FLANN to calculate at least one of the second feature points.

7. The method for generating a three-dimensional object using aerial images according to claim 2, wherein in the step of calculating the height of the building, the calculation device further uses a digital elevation model (DEM) that has been stored in advance to calculate the height of the building.

8. A method for generating a three-dimensional object using aerial imagery according to claim 2, wherein in the modeling step, the computing device performs tessellation on the roof polygon to divide the roof polygon into a plurality of triangles, and generates the initial spatial object based on the plurality of triangles and the height of the building.

9. The method for generating a three-dimensional object using aerial images according to claim 8, wherein in the modeling step, the computing device performs the tessellation using the Sweeping Line Algorithm and the Ear Clipping Algorithm.

10. A method for generating a three-dimensional object using aerial images according to claim 1, wherein in the mapping candidate selection step, the computing device selects the mapping candidate aerial images using the Backface Culling Algorithm.

11. Further including a final texture generation step, The method for generating a three-dimensional object using aerial images according to claim 1, wherein in the final texture generation step, the computing device performs an affine transformation on the intermediate texture and then interpolates it to generate the final texture.

12. The exterior of the aforementioned building is provided in multiple units, Further including the atlas generation step, A method for generating a 3D object using aerial images according to claim 11, wherein in the atlas generation step, the computing device repeatedly performs the mapping candidate selection step, the intermediate texture selection step, and the final texture generation step for each of the multiple exterior surfaces of the multiple buildings to generate a plurality of final textures, and generates an atlas using a Binary Space Partitioning Tree based on the plurality of final textures.

13. Further including the final spatial object generation step, A method for generating a three-dimensional object using aerial images according to claim 12, wherein in the final spatial object generation step, the computing device maps the plurality of final textures to the initial spatial object based on the atlas to generate the final spatial object.

14. Further including an inpainting step, A method for generating a three-dimensional object using aerial images according to claim 13, wherein in the inpainting step, the computing device performs inpainting on the final spatial object using a third AI model.

15. Further including a super-resolution step, In the super-resolution step, the computing device uses the super-resolution model to increase the resolution of the final spatial object. The method for generating a 3D object using aerial images according to claim 13, wherein the Backbone of the super-resolution model is a neural network model, the super-resolution model is trained on a plurality of images having different resolutions, and the resolution of the plurality of images is changed using Attention Mechanism.

16. Further texture editing steps are included, The method for generating a three-dimensional object using aerial imagery according to claim 13, wherein in the texture editing step, one of the mapped plurality of final textures is a first final texture and the other one is a second final texture, and the computing device edits the second final texture using at least a portion of a predetermined texture stored in advance or at least a portion of the first texture.

17. The initial spatial object generation step further includes a preprocessing step, Each of the aforementioned aerial images includes a plurality of pixels arranged on a plane extending in the x-axis and y-axis directions, and the position of each of the plurality of pixels is determined by the x-coordinate and y-coordinate. The first aerial image includes a plurality of polygonal pixels corresponding to the roof polygon, the smallest of the plurality of x-coordinates corresponding to the plurality of polygonal pixels is the first left-hand value, the largest is the first right-hand value, and the smallest of the plurality of y-coordinates corresponding to the plurality of polygonal pixels is the first bottom-hand value, the largest is the first top-hand value. In the first aerial image, the region where the first aerial image overlaps with the second aerial image is the overlapping region, the region where they do not overlap is the non-overlapping region, the overlapping direction is the direction from the centroid of the non-overlapping region toward the centroid of the overlapping region, and the non-overlapping length is the length of the non-overlapping region measured in the overlapping direction. In the aforementioned preprocessing step, If the overlapping direction is aligned in the x-axis direction, the calculation device removes the cropping area from the second aerial image using formula 1 or formula 2; if the overlapping direction is aligned in the y-axis direction, the calculation device removes the cropping area from the second aerial image using formula 3 or formula 4; the cropping area is the outer region of the area enclosed by the line where the x-coordinate is the second left-hand value, the line where the y-coordinate is the second lower-hand value, the line where the x-coordinate is the second right-hand value, and the line where the y-coordinate is the second upper-hand value in the second aerial image. [Mathematics 1] Xmin2=Xmin1+Vmin, Xmax2=Xmax1+Vmax Ymin2=Ymin1, Ymax2=Ymax1 [Mathematics 2] Xmin2=Xmin1-Vmax, Xmax1=Xmax1-Vmin Ymin2=Ymin1, Ymax2=Ymax1 [Mathematics 3] Xmin2=Xmin1, Xmax2=Xmax1 Ymin2=Ymin1+Vmin, Ymax2=Ymax1+Vmax [Math 4] Xmin2=Xmin1, Xmax2=Xmax1 Ymin2=Ymin1-Vmin, Ymax2=Ymax1-Vmax In each of the above formulas 1 to 4, Xmin2 is the second left-hand side value, Ymin2 is the second lower-hand side value, Xmax2 is the second right-hand side value, Ymax2 is the second upper-hand side value, Xmin1 is the first left-hand side value, Xmax1 is the first right-hand side value, Ymin1 is the first lower-hand side value, Ymax1 is the first upper-hand side value, Vmin is the minimum value of the non-overlapping length, and Vmax is the maximum value of the non-overlapping length. A method for generating a three-dimensional object using an aerial image according to claim 2, wherein in the feature point matching step, the second aerial image is the second aerial image from which the crop target area has been removed.

18. In the aforementioned preprocessing step, If the superposition direction is opposite to the x-axis direction, the calculation device removes the cropping area from the second aerial image using formula 1, and if the superposition direction is the same as the x-axis direction, the calculation device removes the cropping area from the second aerial image using formula 2. The method for generating a three-dimensional object using an aerial image according to claim 17, wherein if the superposition direction is opposite to the y-axis direction, the calculation device removes the crop target area from the second aerial image using formula 3, and if the superposition direction is the same as the y-axis direction, the calculation device removes the crop target area from the second aerial image using formula 4.

Images