Method and device for integrating three-dimensional model and geographic information system

Abstract

The present invention provides a method and device for fusing a three-dimensional model with a geographic information system, wherein the method comprises: obtaining a screenshot of a top view of a 3D model and a satellite image to be matched in a geographic information system (GIS); inputting the screenshot of the top view and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model; calculating the mapping area of ​​the 3D model on the satellite image to be matched and the central longitude and latitude of the mapping area based on the matching point pair set; fusing the 3D model with the GIS based on the central longitude and latitude. The present invention expands the types and scope of three-dimensional models in the fusion of three-dimensional models and GIS, and improves the applicability of the fusion method of three-dimensional models and geographic information systems.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for fusing a three-dimensional model with a geographic information system. Background Art

[0002] City Information Modeling (CIM) is an organic complex of urban information in a three-dimensional digital space, based on technologies such as Building Information Modeling (BIM) and Geographic Information System (GIS), integrating multi-dimensional and multi-scale spatial data of the city. The CIM basic platform has become the basic infrastructure of smart cities, which will effectively promote urban transformation and high-quality development. GIS is mainly used to define the scale and geographic space of a city or region, and it does not pay too much attention to details, which makes it difficult to meet the requirements of CIM visualization of multi-dimensional and multi-scale data. Three-dimensional (3D) models naturally have the ability to express detailed information, so researchers in the field of GIS have begun to use 3D models in large quantities and integrate 3D models into GIS to describe spatial objects such as buildings or terrain, which can greatly promote the development of CIM.

[0003] GIS is different from other systems in that its data is based on geographic coordinates. When integrating 3D models, it is necessary to consider the problem of model spatial location mapping in GIS. However, the existing technology is constrained by the complex spatial characteristics of 3D models, and the methods proposed for the spatial location mapping of 3D models in GIS are too limited and only suitable for the mapping of a small number of types of three-dimensional models. Summary of the invention

[0004] The present invention provides a method and device for fusing three-dimensional models and geographic information systems, which are used to solve the defect that the 3D model and GIS mapping method in the prior art is only applicable to some specific types of 3D models, and expand the types and scope of GIS fused three-dimensional models.

[0005] In a first aspect, the present invention provides a method for fusing a three-dimensional model and a geographic information system, comprising:

[0006] Get a screenshot of the top view of the 3D model and the satellite image to be matched in the Geographic Information System (GIS);

[0007] Inputting the top view screenshot and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model;

[0008] Calculate the mapping area of ​​the 3D model on the to-be-matched satellite image and the central longitude and latitude of the mapping area based on the set of matching point pairs;

[0009] Fusing the 3D model with the GIS based on the central longitude and latitude;

[0010] Wherein, the image matching model is constructed based on an image matching algorithm; the matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the top view screenshot, and the second eigenvector corresponds to the satellite image to be matched.

[0011] Optionally, the image matching model includes a feature extraction module and a feature matching module; the feature extraction module is constructed based on a feature extraction algorithm, and the feature matching module is constructed based on a K-nearest neighbor feature point matching algorithm;

[0012] The step of inputting the top view screenshot and the to-be-matched satellite image into an image matching model to obtain a set of matching point pairs output by the image matching model includes:

[0013] Inputting the top view screenshot and the to-be-matched satellite image into the feature extraction module to obtain a first feature vector set and a second feature vector set output by the feature extraction module;

[0014] The first feature vector and the second feature vector are input into the feature matching module to obtain a set of matching point pairs output by the feature matching module.

[0015] Optionally, the calculating, based on the set of matching point pairs, a mapping area of ​​the 3D model on the to-be-matched satellite image and a central longitude and latitude of the mapping area includes:

[0016] The matching point pair set is screened based on the RANSAC algorithm, and a transformation matrix corresponding to the screened matching point pair set is obtained;

[0017] Transform the top view screenshot based on the transformation matrix to obtain a mapping area of ​​the 3D model on the satellite image to be matched;

[0018] The longitude and latitude corresponding to the center point of the mapping area are calculated to obtain the center longitude and latitude.

[0019] Optionally, the step of obtaining the satellite image to be matched includes:

[0020] Acquire multiple candidate satellite image maps; the candidate satellite image maps are obtained by stepping through the full-area satellite image map in the GIS system through a sliding window; the sliding window is predetermined according to the size information of the top view screenshot and the size information of the full-area satellite image map;

[0021] Based on the image matching model, each of the candidate satellite image is matched with the overhead view screenshot in sequence to obtain a set of candidate matching point pairs corresponding to each of the candidate satellite image;

[0022] A satellite image to be matched is determined based on the candidate matching point pair set.

[0023] Optionally, determining a to-be-matched satellite image based on the candidate matching point pair set includes:

[0024] Each of the candidate matching point pair sets is screened based on the RANSAC algorithm, and the candidate satellite image corresponding to the candidate matching point pair set with the largest number of matching point pairs after screening is the satellite image to be matched.

[0025] Optionally, the feature extraction module is constructed based on a feature extraction algorithm, and the feature extraction algorithm is a SIFT algorithm or an ASLFeat algorithm.

[0026] In a second aspect, the present invention further provides a fusion device of a three-dimensional model and a geographic information system, comprising:

[0027] An acquisition module is used to obtain a screenshot of a top view of a 3D model and a satellite image to be matched in a geographic information system (GIS);

[0028] A matching module, used for inputting the top view screenshot and the satellite image to be matched into an image matching algorithm to obtain a set of matching point pairs output by the image matching algorithm;

[0029] A calculation module, used for calculating the mapping area of ​​the 3D model on the to-be-matched satellite image and the central longitude and latitude of the mapping area based on the set of matching point pairs;

[0030] A fusion module, used for fusing the 3D model and the GIS based on the central longitude and latitude;

[0031] The matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the top view screenshot, and the second eigenvector corresponds to the satellite image to be matched.

[0032] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for fusing a three-dimensional model and a geographic information system as described in the first aspect are implemented.

[0033] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for fusing a three-dimensional model and a geographic information system as described in the first aspect.

[0034] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for fusing a three-dimensional model and a geographic information system as described in the first aspect.

[0035] The method and device for fusing a three-dimensional model and a geographic information system provided by the present invention obtains matching point pairs of the two images by taking a screenshot of a top view of a 3D model and a satellite image of the area where the 3D model is located as inputs of an image matching model, and then calculates the mapping area of ​​the 3D model in the GIS based on the matching point pairs and obtains the central longitude and latitude of the corresponding area, and finally maps the 3D model to the corresponding area in the GIS according to the central longitude and latitude. The embodiment of the present application fuses the 3D model and GIS based on image matching, expands the types and scope of three-dimensional models in the fusion of 3D models and GIS, and improves the applicability of the method for fusing the three-dimensional model and the geographic information system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is one of the flow charts of the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of characteristic points provided by an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of calculating a gradient histogram provided by an embodiment of the present invention;

[0040] Figure 4 is a mapping schematic diagram provided by an embodiment of the present invention;

[0041] Figure 5is a schematic diagram of using a sliding window to divide a satellite image map provided by an embodiment of the present invention;

[0042] Figure 6 This is the second flow chart of the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention;

[0043] Figure 7 It is a schematic diagram of the process of screenshot preprocessing provided by an embodiment of the present invention;

[0044] Figure 8 It is a schematic diagram of the structure of a device for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention;

[0045] Fig. 9 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Combine the following Figure 1-Figure 7 The present invention describes a method for fusing a three-dimensional model and a geographic information system provided by an embodiment of the present invention.

[0048] Figure 1 FIG. 1 is one of the flow charts of the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention. Figure 1 As shown, the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention includes:

[0049] Step 110, obtaining a screenshot of the top view of the 3D model and a satellite image to be matched in a geographic information system GIS;

[0050] Step 120, inputting the top view screenshot and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model;

[0051] Step 130, calculating the mapping area of ​​the 3D model on the to-be-matched satellite image and the central longitude and latitude of the mapping area based on the set of matching point pairs;

[0052] Step 140, fusing the 3D model with the GIS based on the central longitude and latitude;

[0053] Wherein, the image matching model is constructed based on an image matching algorithm; the matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the top view screenshot, and the second eigenvector corresponds to the satellite image to be matched.

[0054] The accuracy of satellite images in GIS systems can reach 1m, and the accuracy of models generated by 3D modeling technology is infinitely close to the real thing. Therefore, at similar scales, the satellite image of the area corresponding to the 3D model and the top view screenshot of the 3D model are highly similar. In addition, while ensuring that the orientation of the 3D model is consistent with the orientation of the GIS, you only need to find the central longitude and latitude of the area mapped by the 3D model on the GIS, and then load the 3D model into the corresponding longitude and latitude in the GIS to complete the mapping process between the 3D model and the GIS.

[0055] Therefore, an embodiment of the present invention provides a method for fusing a three-dimensional model and a geographic information system, taking a screenshot of a top view of a 3D model and a satellite image of the area where the 3D model is located as inputs to an image matching model, obtaining matching point pairs of the two images, and then calculating the mapping area of ​​the 3D model in GIS based on the matching point pairs and obtaining the central longitude and latitude of the corresponding area, and finally mapping the 3D model to the corresponding area in GIS according to the central longitude and latitude. The embodiment of the present application fuses the 3D model and GIS based on image matching, and is not restricted by the complex spatial characteristics of the 3D model. The 3D model in the embodiment of the present invention can be a model modeled by 3D software (such as a BIM model), or a three-dimensional model modeled by instruments and equipment (such as an oblique photography model); it can be a three-dimensional model homologous to GIS, or a non-homologous three-dimensional model. The embodiment of the present invention expands the types and scope of GIS fusion of non-homologous three-dimensional models, and improves the applicability of the fusion method of the three-dimensional model and the geographic information system.

[0056] Next, possible implementations of the above steps in specific embodiments are further described.

[0057] Step 110, obtaining a screenshot of the top view of the 3D model and a satellite image to be matched in a geographic information system GIS.

[0058] The 3D model to be mapped is represented as M, the target GIS for model mapping is represented as g, and the top view screenshot of the 3D model is represented as P M , the satellite image of the area corresponding to the 3D model is the satellite image to be matched, denoted as P g .P g contains longitude and latitude information, which can be used to determine P in the GIS system. g The latitude and longitude range corresponding to each pixel point in P M With P gThe orientation is consistent, that is, the top view screenshot of the 3D model and the satellite image to be matched are matched according to the principle of north at the top, south at the bottom, west on the left and east on the right.

[0059] Step 120: input the top view screenshot and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model.

[0060] Wherein, the image matching model is constructed based on an image matching algorithm; the matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the top view screenshot, and the second eigenvector corresponds to the satellite image to be matched.

[0061] Optionally, the image matching model includes a feature extraction module and a feature matching module; the feature extraction module is constructed based on a feature extraction algorithm, and the feature matching module is constructed based on a K-nearest neighbor feature point matching algorithm;

[0062] The step of inputting the top view screenshot and the to-be-matched satellite image into an image matching model to obtain a set of matching point pairs output by the image matching model includes:

[0063] Step 121, inputting the top view screenshot and the to-be-matched satellite image into the feature extraction module, and obtaining a first feature vector set and a second feature vector set output by the feature extraction module;

[0064] Step 122: input the first feature vector and the second feature vector into the feature matching module to obtain a set of matching point pairs output by the feature matching module.

[0065] The feature extraction module is constructed based on a feature extraction algorithm, and the feature extraction algorithm is a SIFT algorithm or an ASLFeat algorithm.

[0066] In one embodiment, the image matching model is constructed based on the SIFT algorithm and the K nearest neighbor feature point matching algorithm.

[0067] The SIFT algorithm is mainly divided into three steps: feature point detection, feature point descriptor generation, and feature point matching. These three steps will be introduced in detail below.

[0068] Feature point detection:

[0069] a. Construct scale space. The establishment of scale space theory is to simulate the multi-scale characteristics of images. Gaussian convolution kernel is the only linear kernel that can achieve scale transformation. The scale space of the image is defined as:

[0070]

[0071] L(x,y,z)=G(x,y,z)*I(x,y);

[0072] Where G(x,y,z) is a scalable Gaussian function. (x,y) is a spatial coordinate and a scale coordinate. The smoothness of the image depends on the size of σ: a coarse scale (low resolution) corresponds to a large σ value, and a fine scale (high resolution) corresponds to a small σ. * represents the convolution operation, and L(x,y,z) represents the scale space of the image I(x,y).

[0073] b. Detecting spatial extreme points. Compare each sampling point with all its adjacent points and compare the size of the image domain and scale domain between them. In this way, the purpose of finding the extreme points in the scale space can be achieved. Figure 2 is a schematic diagram of characteristic points provided by an embodiment of the present invention, such as Figure 2 As shown in the figure, the middle layer in the figure is the current layer, the points marked with white crosses in the middle layer are feature points, and the adjacent points of the feature points in the figure are 8 adjacent points in the current layer and 9 adjacent points in the adjacent previous and next layers. Therefore, when detecting spatial extreme points, a total of 26 adjacent points are detected.

[0074] c. Feature point descriptor generation: Determine the main direction of the feature point by calculating the directional gradient histogram of the feature point.

[0075] After the above steps, SIFT finds feature points of different scales in the corresponding image, and at this time it can be guaranteed that the extracted feature points have scale invariance.

[0076] Feature point descriptor generation:

[0077] After cropping the image, the horizontal and vertical gradient magnitude maps are generated. Figure 3 is a schematic diagram of calculating the gradient histogram provided by an embodiment of the present invention, such as Figure 3 As shown, Figure 3 -a represents a 16×16 cell image of the feature point neighborhood taken in the embodiment of the present invention, and the cell image is divided into 4×4 windows to calculate the gradient histograms (Histograms of oriented gradients, HOG) respectively, Figure 3 -b indicates that each 4×4 window calculates the HOG, and each HOG can be used with a vector v i It means, i=1,2,...,16, Figure 3 -a and Figure 3-b The dot at the center represents the feature point. Calculate the HOG in the 16×16 cells around the feature point. When calculating each cell vector, each feature point will be represented by a 128-dimensional vector. Before calculating the gradient histogram, the coordinate axis is rotated to the main direction of the feature point, so that the extracted features have rotation invariance. Finally, the features are standardized to make the extracted features have illumination invariance. The final vector is the feature vector of the corresponding feature point.

[0078] Based on the SIFT algorithm, the top view screenshot is input into the feature extraction module to obtain a first feature vector set output by the feature extraction module; the satellite image to be matched is input into the feature extraction module to obtain a second feature vector set output by the feature extraction module.

[0079] Feature point matching: k nearest neighbor feature point matching algorithm

[0080] Considering that a large number of feature vectors need to be searched during matching, multiple random kd trees are established, and the specific number is limited by the algorithm operating environment. model The feature vector set representing the feature points of the top view screenshot of the 3D model at a certain scale extracted, that is, the first feature vector set, where f m Indicates F model The eigenvector in F base It represents the set of feature vectors extracted from the satellite image to be matched, that is, the second set of feature vectors. f b Indicates F base Φ represents the set of matching point pairs, which is initialized to an empty set. ψ represents the established kd tree, and σ represents the feature vector v currently to be matched in the k nearest neighbor algorithm. m and the next nearest neighbor vector N next Similarity and nearest neighbor vector N max The similarity ratio is generally set to 0.7, the feature vector With the eigenvector The similarity between Function calculation.

[0081] The specific algorithm is as follows:

[0082]

[0083] In one embodiment, the image matching model is constructed based on the ASLFeat algorithm and the K nearest neighbor feature point matching algorithm.

[0084] In order to improve computational efficiency, ASLFeat mainly uses L2-Net as the benchmark for its network framework, replacing the last three layers of the network with DCN (deformable convolutional network). At the same time, ASLFeat takes advantage of the inherent hierarchy of the network pyramid features and proposes a new multi-scale detection mechanism, that is, a hierarchical scale fusion method, which makes the detected feature points scale-invariant. Compared with the original D2-Net, it changes the judgment criteria of feature points in the channel and can detect more accurate feature points.

[0085] ASLFeat has made some changes to the M function, which can detect feature points and generate feature point descriptors more stably. Γ(I,I ′ ) represents the loss function of the network. I and I ′ Indicates the two pictures that need to be matched, f c and f c ′ represents the descriptor corresponding to the feature point, M(.,.) is the sorting loss of the network. D(.,.) represents the Euclidean distance between two feature points, m p and m n Set their values ​​to 0.2 and 1.0 respectively.

[0086]

[0087]

[0088] P M and P g The feature extraction module is input to the feature extraction module. After iteration of the feature extraction network, the feature points of the two images are finally detected and the descriptors of the corresponding feature points are generated at the same time, that is, the first feature vector set and the second feature vector set are generated. The ASLFeat network has been initialized on the classification task on D2-Net. Compared with the feature points extracted by the traditional SIFT algorithm, the ASLFeat algorithm can extract more and higher-quality feature points, but it has poor adaptability to image rotation and has higher requirements for algorithm running memory and consumes more time.

[0089] The feature matching is the same as the feature point matching based on the k-nearest neighbor feature point matching algorithm described above, and will not be described again here.

[0090] Step 130: Calculate the mapping area of ​​the 3D model on the to-be-matched satellite image and the central longitude and latitude of the mapping area based on the set of matching point pairs.

[0091] Optionally, the calculating, based on the set of matching point pairs, a mapping area of ​​the 3D model on the to-be-matched satellite image and a central longitude and latitude of the mapping area includes:

[0092] Step 131, screening the matching point pair set based on the RANSAC algorithm, and obtaining a transformation matrix corresponding to the screened matching point pair set;

[0093] In this step, the random sample consensus algorithm (RANSAC) is used to screen the set of matching point pairs to obtain the correct matching point pairs. First, several matching point pairs in the set of matching point pairs are randomly selected, and the transformation matrix F is calculated. Then, whether other points satisfy the transformation matrix F is detected, and the inner points that satisfy the transformation matrix are obtained. Based on the iteration of the RANSAC algorithm, multiple transformation matrices are obtained. The transformation matrix with the most inner points is the optimal single mapping transformation matrix based on the two-dimensional point pairs, that is, the transformation matrix corresponding to the set of matching point pairs. By substituting the points in the top view screenshot into the transformation matrix, the corresponding positions of the points in the top view screenshot in the satellite image can be obtained.

[0094] Step 132, transforming the top view screenshot based on the transformation matrix to obtain a mapping area of ​​the 3D model on the satellite image to be matched;

[0095] Figure 4 is a mapping schematic diagram provided by an embodiment of the present invention, such as Figure 4 As shown, Figure 4 The small picture on the upper left is a screenshot of the top view, and the right side is the satellite image to be matched. The line in the picture represents a pair of matching feature points in the top view screenshot and the satellite image to be matched. According to the transformation matrix, the four corners of the top view screenshot are transformed to obtain the corresponding coordinates of the four corners on the satellite image. The mapping positions of the four corners on the satellite image are connected to obtain the mapping area, such as Figure 4 The area selected by the white rectangle in the satellite image is shown in the figure.

[0096] Step 133, calculating the longitude and latitude corresponding to the center point of the mapping area to obtain the center longitude and latitude.

[0097] Calculate the center point in the mapping area, that is, the center pixel of the mapping area. Since the scale is small enough, the longitude and latitude lines on the satellite image can be considered as straight lines. At the same time, the longitude and latitude range of the entire satellite image is known, so the longitude and latitude coordinates corresponding to each pixel of the satellite image can be directly calculated, so the longitude and latitude corresponding to the center pixel of the mapping area can be obtained.

[0098] Step 140: Fusing the 3D model and the GIS based on the central longitude and latitude.

[0099] Since the 3D model contains geographic information, such as the distance information, direction information and scale information of the building, the GIS system also contains the same geographic information. Therefore, after adjusting the scale and orientation of the 3D model and the GIS (usually in accordance with the rule of north at the top, south at the bottom, west at the left and east at the right), it is only necessary to match the position of the 3D model and the GIS system to complete the fusion of the 3D model and GIS. The embodiment of the present invention can determine the mapping position in the 3D model by calculating the center position of the top view screenshot of the 3D model, and the mapping position refers to the position in the 3D model corresponding to the center of the top view screenshot, and move the mapping position of the 3D model to the center longitude and latitude position determined in the GIS, thereby completing the task of accurately mapping the 3D model to the corresponding area of ​​the GIS.

[0100] In the matching process, the satellite image to be matched can be a full-area satellite image saved in the GIS system. The full-area satellite image refers to a complete regional satellite image saved in the GIS system, which can be a region or the entire earth surface. For example, if the GIS system saves a satellite image of City A, and the 3D model corresponds to Street C in District B of City A, then the full-area satellite image of City A is matched with the top view screenshot of the 3D model. On the one hand, the 3D model is mapped to GIS only corresponding to a small area on the full-area satellite image. However, since the full-area satellite image is too large, there are more feature interferences of similar areas during matching, thereby reducing the matching accuracy. On the other hand, if the full-area satellite image is too large, it will also occupy too much memory and fail to complete the task of mapping the 3D model to GIS.

[0101] Optionally, the step of obtaining the satellite image to be matched in the embodiment of the present invention includes:

[0102] Step 510, obtaining a plurality of candidate satellite image maps; the candidate satellite image maps are obtained by stepping through the full-area satellite image map in the GIS system through a sliding window; the sliding window is predetermined according to the size information of the top view screenshot and the size information of the full-area satellite image map;

[0103] The satellite image of the entire area is divided into grids. The size of the grid is determined by the scale of the top view screenshot and the satellite image. By converting the scale, the size of the area mapped by the 3D model on the satellite image can be calculated. M Indicates the scale of the top view screenshot, with S g Indicates the scale of the satellite image, W M , H M Respectively represent the width and height of the top view screenshot, W match , H matchThey represent the width and height of the mapping area of ​​the 3D model on the satellite image. The width and height of the mapping area can be calculated using the following formula.

[0104]

[0105]

[0106] In this way, the satellite image is divided into match and height H match The grid division can ensure that the mapping area of ​​the 3D model is contained in at least one complete grid. The size of the sliding window is determined according to the grid size. Optionally, a sliding window with a 2×2 grid size ensures that the mapping area across the grid can be completely calculated in one feature point matching task, that is, each time P M Match feature points with a sliding window, so there must be a window that includes the mapping area. For the satellite image of the entire area, first, according to the width W match and height H match Divide into grid N is a positive integer, divided into N grids. Then the grids are combined into a sliding window wd j , j = 1, 2, 3, ..., M, M is a positive integer, divided into M windows. The area corresponding to each window is a candidate satellite image.

[0107] In one embodiment, Figure 5 is a schematic diagram of using a sliding window to divide a satellite image map provided by an embodiment of the present invention, such as Figure 5 As shown, Figure 5 The small image on the left shows a screenshot of the top view of the 3D model, and the right shows a satellite image divided into smaller areas by grids. The satellite image of the entire area is divided into 28 grids, and the sliding window size uses a 2×2 grid size. The initial sliding window includes grids 1, 2, 8, and 9. Grids 1, 2, 8, and 9 form a candidate satellite image. The sliding window steps forward with a grid as a step size to enter the next area grid 2, 3, 9, and 10 (such as Figure 5 The first white rectangle in the figure).

[0108] Step 520, matching each of the candidate satellite image with the overhead view screenshot in sequence based on the image matching model to obtain a set of candidate matching point pairs corresponding to each of the candidate satellite image;

[0109] Input the top view screenshot into the feature extraction module of the image matching model to obtain the first feature vector set output by the feature extraction module. Input each candidate satellite image into the feature extraction module of the image matching model to obtain the second feature vector set output by the feature extraction module corresponding to each candidate satellite image. It can be expressed as: wd j , j=1,2,3,……,M.

[0110] The first eigenvector set and wd 1 Input to the feature matching module to obtain the candidate matching point pair set output by the feature matching module; combine the first feature vector set and wd 2 Input to the feature matching module to obtain the candidate matching point pair set output by the feature matching module... and so on, until the top view screenshot is matched with all candidate satellite images. It can be understood that in order to prevent memory overflow, feature extraction and matching can be performed sequentially, and in order to improve efficiency, multiple feature extraction and matching tasks can also be assigned to multiple computers to perform simultaneously.

[0111] The distributed feature point matching algorithm based on multiple candidate satellite images is as follows:

[0112]

[0113]

[0114] Step 530: Determine the satellite image to be matched based on the candidate matching point pair set.

[0115] Each of the candidate matching point pair sets is screened based on the RANSAC algorithm, and the candidate satellite image corresponding to the candidate matching point pair set with the largest number of matching point pairs after screening is the satellite image to be matched.

[0116] The method of screening each of the candidate matching point pairs based on the RANSAC algorithm is the same as the above step 131, which will not be repeated here.

[0117] In one embodiment, there are 8 matching point pairs in the filtered candidate matching point pair set 1, 18 matching point pairs in the filtered candidate matching point pair set 2, and 99 matching point pairs in the filtered candidate matching point pair set 3. Then the candidate satellite image corresponding to the candidate matching point pair set 3 is the satellite image to be matched. Figure 5 As shown, the candidate matching point pair set 3 corresponds to the candidate satellite image composed of grids 11, 12, 18 and 19 (as shown in the second white rectangular box in the figure), and the small white rectangular box between the grids 11, 12, 18 and 19 represents the mapping area of ​​the 3D model in the satellite image.

[0118] It is understandable that after determining the satellite image to be matched, the process can go directly to step 132 without re-performing image matching and matching point pair screening on the top view screenshot and the satellite image to be matched.

[0119] The distributed algorithm-based determination of the satellite image map to be matched provided in the embodiment of the present invention solves the problem of memory overflow caused by the satellite image map of the corresponding area of ​​GIS being too large, and also improves the algorithm operation efficiency, so that the 3D model can be mapped to the specified area in GIS more quickly.

[0120] Figure 6 FIG. 2 is a flow chart of the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention. Figure 6 As shown, the method for integrating a three-dimensional model and a geographic information system provided by an embodiment of the present invention includes:

[0121] 1. Image matching. Input the top view screenshot of the 3D model and the satellite image (hereinafter also referred to as the base map) into the SIFT algorithm. After three small steps: feature point detection; feature point descriptor generation; feature point matching, the algorithm finally outputs a set of feature point vector pairs that correspond to each other between the top view screenshot and the satellite image. For image matching based on deep learning ASLFeat algorithm, the algorithm generates feature point descriptors while detecting feature points. When matching feature points, the K nearest neighbor feature point matching algorithm (the same matching algorithm as SIFT) is used to perform this process. For ease of understanding, Figure 6 Two sizes of top view screenshots are used as examples. The combination of white rectangles and arrows in the figure represents local magnification. After the first selected area in the satellite image of the corresponding GIS area is partially magnified, the satellite image of the first selected area is matched with the larger top view screenshot. At this time, due to the large area of ​​the satellite image, there are more feature interferences in similar areas during matching, which reduces the matching accuracy. The area with dense matching point pairs in the first selected area is selected to obtain the second selected area. The second selected area is partially magnified, and the corresponding 3D model top view screenshot is enlarged accordingly, and the matching accuracy is improved.

[0122] 2. Regional mapping and central longitude and latitude calculation. The RANSAC algorithm is used to calculate the area range of the top view screenshot mapped to the satellite image. If the final area range is the size of the area range mapped on the base map after the top view screenshot is scaled, it can be determined that the mapped area is the correct area for the final 3D model mapping on GIS. After that, the central longitude and latitude of the mapping area are calculated and the longitude and latitude data are returned. The 3D model is loaded to the longitude and latitude on the GIS platform to complete the accurate mapping of the 3D model and GIS.

[0123] The method proposed in the embodiment of the present invention converts the problem of accurate mapping between 3D model and GIS into the matching of two pictures, by matching the screenshot of the top view of the 3D model with the satellite image, and then calculating the specific area of ​​the model screenshot mapped to the satellite image according to the matching result, and at the same time calculating the longitude and latitude of the center of the mapping area, and then loading the 3D model to the corresponding longitude and latitude position on the GIS, thus completing the accurate mapping between the 3D model and GIS. The scene of an area is 3D modeled by 3D modeling technology, and the final model has the fine 3D features of the area, and then the 3D model is mapped to the specified longitude and latitude on the GIS, which makes up for the shortcoming of GIS lacking local detailed 3D information.

[0124] The effects of the embodiments of the present invention are introduced below. The embodiments are all carried out on Google Chrome. First, the 3D model needs to be loaded in the browser and a screenshot of its top view is taken. Because the scale needs to be considered when performing image matching, the screenshot viewing angle height needs to be adjusted when taking a screenshot, and the screenshot also needs to be preprocessed; secondly, because the experiment is carried out on the browser, the SIFT and ASLFeat algorithms encapsulated in the Python language need to be called in the background. Therefore, the experiment is introduced in two parts below. The first part is the preprocessing part: viewing angle height adjustment and screenshot preprocessing; the second part is the part where the SIFT or ASLFeat algorithm returns the result to adjust the model position: algorithm call and parameter passing. For GIS satellite image data, the satellite image of the corresponding area is directly obtained by calling the map API, and its scale is a fixed value.

[0125] Viewing angle adjustment and screenshot preprocessing. Load the 3D model in the browser at different viewing angles and heights. The background automatically calls a function to take a screenshot of the top view of the model. The top view of the 3D model with the largest area is selected as the input image. Because some areas unrelated to the model will be captured when taking screenshots of the model in the browser, the model screenshot needs to be further preprocessed. Design a preprocessing algorithm, first locate the area boundary of the original screenshot (that is, the unprocessed complete screenshot in the browser) and crop it, then binarize the image and retrieve the model screenshot boundary. Figure 7 is a schematic diagram of the screenshot preprocessing process provided by an embodiment of the present invention, such as Figure 7 As shown in the figure, the left picture is a complete screenshot from the browser interface. It can be found that in addition to the top view of the 3D model, there is also the browser interface and most of the background. The right picture is the result returned after calling the preprocessing algorithm. The screenshot after preprocessing removes the browser interface and the background that is not related to the model.

[0126] Algorithm call and parameter passing. Establish a node server and call the image matching model built based on the image matching algorithm by listening to the host port 8086. Specifically, set a button on the model display page, click the button to send a post request to the node server, and the background program will first adjust the scale of the model, then take a screenshot of the screen and pre-process the image, and use this image and the screenshot of the GIS of the corresponding area as parameters to run the image matching model. Specifically, use the child_process.exec() method in node to call the python script. child_process.exec('python test.py') is equivalent to entering python test.py in the command line to achieve the purpose of calling the script. The server completes the algorithm operation and result processing in the background, and finally returns the center longitude and latitude to the web page, automatically completing the movement of the model on the front-end page.

[0127] In the embodiment of the present invention, the more correctly matched feature points there are, the more accurately the 3D model is mapped to the GIS; the less time the algorithm takes, the better its performance. The feature point logarithm represents the feature points extracted from the top view screenshot of the 3D model and the base map that meet the matching accuracy requirements.

[0128] This embodiment mainly uses 3D models (oblique photography model and BIM model) to map to the corresponding GIS to test the algorithm, where models 1 to 3 are oblique photography models, model 4 is a BIM model, and the satellite image of the corresponding GIS area is divided into six scales: L15, L16, L17, L18, L19, and L20. The information of different scales is shown in Table 1.

[0129] Table 1. Correspondence between scale and ratio

[0130]

[0131] The top view screenshot data of the 3D model is shown in Table 2.

[0132] Table 2. 3D model top view screenshot information

[0133]

[0134] Table 3 shows the experimental results of model 1 and base maps of area 1 at different scales, Table 4 shows the experimental results of model 2 and base maps of area 2 at different scales, Table 5 shows the experimental results of model 3 and base maps of area 3 at different scales, and Table 6 shows the experimental results of model 4 and base maps of area 3 at different scales.

[0135] In the original algorithm and the original algorithm after using distributed matching in Table 3, the 3D model failed to match the base map of L15 and L16 scales successfully. The performance difference between the original algorithm and the original algorithm using distributed technology is not significant, but the ASLFeat algorithm using distributed matching can match successfully, which shows that its ability to extract feature points is better than the SIFT algorithm. At the L17 scale, all three algorithms can match successfully on the host with the same configuration, but because the network needs to be trained and a large number of parameters need to be calculated, although the number of matching points is large, it takes too long to reach the standard under the industrial production environment. The scale of the 3D model screenshot is close to the scale of the L16 level base map, with the most matching points and higher matching reliability. In this embodiment, the two algorithms are subjected to matching experiments of different 3D model screenshot models and base maps of different scales. The algorithm has a certain scale adaptability, but in general, the matching effect is best only when the 3D model screenshot and the base map scale are close. If the scale difference is too large, it will cause the situation that the feature difference is large and cannot be matched, and the matching effect is better when the picture scale and pixel are larger. As the number of image pixels increases, the computational complexity of feature point detection and matching increases, and the algorithm time consumption also increases. Whether it is the SIFT algorithm or the ASLFeat algorithm, after adopting distributed technology, when performing image matching, it is not necessary to load the entire GIS base map into memory for matching, which solves the problem of memory overflow due to the large 3D GIS base map.

[0136] Table 3. Model 1 matching experimental results

[0137]

[0138] Table 4. Model 2 matching experimental results

[0139]

[0140]

[0141] Table 5. Model 3 matching experimental results

[0142]

[0143] Table 6. Model 4 matching experimental results

[0144]

[0145] The embodiment of the present invention solves the problem of mapping a 3D model in a corresponding 3DGIS area in practical applications. Using the feature point matching algorithm in image processing, feature points are extracted from the top view screenshots of the 3D model at different scales and the satellite image of the area corresponding to the 3D model, and then the extracted feature points are described using the gradient histogram. Finally, the feature vector of the top view screenshot of the 3D model is matched with the feature vector of the corresponding satellite image using the k-nearest neighbor matching algorithm. The embodiment of the present invention also calculates the size of the area where the screenshot of the 3D model is mapped on the base map according to the scale and size of the 3D model screenshot and the base map, and divides the base map into a larger grid (such as four times the range). In this way, the 3D model screenshot and the divided base map are matched in parallel, which can reduce the feature interference of other similar areas during matching on the one hand; on the other hand, because the base map is divided, the algorithm can perform image matching tasks in parallel on multiple machines, thereby improving the matching efficiency.

[0146] The following is a description of the fusion device of the three-dimensional model and the geographic information system provided by the present invention. The fusion device of the three-dimensional model and the geographic information system described below and the fusion method of the three-dimensional model and the geographic information system described above can be referenced to each other.

[0147] Figure 8 is a schematic diagram of the structure of a fusion device of a three-dimensional model and a geographic information system provided by an embodiment of the present invention, such as Figure 8 As shown, the fusion device of the three-dimensional model and the geographic information system provided in the embodiment of the present invention includes: an acquisition module 810, a matching module 820, a calculation module 830 and a fusion module 840;

[0148] An acquisition module 810 is used to acquire a screenshot of a top view of a 3D model and a satellite image to be matched in a geographic information system GIS;

[0149] A matching module 820 is used to input the top view screenshot and the satellite image to be matched into an image matching algorithm to obtain a set of matching point pairs output by the image matching algorithm;

[0150] A calculation module 830, configured to calculate a mapping area of ​​the 3D model on the to-be-matched satellite image and a central longitude and latitude of the mapping area based on the set of matching point pairs;

[0151] A fusion module 840 is used to fuse the 3D model and the GIS based on the central longitude and latitude;

[0152] The matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the top view screenshot, and the second eigenvector corresponds to the satellite image to be matched.

[0153] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0154] Fig. 9 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a method for fusing a three-dimensional model and a geographic information system, the method comprising: obtaining a top view screenshot of a 3D model and a satellite image to be matched in a geographic information system GIS; inputting the top view screenshot and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model; calculating the mapping area of ​​the 3D model on the satellite image to be matched and the central longitude and latitude of the mapping area based on the matching point pair set; fusing the 3D model and the GIS based on the central longitude and latitude; wherein the image matching model is constructed based on an image matching algorithm; the matching point pair includes a first feature vector and a second feature vector, the first feature vector corresponds to the top view screenshot, and the second feature vector corresponds to the satellite image to be matched.

[0155] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fusion method of the three-dimensional model and the geographic information system provided by the above methods, the method including: obtaining a screenshot of a top view of the 3D model and a satellite image to be matched in a geographic information system GIS; inputting the screenshot of the top view and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model; calculating the mapping area of ​​the 3D model on the satellite image to be matched and the central longitude and latitude of the mapping area based on the set of matching point pairs; fusing the 3D model and the GIS based on the central longitude and latitude; wherein the image matching model is constructed based on an image matching algorithm; the matching point pair includes a first eigenvector and a second eigenvector, the first eigenvector corresponds to the screenshot of the top view, and the second eigenvector corresponds to the satellite image to be matched.

[0157] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fusion method of the three-dimensional model and the geographic information system provided by the above-mentioned methods, the method comprising: obtaining a screenshot of a top view of the 3D model and a satellite image to be matched in a geographic information system GIS; inputting the screenshot of the top view and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model; calculating the mapping area of ​​the 3D model on the satellite image to be matched and the central longitude and latitude of the mapping area based on the set of matching point pairs; fusing the 3D model and the GIS based on the central longitude and latitude; wherein the image matching model is constructed based on an image matching algorithm; the matching point pair comprises a first eigenvector and a second eigenvector, the first eigenvector corresponds to the screenshot of the top view, and the second eigenvector corresponds to the satellite image to be matched.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for integrating a three-dimensional model and a geographic information system, characterized in that: include: Get a screenshot of the top view of the 3D model and the satellite image to be matched in the Geographic Information System (GIS); Inputting the top view screenshot and the satellite image to be matched into an image matching model to obtain a set of matching point pairs output by the image matching model; Calculate the mapping area of ​​the 3D model on the to-be-matched satellite image and the central longitude and latitude of the mapping area based on the set of matching point pairs; Fusing the 3D model with the GIS based on the central longitude and latitude; The image matching model is constructed based on an image matching algorithm; the matching point pair includes a first feature vector and a second feature vector, the first feature vector corresponds to the top view screenshot, and the second feature vector corresponds to the satellite image to be matched; The calculating, based on the set of matching point pairs, a mapping area of ​​the 3D model on the to-be-matched satellite image and a central longitude and latitude of the mapping area includes: The matching point pair set is screened based on the RANSAC algorithm, and a transformation matrix corresponding to the screened matching point pair set is obtained; Transform the top view screenshot based on the transformation matrix to obtain a mapping area of ​​the 3D model on the satellite image to be matched; The longitude and latitude corresponding to the center point of the mapping area are calculated to obtain the center longitude and latitude.

2. The method for integrating a three-dimensional model and a geographic information system according to claim 1, characterized in that: The image matching model includes a feature extraction module and a feature matching module; The feature extraction module is constructed based on a feature extraction algorithm, and the feature matching module is constructed based on a K-nearest neighbor feature point matching algorithm; The step of inputting the top view screenshot and the to-be-matched satellite image into an image matching model to obtain a set of matching point pairs output by the image matching model includes: Inputting the top view screenshot and the to-be-matched satellite image into the feature extraction module to obtain a first feature vector set and a second feature vector set output by the feature extraction module; The first feature vector and the second feature vector are input into the feature matching module to obtain a set of matching point pairs output by the feature matching module.

3. The method for integrating a three-dimensional model and a geographic information system according to claim 1 or 2, characterized in that: The step of obtaining the satellite image to be matched includes: Acquire multiple candidate satellite image maps; the candidate satellite image maps are obtained by stepping through the full-area satellite image map in the GIS system through a sliding window; the sliding window is predetermined according to the size information of the top view screenshot and the size information of the full-area satellite image map; Based on the image matching model, each of the candidate satellite image is matched with the overhead view screenshot in sequence to obtain a set of candidate matching point pairs corresponding to each of the candidate satellite image; A satellite image to be matched is determined based on the candidate matching point pair set.

4. The method for integrating a three-dimensional model and a geographic information system according to claim 3, characterized in that: The determining of the to-be-matched satellite image map based on the candidate matching point pair set includes: Each of the candidate matching point pair sets is screened based on the RANSAC algorithm, and the candidate satellite image corresponding to the candidate matching point pair set with the largest number of matching point pairs after screening is the satellite image to be matched.

5. The method for integrating a three-dimensional model and a geographic information system according to claim 2, characterized in that: The feature extraction module is constructed based on a feature extraction algorithm, and the feature extraction algorithm is a SIFT algorithm or an ASLFeat algorithm.

6. A fusion device of a three-dimensional model and a geographic information system, characterized in that: include: An acquisition module is used to obtain a screenshot of a top view of a 3D model and a satellite image to be matched in a geographic information system (GIS); A matching module, used for inputting the top view screenshot and the satellite image to be matched into an image matching algorithm to obtain a set of matching point pairs output by the image matching algorithm; A calculation module, used for calculating the mapping area of ​​the 3D model on the satellite image to be matched and the central longitude and latitude of the mapping area based on the set of matching point pairs; A fusion module, used for fusing the 3D model with the GIS based on the central longitude and latitude; The matching point pair includes a first feature vector and a second feature vector, the first feature vector corresponds to the top view screenshot, and the second feature vector corresponds to the satellite image to be matched; The calculating, based on the set of matching point pairs, a mapping area of ​​the 3D model on the to-be-matched satellite image and a central longitude and latitude of the mapping area includes: The matching point pair set is screened based on the RANSAC algorithm, and a transformation matrix corresponding to the screened matching point pair set is obtained; Transform the top view screenshot based on the transformation matrix to obtain a mapping area of ​​the 3D model on the satellite image to be matched; The longitude and latitude corresponding to the center point of the mapping area are calculated to obtain the center longitude and latitude.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for fusing a three-dimensional model and a geographic information system as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for fusing a three-dimensional model and a geographic information system as claimed in any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for fusing a three-dimensional model and a geographic information system as claimed in any one of claims 1 to 5 are implemented.

Images