A mountainous area image geolocation method and system based on a scene semantic graph

Abstract

The application discloses a mountainous area image geographical positioning method and system based on a scene semantic graph. In view of the problems of lack of reference images in mountainous areas and weak target texture of mountainous area images and the characteristics that it is difficult to match and position, a mountainous area image scene information extraction and modeling method is provided to obtain a mountainous area scene semantic graph; a mountainous area scene semantic graph matching method is provided to realize mountainous area image geographical positioning. Compared with the prior art, the application constructs a reference image library of mountainous area rendering images, and provides a mountainous area scene semantic graph similarity matching and positioning algorithm based on a graph editing distance, so that the positioning precision and speed are good, and the application has wide application value.

Description

Technical Field

[0001] This invention belongs to the field of image geolocation and is a method and system for geolocation of mountainous images that combines semantic graph modeling of mountainous scenes with distance matching of approximate graph editing. Background Technology

[0002] Image geolocation is a task that involves identifying and locating the image's capture location given an unlabeled image. Mountain image geolocation is an important branch of image geolocation, with significant application prospects in areas such as search and rescue and tourism development. While some research has been conducted on mountain image geolocation, it still faces several challenges: the number of mountain images is relatively small, making it difficult to cover most mountainous areas and thus hindering the construction of a mountain image database using existing images; mountain scene images often only contain weakly textured targets such as mountains, sky, and vegetation, making image matching based on local texture features difficult, and semantic and relational information is not effectively utilized. These issues pose considerable difficulties for mountain image geolocation. Therefore, exploring mountain image geolocation based on scene semantic maps, effectively utilizing the aforementioned semantic and relational information to construct a retrieval database and perform mountain image matching and localization, provides an alternative approach to mountain image geolocation and is of significant research importance. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method for geolocation of mountainous images based on scene semantic graphs. This invention extracts geometric features from rendered depth images, integrates mountainous semantic information and relational information, establishes a mountainous scene semantic graph retrieval database, and then uses an approximate graph editing distance algorithm for graph matching to obtain the geolocation results of mountainous images.

[0004] To achieve the above objectives, the technical solution provided by this invention is: a method for geolocation of mountainous images based on scene semantic maps, comprising the following steps:

[0005] Step 1: Render depth images of mountainous area scene images based on digital elevation data to obtain a reference image library;

[0006] Step 2: Extract the geometric features of the mountains from the images in the reference image library;

[0007] Step 3: Semantic graph modeling for mountainous scenes;

[0008] The semantic graph is based on an undirected graph structure. Based on the mountain geometric features extracted from the above mountain images, all the mountain polygons in each image and the relationships between them are modeled as graph G(V,E). Graph G consists of a vertex set V and an edge set E. Each mountain polygon constitutes a vertex, and the relationships between mountain polygons constitute edges. Corresponding attribute features are designed for vertices and edges to calculate the edit distance between graphs.

[0009] Step 4: Calculate the edit distance between two semantic graphs based on the algorithm of approximate graph edit distance, calculate the edit path to transform query graph g1 into matching graph g2, and output the edit path and the edit distance of the path. The smaller the edit distance result, the more similar the two graphs are. When the distance is within a certain error range, the localization is considered successful.

[0010] Furthermore, the specific implementation method of step 1 is as follows;

[0011] First, the elevation information of the DEM data is converted into a model with LOD triangulation using the OSG toolset;

[0012] Using the constructed LOD triangulation model, the generation process of real scene images is simulated to create depth-rendered images. In the depth-rendered images, each pixel represents the distance from the shooting position to the real position corresponding to that pixel. Because the range of real depth recorded by the image is limited, the depth normalized value is obtained, with the value of the farthest point being 1 and the value of the closest point being 0.

[0013] Furthermore, the specific implementation method of step 2 is as follows;

[0014] The geometric features of a mountain include seven items, the first six of which are basic features and the seventh is a moment feature.

[0015] ① Area and perimeter

[0016] The area represents the pixel size of the polygon, and the perimeter is the pixel length of the polygon's boundary.

[0017] ② Boundary Description

[0018] Boundaries are described using chain codes, which are a method of representing boundaries by encoding the directions of eight neighboring boundaries.

[0019] ③ Circumscribed rectangle and convex hull

[0020] The circumscribed rectangle completely encloses the polygon. The convex hull is a convex polygon that contains all polygons and has the smallest area. Graham's method can be used to solve this problem.

[0021] ④ Eccentricity and elongation

[0022] The eccentricity and elongation of a polygon are related to its major and minor axes. The major axis is the longest line segment that can be accommodated within the polygon, and the minor axis is the longest line segment perpendicular to the major axis. The eccentricity is the ratio of the minor axis to the major axis of the polygon; the elongation is the ratio of the width to the length of the circumscribed rectangle.

[0023] ⑤ Circularity

[0024] The circularity of a polygon refers to the degree to which it approximates a circle, including: compactness, sphericity, and convexity;

[0025] ⑥ Degree of shape change

[0026] The degree of shape variation represents the difference between a polygon and a standard shape. The calculation process is as follows: calculate the distance from each point on the polygon to the boundary of a circle with equal area, calculate the sum of the squares of all distances, and then divide by the number of points to obtain the mean square error of the distance, which is the degree of shape variation of the mountain polygon.

[0027] ⑦ Moment characteristics

[0028] Hu moments are used to systematically describe the characteristics of polygons. Hu moments construct seven invariant feature quantities. When a polygon undergoes only arbitrary translation, arbitrary scaling, and arbitrary rotation operations, the Hu moments remain unchanged.

[0029] Because the above seven invariant characteristic quantities have inconsistent dimensions and large range variations, they cannot be directly used to calculate the shape differences between polygons. Therefore, a log transformation is first performed on each characteristic value:

[0030] H i =-sign(h i )log(|h i |)

[0031] Where sign is the sign function, if H i If the value is greater than 0, it is 1; if the value is less than 0, it is -1.

[0032] Furthermore, the specific implementation method of step 3 is as follows;

[0033] (1) Scene semantic graph vertex construction

[0034] Vertex construction involves modeling the mountain polygons and scene information in the image as vertices in graph G. The vertices in graph G are represented by V = {v1, v2, ..., v...}. N} represents the set of vertices formed by the mountain polygons, N is the number of mountains in the image, each mountain polygon constitutes a vertex v, and the features of each vertex include the boundary, position, area, principal axis angle and Hu moment features of the mountain polygon;

[0035] (2) Scene semantic graph edge construction

[0036] Edge construction models the relationships between polygons in a mountain landscape as edges in a semantic graph. Each pair of vertices forms an edge, and edges are represented by E = {e...} 11 ,e 12 ...,e ij ...,e NN} indicates that e ij Indicates connection to vertex v i and v j The edges are formed by connecting every pair of vertices, so the number of edges is . The relative relationships between vertices are considered as attributes of edges. Edge attributes include: length and angle.

[0037] Furthermore, the specific implementation method of mountain image localization based on approximate graph editing distance in step 4 is as follows;

[0038] The approximate graph edit distance algorithm is based on ordered trees. When constructing new nodes, it solves the edit path using a bipartite graph matching algorithm. The solution process is as follows: Let the query graph g1 and the matching graph g2 have n and m vertices respectively. All n+m vertices are considered as a new graph g. new For each vertex in g1 and g2, construct an (n+m)×(n+m) edit distance matrix. Perform the same operation on the edges in g1 and g2. Use the bipartite graph matching algorithm to find the optimal vertex edit for the (n+m)×(n+m) edit distance matrix, and then determine the edit of the associated edges based on the optimal vertex edit.

[0039] First, construct the edit distance matrices C for vertices and edges respectively. v and C e μ represents the vertex attribute, ζ represents the edge attribute, and two basic graphs are defined: g1 = (V1, E1, μ1, ζ1) and g2 = (V2, E2, μ2, ζ2); V1 = (u1, ..., u1)... n V2 = (v1, ... v) m V1 and V2 represent the vertex sets of the query graph g1 and the matching graph g2, respectively; E1 and E2 represent the edge sets of the query graph g1 and the matching graph g2, respectively; C v This represents the vertex edit distance matrix of the new graph, with dimensions (x+y)×(x+y), where x=n and y=m; C e This represents the edge edit distance matrix of the new graph, with dimensions (x'+y')×(x'+y'), where... C v With C e The construction principle is as follows:

[0040]

[0041] Define an intermediate variable set ε, which allows you to revert points to ε or insert points from ε into the graph. Use (v→ε) to represent deleting a vertex and (ε→v) to represent inserting a vertex; therefore, in the graph above, c ij This means replacing the i-th vertex of g1 with the j-th vertex of g2 (u i →v j Distance, c iε This indicates that the i-th vertex of g1 is deleted (u i →ε) distance, c εj This indicates that the j-th vertex of g2 is inserted (v j →ε) distance; the lower right (m×n) part of the matrix represents (ε→ε), so all are 0, and the position of the ∞ symbol indicates an invalid editing operation;

[0042] Next, the ordered tree is constructed and explored in a depth-first order, where node p is visited before its child nodes children(p). Then, other paths in the ordered tree are traversed backtracking. The root node of the ordered tree is r, which satisfies the condition of the set of matched vertices. Matched edge set pending-vertices 1,2 (r) = V1∪V2, pending-edges 1,2 (r) = E1∪E2, edited distance g(r) = ∞, unedited distance lb(r) = ∞, initialization The OPEN set contains nodes of an ordered tree, which is a partial set of edit paths;

[0043] Add the root node r to the OPEN set, and then construct the next node of the ordered tree. The construction criterion is: sort the vertex set V1 of g1, and each time start constructing the next node of the ordered tree from the vertex with the most potential and its edit operation; the vertex sorting process is: find a vertex edit path with the minimum cost between bipartite graphs, and then take the vertex with the minimum row coordinate where the edit operation is located as the most potential vertex. This gives all the child nodes children(r) of node r. Then add children(r) to the OPEN set, calculate the cost value g(p) + lb(p) for all ordered tree nodes in the OPEN set, i.e., the partial edit paths, and find the p that minimizes it. min If p min If it is a complete edit path, then return the result; if p min If the value is empty, then backtrack and traverse the other child nodes of the current node's parent node.

[0044] Furthermore, editing operations include replacement, insertion, and deletion, and the objects of these operations include vertices and edges.

[0045] Furthermore, the calculation formula for obtaining the principal axis angle of the polygon is as follows:

[0046]

[0047] μ 20 μ 02 and μ 11 All of these are characteristic quantities constructed from second-order Hu moments.

[0048] This invention provides a mountainous image geolocation system based on scene semantic graphs, comprising the following modules:

[0049] The image library acquisition module is used to render depth images of mountainous area scene pictures based on digital elevation data to obtain a reference image library;

[0050] The feature extraction module is used to extract the geometric features of mountains from images in the reference image library;

[0051] The semantic graph construction module is used for semantic graph modeling of mountainous scenes;

[0052] The semantic graph is based on an undirected graph structure. Based on the mountain geometric features extracted from the above mountain images, all the mountain polygons in each image and the relationships between them are modeled as graph G(V,E). Graph G consists of a vertex set V and an edge set E. Each mountain polygon constitutes a vertex, and the relationships between mountain polygons constitute edges. Corresponding attribute features are designed for vertices and edges to calculate the edit distance between graphs.

[0053] The localization module is used to calculate the edit distance between two semantic graphs based on the algorithm of approximate graph edit distance. It calculates the edit path to transform the query graph g1 into the matching graph g2, and outputs the edit path and the edit distance of the path. The smaller the edit distance result, the more similar the two graphs are. When the distance is within a certain error range, the localization is considered successful.

[0054] Compared with existing technologies, the advantages and beneficial effects of this invention are as follows: This paper proposes, for the first time, a geolocation method for mountainous images based on scene semantic maps, specifically addressing the problem of geolocation in mountainous areas. Addressing the lack of reference images in mountainous regions and the difficulty in matching and locating targets in mountainous images due to weak target texture, this invention proposes methods for extracting and modeling scene information from mountainous images, and a geolocation method for mountainous images based on graph matching. First, a depth-rendered image with a horizontal viewpoint is obtained using mountainous terrain data. Second, features and semantic relationships are extracted from the mountainous depth-rendered image to construct a mountainous scene semantic map. Finally, geolocation is achieved through semantic map matching. Compared with existing technologies, this invention constructs a reference image library for mountainous rendered images and proposes a mountainous scene semantic map similarity matching algorithm based on graph edit distance, which can effectively achieve geolocation of mountainous images and has broad application value. Attached Figure Description

[0055] Figure 1 For technology roadmap;

[0056] Figure 2 This is a partial detail image of the LOD triangular mesh model;

[0057] Figure 3 A schematic diagram of depth image rendering;

[0058] Figure 4 Examples of mountain polygons and their rectangular features;

[0059] Figure 5 A schematic diagram of semantic graph construction for mountainous scenes;

[0060] Figure 6 A schematic diagram of scene semantic graph modeling for experimental data;

[0061] Figure 7 Examples of edit paths and edit distances between semantic graphs of experimental data;

[0062] Figure 8 A visualization of the distance matching results for editing;

[0063] Figure 9 This is a display of geolocation results in mountainous areas based on scene semantic maps. Detailed Implementation

[0064] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0065] This invention extracts geometric features from rendered depth images, integrates semantic and relational information about mountainous areas, establishes a semantic graph retrieval database for mountainous scenes, and then uses an approximate graph editing distance algorithm for graph matching to obtain geolocation results for mountainous images. The technical approach of this paper mainly includes two parts: a method for extracting and modeling scene information from mountainous images, and geolocation of mountainous images based on graph matching. The overall technical approach is attached. Figure 1 As shown.

[0066] 1. Mountain Depth Image Rendering Based on Digital Elevation Data

[0067] For scene images of mountainous areas, due to the lack of sufficient texture information and reference images, target localization is achieved using remote sensing geographic data, namely Digital Elevation Models (DEMs). DEMs are typically stored as raster data and provide geometric information from a downward vertical perspective. However, scene images are taken from a near-horizontal perspective. Therefore, it is necessary to first construct a Level of Depth (LOD) triangulation from the DEM, then render the image from the user's perspective, and finally create a reference image library.

[0068] (1) Constructing a LOD triangulation model based on DEM

[0069] The Level of Detail (LOD) model is based on a heuristic algorithm and uses a tree structure to encode the model. Each tree node represents a level of detail for a geographic object, with leaf nodes having the richest hierarchy. Considering the large coverage and volume of remote sensing geographic data, the 3D rendering platform needs to support the rendering capability of massive datasets. This invention uses the open-source 3D development toolset OSG to construct a 3D rendering scene and achieve simulated image generation: First, using the OSG toolset, the elevation information of the DEM data is converted into a triangulation model with LOD. (Appendix) Figure 2 This is a partial detail image of the LOD triangular mesh model.

[0070] (2) Rendering simulated depth images based on LOD model

[0071] This invention uses a LOD (Level of Detail) model to simulate the generation process of real-world scene images, creating depth-rendered images. In the depth-rendered image, each pixel represents the distance from the shooting position to its corresponding real-world location. Because the range of real-world depth recorded by an image is limited, this invention obtains a normalized depth value, with a value of 1 for the farthest point and a value of 0 for the closest point.

[0072] Appendix Figure 3 This is a horizontal perspective depth rendering image built based on a Level of Dimension (LOD) model. As you can see, the overall depth rendering image is black, white, and gray. The farther away from the camera, the whiter the color, and vice versa. Details closer to the camera are more clearly visible, while details on distant mountains are more blurred and lack clarity.

[0073] 2. Extraction of geometric features of mountains from depth images in mountainous areas

[0074] The mountain-filled contour in the mountain depth rendering image obtained in step (2) above is a mountain semantic entity. This semantic entity is a polygon, and its geometric features include 7 items. The first six items belong to basic features, and the seventh item belongs to moment features:

[0075] ① Area and perimeter

[0076] Area represents the pixel size of a polygon. Perimeter is the pixel length along the polygon's boundary.

[0077] ② Boundary Description

[0078] Boundaries are described using chain codes, which are a method of representing boundaries by encoding the directions of eight neighboring boundaries.

[0079] ③ Circumscribed rectangle and convex hull

[0080] The circumscribed rectangle completely encloses the polygon. The convex hull is a convex polygon that contains all other polygons and has the smallest area; it is solved using Graham's method.

[0081] ④ Eccentricity and elongation

[0082] The eccentricity and elongation of a polygon are related to its major and minor axes. The major axis is the longest line segment that can fit within the polygon, and the minor axis is the longest line segment perpendicular to the major axis. Eccentricity is the ratio of the minor axis to the major axis of the polygon; elongation is the ratio of the width to the length of the circumscribed rectangle.

[0083] ⑤ Circularity

[0084] The circularity of a polygon refers to the degree to which it approximates a circle, including: compactness, sphericity, and convexity.

[0085] ⑥ Degree of shape change

[0086] The degree of shape variation represents the difference between a polygon and a standard shape. The calculation process is as follows: calculate the distance from each point on the polygon to the boundary of a circle of equal area, calculate the sum of the squares of all distances and then divide by the number of points to obtain the mean square error of the distance, which is the degree of shape variation of the mountain polygon.

[0087] ⑦ Moment characteristics

[0088] Hu moments can be used to systematically describe the characteristics of polygons. Hu moments construct seven invariant characteristic quantities. When a polygon undergoes only arbitrary translation, arbitrary scaling, and arbitrary rotation operations, the Hu moments remain unchanged.

[0089] It should be noted that the above seven invariant eigenvalues, due to their inconsistent dimensions and large range variations, cannot be directly used to calculate the shape differences between polygons. Therefore, a logarithmic transformation is first performed on each eigenvalue:

[0090] H i =-sign(h i )log(|h i |)

[0091] Where sign is the sign function, if H i If the value is greater than 0, it is 1; if the value is less than 0, it is -1.

[0092] Appendix Figure 4 This is an example of a mountain polygon, with the basic moment features and Hu moments extracted with its lower left corner as the origin, and the centroid and angles calculated based on the moment features.

[0093] 3. Semantic Graph Modeling for Mountainous Scenes

[0094] Appendix Figure 5This is a schematic diagram of semantic graph construction for a mountainous scene. The semantic graph is mainly based on an undirected graph structure, modeling all the mountain polygons in each image and the relationships between them as a graph G(V, E), obtained from the extraction of the mountain geometric features of the aforementioned mountain images. Graph G consists of a vertex set V and an edge set E. Each mountain polygon constitutes a vertex. The relationships between mountain polygons constitute edges. Corresponding attribute features are designed for vertices and edges to facilitate the calculation of edit distances between graphs.

[0095] (1) Scene semantic graph vertex construction

[0096] Vertex construction involves modeling the mountain polygons and scene information (such as boundaries, positions, and areas) in the image as vertices in graph G. The vertices in graph G are represented by V = {v1, v2, ..., vN}, where V is the number of mountains in the image, and each mountain polygon constitutes a vertex v. Ultimately, each vertex's features include five items: the boundary, position, area, principal axis angle, and Hu moment of the mountain polygon. The vertex attributes and descriptions of the scene semantic graph are shown in the table below:

[0097] Table 1 Vertex Attributes of Scene Semantic Graph

[0098] Vertex attributes illustrate boundary Stored in array format Location Centroid of the mountain polygon area Number of pixels within the boundary spindle angle Based on Hu moments, the range is [-180°, 180°]. Hu Ju Hu moments after log transformation

[0099] The formula for calculating the principal axis angle of the polygon is as follows:

[0100]

[0101] μ 20 μ 02 and μ 11 All of these are characteristic quantities constructed from second-order Hu moments.

[0102] (2) Scene semantic graph edge construction

[0103] Edge construction models the relationships between polygons in an image as edges in a semantic graph. Each pair of vertices forms an edge. Edges are represented using E = {e...} 11 ,e 12 ...,e NN} represents. Where e ij Indicates connection to vertex v i and v j The edges. An edge is formed by connecting every pair of vertices, so the number of edges is . The relative relationships between vertices are considered as attributes of edges. The main attributes of edges include length and angle. The edge attributes of the scene semantic graph are shown in the table below:

[0104] Table 2 Scene Semantic Graph Edge Attributes

[0105]

[0106] 4. Image localization in mountainous areas based on approximate graph editing distance

[0107] (1) Introduction to the principle

[0108] The approximate graph edit distance algorithm is based on ordered trees and solves the edit path using a bipartite graph matching algorithm when constructing new nodes. The solution process is as follows: Let graphs g1 and g2 have n and m vertices respectively. All n+m vertices are considered as a new graph g. new For each vertex in the graph, an (n+m)×(n+m) edit distance matrix is ​​constructed. The same operation is performed on the edges in g1 and g2. A bipartite graph matching algorithm is used to find the optimal vertex edit based on the (n+m)×(n+m) edit distance matrix, and then the edits of the associated edges are determined based on the optimal vertex edit. Basic edit operations include replacement, insertion, and deletion, and the objects of these operations include vertices and edges.

[0109] First, construct the edit distance matrices C for vertices and edges respectively. v and C e μ represents the vertex attribute, and ζ represents the edge attribute. Define two basic graphs g1 = (V1, E1, μ1, ζ1) and g2 = (V2, E2, μ2, ζ2). V1 = (u1, ..., u1) n V2 = (v1, ... v) m V1 and V2 represent the vertex sets of the query graph g1 and the matching graph g2, respectively, and E1 and E2 represent the edge sets of the query graph g1 and the matching graph g2, respectively. C v This represents the vertex edit distance matrix of the new graph, with dimensions (x+y)×(x+y), where x=n and y=m; C e This represents the edge edit distance matrix of the new graph, with dimensions (x'+y')×(x'+y'), where... C v With C e The construction principle is as follows:

[0110]

[0111] Define an intermediate variable set ε, which allows us to revert points to ε or insert points from ε into the graph. Use (v→ε) to represent deleting a vertex and (ε→v) to represent inserting a vertex. Therefore, in the graph above, c ij This means replacing the i-th vertex of g1 with the j-th vertex of g2 (u i →v j Distance, c iε This indicates that the i-th vertex of g1 is deleted (u i →ε) distance, c εj This indicates that the j-th vertex of g2 is inserted (v j→ε) distance. The lower right (m×n) part of the matrix represents (ε→ε), so it is all 0. The position of the ∞ symbol indicates an invalid editing operation.

[0112] Next, we construct and explore the ordered tree. The construction and exploration of the ordered tree follow a depth-first order, meaning that node p is visited before its child nodes (children(p)). This also means that the construction and exploration of the ordered tree will first explore as deep as possible, and then backtrack to visit other paths in the ordered tree. The root node of the ordered tree is r, and the set of matched vertices is... Matched edge set The set of pending vertices from vertex set V1 to V2 1,2 (r) = V1∪V2, the pending edges from edge set E1 to E2. 1,2 (r) = E1∪E2, edited distance g(r) = ∞, unedited distance lb(r) = ∞. Initialization The OPEN set contains nodes of an ordered tree, which is a partial set of edit paths.

[0113] First, add the root node *r* to the OPEN set, and then construct the next node of the ordered tree. The construction criterion is: sort the vertex set *V1* of *g1*, and each time start constructing the next node of the ordered tree from the vertex with the most potential and its edit operation, thus accelerating the construction process of the ordered tree. The vertex sorting process is: find a vertex edit path with the minimum cost between bipartite graphs, and then take the vertex with the minimum row coordinate where the edit operation is located as the most potential vertex. This gives all the child nodes *children(r)* of node *r*, and then adds *children(r)* to the OPEN set. Then, calculate the cost value *g(p)* + *lb(p)* for all ordered tree nodes (i.e., partial edit paths) in the OPEN set, and find the p that minimizes it. min If p min If it is a complete edit path, then return the result. If p min If the value is empty, then backtrack and traverse the other child nodes of the current node's parent node.

[0114] (2) Experiment on semantic graph matching and localization of images in mountainous areas

[0115] The experiment specifically uses the graph edit distance-based algorithm described above to calculate the edit distance between two semantic graphs, calculating the edit path that transforms the query graph g1 into the matching graph g2. The output edit path (p) is then used. min The distance between the graphs and the path (the minimum cost g(p) + lb(p)). A smaller distance indicates greater similarity between the two graphs. (See attached image.) Figure 6As shown, semantic graph modeling is performed on two mountain images: semantic entities of mountains are obtained after rendering, geometric features of mountains are extracted, and semantic graphs of mountain scenes are established.

[0116] Calculate the distance between semantic graphs g1 and g2 constructed from the two experimental images, respectively. g1 has 5 vertices and 10 edges; g2 has 2 vertices and 1 edge. The results of calculating the distance between g1 and g2 are shown in the table below:

[0117] Table 3 shows the edit distance output results when g1 and g2 are completely different.

[0118]

[0119] As attached Figure 7 As shown, the edit distance between graphs g1 and g2 is 26172.22. The vertex edit path contains 5 pairs of vertices. (0→0) and (2→1) represent replacing vertices 0 and 2 of the query graph g1 with vertices 0 and 1 of the matching graph g4, respectively. (1→ε), (3→ε), and (4→ε) represent deleting all other vertices of g1. Finally, the similarity between images is reflected by the edit distance between semantic graphs. The entire process takes 0.18 seconds and has certain potential for widespread application.

[0120] As attached Figure 8 As shown in the mountain images, the examples of mountains in the mountainous areas show that the mountains 0 and 2 in the left image and the mountains 0 and 1 in the right image are a good match in terms of both area and relative orientation.

[0121] After scene semantic graph matching, a reference result sequence is obtained for each query image. Both the query image and the reference image have latitude and longitude coordinates. The distance between the query image and the reference image can be calculated. If the distance is within a certain error range, the localization is considered successful. The localization result of the query image is as follows: Figure 9 As shown.

[0122] The present invention also provides a mountainous image geolocation system based on scene semantic graphs, comprising the following modules:

[0123] The image library acquisition module is used to render depth images of mountainous area scene pictures based on digital elevation data to obtain a reference image library;

[0124] The feature extraction module is used to extract the geometric features of mountains from images in the reference image library;

[0125] The semantic graph construction module is used for semantic graph modeling of mountainous scenes;

[0126] The semantic graph is based on an undirected graph structure. Based on the mountain geometric features extracted from the above mountain images, all the mountain polygons in each image and the relationships between them are modeled as graph G(V,E). Graph G consists of a vertex set V and an edge set E. Each mountain polygon constitutes a vertex, and the relationships between mountain polygons constitute edges. Corresponding attribute features are designed for vertices and edges to calculate the edit distance between graphs.

[0127] The localization module is used to calculate the edit distance between two semantic graphs based on the algorithm of approximate graph edit distance. It calculates the edit path to transform the query graph g1 into the matching graph g2, and outputs the edit path and the edit distance of the path. The smaller the edit distance result, the more similar the two graphs are. When the distance is within a certain error range, the localization is considered successful.

[0128] The specific implementation methods of each module are the same as those of each step, and will not be described in this invention.

[0129] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A mountainous image geolocalization method based on a scene semantic graph, characterized in that, Comprise the following steps: Step 1, based on the digital elevation data of mountain area scene picture depth image rendering, obtain reference image library; Step 2, the image in the reference image library is extracted; Step 3, mountainous area scene semantic graph modeling; The semantic graph is based on the undirected graph structure, and the relationship between all the mountain polygons and the relationship between them is modeled as a graph G(V, E) based on the above mountain image mountain polygon feature extraction of each image. The graph G is composed of vertex set V and edge set E. Each mountain polygon constitutes a vertex, and the relationship between the mountain polygons and the polygons constitutes an edge. The corresponding attribute features are designed for the vertices and edges respectively, so as to calculate the edit distance between the graphs. The specific implementation mode of step 3 is as follows: (1) scene semantic graph vertex construction Vertex construction is to model the mountain polygons in the image and the scene information as vertices in the graph G, the vertices in the graph G are represented by V = {v1, v2…, vN}, N is the number of mountains in the image, each mountain polygon constitutes a vertex v, and the features of each vertex include the boundary, position, area, principal axis angle and Hu moment features of the mountain polygon. N ​ (2) scene semantic graph edge construction Edge construction is to model the relationship between the mountain polygons in the image as the edge of the semantic graph, and the two vertices constitute an edge between each other, and the edge uses E={e 11 , e 12 ...,e ij ..., e NN} represents, where e ij represents the edge connecting the vertices v i and v j , and each two vertices form an edge, so the number of edges is , and the relative relationship between the vertices is taken as the attribute of the edge, and the attributes of the edge include: length, angle; Step 4, based on the approximate graph edit distance algorithm to calculate the edit distance of two semantic graphs, calculate the edit path of the query graph g1 to the matching graph g2, output the edit path and the edit distance of the path, the smaller the edit distance result, the more similar the two graphs are. When the distance is within a certain error range, it is considered that the positioning is successful.

2. The mountainous image geolocalization method based on scene semantic graph according to claim 1, characterized in that: The specific implementation mode of step 1 is as follows: Firstly, the elevation information of DEM data is converted into LOD triangular net model by using OSG tool set; Using the constructed LOD triangular net model, the generation process of the real scene picture is simulated, and the depth rendering picture is made. In the depth rendering picture, each pixel represents the distance from the corresponding real position to the shooting position. Because the image records the real depth range, the obtained is the depth normalized value, the value of the farthest place is 1, and the value of the nearest place is 0.

3. The mountainous image geolocalization method based on scene semantic graph according to claim 1, characterized in that: The specific implementation mode of step 2 is as follows: The mountain geometric features include 7 items, the first six items belong to basic features, and the seventh item belongs to the matrix feature; ① Area and perimeter The area represents the pixel size of the polygon, and the perimeter is the pixel length on the boundary of the polygon; ② Boundary description The boundary is described by chain code, which is a kind of boundary description by encoding the boundary direction of eight neighborhood; ③ Circumscribed rectangle and convex hull The circumscribed rectangle completely encloses the polygon, and the convex hull is a convex polygon that satisfies that all the polygons fall within it and has the minimum area, which is solved by Graham method; ④ Eccentricity and elongation The eccentricity and elongation of the polygon are related to the long axis and the short axis. The long axis is the longest line segment that can be contained in the polygon, and the short axis is the longest line segment perpendicular to the long axis. The eccentricity is the ratio of the short axis to the long axis; The elongation is the ratio of the width to the length of the circumscribed rectangle; ⑤ Circular degree The circular degree of the polygon represents the degree of approximation to the circle, including: compactness, sphericity, convexity; ⑥ Shape change degree The shape change degree represents the difference between the polygon and the standard shape. The calculation process is as follows: calculate the distance from each point on the polygon to the boundary of the equal-area circle, calculate the sum of the squares of all distances and then divide by the number of points to obtain the mean square error of the distance, which is the shape change degree of the mountain polygon; ⑦ Matrix feature Hu moments are used to systematically describe the characteristics of polygons, and 7 invariant characteristic quantities are constructed when the polygons are only subjected to arbitrary translation, arbitrary scaling and arbitrary rotation operations, and the Hu moments remain unchanged; Since the above 7 invariant characteristic quantities are inconsistent in dimension and vary greatly in range, they cannot be directly used to calculate the shape difference between polygons, so the log transformation is first performed on each characteristic value: where sign is the sign function, which is 1 if greater than 0 and -1 if less than 0.

4. The mountainous image geolocalization method based on scene semantic graph according to claim 1, characterized in that: The specific implementation of mountain image positioning based on approximate graph edit distance in step 4 is as follows: The approximate graph edit distance algorithm is based on an ordered tree, and an edit path is solved according to a bipartite graph matching algorithm when a new node is constructed. The solving process is as follows: assuming that a query graph g1 and a matching graph g2 have n and m vertices respectively, all the n+m vertices are regarded as a new graph g new An (n+m) x (n+m) edit distance matrix is constructed for the vertices, and the same operation is performed for the edges in g1 and g2; the bipartite graph matching algorithm is used to find the optimal vertex edit for the (n+m) x (n+m) edit distance matrix, and then the edit of the associated edges is determined according to the optimal vertex edit. First, the edit distance matrix C of vertices and edges are constructed respectively v and C e , μ represents the vertex attribute, ζ represents the edge attribute, and two basic graphs g1= (V1, E1, μ1, ζ1) and g2= (V2, E2, μ2, ζ2) are defined; V1= (u1, …u n n), V2= (v1, …v m m); V1 and V2 represent the vertex set of the query graph g1 and the matching graph g2 respectively, E1 and E2 represent the edge set of the query graph g1 and the matching graph g2 respectively, C v represents the vertex edit distance matrix of the new graph, and the matrix dimension is (x+y) × (x+y), where x=n, y=m; C e represents the edge edit distance matrix of the new graph, and the matrix dimension is (x’+y’) × (x’+y’), where x’= , y’= ; The construction principle of C v and C e is as follows: C = Define a set of intermediate variables ε, which can be used to recover a point to ε or to insert a point from ε into the graph. Let (v→ε) denote the deletion of a vertex and (ε→v) denote the insertion of a vertex; thus in the above graph, c ij represents the (u i →v j ) distance, c iε represents the (u i →ε) distance, c εj represents the (v j →ε) distance; the (m x n) lower right corner of the matrix represents (ε→ε), so all are 0, and the position of the symbol ∞ represents an invalid edit operation; Next, the ordered tree is constructed and explored in a depth-first order, where node p is visited before its children (p). Then, other paths in the ordered tree are explored backtracking. The root node of the ordered tree is r, and the set of matched vertices (r) is equal to {...}. }, the set of matched edges matched-edges(r) = { }, pending-vertices 1,2 (r) = ∪ pending-edges 1,2 (r) = Edited distance g(r) = ∞, unedited distance lb(r) = ∞, initialization OPEN = { The OPEN set contains nodes of an ordered tree, which is a partial set of edit paths; Add the root node r to the OPEN set, then build the next node of the ordered tree, the criterion of building is: sort the vertex set V1 of g1, build the next node of the ordered tree from the most potential vertex and its editing operation each time; the process of vertex sorting is: find the vertex editing path with the minimum cost between the bipartite graph, then take the vertex with the editing operation of the minimum row coordinate as the most potential vertex, thus all the children(r) of the r node are obtained, then add children(r) to the OPEN set, calculate the cost value g(p) + lb(p) of all the ordered tree nodes in the OPEN set, find the minimum p min If p min is a complete editing path, return the result, if p min is empty, backtrack through the other children of the current node.

5. The mountainous image geolocalization method based on scene semantic graph according to claim 4, characterized in that: The editing operations include replacement, insertion and deletion, and the operation objects include vertices and edges.

6. The mountainous image geolocalization method based on scene semantic graph according to claim 3, characterized in that: The calculation formula of the polygon principal axis angle acquisition method is: μ 20、 μ 02 and μ 11 are characteristic quantities constructed from the second-order Hu moments.

7. A mountainous image geolocalization system based on scene semantic graph, characterized in that, Comprise the following modules: The image library acquisition module is used to render the depth image of the mountain region scene picture based on the digital elevation data, and obtain the reference image library; The feature extraction module is used for mountain geometry feature extraction of images in the reference image library; The semantic graph construction module is used for mountain region scene semantic graph modeling; The semantic graph is based on an undirected graph structure, and all mountain polygons and their relationships between each image obtained by the above mountain image mountain geometry feature extraction are modeled as a graph G(V, E), the graph G is composed of a vertex set V and an edge set E, each mountain polygon constitutes a vertex, and the relationship between the mountain polygons and the polygons constitutes an edge, and corresponding attribute features are designed for the vertices and edges respectively, so as to calculate the edit distance between the graphs. The specific implementation is as follows: (1) Scene semantic graph vertex construction Vertex construction is to model the mountain polygons in the image and the scene information as vertices in the graph G, the vertices in the graph G are represented by V = {v1, v2…, vN}, N is the number of mountains in the image, each mountain polygon constitutes a vertex v, and the features of each vertex include the boundary, position, area, principal axis angle and Hu moment features of the mountain polygon. N ​ (2) Scene semantic graph edge construction Edge construction is to model the relationship between the mountain polygons in the image as the edge of the semantic graph, and the two vertices constitute an edge between each other, and the edge uses E={e 11 , e 12 ...,e ij ..., e NN} represents, where e ij represents the edge connecting vertex v i and v j , and each two vertices form an edge, so the number of edges is , and the relative relationship between the vertices is taken as the attribute of the edge, and the attributes of the edge include: length, angle; The positioning module is used to calculate the edit distance of two semantic graphs based on the approximate graph edit distance algorithm, calculate the edit path of converting the query graph g1 into the matching graph g2, output the edit path and the edit distance of the path, and the smaller the edit distance result, the more similar the two graphs are. When the distance is within a certain error range, it is considered that the positioning is successful.

Images