Multi-modal geographic positioning method based on knowledge graph

Abstract

The invention provides a multi-modal geographic positioning method based on a knowledge graph. The method comprises the following steps: graph construction: constructing a geographic knowledge graph; image positioning: calculating the entity similarity between the to-be-queried image and each sub-image so as to select an image positioning candidate sub-image, calculating the image matching similarity between the to-be-queried image and each image positioning candidate sub-image so as to determine an image positioning target sub-image, and taking the latitude and longitude coordinates of each image positioning target sub-image as a positioning result; text positioning: converting a to-be-queried text into map representation, calculating the map similarity between a text query sub-graph and each sub-graph to select a text positioning candidate sub-graph, and calculating the semantic similarity between the to-be-queried text and each text positioning candidate sub-graph, and calculating a text matching similarity corresponding to each text positioning candidate sub-graph based on the graph similarity and the semantic similarity corresponding to each text positioning candidate sub-graph so as to determine a text positioning target sub-graph, and taking latitude and longitude coordinates of each text positioning target sub-graph as a positioning result.

Description

Technical Field

[0001] The present invention relates to the field of geolocation technology, specifically to geolocation technology based on knowledge graphs, and more specifically to a multimodal geolocation method based on knowledge graphs. Background Art

[0002] Geolocation technology generally refers to estimating the geographic location of an image given an image without metadata. Geolocation technology has been widely used in many fields, such as outdoor positioning, disaster relief, and intelligence acquisition.

[0003] Geolocation technology typically relies on images with known geographic locations. Early approaches primarily used handcrafted features, while in the era of deep learning, models are increasingly based on convolutional neural networks. Recent research has focused on improving geolocation performance using pre-trained models based on the Transformer architecture. Representative approaches include classification-based methods, retrieval-based methods, and methods based on open-source map data.

[0004] Among them, the classification-based approach is discussed in the paper "Lukas Haas, Michal Skreta, Silas Alberti, Chelsea Finn. PIGEON: Predicting Image Geolocations. Computer Vision and Pattern Recognition, 2024." This paper proposes modeling the geolocation problem as a classification problem by constructing geographic units to divide the regional surface into a fixed number of categories. This geolocation approach uses a pre-trained CLIP model as the backbone network and adds a linear layer to the visual encoder to predict geographic units. Furthermore, it synthesizes descriptive data related to season, climate, and direction, and designs a multi-task learning framework to achieve parameter sharing and feature enhancement. It also uses the OPTICS clustering algorithm to generate location cluster representations, and combines this with a hierarchical retrieval mechanism to optimize the geolocation results. While this geolocation approach achieves efficient location positioning over a large area, its precision positioning capability is limited in smaller areas, making it difficult to meet high-precision requirements. Furthermore, this approach relies on a large-scale pre-trained model and lacks explicit geographic knowledge guidance, resulting in a limited understanding of specific areas or scenes.

[0005] A retrieval-based approach is presented in the paper "Sergio Izquierdo, Javier Civera. Optimal Transport Aggregation for Visual Place Recognition. Computer Vision and Pattern Recognition, 2024." This paper proposes using images with known geographic locations as reference data to achieve geolocation by extracting and matching visual features between the query and reference images. This geolocation method uses the DINOv2 backbone network and undergoes fine-tuning. It reformulates the soft assignment problem of local features to clusters as an optimal transport problem. It also considers bidirectional relationships from features to clusters and from clusters to features to optimize the feature aggregation process and improve feature quality, thereby achieving better retrieval and positioning results. This geolocation method relies on single-modal visual data and lacks the integration of multimodal information such as text and geographic knowledge. This limits its robustness in complex scenarios and can lead to mislocalization when dealing with seasonal changes or similar scenes.

[0006] A method based on open-source map data is presented in the paper "Alishiba Dsouza, Nicolas Tempelmeier, Ran Yu, Simon Gottschalk, Elena Demidova. WorldKG: A World-Scale Geographic Knowledge Graph. International Conference on Information & Knowledge Management, 2021." This paper proposes constructing a global geographic knowledge graph to provide a comprehensive semantic representation of geographic entities in Open Street View, thereby enabling geolocation. This geolocation method first designs an ontology structure as the backbone of the semantic dataset, automatically extracts the ontology and geographic entities from Open Street View, and enhances the annotation of the entities, resulting in a large-scale and high-precision semantic geographic knowledge graph, thereby enabling geolocation. Although this geolocation method improves the structuredness of the data through ontology constraints, it lacks the support of multimodal information and has certain limitations in processing cross-modal information fusion, which limits its application in downstream tasks.

[0007] In summary, existing geolocation technologies mostly rely on the visual features of the image itself and lack knowledge reasoning capabilities, which affects the accuracy of geolocation in complex environments. Moreover, existing geolocation technologies lack the fusion of multimodal data and fail to fully tap their application potential in geolocation tasks.

[0008] It should be noted that this background information is provided solely to introduce relevant information of the present invention to facilitate understanding of the technical solution of the present invention. It does not necessarily constitute prior art. In the absence of evidence demonstrating that the relevant information was disclosed prior to the filing date of the present invention, the relevant information should not be considered prior art. Summary of the Invention

[0009] Therefore, the purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a multimodal geolocation method based on knowledge graph.

[0010] The objectives of the present invention are achieved through the following technical solutions.

[0011] According to a first aspect of the present invention, a multimodal geographic positioning method based on a knowledge graph is provided for geolocating a query image and a query text, the method comprising: graph construction: constructing a geographic knowledge graph, the graph comprising a plurality of subgraphs, each subgraph comprising a coordinate node and a plurality of entity nodes; wherein each coordinate node corresponds to a latitude and longitude coordinate and a plurality of street view images, and each entity node corresponds to a geographic entity; image positioning: calculating the entity similarity between the query image and each subgraph to select a plurality of image positioning candidate subgraphs; and calculating the image matching similarity between the query image and each image positioning candidate subgraph to determine a plurality of image positioning target subgraphs, and taking the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the query image; wherein the entity similarity indicates the entity similarity between the query image and each subgraph. The similarity between all geographic entities contained in the query image and all geographic entities in the subgraph; the image matching similarity indicates the similarity between the query image and all street view images corresponding to the subgraph; text positioning: the query text is converted into a graph representation to obtain a text query subgraph, and the graph similarity between the text query subgraph and each subgraph is calculated to select multiple text positioning candidate subgraphs; and the semantic similarity between the query text and each text positioning candidate subgraph is calculated. Based on the graph similarity and semantic similarity corresponding to each text positioning candidate subgraph, the text matching similarity corresponding to each text positioning candidate subgraph is calculated to determine multiple text positioning target subgraphs, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text; wherein, the graph similarity indicates the similarity between the text query subgraph and the subgraph.

[0012] In some embodiments of the present invention, the method also includes comprehensive positioning, which includes: determining multiple image positioning target subgraphs corresponding to the image to be queried, and the image matching similarity between the image to be queried and each image positioning target subgraph; determining multiple text positioning target subgraphs corresponding to the text to be queried that matches the image to be queried, and the text matching similarity between the text to be queried and each text target subgraph; taking the union of all image positioning target subgraphs and all text positioning target subgraphs to obtain a target subgraph set; performing weighted summation of the image matching similarity and text matching similarity corresponding to each subgraph in the target subgraph set to obtain a comprehensive similarity corresponding to each subgraph; sorting the comprehensive similarities corresponding to all subgraphs in the target subgraph set in descending order, and selecting the subgraph with the largest comprehensive similarity as the comprehensive positioning target subgraph, and using the latitude and longitude coordinates corresponding to the comprehensive positioning target subgraph as the comprehensive positioning result of the query image and the text to be queried that matches itself.

[0013] In some embodiments of the present invention, the atlas construction includes: performing raster sampling on the target area to obtain multiple longitude and latitude coordinates, and obtaining multiple street view images corresponding to each longitude and latitude coordinate; determining all geographic entities within a preset range centered on each longitude and latitude coordinate and the entity relationship between any two geographic entities; wherein the entity relationship includes a distance relationship, an orientation relationship and a topological relationship, and the topological relationship is adjacent, overlapping or intersecting; the distance relationship represents the spherical distance between geographic entities; the orientation relationship represents the relative orientation angle between geographic entities; when the topological relationship is adjacent, it means that the spherical distance between geographic entities is less than or equal to a preset threshold, when the topological relationship is overlapping, it means that one geographic entity contains another geographic entity, and when the topological relationship is intersecting, it means that the two geographic entities have overlapping parts in space; all longitude and latitude coordinates and all geographic entities are abstracted as coordinate nodes and entity nodes respectively, and based on each A subgraph corresponding to each coordinate node is constructed based on the entity relationship between the longitude and latitude coordinates and all geographic entities within a preset range centered on the longitude and latitude coordinates; wherein, each subgraph includes a coordinate node, multiple entity nodes and multiple edges connecting two nodes, the edges between the coordinate nodes and the entity nodes in each subgraph indicate that the corresponding geographic entity is within the preset range centered on the longitude and latitude coordinates corresponding to the coordinate node, and the edges between the entity nodes in each subgraph indicate the entity relationship between the corresponding geographic entities; based on the longitude and latitude coordinates, the edges between the coordinate nodes are determined to connect all the subgraphs to obtain a geographic knowledge graph; wherein, the edges between the coordinate nodes indicate that two longitude and latitude coordinates are adjacent; wherein, each entity node in the geographic knowledge graph is configured with one or more detection boxes to indicate the location area of ​​the geographic entity indicated by itself in the corresponding street view image, and each entity node is also configured with the entity type corresponding to the geographic entity indicated by itself.

[0014] In some embodiments of the present invention, the image positioning includes: determining all geographic entities contained in the image to be queried, the entity type corresponding to each geographic entity, and the location area of ​​each geographic entity in the image to be queried, and calculating the entity similarity between the image to be queried and each sub-graph based on all geographic entities contained in the image to be queried and the entity type corresponding to each geographic entity and all entity nodes in each sub-graph and the entity type corresponding to each entity node; sorting the entity similarities corresponding to all sub-graphs in descending order, and selecting multiple sub-graphs ranked first as image positioning candidate sub-graphs based on a first preset value; calculating the similarity between the image to be queried and all street view images corresponding to each image positioning candidate sub-graph, so as to obtain the full similarity between the image to be queried and each image positioning candidate sub-graph. local feature similarity; and calculate the similarity between the location area of ​​each geographic entity in the query image and all detection boxes corresponding to all entity nodes of the same entity type in each image positioning candidate subgraph to obtain the local feature similarity between the query image and each image positioning candidate subgraph; perform weighted summation of the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph to obtain the image matching similarity between the query image and each image positioning candidate subgraph; sort the image matching similarities corresponding to all image positioning candidate subgraphs in descending order, and select multiple image positioning candidate subgraphs ranked first as image positioning target subgraphs based on a second preset value, and use the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the query image.

[0015] In some embodiments of the present invention, in the image localization, the entity similarity between the query image and each sub-image is calculated as follows:

[0016]

[0017] in, Represents the query image With subgraph Entity similarity, represents the query image, Indicates the sub-graphs, Represents the query image The entity type set consisting of the entity types corresponding to all geographic entities in , Represented by subgraph The entity type set consisting of the entity types corresponding to all geographic entities in , Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The intersection of Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The union of .

[0018] In some embodiments of the present invention, in the image positioning, the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph are calculated as follows: a preset visual model is used to extract the global feature vector of the query image and the global feature vectors of all street view images corresponding to the current image positioning candidate subgraph; the similarity between the global feature vector of the query image and the global feature vector of each street view image corresponding to the current image positioning candidate subgraph is calculated; and the average of all the calculated similarities is taken to obtain the global feature similarity between the query image and the current image positioning candidate subgraph, or the maximum similarity is selected from all the calculated similarities as the global feature similarity between the query image and the current image positioning candidate subgraph. Feature similarity; a preset visual model is used to extract the local feature vector of the location area of ​​each geographic entity in the query image, as well as the local feature vectors of all detection boxes corresponding to each entity node in the current image positioning candidate subgraph; the similarity between the local feature vector of the location area of ​​each geographic entity in the query image and the local feature vectors of all detection boxes corresponding to all entity nodes of the same entity type in the current image positioning candidate subgraph is calculated and the average is calculated to obtain the local similarity between each geographic entity in the query image and the current image positioning candidate subgraph; the average of the local similarities of all geographic entities in the query image and the current image positioning candidate subgraph is calculated to obtain the local feature similarity between the query image and the current image positioning candidate subgraph.

[0019] In some embodiments of the present invention, the text positioning includes: converting the text to be queried into a text query subgraph based on the relationship between different geographic entities described in the text to be queried; wherein the text query subgraph includes multiple entity nodes and multiple edges connecting two nodes, and each edge represents the relationship between the entity nodes; calculating the matching degree of all entity nodes in the text query subgraph with all entity nodes in each subgraph to obtain the node matching degree of the text query subgraph with each subgraph, and the matching degree of all edges in the text query subgraph with all edges in each subgraph to obtain the edge matching degree of the text query subgraph with each subgraph, adding the node matching degree and the edge matching degree of the same subgraph to obtain the graph similarity between the text to be queried and each subgraph; sorting the graph similarities corresponding to all subgraphs in order from high to low, and selecting multiple subgraphs ranked first as text positioning candidate subgraphs based on a first preset value; A preset multimodal model is used to extract the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text positioning candidate subgraph; the similarity between the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text positioning candidate subgraph is calculated and summed to obtain the semantic similarity between the query text and the current text positioning candidate subgraph; the semantic similarity and graph similarity between the query text and each text positioning candidate subgraph are weighted and summed to obtain the text matching similarity between the query text and each text positioning candidate subgraph; the text matching similarities corresponding to all text positioning candidate subgraphs are sorted in descending order, and multiple text positioning candidate subgraphs ranked first are selected as text positioning target subgraphs based on a second preset numerical value, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text.

[0020] In some embodiments of the present invention, in the text localization, the node matching degree between the text query subgraph and each subgraph is calculated as follows:

[0021]

[0022] in, Represents a text query subgraph With subgraph The node matching degree, represents the text query subgraph, Represents a subgraph queried by text The entity set consisting of all entity nodes in , Represented by subgraph The entity set consisting of all entity nodes in , Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The intersection of Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The union of

[0023] The edge matching degree between the text query subgraph and each subgraph is calculated as follows:

[0024]

[0025] in,

[0026]

[0027] Representing a text query subgraph With subgraph Edge matching degree; Representing a text query subgraph Any edge in , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a text query subgraph The set of middle edges; Representing a subgraph Any edge of , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a subgraph The set of middle edges; Represents an edge With edge The matching score, Represents the indicator function.

[0028] Compared with the existing technology, the advantages of the present invention are: (1) the constructed geographic knowledge graph encodes the spatial relationship between geographic entities through orientation relationship, distance relationship and topological relationship, and introduces street view images to enhance the expression ability of the geographic knowledge graph, thereby providing rich data support for geographic positioning; (2) it supports multimodal geographic positioning based on text, image or a combination of the two, and can flexibly cope with a variety of practical application scenarios; (3) in image positioning and text positioning, it fully utilizes the reasoning ability of the geographic knowledge graph, realizes image positioning by matching the query image with the street view image, and realizes text positioning by parsing the query text into a text query subgraph and then matching it with the subgraph, which not only significantly improves the accuracy of geographic positioning, but also enhances the interpretability of geographic positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0030] Figure 1 1 is a flow chart of a multimodal geolocation method according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of an example of a geographic knowledge graph according to an embodiment of the present invention;

[0032] Figure 3 2 is a schematic diagram of a multimodal geolocation framework flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] As mentioned in the background technology section, existing geolocation technologies mostly rely on the visual features of the image itself and lack knowledge reasoning capabilities, which affects the geolocation accuracy in complex environments. Moreover, existing geolocation technologies lack the fusion of multimodal data and fail to fully tap their application potential in geolocation tasks.

[0035] In order to solve the above problems, the inventors analyzed the existing technologies mentioned in the background technology and proposed to construct a geographic knowledge graph that integrates street view images and spatial relationships of geographic entities, so as to enrich the multimodal expression of the geographic knowledge graph and provide rich data support for geographic positioning; and based on the multimodal information and knowledge reasoning ability in the geographic knowledge graph, match images or texts with the geographic knowledge graph to achieve geographic positioning, thereby overcoming the problems of existing technologies that only rely on the visual features of the image itself and lack knowledge reasoning ability, and significantly improve the positioning accuracy; not only that, it is also proposed that the positioning results of the image and the positioning results of the text corresponding to the image can be integrated to obtain a comprehensive positioning result, so as to further improve the positioning accuracy.

[0036] Based on the analysis of the prior art, the present invention proposes a geographic positioning method based on knowledge graph, which includes graph construction, image positioning, text positioning and comprehensive positioning. Graph construction: constructing a geographic knowledge graph containing multiple subgraphs, and each subgraph corresponds to a latitude and longitude coordinate and multiple street view images; image positioning: calculating the entity similarity between the image to be queried and each subgraph to select multiple image positioning candidate subgraphs, and calculating the image matching degree between the image to be queried and each image positioning candidate subgraph to determine multiple image positioning target subgraphs, and taking the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the image to be queried; text positioning: converting the text to be queried into a graph representation to obtain a text query subgraph, calculating the graph similarity between the text query subgraph and each subgraph to select multiple text positioning candidate subgraphs; and calculating the language similarity between the text to be queried and each text positioning candidate subgraph. Semantic similarity: Based on the graph similarity and semantic similarity corresponding to each text positioning candidate subgraph, the text matching similarity corresponding to each text positioning candidate subgraph is calculated to determine multiple text positioning target subgraphs, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text; Comprehensive positioning: Based on the image matching similarity between the query image and each image positioning target subgraph, and the text matching similarity between the query text matched by the query image and each text target subgraph, the comprehensive similarity is calculated and the subgraph with the largest comprehensive similarity is selected as the comprehensive positioning target subgraph, and the latitude and longitude coordinates corresponding to the comprehensive positioning target subgraph are used as the comprehensive positioning result of the query image and the query text matched by itself.

[0037] In summary, if Figure 1As shown, the present invention provides a multimodal geographic positioning method based on knowledge graph, which is used for geographic positioning of query images and query texts, and the method includes: graph construction: constructing a geographic knowledge graph, which includes multiple subgraphs, each subgraph includes a coordinate node and multiple entity nodes; wherein each coordinate node corresponds to a latitude and longitude coordinate and multiple street view images, and each entity node corresponds to a geographic entity; image positioning: calculating the entity similarity between the query image and each subgraph to select multiple image positioning candidate subgraphs; and calculating the image matching similarity between the query image and each image positioning candidate subgraph to determine multiple image positioning target subgraphs, and taking the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the query image; wherein the entity similarity indicates the entity similarity of the query image. The similarity between all geographic entities contained in the image and all geographic entities in the subgraph; the image matching similarity indicates the similarity between the query image and all street view images corresponding to the subgraph; text positioning: the query text is converted into a graph representation to obtain a text query subgraph, and the graph similarity between the text query subgraph and each subgraph is calculated to select multiple text positioning candidate subgraphs; and the semantic similarity between the query text and each text positioning candidate subgraph is calculated. Based on the graph similarity and semantic similarity corresponding to each text positioning candidate subgraph, the text matching similarity corresponding to each text positioning candidate subgraph is calculated to determine multiple text positioning target subgraphs, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text; wherein, the graph similarity indicates the similarity between the text query subgraph and the subgraph.

[0038] In order to better understand the present invention, the map construction, image positioning, text positioning and comprehensive positioning involved in the multimodal geographic positioning method proposed in the present invention are described in detail below in combination with specific embodiments.

[0039] 1. Graph Construction

[0040] The graph construction instruction constructs a geographic knowledge graph for geographic positioning; the graph includes multiple subgraphs, each subgraph includes a coordinate node and multiple entity nodes; wherein each coordinate node corresponds to a latitude and longitude coordinate and multiple street view images, and each entity node corresponds to a geographic entity.

[0041] According to one embodiment of the present invention, the atlas construction includes: performing raster sampling on the target area to obtain multiple longitude and latitude coordinates, and obtaining multiple street view images corresponding to each longitude and latitude coordinate; determining all geographic entities within a preset range centered on each longitude and latitude coordinate and the entity relationship between any two geographic entities; wherein the entity relationship includes a distance relationship, an orientation relationship and a topological relationship, and the topological relationship is adjacent, overlapping or intersecting; the distance relationship represents the spherical distance between geographic entities; the orientation relationship represents the relative orientation angle between geographic entities; when the topological relationship is adjacent, it means that the spherical distance between geographic entities is less than or equal to a preset threshold, when the topological relationship is overlapping, it means that one geographic entity contains another geographic entity, and when the topological relationship is intersecting, it means that the two geographic entities have overlapping parts in space; all longitude and latitude coordinates and all geographic entities are abstracted as coordinate nodes and entity nodes respectively, and based on each A subgraph corresponding to each coordinate node is constructed based on the entity relationship between the longitude and latitude coordinates and all geographic entities within a preset range centered on the longitude and latitude coordinates; wherein, each subgraph includes a coordinate node, multiple entity nodes and multiple edges connecting two nodes, the edges between the coordinate nodes and the entity nodes in each subgraph indicate that the corresponding geographic entity is within the preset range centered on the longitude and latitude coordinates corresponding to the coordinate node, and the edges between the entity nodes in each subgraph indicate the entity relationship between the corresponding geographic entities; based on the longitude and latitude coordinates, the edges between the coordinate nodes are determined to connect all the subgraphs to obtain a geographic knowledge graph; wherein, the edges between the coordinate nodes indicate that two longitude and latitude coordinates are adjacent; wherein, each entity node in the geographic knowledge graph is configured with one or more detection boxes to indicate the location area of ​​the geographic entity indicated by itself in the corresponding street view image, and each entity node is also configured with the entity type corresponding to the geographic entity indicated by itself.

[0042] During the map construction process, the distance relationship and orientation relationship between geographic entities can be calculated as follows.

[0043]

[0044] in, Representing geographic entities With geographic entities The spherical distance between Representing geographic entities The longitude and latitude coordinates of Representing geographic entities The longitude and latitude coordinates of Represents the distance algorithm.

[0045]

[0046]

[0047] in, Representing geographic entities With geographic entities The relative azimuth between .

[0048] During the map construction process, the 4-intersection model, 9-intersection model, regional link method, relationship matrix table, extended 9-intersection model, 16-intersection model and other methods can be used to calculate whether the topological relationship between geographic entities is overlapping or intersecting.

[0049] In order to better understand the geographic knowledge graph constructed by the present invention, Figure 2 The geographic knowledge graph shown is used as an example for illustration.

[0050] Figure 2 The displayed geographic knowledge graph can be represented as ,in, Represents a node set, including all entity nodes (nodes represented by circles) and all coordinate nodes (nodes represented by hexagons); Represents a set of edges. An edge represents the relationship between the two nodes it connects, including the distance, orientation, and topological relationships between physical nodes, the proximity relationship between coordinate nodes and physical nodes (indicating that the corresponding geographic entity is within a preset range centered on the latitude and longitude coordinates corresponding to the coordinate node), and the adjacent relationship between coordinate nodes.

[0051] Combining the above embodiments and Figure 2 It can be seen that each coordinate node (node ​​represented by a hexagon) corresponds to a latitude and longitude coordinate and multiple street view images. In addition, each coordinate node is also set with an ID number as the globally unique identifier of the coordinate node. Figure 2 The latitude and longitude coordinates corresponding to the leftmost coordinate node are , this coordinate node corresponds to multiple street view images as shown in its lower left corner, and the ID number of this coordinate node is 357 (indicating area 357). Figure 2 The longitude and latitude coordinates corresponding to the coordinate node in the middle are , this coordinate node corresponds to multiple street view images as shown in its lower left corner, and the ID number of this coordinate node is 2801 (indicating area 2801). Figure 2 The latitude and longitude coordinates corresponding to the rightmost coordinate node are The ID number corresponding to this coordinate node is 2802 (indicating area 2802). Due to space limitations, the multiple street view images corresponding to this coordinate node are not included in this article. Figure 2 Shown in.

[0052] Combining the above embodiments and Figure 2It can be seen that each entity node corresponds to a specific geographical entity, such as XXX Library, XXX Park, XXX Shopping Mall. Each entity node is configured with one or more detection boxes to indicate the location area of ​​the geographical entity indicated by itself in the corresponding street view image. For example, an entity node corresponds to a detection box , the detection box corresponds to a coordinate in the street view image The selected area. Each entity node is also configured with the entity type corresponding to the geographic entity it indicates, such as traffic lights, gas stations, scenic spots, lakes, roads, subway stations, hotels, etc. In addition, each entity node also includes the following information: ID number, which is the globally unique identifier of the entity node; geographic entity name (name), which indicates the name of the specific geographic entity corresponding to the entity node. For example, in Figure 2 In the example, the entity node with ID number 07433 represents XXX Park, and the entity type of this entity node is Scenic Area; the entity node with ID number 07569 represents XXX Lake, and the entity type of this entity node is Lake. This entity node corresponds to a detection box, and the detection box corresponds to the area represented by the picture connected to the upper right corner of the entity node (the picture corresponds to a specific area in a street view image); the entity node with ID number 08018 represents XX Street, and the entity type of this entity node is Road. Figure 2 In the figure, due to space limitations, the detection frames corresponding to some entity nodes are not shown. The detection frames corresponding to some entity nodes are represented in the form of pictures containing the coordinate information of the street view image where the detection frames are located. For example, the detection frame corresponding to the entity node represented by ID No. 07569 is the picture containing the coordinate information of the street view image where the detection frame is located, which is connected to the upper right corner; for another example, the detection frame corresponding to the entity node represented by ID No. 07844 is the picture containing the coordinate information of the street view image where the detection frame is located, which is connected to the lower left corner; for another example, the detection frame corresponding to the entity node represented by ID No. 36034 is the picture containing the coordinate information of the street view image where the detection frame is located, which is connected to the lower left corner.

[0053] It should be noted that the one or more detection boxes corresponding to each entity node are determined by performing target detection on all street view images in the subgraph containing the entity node using a pre-trained neural network model. This means that the pre-trained neural network model is used to detect all geographic entities in all street view images, and the entity category and detection box position corresponding to each geographic entity in each street view image are recorded, so that the detection box position corresponding to each geographic entity is assigned to the corresponding entity node. The neural network model can be a YOLO-World model, a GLIP model, a GroundingDINO model, an OV-DINO model, or the like, and is not specifically limited in this disclosure.

[0054] It should also be noted that in order to further enrich the information in the geographic knowledge graph, each entity node can also contain the geometry of the geographic entity, which represents the geometric representation of the specific geographic entity corresponding to the entity node. The geometric representation is divided into three types: point, line, and surface. Points represent single geographic entities (such as traffic lights, gas stations, etc.), lines represent linear geographic entities (such as roads, rivers, etc.), and surfaces represent regional geographic entities (such as buildings, parks, lakes, etc.); the category to which the entity belongs (source) represents the category to which the geographic entity belongs. For example, the category to which the traffic light belongs is "traffic-point", the category to which XX Park belongs is "scenic area-surface", the category to which XX Library belongs is "building-surface", and the category to which XX gas station belongs is "energy-point".

[0055] Based on the above embodiments, it can be known that each edge represents the relationship between the two nodes it connects. Therefore, each edge can be represented by a triple as <head entity (head), relationship type (relation_type), tail entity (tail)>, where the head entity corresponds to the ID number of one of the entity nodes connected by the edge, the tail entity corresponds to the ID number of the other entity node connected by the edge, and the relationship type represents the type of relationship between the two nodes connected by the edge, such as the distance relationship, orientation relationship and topological relationship between entity nodes, the adjacent relationship between coordinate nodes, and the proximity relationship between coordinate nodes and entity nodes.

[0056] Based on the above embodiments and Figure 2 As can be seen from the example of the geographic knowledge graph shown, the geographic knowledge graph constructed by the present invention contains coordinate nodes and entity nodes, wherein the entity nodes contain rich geographic information, and calculate the distance relationship, orientation relationship and topological relationship between different geographic entities through spatial relationships and geometry, so as to accurately encode the relationship between geographic entities; at the same time, street view images are introduced for the coordinate nodes, and the visual information is associated with the coordinate nodes, which enriches the multimodal expression of the geographic knowledge graph and enhances the description ability of complex geographic scenes.

[0057] 2. Image Positioning

[0058] The image positioning instruction determines the geolocation result of the query image based on the geographic knowledge graph. Image positioning selects multiple candidate image positioning subgraphs by calculating the entity similarity between the query image and each subgraph; and then calculates the image matching similarity between the query image and each candidate image positioning subgraph to determine multiple image positioning target subgraphs, with the latitude and longitude coordinates corresponding to each image positioning target subgraph being used as the positioning result of the query image. Entity similarity indicates the degree of similarity between all geographic entities contained in the query image and all geographic entities in the subgraph; and image matching similarity indicates the degree of similarity between the query image and all street view images corresponding to the subgraph.

[0059] According to one embodiment of the present invention, the image positioning includes: determining all geographic entities contained in the query image, the entity type corresponding to each geographic entity, and the location area of ​​each geographic entity in the query image, and calculating the entity similarity between the query image and each sub-graph based on all geographic entities contained in the query image and the entity type corresponding to each geographic entity and all entity nodes in each sub-graph and the entity type corresponding to each entity node; sorting the entity similarities corresponding to all sub-graphs in descending order, and selecting multiple sub-graphs ranked first as image positioning candidate sub-graphs based on a first preset value; calculating the similarity between the query image and all street view images corresponding to each image positioning candidate sub-graph to obtain the full similarity between the query image and each image positioning candidate sub-graph. local feature similarity; and calculate the similarity between the location area of ​​each geographic entity in the query image and all detection boxes corresponding to all entity nodes of the same entity type in each image positioning candidate subgraph to obtain the local feature similarity between the query image and each image positioning candidate subgraph; perform weighted summation of the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph to obtain the image matching similarity between the query image and each image positioning candidate subgraph; sort the image matching similarities corresponding to all image positioning candidate subgraphs in descending order and select the top-ranked multiple image positioning candidate subgraphs as the image positioning target subgraph based on a second preset value, and use the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the query image.

[0060] According to one embodiment of the present invention, in the image localization, the entity similarity between the query image and each sub-image is calculated as follows:

[0061]

[0062] in, Represents the query image With subgraph Entity similarity, represents the query image, Indicates the sub-graphs, Represents the query image The entity type set consisting of the entity types corresponding to all geographic entities in , Represented by subgraph The entity type set consisting of the entity types corresponding to all geographic entities in , Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The intersection of Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The union of .

[0063] According to one embodiment of the present invention, in the image positioning, the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph are calculated as follows: a preset visual model is used to extract the global feature vector of the query image and the global feature vectors of all street view images corresponding to the current image positioning candidate subgraph; the similarity between the global feature vector of the query image and the global feature vector of each street view image corresponding to the current image positioning candidate subgraph is calculated; and the average of all the calculated similarities is taken to obtain the global feature similarity between the query image and the current image positioning candidate subgraph, or the maximum similarity is selected from all the calculated similarities as the global feature similarity between the query image and the current image positioning candidate subgraph. Feature similarity; a preset visual model is used to extract the local feature vector of the location area of ​​each geographic entity in the query image, as well as the local feature vectors of all detection boxes corresponding to each entity node in the current image positioning candidate subgraph; the similarity between the local feature vector of the location area of ​​each geographic entity in the query image and the local feature vectors of all detection boxes corresponding to all entity nodes of the same entity type in the current image positioning candidate subgraph is calculated and the average is calculated to obtain the local similarity between each geographic entity in the query image and the current image positioning candidate subgraph; the average of the local similarities of all geographic entities in the query image and the current image positioning candidate subgraph is calculated to obtain the local feature similarity between the query image and the current image positioning candidate subgraph.

[0064] It should be noted that, in the process of image positioning, existing similarity calculation methods may be used to calculate global feature similarity and local feature similarity, such as cosine similarity, and the present invention does not impose any special limitation.

[0065] In order to better understand the image positioning process, the following is combined with the above embodiment and Figure 3 The multimodal geolocation framework flow chart shown in FIG is used for detailed description. Figure 3 In the figure, the upper part shows the process of image positioning of the query image shown on the left; the lower part shows the process of text positioning of the query text matching the query image shown on the left; Figure 3 The upper and lower parts are the process of comprehensive positioning.

[0066] Step 1: Determine all geographic entities contained in the query image, the entity type corresponding to each geographic entity, and the location area of ​​each geographic entity in the query image. Figure 3 As shown, the query image shown in the figure includes two geographical entities whose entity type is roads, two geographical entities whose entity type is buildings, one geographical entity whose entity type is streetlights, one geographical entity whose entity type is sidewalks, and one geographical entity whose entity type is rivers.

[0067] Step 2: Calculate the entity similarity between the query image and each subgraph based on all geographic entities contained in the query image and the entity type corresponding to each geographic entity, as well as all entity nodes in each subgraph and the entity type corresponding to each entity node. The entity similarity between the query image and each subgraph is calculated as follows:

[0068]

[0069] in, Represents the query image With subgraph Entity similarity, represents the query image, Indicates the sub-graphs, Represents the query image The entity type set consisting of the entity types corresponding to all geographic entities in , Represented by subgraph The entity type set consisting of the entity types corresponding to all geographic entities in , Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The intersection of Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The union of .

[0070] Step 3: Sort the entity similarities corresponding to all sub-images from highest to lowest, and select the top-ranked sub-images as candidate sub-images for image localization based on a first preset value. It should be noted that the first preset value is determined based on actual needs, for example, the first preset value is 60, 50, 40, or 30, and the present invention does not impose any particular limitation.

[0071] Step 4: Calculate the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph; perform weighted summation of the global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph to obtain the image matching similarity between the query image and each image positioning candidate subgraph. The image matching similarity calculation process can be expressed as:

[0072]

[0073] in, Represents the query image and the image localization candidate subgraph Image matching similarity, Represents the weight coefficient corresponding to the global feature similarity, Represents the query image and the image localization candidate subgraph The global feature similarity of Represents the weight coefficient corresponding to the local feature similarity, Represents the query image and the image localization candidate subgraph The local feature similarity.

[0074] The global feature similarity between the query image and each image localization candidate subgraph is calculated as follows: a preset visual model is used to extract the global feature vector of the query image and the global feature vectors of all street view images corresponding to the current image localization candidate subgraph; the similarity between the global feature vector of the query image and the global feature vector of each street view image corresponding to the current image localization candidate subgraph is calculated; and all calculated similarities are averaged to obtain the global feature similarity between the query image and the current image localization candidate subgraph, or the maximum similarity is selected from all calculated similarities as the global feature similarity between the query image and the current image localization candidate subgraph. It should be noted that the preset visual model can be a pre-trained DINOv2Salad model.

[0075] The local feature similarity between the query image and each image positioning candidate subgraph is calculated as follows: a preset visual model is used to extract the local feature vector of the location area of ​​each geographic entity in the query image, as well as the local feature vectors of all detection boxes corresponding to each entity node in the current image positioning candidate subgraph; the similarity between the local feature vector of the location area of ​​each geographic entity in the query image and the local feature vectors of all detection boxes corresponding to all entity nodes of the same entity type in the current image positioning candidate subgraph is calculated and averaged to obtain the local similarity between each geographic entity in the query image and the current image positioning candidate subgraph; the average of the local similarities between all geographic entities in the query image and the current image positioning candidate subgraph is calculated to obtain the local feature similarity between the query image and the current image positioning candidate subgraph.

[0076] Since the calculation process of local feature similarity is relatively complicated, in order to better understand the calculation process, Figure 3 The query image shown in the figure contains seven geographic entities (entity types are buildings, roads, street lights, sidewalks and rivers) as an example to illustrate how to calculate the local feature similarity between the query image and an image positioning candidate subgraph. The specific calculation process is as follows: first calculate the local feature vector of the area where a geographical entity with the entity type of road is located in the query image, and then calculate the similarity between this local feature vector and the local feature vectors of all detection boxes corresponding to all geographical entities with the entity type of road in the current image positioning candidate subgraph, and calculate the average to obtain the local similarity between the geographical entity and the current image positioning candidate subgraph; if there are two geographical entities with the entity type of road in the current image positioning candidate subgraph, and each geographical entity contains two detection boxes, then it is necessary to calculate the similarity between the local feature vector of the area where the geographical entity is located in the query image and the local feature vectors of the four detection boxes corresponding to these two geographical entities, and calculate the average to obtain the local similarity between the geographical entity in the query image and the current image positioning candidate subgraph; then calculate the local similarity between the remaining 6 geographical entities and the current image positioning candidate subgraph according to the above calculation method; finally, calculate the average of the local similarities between the 7 geographical entities and the current image positioning candidate subgraph to obtain the local feature similarity between the query image and the current image positioning candidate subgraph.

[0077] Step 5: Sort the image matching similarities corresponding to all image positioning candidate subgraphs in descending order and select the top-ranked multiple image positioning candidate subgraphs as image positioning target subgraphs based on the second preset value, and use the latitude and longitude coordinates corresponding to each image positioning target subgraph as the positioning result of the image to be queried. It should be noted that the second preset value is determined based on actual needs. For example, the second preset value is 10, 8 or 5, and the present invention does not impose any special restrictions.

[0078] Based on the foregoing embodiments, it can be seen that the image positioning method proposed in the present invention can make full use of the reasoning ability of the knowledge graph, and screen multiple image positioning candidate subgraphs by extracting the geographic entities in the query image and matching them with the subgraphs in the constructed geographic knowledge graph; and by calculating the image matching degree of all street view images corresponding to the query image and each image positioning candidate subgraph, so as to fully mine the geographic entities and their spatial relationship information, thereby improving the reliability of image positioning.

[0079] 3. Text Positioning

[0080] The text positioning instruction determines the geographic positioning result of the query text based on the geographic knowledge graph. The text positioning is performed by converting the query text into a graph representation to obtain a text query subgraph, calculating the graph similarity between the text query subgraph and each subgraph to select multiple text positioning candidate subgraphs; and calculating the semantic similarity between the query text and each text positioning candidate subgraph. Based on the graph similarity and semantic similarity corresponding to each text positioning candidate subgraph, the text matching similarity corresponding to each text positioning candidate subgraph is calculated to determine multiple text positioning target subgraphs, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text. The graph similarity indicates the degree of similarity between the text query subgraph and the subgraph.

[0081] According to one embodiment of the present invention, the text positioning includes: converting the text to be queried into a text query subgraph based on the relationship between different geographical entities described in the text to be queried; wherein the text query subgraph includes multiple entity nodes and multiple edges connecting two nodes, and each edge represents the relationship between the entity nodes; calculating the matching degree of all entity nodes in the text query subgraph with all entity nodes in each subgraph to obtain the node matching degree of the text query subgraph with each subgraph, and the matching degree of all edges in the text query subgraph with all edges in each subgraph to obtain the edge matching degree of the text query subgraph with each subgraph, adding the node matching degree and the edge matching degree of the same subgraph to obtain the graph similarity between the text to be queried and each subgraph; sorting the graph similarities corresponding to all subgraphs in order from high to low, and selecting multiple subgraphs ranked first as text positioning candidate subgraphs based on a first preset value; A preset multimodal model is used to extract the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text positioning candidate subgraph; the similarity between the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text positioning candidate subgraph is calculated and summed to obtain the semantic similarity between the query text and the current text positioning candidate subgraph; the semantic similarity and graph similarity between the query text and each text positioning candidate subgraph are weighted and summed to obtain the text matching similarity between the query text and each text positioning candidate subgraph; the text matching similarities corresponding to all text positioning candidate subgraphs are sorted in descending order, and based on a second preset value, multiple text positioning candidate subgraphs ranked first are selected as text positioning target subgraphs, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the query text. It should be noted that the preset multimodal model is set according to actual needs. For example, a CLIP model can be used, and the present invention does not impose any special restrictions.

[0082] According to one embodiment of the present invention, in the text localization, the node matching degree between the text query subgraph and each subgraph is calculated as follows:

[0083]

[0084] in, Representing a text query subgraph With subgraph The node matching degree, represents the text query subgraph, Represents a subgraph queried by text The entity set consisting of all entity nodes in , Represented by subgraph The entity set consisting of all entity nodes in , Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The intersection of Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The union of

[0085] The edge matching degree between the text query subgraph and each subgraph is calculated as follows:

[0086]

[0087] in,

[0088]

[0089] Representing a text query subgraph With subgraph Edge matching degree; Representing a text query subgraph Any edge in , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a text query subgraph The set of middle edges; Representing a subgraph Any edge of , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a subgraph The set of middle edges; Represents an edge With edge The matching score, Represents the indicator function.

[0090] In order to better understand the text positioning process, the following Figure 3 The multimodal geolocation process diagram shown in FIG is used for detailed description.

[0091] Step 1: Convert the query text into a text query subgraph based on the relationships between different geographic entities described in the query text; wherein, the text query subgraph includes multiple entity nodes and multiple edges connecting two nodes, and each edge represents the relationship between entity nodes. The specific conversion process is: first parse the query text through a semantic analyzer, and then use SpaCy or other language models to perform word segmentation, named entity recognition and relationship extraction to extract the geographic entities and the relationships between geographic entities in the query text; then align the geographic entities and relationships in the query text to the predefined geographic elements and topological relationship categories in the knowledge graph, and convert the query text into a structure similar to the geographic knowledge graph, such as Figure 3 The text query subgraph shown in .

[0092] Step 2: Calculate the matching degree of all entity nodes in the text query subgraph with all entity nodes in each subgraph to obtain the node matching degree between the text query subgraph and each subgraph, and the matching degree of all edges in the text query subgraph with all edges in each subgraph to obtain the edge matching degree between the text query subgraph and each subgraph. Add the node matching degree and edge matching degree of the same subgraph to obtain the graph similarity between the query text and each subgraph. The graph similarity calculation process can be expressed as:

[0093]

[0094] in, Indicates the graph similarity, represents the node matching degree, Indicates the edge matching degree.

[0095] The matching degree between the text query subgraph and each subgraph node is calculated as follows:

[0096]

[0097] in, Representing a text query subgraph With subgraph The node matching degree, represents the text query subgraph, Represents a subgraph queried by text The entity set consisting of all entity nodes in , Represented by subgraph The entity set consisting of all entity nodes in , Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The intersection of Representing a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The union of .

[0098] The edge matching degree between the text query subgraph and each subgraph is calculated as follows:

[0099]

[0100] in,

[0101]

[0102] Representing a text query subgraph With subgraph Edge matching degree; Representing a text query subgraph Any edge in , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a text query subgraph The set of middle edges; Representing a subgraph Any edge of , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a subgraph The set of middle edges; Represents an edge With edge The matching score, represents the indicator function. is an indicator function, if but ,otherwise , for example, if ,but ,otherwise .

[0103] Step 3: Sort the graph similarities corresponding to all sub-graphs in descending order, and select the top sub-graphs as candidate sub-graphs for text localization based on a first preset value.

[0104] Step 4: Use the preset multimodal model to extract the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text location candidate subgraph; calculate the similarity between the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text location candidate subgraph and sum them to obtain the semantic similarity between the query text and the current text location candidate subgraph. The semantic similarity calculation process can be expressed as:

[0105]

[0106] in, Represents the query text and text location candidate subgraph The semantic similarity of Represents a candidate subgraph for text localization the number of corresponding street view images; Represents the global feature vector of the query text; Represents a candidate subgraph for text localization The corresponding Global feature vector of a street view image.

[0107] Step 5: Weighted sum of the semantic similarity and graph similarity between the query text and each text location candidate subgraph to obtain the text matching similarity between the query text and each text location candidate subgraph. The calculation process of text matching similarity can be expressed as:

[0108]

[0109] in, Represents the query text and text location candidate subgraph The text matching similarity of Represents the query text and text location candidate subgraph The weight coefficient of the graph similarity, Represents the query text and text location candidate subgraph The graph similarity of Represents the query text and text location candidate subgraph The weight coefficient of the semantic similarity, Represents the query text and text location candidate subgraph semantic similarity.

[0110] Step 6: Sort the text matching similarities corresponding to all text positioning candidate subgraphs in descending order, and select multiple text positioning candidate subgraphs ranked first as text positioning target subgraphs based on the second preset value, and use the latitude and longitude coordinates corresponding to each text positioning target subgraph as the positioning result of the text to be queried.

[0111] Based on the aforementioned embodiments, the text positioning method proposed in the present invention parses the text to be queried into a text query subgraph represented by a graph structure, and matches the text query subgraph with the subgraph in the geographic knowledge graph to screen multiple text positioning candidate subgraphs; and calculates the text matching degree between the text to be queried and all street view images corresponding to each text positioning candidate subgraph to fully mine geographic entities and their spatial relationship information, thereby improving the reliability of text positioning.

[0112] 4. Comprehensive Positioning

[0113] The comprehensive positioning instruction determines the geographic positioning result corresponding to the image to be queried and the text to be queried that matches it based on the geographic knowledge graph.

[0114] According to one embodiment of the present invention, the comprehensive positioning includes: determining multiple image positioning target subgraphs corresponding to the query image, and the image matching similarity between the query image and each image positioning target subgraph; determining multiple text positioning target subgraphs corresponding to the query text matching the query image, and the text matching similarity between the query text and each text target subgraph; taking the union of all image positioning target subgraphs and all text positioning target subgraphs to obtain a target subgraph set; performing weighted summation of the image matching similarity and the text matching similarity corresponding to each subgraph in the target subgraph set to obtain a comprehensive similarity corresponding to each subgraph; sorting the comprehensive similarities corresponding to all subgraphs in the target subgraph set in descending order, selecting the subgraph with the largest comprehensive similarity as the comprehensive positioning target subgraph, and using the latitude and longitude coordinates corresponding to the comprehensive positioning target subgraph as the comprehensive positioning result of the query image and the query text matching itself.

[0115] Among them, the calculation process of comprehensive similarity can be expressed as:

[0116]

[0117] in, Representing a subgraph The corresponding comprehensive similarity is Representing a subgraph The corresponding weight coefficient of image matching similarity, Representing a subgraph The corresponding image matching similarity, Representing a subgraph The corresponding weight coefficient of text matching similarity, Representing a subgraph The corresponding text matching similarity.

[0118] It should be noted that the image positioning target subgraph determined by image positioning and the text positioning target subgraph determined by text positioning are not necessarily in a one-to-one correspondence. For example, the image positioning target subgraph includes subgraphs. Figure 1 ,son Figure 3 , sub-image 5 and sub-image 7, the text positioning target sub-image includes sub- Figure 1 ,son Figure 2 , sub-image 6 and sub-image 7; in this case, when calculating the comprehensive similarity of each sub-image, the text matching similarity or image matching similarity corresponding to some sub-images can be set to 0 to calculate the comprehensive similarity.

[0119] Based on the above embodiments, it can be seen that compared with the existing technology, the geographic positioning method proposed in the present invention can support image positioning, text positioning and comprehensive positioning of image and text through the constructed geographic knowledge graph. Among them, for image positioning, multiple image positioning candidate subgraphs are screened by extracting geographic entities in the image to be queried and matching them with subgraphs in the constructed geographic knowledge graph; and by calculating the image matching degree of all street view images corresponding to the query image and each image positioning candidate subgraph, the geographic entities and their spatial relationship information are fully mined, thereby improving the reliability of image positioning; for text positioning: by parsing the query text into a graph structure and achieving the geographic positioning of the query text through subgraph matching; for comprehensive positioning: by using a weighted voting mechanism to aggregate the image positioning combination and text positioning results, the accuracy of geographic positioning can be significantly improved.

[0120] The beneficial effects of the present invention are as follows: (1) the constructed geographic knowledge graph encodes the spatial relationship between geographic entities through orientation relationship, distance relationship and topological relationship, and introduces street view images to enhance the expression ability of the geographic knowledge graph, thereby providing rich data support for geographic positioning; (2) it supports multimodal geographic positioning based on text, image or a combination of the two, and can flexibly cope with a variety of practical application scenarios; (3) in image positioning and text positioning, the reasoning ability of the geographic knowledge graph is fully utilized, and image positioning is achieved by matching the image to be queried with the street view image, and text positioning is achieved by parsing the text to be queried into a text query subgraph and then matching it with the subgraph, which not only significantly improves the accuracy of geographic positioning, but also enhances the interpretability of geographic positioning.

[0121] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0122] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0123] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0124] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal geolocation method based on knowledge graph for geolocating query images and query texts, characterized in that: The method comprises: Graph construction: Build a geographic knowledge graph, which includes multiple subgraphs. Each subgraph includes a coordinate node and multiple entity nodes. Each coordinate node corresponds to a latitude and longitude coordinate and multiple street view images, and each entity node corresponds to a geographic entity. Image localization: Calculate the entity similarity between the query image and each subgraph to select multiple candidate subgraphs for image localization. Calculate the image matching similarity between the query image and each candidate subgraph to determine multiple target subgraphs for image localization. The longitude and latitude coordinates corresponding to each target subgraph are used as the localization result for the query image. Entity similarity indicates the degree of similarity between all geographic entities contained in the query image and all geographic entities in the subgraph. Image matching similarity indicates the degree of similarity between the query image and all street view images corresponding to the subgraph. Text localization: The query text is converted into a graph representation to obtain a text query subgraph, and the graph similarity between the text query subgraph and each subgraph is calculated to select multiple text localization candidate subgraphs. The semantic similarity between the query text and each text localization candidate subgraph is also calculated. Based on the graph similarity and semantic similarity corresponding to each text localization candidate subgraph, the text matching similarity corresponding to each text localization candidate subgraph is calculated to determine multiple text localization target subgraphs. The latitude and longitude coordinates corresponding to each text localization target subgraph are used as the localization result of the query text. The graph similarity indicates the degree of similarity between the text query subgraph and the subgraph.

2. The method according to claim 1, characterized in that The method further includes comprehensive positioning, which includes: Determining multiple image positioning target subgraphs corresponding to the query image and image matching similarities between the query image and each image positioning target subgraph; Determining a plurality of text location target subgraphs corresponding to the query text that matches the query image, and a text matching similarity between the query text and each text target subgraph; Calculate the union of all image localization target subgraphs and all text localization target subgraphs to obtain the target subgraph set; Perform weighted summation of the image matching similarity and text matching similarity corresponding to each sub-graph in the target sub-graph set to obtain the comprehensive similarity corresponding to each sub-graph; The comprehensive similarities corresponding to all sub-graphs in the target sub-graph set are sorted in descending order, and the sub-graph with the largest comprehensive similarity is selected as the comprehensive positioning target sub-graph, and the latitude and longitude coordinates corresponding to the comprehensive positioning target sub-graph are used as the comprehensive positioning result of the query image and the query text matching itself.

3. The method according to claim 1, characterized in that The map construction includes: Perform raster sampling on the target area to obtain multiple longitude and latitude coordinates, and obtain multiple street view images corresponding to each longitude and latitude coordinate; Determine all geographic entities within a preset range centered on each longitude and latitude coordinate, as well as the entity relationships between any two geographic entities; wherein entity relationships include distance relationships, orientation relationships, and topological relationships, and topological relationships are adjacent, overlapping, or intersecting; distance relationships represent the spherical distance between geographic entities; orientation relationships represent the relative azimuths between geographic entities; when the topological relationship is adjacent, it means that the spherical distance between the geographic entities is less than or equal to a preset threshold; when the topological relationship is overlapping, it means that one geographic entity contains another geographic entity; and when the topological relationship is intersecting, it means that the two geographic entities have overlapping parts in space; All longitude and latitude coordinates and all geographic entities are abstracted as coordinate nodes and entity nodes, respectively. A subgraph corresponding to each coordinate node is constructed based on the entity relationship between each longitude and latitude coordinate and all geographic entities within a preset range centered on the longitude and latitude coordinate. Each subgraph includes a coordinate node, multiple entity nodes, and multiple edges connecting two nodes. The edges between the coordinate nodes and the entity nodes in each subgraph indicate that the corresponding geographic entity is within the preset range centered on the longitude and latitude coordinates corresponding to the coordinate node. The edges between the entity nodes in each subgraph indicate the entity relationship between the corresponding geographic entities. Based on the longitude and latitude coordinates, edges between coordinate nodes are determined to connect all subgraphs to obtain a geographic knowledge graph; wherein, edges between coordinate nodes indicate that two longitude and latitude coordinates are adjacent; Among them, each entity node in the geographic knowledge graph is configured with one or more detection boxes to indicate the location area of ​​the geographic entity it indicates in the corresponding street view image, and each entity node is also configured with the entity type corresponding to the geographic entity it indicates.

4. The method according to claim 3, characterized in that The image positioning includes: Determine all geographic entities contained in the query image, the entity type corresponding to each geographic entity, and the location area of ​​each geographic entity in the query image, and calculate the entity similarity between the query image and each subgraph based on all geographic entities contained in the query image and the entity type corresponding to each geographic entity and all entity nodes in each subgraph and the entity type corresponding to each entity node; Sort the entity similarities corresponding to all sub-images in descending order, and select the top-ranked sub-images as candidate sub-images for image positioning based on a first preset value; Calculate the similarity between the query image and all street view images corresponding to each image positioning candidate subgraph to obtain the global feature similarity between the query image and each image positioning candidate subgraph; and calculate the similarity between the location area of ​​each geographic entity in the query image and all detection boxes corresponding to all entity nodes of the same entity type in each image positioning candidate subgraph to obtain the local feature similarity between the query image and each image positioning candidate subgraph; The global feature similarity and local feature similarity between the query image and each image positioning candidate subgraph are weightedly summed to obtain the image matching similarity between the query image and each image positioning candidate subgraph; The image matching similarities corresponding to all image positioning candidate subgraphs are sorted in descending order, and multiple image positioning candidate subgraphs ranked first are selected as image positioning target subgraphs based on a second preset value, and the latitude and longitude coordinates corresponding to each image positioning target subgraph are used as the positioning result of the image to be queried.

5. The method according to claim 4, characterized in that In the image localization, the entity similarity between the query image and each sub-image is calculated as follows: in, Represents the query image With subgraph Entity similarity, represents the query image, Indicates the sub-graphs, Represents the query image The entity type set consisting of the entity types corresponding to all geographic entities in , Represented by subgraph The entity type set consisting of the entity types corresponding to all geographic entities in , Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The intersection of Represents the query image Corresponding entity type collection With subgraph Corresponding entity type collection The union of .

6. The method according to claim 5, characterized in that In the image localization, the global feature similarity and local feature similarity between the query image and each image localization candidate sub-image are calculated as follows: A preset visual model is used to extract the global feature vector of the query image and the global feature vectors of all street view images corresponding to the candidate sub-image of the current image; Calculate the similarity between the global feature vector of the query image and the global feature vector of each street view image corresponding to the current image positioning candidate subgraph; and average all the calculated similarities to obtain the global feature similarity between the query image and the current image positioning candidate subgraph, or select the largest similarity from all the calculated similarities as the global feature similarity between the query image and the current image positioning candidate subgraph; Use the preset visual model to extract the local feature vector of the location area of ​​each geographic entity in the query image, as well as the local feature vectors of all detection boxes corresponding to each entity node in the current image positioning candidate subgraph; Calculate the similarity between the local feature vector of the location area of ​​each geographic entity in the query image and the local feature vectors of all detection boxes corresponding to all entity nodes of the same entity type in the current image positioning candidate subgraph, and calculate the average to obtain the local similarity between each geographic entity in the query image and the current image positioning candidate subgraph; The average of the local similarities between all geographic entities in the query image and the candidate subgraphs of the current image is calculated to obtain the local feature similarity between the query image and the candidate subgraphs of the current image.

7. The method according to claim 6, characterized in that The text positioning includes: The query text is converted into a text query subgraph according to the relationships between different geographic entities described in the query text; wherein the text query subgraph includes multiple entity nodes and multiple edges connecting two nodes, and each edge represents the relationship between the entity nodes; Calculate the matching degree of all entity nodes in the text query subgraph with all entity nodes in each subgraph to obtain the node matching degree between the text query subgraph and each subgraph, and the matching degree of all edges in the text query subgraph with all edges in each subgraph to obtain the edge matching degree between the text query subgraph and each subgraph. Add the node matching degree and edge matching degree of the same subgraph to obtain the graph similarity between the query text and each subgraph. Sort the graph similarities corresponding to all subgraphs in descending order, and select the top subgraphs as candidate subgraphs for text localization based on a first preset value; A preset multimodal model is used to extract the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text location candidate sub-image; Calculate the similarity between the global feature vector of the query text and the global feature vector of each street view image corresponding to the current text positioning candidate subgraph and sum them up to obtain the semantic similarity between the query text and the current text positioning candidate subgraph; The semantic similarity and graph similarity between the query text and each text positioning candidate subgraph are weighted and summed to obtain the text matching similarity between the query text and each text positioning candidate subgraph; The text matching similarities corresponding to all text positioning candidate subgraphs are sorted in descending order, and multiple text positioning candidate subgraphs ranked first are selected as text positioning target subgraphs based on a second preset value, and the latitude and longitude coordinates corresponding to each text positioning target subgraph are used as the positioning result of the text to be queried.

8. The method according to claim 7, characterized in that In the text localization, the node matching degree between the text query subgraph and each subgraph is calculated as follows: in, Represents a text query subgraph With subgraph The node matching degree, represents the text query subgraph, Represents a subgraph queried by text The entity set consisting of all entity nodes in , Represented by subgraph The entity set consisting of all entity nodes in , Represents a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The intersection of Represents a text query subgraph Corresponding entity collection With subgraph Corresponding entity collection The union of The edge matching degree between the text query subgraph and each subgraph is calculated as follows: in, Represents a text query subgraph With subgraph Edge matching degree; Represents a text query subgraph Any edge in , and , and Represents an edge The two connected entity nodes, express and the relationship between; Represents a text query subgraph The set of middle edges; Representing a subgraph Any edge of , and , and Represents an edge The two connected entity nodes, express and the relationship between; Representing a subgraph The set of middle edges; Represents an edge With edge The matching score, Represents the indicator function.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1-8.

10. An electronic device, characterized in that: include: One or more processors, and memory, wherein the memory is configured to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images