Graph matching method for locating a geographical position in an urban scene

Abstract

The embodiment of the present application provides a kind of urban scene geographic position positioning method of pattern matching, comprising: constructing city element extraction database step;Element target detection step;Element component segmentation step;Element refinement identification step;Scene element knowledge set construction step;Scene layout pattern construction step;Cross view image search step.The embodiment of the present application relies on digital map, extracts the element information in urban scene, and constructs the spatial layout pattern of element, using the retrieval technology based on vector, can establish the mapping relationship between it and digital map for the city video / image data without landmark feature, provides a kind of solution based on pattern reasoning for urban scene geographic position estimation.

Description

Urban Scene Geographic Location Method Based on Graphic Matching Technical Field

[0001] This invention belongs to the field of visual scene geolocation technology, specifically relating to a method for geolocation in urban scenes using graphic matching. Background Technology

[0002] Visual scene geolocation is a scene location recognition technology based on machine learning methods. It is used to mine location information of images / videos to help decision-makers interpret the activities of target scenes / people. With the development of technology and the impetus of social needs, data analysis has entered an era marked by intelligent services. "City, visual data, and geolocation" have become the three basic elements of contemporary big data information mining. However, due to factors such as the lack of location information, insignificant indicative features, and diverse element categories in massive visual data, the massive data in the current Internet era cannot fully leverage its information advantages to effectively assist decision-makers. Therefore, research on scene geolocation technology based on urban visual data, addressing the needs of urban data information mining, can assist in the long-term tracking of the dynamics of people / regions and has significant practical application value.

[0003] In terms of current research, early visual scene geolocation technologies primarily focused on constructing hierarchical tree models of cities, districts, and streets to find solutions. However, these methods are limited to small-scale scenes and cannot meet the task requirements of large-scale, complex scenes within a specified city. In recent years, with the rapid development of deep learning technology, intelligent visual scene geolocation technologies have emerged in two categories: image retrieval-based scene geolocation and image classification-based scene geolocation. Image retrieval-based scene geolocation works by retrieving the most similar reference image from a city-level database, thereby estimating the geographical location of the target image based on the reference image's location. From an image retrieval perspective, prior knowledge of the location of interest to the task is represented as a set of images (database). Each image in the database is labeled with its location identifier, such as a landmark name or GPS coordinates. When a new image needs to be localized (queried), the location recognition system searches the database for similar images. If similar images are found, their labeled locations are used to infer the queried location. This retrieval process is typically implemented as a three-stage pipeline: 1) an encoding process extracts a vector representation of each image's content (image representation); 2) a similarity search performs pairwise comparisons between the query representation and the representations of each image in the database based on a scoring function (e.g., Euclidean distance or cosine similarity), returning the best match; 3) a post-processing stage refines the results of the similarity search. For example, Per-Bundle VLAD uses the classic SIFT operator to determine whether two images have corresponding points, but its drawback is that it is currently limited to image geolocation at the city scale and is difficult to extend to a larger geographic space. Li J et al. investigated how to use image visual features and their similarity to predict the geographic location of images. They employed a method of refining global and local features to improve prediction accuracy, while using a hierarchical structure to reduce the computational cost of local features. Their research shows that using both global and local features can better describe image content, thereby improving the accuracy of similar image matching. However, they also pointed out that it is still difficult to predict the geographic location of images without landmarks, such as beaches or empty mine streets.

[0004] Image-based scene geolocation technology divides the world spatially into multiple geographic units according to latitude and longitude, and trains deep networks to predict the geographic unit to which a given image belongs, thus obtaining its geographic location estimate. In 2016, Google proposed the PlaNet model, which treats image geolocation as a classification problem by subdividing the Earth's surface into thousands of geographic units and training a deep network using millions of geotagged images. It achieved location recognition for natural scenes such as city landmarks, street views, mountains, waterfalls, or beaches. These methods rely on a large number of training images—careful examination of their data yields many images of indoor scenes, food, and pets. While PlaNet correctly classified 28.4% of the test data at the country level, its value error in the classic geolocation estimation dataset IM2GPS test set was 500 kilometers. In 2018, Carnegie Mellon University proposed the DeepGeo model, using deep neural networks to achieve photo geolocation. This work created the open-source dataset 50States10K—containing 500,000 street view images from Google in the US—to train a deep neural network with a residual structure. Four different strategies incorporating low-level cardinality information were proposed, resulting in a performance that outperformed humans in four out of five rounds of the GeoGuesser game. However, this cell partitioning is performed using adaptive subdivision, meaning it is recursively partitioned if each cell contains more than a certain number of images. Cells with too few samples are eventually discarded. Such subdivision allows for balanced classes while allocating more classes (or equivalently, more network parameter space) to regions with higher image density. One drawback of this solution is that the size of the region determines the maximum achievable accuracy. Using finer subdivisions within cells to improve classification accuracy is tempting, but this doesn't work well because if the geographic cells are very fine, the number of classifier parameters increases dramatically, leading to scalability and generalization problems.

[0005] In summary, current scene geolocation technology is still in its early stages of development, and there is an urgent need for new urban scene geolocation methods to solve the problems of difficulty in predicting the geolocation of images without landmark features, and the fact that the accuracy of geolocation estimation depends on the division of geographical units in the training set. Summary of the Invention

[0006] The purpose of this invention is to address the problem of missing geographic location information in urban visual data analysis. It overcomes the shortcomings of current geographic location positioning technologies, which rely heavily on massive databases for accuracy and struggle to estimate the geographic location of images lacking distinctive features. This invention provides a graphical matching method for urban scene geographic location positioning. Based on digital maps, it extracts element information from urban scenes, constructs spatial layout graphics of these elements, and utilizes vector similarity retrieval technology to estimate the geographic location of any urban visual data within a specific region. This invention can establish a mapping relationship between urban video / image data lacking distinctive features and digital maps, providing a graph-based reasoning solution for urban scene geographic location estimation.

[0007] To achieve this objective, the present invention adopts the following technical solution:

[0008] A method for geographic location positioning in urban scenes using image matching, the method comprising:

[0009] For a specified urban area, based on the characteristics of the artificial and natural features of the urban area, the categories of urban elements are determined, and an urban element extraction database is constructed.

[0010] Based on the input visual data and the city element extraction database, a target element detection information set is generated using a target detection algorithm.

[0011] Based on the target feature detection information set, a target feature segmentation map set is obtained using a feature component segmentation algorithm;

[0012] Based on the target element segmentation map, a set of scene element semantic information is obtained using an element refinement recognition algorithm;

[0013] Based on the target element detection information set, the target element segmentation atlas, and the scene element semantic information set, an index relationship is established according to specified rules to obtain a scene element knowledge set;

[0014] Based on the scene element knowledge set, a scene graph generation algorithm is used to obtain the scene layout graph of the input visual data, and to generate a set of scene spatial layout graphs of the target area in the digital map.

[0015] Based on the scene spatial layout graphic set, similar images in the target area scene spatial layout graphic set are retrieved using vector similarity retrieval technology to obtain the geographical location estimate of the input visual data.

[0016] Optionally, for a specified urban area, based on the characteristics of the artificial and natural features of the urban area, the urban element categories are determined, and an urban element extraction database is constructed, including:

[0017] Based on the typical land cover types involved in the specified urban area, T categories are given, with the category numbers ranging from 1 to T, where T is a positive integer; wherein, the typical land cover types include artificial land cover types and natural land cover types;

[0018] For each land feature category t, visual data of the urban area is collected using manual collection or web crawling. The annotation information involved in the visual data includes: detection borders, category information, and scene layout graph information for each land feature, ultimately forming the urban element extraction database A.

[0019] Optionally, the step of generating a target element detection information set based on the input visual data, using the urban element extraction database, and employing a target detection algorithm, includes:

[0020] Based on the type of the input visual data M, select the corresponding type of data and its corresponding detection bounding box annotation in the urban element extraction database A to form the urban element target detection dataset A1.

[0021] Based on the urban element target detection dataset A1, train a deep learning-based target detection neural network f. θ , where θ are the network parameters after training;

[0022] Based on the trained target detection neural network f θ The target feature detection information set is generated by applying the input visual data M. Here, set B consists of the top-left corner of the detected features within the area occupied by the image. and the bottom right corner The coordinates are constructed, and are based on the coordinates of the top left corner. The elements are sorted in order of size, and N represents the number of detected elements.

[0023] Optionally, obtaining a target feature segmentation map set based on the target feature detection information set using a feature component segmentation algorithm includes:

[0024] On the input visual data M, traverse the target feature detection information set B, based on the top left corner point. and the bottom right corner By cropping a local image using coordinates, a segmentation atlas of the target features is obtained. Where M i This refers to the image extracted from the input visual data based on the coordinates.

[0025] Optionally, the step of obtaining a scene element semantic information set based on the target element segmentation atlas and using an element refinement recognition algorithm includes:

[0026] Based on the type of the input visual data M, select the corresponding type of data and its corresponding category information from the urban element extraction database A to form the urban element target recognition dataset A2.

[0027] Based on the urban element target recognition dataset A2, train a deep learning-based target recognition neural network. in These are the network parameters after training;

[0028] Based on the trained target recognition neural network The target feature segmentation atlas C is applied to generate the scene feature semantic information set. Wherein, the set D is composed of category labels of the target feature segmentation map, and L i M represents the segmentation graph of the i-th feature. i Category tags.

[0029] Optionally, the step of establishing an index relationship according to specified rules to obtain a scene element knowledge set based on the target element detection information set, the target element segmentation atlas, and the scene element semantic information set includes:

[0030] Based on the scene element semantic information set D, the elements are categorized according to the semantic categories contained in the scene element semantic information set D; that is, assuming that D contains P different semantic element information classes, the categorized semantic information set is obtained. Where L′ j ={L i |i∈{1,...,N},L i =j} is the set of all semantic information in D with the semantic value j;

[0031] Based on the categorized semantic information set D', and combined with the target element detection information set B and the target element segmentation map set C, the scene element knowledge set is constructed.

[0032] Optionally, the step of obtaining a scene layout graphic of the input visual data and generating a set of scene spatial layout graphics of the target area in the digital map based on the scene element knowledge set and using a scene graph generation algorithm includes:

[0033] Based on the type of the input visual data M, the trained target detection neural network f is applied. θ and target recognition neural network Obtain the scene element knowledge set of the corresponding type of data in the city element extraction database A, and together with the corresponding scene layout map information, form the city image layout dataset A3.

[0034] Based on the city image layout dataset A3, a deep learning-based relation regularization network h is trained. μ Where μ represents the trained network parameters, and the relational regularization network includes the instance label refiner h′. μ1 and instance relation generator h″ μ1 ;

[0035] Based on the trained deep learning-based relational regularization network h μ The scene layout graph F is applied to the scene element knowledge set E to obtain the scene layout graph F of the input visual data M; wherein, the scene layout graph F is an undirected graph, the vertices are the land cover category labels, and the edges are the positional relationships between the vertices.

[0036] For the specified city area, select the tile data of the corresponding area, divide the area into multiple areas according to preset rules, with each area being a cell of a preset size, and slide sequentially to divide the area into different geographical units to obtain the cell data H of the city area;

[0037] Based on the cell data H, the trained target detection neural network f is applied. θ and target recognition neural network Obtain the scene knowledge element set of the urban area, and apply the trained relation regularization network h. μ Obtain the cell scene layout graphic set G for the city area.

[0038] Optionally, the instance label refiner h′ μ1 It includes two modules: a stacked bidirectional long short-term memory network and a relation-based convolutional network; and / or,

[0039] The instance relation generator h″ μ2 It includes two modules: a relation encoder and a relation decoder for relation regularization.

[0040] Optionally, the preset rule is:

[0041] Based on the minimum latitude and longitude of the urban area [x min y min ] and maximum latitude and longitude [x max y max The coordinates of the multiple regions are obtained by using a preset length as an interval.

[0042] Optionally, the step of retrieving similar images in the scene spatial layout graphic set in the target area using vector similarity retrieval technology, based on the scene spatial layout graphic set, to obtain the geographical location estimate of the input visual data, includes:

[0043] Based on the scene layout graphic F of the visual data, the set of cell scene layout graphics G, the adjacency matrix of the root graphic, and the feature category labels, the scene layout vector F' = (F1, F2) of the input visual data is obtained, where F1 is the vector composed of feature category labels in F, and F2 is the vector after expanding the adjacency matrix of F; similarly, the set of cell scene layout vectors G' is obtained based on the set of cell scene layout graphics G.

[0044] Based on the scene layout vector F' and cell scene layout vector set G' of the input visual data, using scene layout vector F' as the query vector and cell scene layout vector set G' as the query library, the Faiss vector search framework is used to find the geographic cell W in cell scene layout vector set G' that is most similar to scene layout vector F'.

[0045] Based on the number of the geographic cell W, the latitude and longitude of the center point are queried on the corresponding digital map to serve as the geographical location of the scene in the input visual data.

[0046] This invention discloses a graph-matching method for urban scene geolocation. For typical areas, it constructs an urban element extraction database; generates a target element detection dataset using an object detection algorithm; generates a target element segmentation atlas using an element component segmentation algorithm; generates a scene element semantic information set using an element refinement recognition algorithm and establishes a scene element knowledge set according to rules; generates a scene layout graph and a set of urban cell scene layout graphs using a relational regularization network; and determines the geographic cell to which the input visual data belongs using vector similarity retrieval technology to obtain the final estimated latitude and longitude of the scene geolocation. Based on digital maps, this invention extracts element information from urban scenes and constructs spatial layout graphs of elements. Utilizing vector-based retrieval technology, it can establish a mapping relationship between urban video / image data without distinctive features and digital maps, providing a graph-reasoning-based solution for urban scene geolocation estimation. Attached Figure Description

[0047] Figure 1 is a flowchart of a method for locating urban scenes using graphic matching according to an embodiment of the present invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] As shown in Figure 1, an embodiment of the present invention relates to a method for geographic location positioning in urban scenes using image matching, the method comprising:

[0050] Step S110: Constructing an urban element extraction database: For a specified urban area, based on the characteristics of the artificial and natural features of the urban area, determine the categories of urban elements and construct an urban element extraction database.

[0051] Specifically, in this step, based on the typical land cover types involved in the specified urban area, T categories are given, and the category numbers are 1 to T, where T is a positive integer; wherein, the typical land cover types include artificial land cover types and natural land cover types;

[0052] For each land feature category t, visual data of the urban area is collected using manual collection or web crawling. The annotation information involved in the visual data includes: detection borders, category information, and scene layout graph information for each land feature, ultimately forming the urban element extraction database A.

[0053] Step S120, Target Detection: Based on the input visual data and the city element extraction database, a target element detection information set is generated using a target detection algorithm.

[0054] Specifically, in this step, based on the type of the input visual data M (image or video), the corresponding type of data and its corresponding detection bounding box annotations are selected from the urban element extraction database A to form the urban element target detection dataset A1.

[0055] Based on the urban element target detection dataset A1, train a deep learning-based target detection neural network f. θ , where θ represents the trained network parameters. The deep learning object detection neural network used here can be a classic deep neural network model such as Faster R-CNN, YOLO series models, CornerNet, RetinaNet, etc.

[0056] Based on the trained target detection neural network f θ The target feature detection information set is generated by applying the input visual data M. Here, set B consists of the top-left corner of the detected features within the area occupied by the image. and the bottom right corner The coordinates are constructed, and are based on the coordinates of the top left corner. The elements are sorted in order of size, and N represents the number of detected elements.

[0057] Step S130, Feature component segmentation: Based on the target feature detection information set, the feature component segmentation algorithm is used to obtain the target feature segmentation map set.

[0058] Specifically, in this step, on the input visual data M, the target element detection information set B is traversed, based on the top left corner point. and the bottom right corner By cropping a local image using coordinates, a segmentation atlas of the target features is obtained. Where M i This refers to the image extracted from the input visual data based on the coordinates.

[0059] Step S140, Refined Feature Recognition: Based on the target feature segmentation map, a set of semantic information of scene features is obtained using a refined feature recognition algorithm.

[0060] Specifically, in this step, based on the type of the input visual data M, the corresponding type of data and its corresponding category information are selected from the urban element extraction database A to form the urban element target recognition dataset A2.

[0061] Based on the urban element target recognition dataset A2, train a deep learning-based target recognition neural network. in These are the network parameters after training. The deep learning neural network used for object recognition here can be a classic deep neural network model such as GoogLeNet or ResNet.

[0062] Based on the trained target recognition neural network The target feature segmentation atlas C is applied to generate the scene feature semantic information set. Wherein, the set D is composed of category labels of the target feature segmentation map, and L i M represents the segmentation graph of the i-th feature. i Category tags.

[0063] Step S150: Scene element knowledge set construction: Based on the target element detection information set, the target element segmentation map set, and the scene element semantic information set, establish index relationships according to specified rules to obtain the scene element knowledge set.

[0064] Specifically, in this step, based on the scene element semantic information set D, the elements are categorized according to the semantic categories contained in the scene element semantic information set D; that is, assuming that D contains P different semantic element information classes, the categorized semantic information set is obtained. Where L′ j ={L i |i∈{1,...,N},L i =j} is the set of all semantic information in D with the semantic value j;

[0065] Based on the categorized semantic information set D', and combined with the target element detection information set B and the target element segmentation map set C, the scene element knowledge set is constructed.

[0066] Step S160: Scene layout graphics construction: Based on the scene element knowledge set, a scene graph generation algorithm is used to obtain the scene layout graphics of the input visual data, and to generate a set of scene spatial layout graphics of the target area in the digital map.

[0067] Specifically, in this step, based on the type of the input visual data M, the trained target detection neural network f is applied. θ and target recognition neural network Obtain the scene element knowledge set of the corresponding type of data in the city element extraction database A, and together with the corresponding scene layout map information, form the city image layout dataset A3.

[0068] Based on the city image layout dataset A3, a deep learning-based relation regularization network h is trained. μ Where μ represents the trained network parameters, and the relational regularization network includes the instance label refiner h′. μ1 and instance relation generator h″ μ2 .

[0069] Based on the trained deep learning-based relational regularization network h μ The scene layout graph F is applied to the scene element knowledge set E to obtain the scene layout graph F of the input visual data M; wherein, the scene layout graph F is an undirected graph, the vertices are the land cover category labels, and the edges are the positional relationships between the vertices.

[0070] For the specified city area, select the corresponding area's tile data (including satellite tiles and street roadmap tile data at level 19), and divide the area into four regions Q1, Q2, Q3, and Q4 according to preset rules. Each region is divided into 500m × 500m cells. The area is divided into different geographic units by sliding sequentially to obtain the cell data H of the city area.

[0071] Based on the cell data H, the trained target detection neural network f is applied. θ and target recognition neural network Obtain the scene knowledge element set of the urban area, and apply the trained relation regularization network h. μ Obtain the cell scene layout graphic set G for the city area.

[0072] Optionally, the instance label refiner h′μ1 It includes two modules. One module is a stacked bidirectional long short-term memory network (LSTMRNN) for capturing global contextual relationships in image data. Optionally, the LSMRNN can be either LSTMRNN or Bidirectional RNN. The other module is a relation-based convolutional network to fully utilize instance relational contextual information; the optional relational convolutional network can be a graph neural network (GNN). The instance relation generator h″ μ2 It comprises two modules. One module is a relation encoder for relation regularization, which uses a deep neural network (such as Faster R-CNN) and the aforementioned relation regularization label generation module to transform an image into a set of optimized instance features and refined labels. The other module is a relation decoder, which can be a deep neural network. It maps the output of the relation generator to two feature spaces, the subject domain and the object domain, to obtain the relations between instances.

[0073] Optionally, the preset rule is:

[0074] Based on the minimum latitude and longitude of the urban area [x min y min ] and maximum latitude and longitude [x max y max The coordinates of the multiple regions are obtained at 300-meter intervals, that is, the coordinates of Q1 are [x...]. min -300, y min -300] and [x max +300, y max +300], the coordinates of Q2 are [x min +300, y min -300] and [x max -300, y max +300], the coordinates of Q3 are [x min -300, y min +300] and [x max +300, y max -300], the coordinates of Q4 are [x min +300, y min +300] and [x max -300, y max -300].

[0075] Step S170, cross-view image search: Based on the scene spatial layout graphic set, use vector similarity retrieval technology to retrieve similar images in the scene spatial layout graphic set of the target area, and obtain the geographical location estimate of the input visual data.

[0076] Specifically, in this step, based on the scene layout graphic F of the visual data, the cell scene layout graphic set G, the adjacency matrix of the root graphic, and the feature category labels, the scene layout vector F' = (F1, F2) of the input visual data is obtained, where F1 is the vector composed of feature category labels in F, and F2 is the vector after expanding the adjacency matrix of F; similarly, the cell scene layout vector set G' is obtained based on the cell scene layout graphic set G.

[0077] Based on the scene layout vector F' and cell scene layout vector set G' of the input visual data, using scene layout vector F' as the query vector and cell scene layout vector set G' as the query library, the Faiss framework is used to find the geographic cell W in cell scene layout vector set G' that is most similar to scene layout vector F'.

[0078] Based on the number of the geographic cell W, the latitude and longitude of the center point are queried on the corresponding digital map to serve as the geographical location of the scene in the input visual data.

[0079] This invention discloses a graph-matching method for urban scene geolocation. For typical areas, it constructs an urban element extraction database; generates a target element detection dataset using an object detection algorithm; generates a target element segmentation atlas using an element component segmentation algorithm; generates a scene element semantic information set using an element refinement recognition algorithm and establishes a scene element knowledge set according to rules; generates a scene layout graph and a set of urban cell scene layout graphs using a relational regularization network; and determines the geographic cell to which the input visual data belongs using vector similarity retrieval technology to obtain the final estimated latitude and longitude of the scene geolocation. Based on digital maps, this invention extracts element information from urban scenes and constructs spatial layout graphs of elements. Utilizing vector-based retrieval technology, it can establish a mapping relationship between urban video / image data without distinctive features and digital maps, providing a graph-reasoning-based solution for urban scene geolocation estimation.

[0080] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for geographic location positioning in urban scenes using image matching, characterized in that, The method includes: for a specified urban area, determining urban element categories based on the characteristics of artificial and natural features in the urban area, and constructing an urban element extraction database; based on input visual data and the urban element extraction database, generating a target element detection information set using a target detection algorithm; obtaining a target element segmentation map set using an element component segmentation algorithm based on the target element segmentation map set; obtaining a scene element semantic information set using an element refinement recognition algorithm based on the target element segmentation map set; and establishing an index relationship according to specified rules to obtain scene element knowledge based on the target element detection information set, the target element segmentation map set, and the scene element semantic information set. The process involves: 1) Based on the scene element knowledge set, using a scene graph generation algorithm, obtaining a scene layout graphic of the input visual data and generating a set of scene spatial layout graphics of the target area in a digital map; 2) Based on the scene spatial layout graphics set, using vector similarity retrieval technology, retrieving similar images in the scene spatial layout graphics set of the target area to obtain a geographical location estimate of the input visual data; 3) Based on the scene element knowledge set, using a scene graph generation algorithm, obtaining a scene layout graphic of the input visual data and generating a set of scene spatial layout graphics of the target area in a digital map, including: 4) According to the type of the input visual data M, applying a trained object detection neural network... and target recognition neural network Obtain the scene element knowledge set of the corresponding type of data in the city element extraction database A, and combine it with the corresponding scene layout map information to form the city image layout dataset A3; based on the city image layout dataset A3, train a deep learning-based relational regularization network. ,in The relational regularization network includes an instance label refiner, which serves as the network parameters after training. and instance relationship generator Based on the trained deep learning-based relation regularization network The scene layout graph F is obtained by applying the scene element knowledge set E to the input visual data M; wherein, the scene layout graph F is an undirected graph, the vertices are the land cover category labels, and the edges are the positional relationships between the vertices; for the specified city area, the tile data of the corresponding area is selected, and the area is divided into multiple areas according to preset rules. Each area is a cell with a preset size, and the area is divided into different geographical units by sequential sliding to obtain the cell data H of the city area; based on the cell data H, the trained object detection neural network is applied. and target recognition neural network The scene knowledge element set of the urban area is obtained, and a trained relation regularization network is used. Obtain the cell scene layout graphic set G for the city area.

2. The method according to claim 1, characterized in that, For a specified urban area, based on the characteristics of artificial and natural features in the urban area, urban element categories are determined, and an urban element extraction database is constructed. This includes: Based on the typical feature types involved in the specified urban area, T categories are given, with category numbers ranging from 1 to T, where T is a positive integer; wherein the typical feature types include artificial feature types and natural feature types; for each feature category t, visual data of the urban area is collected using manual collection or web crawling; wherein the annotation information involved in the visual data includes: detection borders, category information, and scene layout graph information for each type of feature, ultimately forming the urban element extraction database A.

3. The method according to claim 2, characterized in that, The step of generating a target element detection information set based on the input visual data and the urban element extraction database using a target detection algorithm includes: selecting data of the corresponding type and its corresponding detection bounding box annotations from the urban element extraction database A according to the type of the input visual data M, thus forming an urban element target detection dataset A1; and training a deep learning-based target detection neural network based on the urban element target detection dataset A1. ,in These are the network parameters after training; based on the trained target detection neural network... The target feature detection information set is generated by applying the input visual data M. Here, set B consists of the top-left corner points of the detected features within the image's area. and the bottom right corner The coordinates are constructed, and are based on the coordinates of the top left corner. The elements are sorted in order of size, and N represents the number of detected elements.

4. The method according to claim 3, characterized in that, The step of obtaining a target feature segmentation map set based on the target feature detection information set and using a feature component segmentation algorithm includes: traversing the target feature detection information set B on the input visual data M, and based on the top left corner point... and the bottom right corner By cropping a local image using coordinates, a segmentation atlas of the target features is obtained. ;in This refers to the image extracted from the input visual data based on the coordinates.

5. The method according to claim 4, characterized in that, The step of obtaining a scene element semantic information set based on the target element segmentation map and using an element refinement recognition algorithm includes: selecting data of the corresponding type and its corresponding category information from the urban element extraction database A according to the type of the input visual data M, to form an urban element target recognition dataset A2; and training a deep learning-based target recognition neural network based on the urban element target recognition dataset A2. ,in These are the network parameters after training; based on the trained target recognition neural network... The target feature segmentation map set C is applied to generate the scene feature semantic information set. Wherein, the set D consists of the category labels of the target feature segmentation map. Represents the segmentation map of the i-th element. Category tags.

6. The method according to claim 5, characterized in that, The step of establishing an index relationship based on the target element detection information set, the target element segmentation atlas, and the scene element semantic information set according to specified rules to obtain a scene element knowledge set includes: classifying the scene element semantic information set D according to the semantic categories contained in the scene element semantic information set D; that is, assuming that D contains P different semantic element information classes, the classified semantic information set D' = ,in It is the set of all semantic information with semantic value j in D; based on the classified semantic information set D', combined with the target element detection information set B and the target element segmentation map set C, the scene element knowledge set E is constructed. = 。 7. The method according to claim 6, characterized in that, The instance label refiner It includes two modules: a stacked bidirectional long short-term memory network and a relation-based convolutional network; and / or, the instance relation generator. It includes two modules: a relation encoder and a relation decoder for relation regularization.

8. The method according to claim 6, characterized in that, The preset rule is: based on the minimum latitude and longitude of the urban area. and the largest latitude and longitude The coordinates of the multiple regions are obtained at intervals of a preset length.

9. The method according to claim 6, characterized in that, The step of retrieving similar images in the scene spatial layout graphic set of the target area using vector similarity retrieval technology to obtain the geographical location estimate of the input visual data includes: obtaining the scene layout vector F' = (F1, F2) of the input visual data based on the scene layout graphic F of the visual data, the cell scene layout graphic set G, the adjacency matrix of the root graphic, and the feature category labels, where F1 is the vector composed of feature category labels in F, and F2 is the vector after expanding the adjacency matrix of F; similarly, obtaining the cell scene layout vector set G' based on the cell scene layout graphic set G; using the scene layout vector F' and the cell scene layout vector set G' of the input visual data, using the scene layout vector F' as the query vector and the cell scene layout vector set G' as the query library, using the Faiss vector search framework, finding the geographical cell W in the cell scene layout vector set G' that is most similar to the scene layout vector F'; and querying the latitude and longitude of the center point on the corresponding digital map based on the number of the geographical cell W as the geographical location of the scene in the input visual data.

Images