Multi-modal matching geographic positioning method

Abstract

The invention provides a geographic positioning method based on multi-modal matching. The geographic positioning method comprises the following steps: S1, selecting a plurality of streetscape images to construct a streetscape image data set; s2, calculating the image-text similarity between the to-be-positioned streetscape image and the text description corresponding to each streetscape image so as to select a plurality of streetscape images as image-text matching candidate images; s3, calculating the graph similarity between the to-be-positioned streetscape image and each streetscape image so as to select a plurality of streetscape images as graph matching candidate images; s4, obtaining a union set of all the image-text matching candidate images and all the image matching candidate images to obtain a target candidate set, and carrying out weighted summation on the image-text similarity and the image similarity corresponding to each streetscape image in the target candidate set to obtain a comprehensive similarity; and S5, selecting one or more streetscape images as target streetscape images according to the comprehensive similarity corresponding to each streetscape image, and taking the geographic coordinates of all the target streetscape images as a positioning result of the streetscape image to be positioned.

Description

Technical Field

[0001] The present invention relates to the field of geolocation technology, and in particular to a geolocation technology based on multimodal matching of street view images, and more particularly to a geolocation method based on multimodal matching. Background Art

[0002] Geolocation technology is widely used in modern society, especially in navigation, map services, intelligent transportation systems and other fields. Geolocation technology can accurately determine the location of objects or places, supporting real-time location tracking and route planning. In traditional geolocation technology, positioning is usually achieved by obtaining geographic coordinates, but in urban environments, indoor scenes or areas with limited signals, the accuracy and availability of positioning are often limited. With the development of artificial intelligence, computer vision and big data technology, existing geolocation technology can achieve geolocation through image recognition and matching. Representative geolocation methods in the existing technology include: positioning methods based on local aggregated description vectors, positioning methods based on global feature aggregation and positioning methods based on foundation models.

[0003] Among them, the localization method based on aggregated local description vectors is described in the paper "Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5297–5307, 2016." This method fuses local descriptors obtained from key points in the image into a vectorized form by calculating the difference between each descriptor and a set of predefined visual clusters. It also uses AlexNet and VGG-16 as the basic framework for geolocation. Although this method can achieve geolocation, it requires weak supervision information as auxiliary input during geolocation.

[0004] A localization method based on global feature aggregation is described in the paper "A. Ali-bey, B. Chaib-Draa, and P. Giguére, "MixVPR: Feature mixing for visual place recognition," in Proc. IEEE / CVF Winter Conf. Appl. Comput. Vis., 2023, pp. 2998–3007." This method uses the feature maps from the pre-trained backbone as a set of global features. It then integrates the global relationships between elements in each feature map in a cascade of feature mixing to achieve geolocation. While this method is capable of achieving geolocation, it lacks interpretability due to its inability to explain the localization results in natural language.

[0005] This method uses a foundation model-based positioning method. The related paper is "Keetha N., Mishra A., Karhade J., Jatavallabhula KM, Scherer S., Krishna M., Garg S. Anyloc: Towards universal visual place recognition. IEEE Robot. Autom. Lett. (2023)." This method uses universal features derived from an existing self-supervised foundation model (DINOv2) to achieve geolocation in unstructured scenes without training.

[0006] Although existing geolocation methods can achieve geolocation through image recognition and matching, they still have three shortcomings. First, existing geolocation methods often rely on direct matching of image features, which makes existing geolocation methods less accurate in the face of diverse environmental changes, time changes, and changes in perspective. Second, most existing geolocation methods lack adaptability across environments and time and space. In particular, in the processing of image descriptions and multimodal data, they are often unable to effectively combine image and text information for comprehensive positioning. Third, existing geolocation methods often lack explainability, making it difficult to provide users with a clear reasoning process about positioning results, resulting in low transparency and reliability of geolocation.

[0007] It should be noted that this background information is provided solely to introduce relevant information of the present invention to facilitate understanding of the technical solution of the present invention. It does not necessarily constitute prior art. In the absence of evidence demonstrating that the relevant information was disclosed prior to the filing date of the present invention, the relevant information should not be considered prior art. Summary of the Invention

[0008] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a multi-modal matching geolocation method.

[0009] The objectives of the present invention are achieved through the following technical solutions.

[0010] According to a first aspect of the present invention, a multimodal matching geolocation method is provided for geolocating a street view image to be located by combining image information and text information. The method comprises: step S1, selecting multiple street view images from an existing database to construct a street view image dataset; wherein each street view image corresponds to geographic coordinates, a text description, and a language feature vector of the text description, and the text description indicates geographic entities contained in the street view image and the relationships between the geographic entities; step S2, calculating the image-text similarity between the street view image to be located and the text description corresponding to each street view image, and selecting multiple street view images as image-text matching candidate images based on the image-text similarity corresponding to each street view image; step S3, calculating the image-text similarity between the street view image to be located and the text description corresponding to each street view image; The method further comprises the steps of: determining the image-image similarity of each street view image, and selecting multiple street view images as image-image matching candidate images based on the image-image similarity corresponding to each street view image; obtaining a target candidate set by taking the union of all image-text matching candidate images and all image-image matching candidate images, and performing a weighted summation of the image-text similarity and the image-image similarity corresponding to each street view image in the target candidate set to obtain a comprehensive similarity between the street view image to be located and each street view image; and sorting the comprehensive similarities corresponding to all street view images in the target candidate set from high to low, selecting one or more street view images ranked first as target street view images, and using the geographic coordinates corresponding to all target street view images as the positioning results of the street view image to be located.

[0011] In some embodiments of the present invention, step S2 includes: using a preset first multimodal model to generate a visual feature vector of the street view image to be located, and calculating the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image to obtain the image-text similarity between the street view image to be located and the text description corresponding to each street view image; sorting the image-text similarities corresponding to all street view images in descending order, and selecting multiple street view images ranked first as candidate images for image-text matching based on a preset first numerical value.

[0012] In some embodiments of the present invention, in step S2, the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image is calculated as follows:

[0013]

[0014] in, Represents the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to the current street view image, Represents the visual feature vector of the street view image to be located, The language feature vector representing the current street view image.

[0015] In some embodiments of the present invention, step S3 includes: using a pre-trained feature extractor to extract global feature vectors of the street view image to be located and each street view image, and calculating the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image to obtain the image-to-image similarity between the street view image to be located and each street view image; sorting the image-to-image similarity corresponding to each street view image in descending order, and selecting multiple street view images ranked first as image-to-image matching candidate images based on a preset first numerical value.

[0016] In some embodiments of the present invention, in step S3, the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image is calculated as follows:

[0017]

[0018] in, Indicates the similarity between the global feature vector of the street view image to be located and the global feature vector of the current street view image, Represents the first global feature vector of the street view image to be located dimensional eigenvalues, Represents the first global feature vector of the current street view image dimensional eigenvalues.

[0019] In some embodiments of the present invention, the method further includes: step S6, using a preset second multimodal model to match the description statement of the specific geographic entity in the text description corresponding to each target street view image with the specific area in the street view image to be located, so as to determine the matching relationship between the street view image to be located and the text description corresponding to each target street view image.

[0020] In some embodiments of the present invention, the method further includes: step S7, using a pre-trained feature extractor to extract local feature vectors of the street view image to be located and each target street view image; wherein the local feature vector is composed of a grid map of fixed size in multiple dimensions, each dimension of the grid map includes multiple grids and each grid corresponds to a feature value, the grids at the same position in all dimensions constitute a key point, and each key point corresponds to a specific area in the street view image to be located or the target street view image; based on the local feature vectors of the street view image to be located and each target street view image, calculating the local feature vector of each key point in the local feature vector of the street view image to be located and each target street view image. The similarity of all key points in the local feature vector of the street view image is used to determine all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image; wherein a mutually matching key point pair indicates that the two key points are mutually corresponding key points with the greatest similarity; based on all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image, a specific area in the street view image to be located indicated by the key point pair is matched with a specific area in the corresponding target street view image to determine a matching relationship between the street view image to be located and each target street view image.

[0021] In some embodiments of the present invention, the pre-trained feature extractor is trained in the following manner: obtaining a training set and a pre-trained DINOv2 model; wherein the DINOv2 model includes multiple transformer modules; setting an adapter in each transformer module of the DINOv2 model; wherein the adapter includes a fully connected layer and an activation layer; using the training set to perform multiple rounds of iterative training on the processed DINOv2 model until convergence, and using the converged DINOv2 model as a feature extractor; wherein, in each round of iterative training, a preset loss function is used to update the parameters of the adapter set in each transformer module in the model.

[0022] Compared with the prior art, the advantages of the present invention are: (1) it introduces a text description containing all the geographical entities in the image and the relationships between the geographical entities, as well as the language feature vectors corresponding to the text description, into the street view image, so as to provide a variety of reliable data information for image-text matching and image-image matching, thereby achieving accurate geolocation; (2) it combines image-image matching and image-text matching to achieve geolocation, thereby improving the accuracy and robustness of geolocation; (3) it provides a visualization means to determine the matching relationship between the street view image to be located and the target street view image, as well as the matching relationship between the text description of the street view image to be located and the target street view image, so that the geolocation result is interpretable. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0024] Figure 1 A schematic diagram of a geographic positioning method according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of an example of a street view image dataset according to an embodiment of the present invention;

[0026] Figure 3 An example schematic diagram is generated for interpretable basis of image-image matching and image-text matching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] As mentioned in the background technology section, although existing geolocation methods can achieve geolocation through image recognition and matching, existing geolocation methods still have three defects. First, existing geolocation methods usually rely on direct matching of image features, which makes the positioning accuracy of existing geolocation methods poor when facing diverse environmental changes, time changes, and perspective changes; second, most existing geolocation methods lack adaptability across environments and across time and space, especially in the processing of image descriptions and multimodal data, and often cannot effectively combine images and text information for comprehensive positioning; third, existing geolocation methods usually lack explainability, making it difficult to provide users with a clear reasoning process about the positioning results, resulting in low transparency and reliability of geolocation.

[0029] In order to solve the above problems, the inventors studied the existing geolocation methods and proposed that a text description containing all the geographical entities in the image and the relationships between the geographical entities and the language feature vectors corresponding to the text description can be introduced into the street view image as prior knowledge to assist geolocation, so as to avoid the defect of focusing only on image features; it is also proposed that image-image matching and image-text matching can be performed based on the visual features and language description features of the street view image itself, and the image-image matching results and image-text matching results can be combined to realize the geographic coordinate inference of the street view image to be located. This not only can effectively combine the image and text information to achieve geolocation, but also can improve the positioning accuracy of the street view image to be located; further, in order to provide a clear reasoning process for the street view image to be located, it is also proposed that a picture-text matching relationship between the street view image to be located and the street view image that matches itself can be generated based on the comprehensive positioning results, thereby providing an explainable basis for the geolocation process of the street view image to be located.

[0030] In summary, if Figure 1 As shown, the present invention provides a multimodal matching geolocation method for combining image information and text information to achieve geolocation of a street view image to be located, the method comprising: step S1, selecting multiple street view images from an existing database to construct a street view image dataset; wherein each street view image corresponds to geographic coordinates, a text description, and a language feature vector of the text description, and the text description indicates the geographic entities contained in the street view image and the relationship between the geographic entities; step S2, calculating the image-text similarity between the street view image to be located and the text description corresponding to each street view image, and selecting multiple street view images as image-text matching candidate images based on the image-text similarity corresponding to each street view image; step S3, calculating the image-text similarity between the street view image to be located and the text description corresponding to each street view image; The method further comprises the following steps: step S4, obtaining a target candidate set by taking the union of all the image-text matching candidate images and all the image-image matching candidate images, and performing a weighted summation of the image-text similarity and the image-image similarity corresponding to each street view image in the target candidate set to obtain a comprehensive similarity between the street view image to be located and each street view image; and step S5, sorting the comprehensive similarities corresponding to all the street view images in the target candidate set from high to low, selecting one or more street view images ranked first as the target street view images, and taking the geographic coordinates corresponding to all the target street view images as the positioning results of the street view image to be located.

[0031] The geolocation method further includes steps S6 and S7. Step S6 is used to determine a matching relationship between the street view image to be located and the text description corresponding to each target street view image, so as to provide an interpretable basis for text matching for the geolocation result. Step S7 is used to determine a matching relationship between the street view image to be located and each target street view image, so as to provide an interpretable basis for image matching for the geolocation result.

[0032] In order to better understand the present invention, each step will be described in detail below in conjunction with specific embodiments.

[0033] 1. Step S1

[0034] In step S1, a plurality of street view images are selected from an existing database to construct a street view image dataset; wherein each street view image corresponds to geographic coordinates, a text description, and a language feature vector of the text description, and the text description indicates the geographic entities contained in the street view image and the relationship between the geographic entities. The street view image dataset constructed in step S1 is as follows: Figure 2 shown.

[0035] It should be noted that the text description corresponding to each street view image and the language feature vector of the text description are generated using the preset first multimodal model.

[0036] The preset first multimodal model includes a visual encoder, a projector, and a language model. The visual encoder uses a visual transformer (ViT) structure to segment the input street view image into fixed-size patches and extract global visual features of the street view image through a series of self-attention mechanisms. After processing by the visual encoder, the street view image is converted into a set of high-dimensional feature vectors that retain the spatial relationships and semantic information of each region in the street view image. The projector uses a multi-layer perceptron (MLP) architecture to map the high-dimensional feature vector of the street view image output by the visual encoder into a vector of the same dimension as the word embedding vector input to the language model, and uses the mapped high-dimensional feature vector of the street view image as input to the language model. The language model uses a pre-trained language model (LLM) architecture to receive the high-dimensional feature vector of the mapped street view image output by the projector and generate a text description of the street view image based on a variety of preset prompts.

[0037] Specifically, the text description corresponding to each street view image is generated by performing multiple rounds of description optimization based on multiple preset prompts using a preset first multimodal model. After multiple rounds of description optimization, each street view image can produce text that accurately describes the key geographic elements (geographic entities) contained in it and the relationships between geographic elements. The multiple rounds of description optimization can be expressed as:

[0038]

[0039]

[0040] in, represents a street view image, represents the preset first multimodal model, Indicates the The text description generated by the first multimodal model preset in the round description optimization process, Indicates various prompts set. Indicates the Prompts used during the round description optimization process. It should be noted that the set prompts focus on guiding the model to generate text descriptions that include geographic elements in street view imagery (roads, surface buildings, traffic signs) while ignoring unstable factors that change over time (pedestrians, weather conditions, and vehicle lights). This ensures that the generated text descriptions cover the key geographic elements in street view imagery.

[0041] Specifically, the language feature vector of the text description corresponding to each street view image is generated using the language model in the preset first multimodal model. That is, the text description corresponding to each street view image is input into the language model, and the output of the embedding layer in the language model is used as the language feature vector of the text description corresponding to the street view image.

[0042] Based on the aforementioned embodiment, it can be seen that step S1 selects multiple street view images from an existing database to construct a street view image dataset, and uses a preset first multimodal model to generate a corresponding text description and a language feature vector of the text description for each street view image, so as to provide a variety of reliable data information for image-text matching and image-image matching in subsequent steps, thereby achieving accurate geographic positioning.

[0043] 2. Step S2

[0044] In step S2, the image-text similarity between the street view image to be located and the text description corresponding to each street view image is calculated, and multiple street view images are selected as image-text matching candidate images based on the image-text similarity corresponding to each street view image.

[0045] According to one embodiment of the present invention, step S2 includes: using a preset first multimodal model to generate a visual feature vector of the street view image to be located, and calculating the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image to obtain the image-text similarity between the street view image to be located and the text description corresponding to each street view image; sorting the image-text similarities corresponding to all street view images in descending order, and selecting multiple street view images ranked first as candidate images for image-text matching based on a preset first numerical value. It should be noted that the visual feature vector of the street view image to be located is obtained in the following manner: inputting the street view image to be located into the preset first multimodal model, performing feature processing on the street view image to be located using a visual encoder to obtain a high-dimensional feature vector of the street view image to be located, and then mapping the high-dimensional feature vector of the street view image to be located using a projector to obtain the visual feature vector of the street view image to be located.

[0046] According to one embodiment of the present invention, the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image is calculated as follows:

[0047]

[0048] in, Represents the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to the current street view image, Represents the visual feature vector of the street view image to be located, The language feature vector representing the current street view image.

[0049] Step S3

[0050] In step S3, the image-to-image similarity between the street view image to be located and each street view image is calculated, and multiple street view images are selected as image-to-image matching candidate images based on the image-to-image similarity corresponding to each street view image.

[0051] According to one embodiment of the present invention, step S3 includes: using a pre-trained feature extractor to extract global feature vectors of the street view image to be located and each street view image, and calculating the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image to obtain the image-to-image similarity between the street view image to be located and each street view image; sorting the image-to-image similarity corresponding to each street view image in descending order, and selecting multiple street view images ranked first as image-to-image matching candidate images based on a preset first numerical value.

[0052] According to one embodiment of the present invention, the pre-trained feature extractor is trained in the following manner: obtaining a training set and a pre-trained DINOv2 model; wherein the DINOv2 model includes multiple transformer modules; providing an adapter in each transformer module of the DINOv2 model; wherein the adapter includes a fully connected layer and an activation layer; using the training set to perform multiple rounds of iterative training on the processed DINOv2 model until convergence, and using the converged DINOv2 model as the feature extractor; wherein, in each round of iterative training, a preset loss function is used to update the parameters of the adapter provided in each transformer module in the model. The preset loss function is:

[0053]

[0054]

[0055] in, Indicates loss, represents the global feature vector of the query image (street view image), Represents the global feature vector of the positive sample corresponding to the query sample (street view images within 10 meters of the query image are positive samples), Indicates the query sample corresponding to the The global feature vector of negative samples (street view images more than 25 meters away from the query image are negative samples), represents the loss function, instruct .

[0056] Among them, the training set can use public visual location recognition benchmark datasets, mainly including the Tokyo24 / 7 dataset, the MSLS dataset and the Pitts30k dataset; the Tokyo24 / 7 dataset includes about 76,000 database images and 315 query images. The query images are selected from 1,125 images taken at 125 different locations, 3 different perspectives and 3 different times. This dataset mainly shows changes in perspective and drastic changes in conditions (day and night changes); the MSLS dataset contains more than 1.6 million images marked with GPS coordinates and compass angles. These images are from urban, suburban and natural scenes in 30 cities. This dataset covers various challenging visual changes caused by lighting, weather, seasons, perspectives, etc.; the Pitts30k dataset is collected from Google Street View panoramas, providing 24k query street view images, and each place has a different perspective.

[0057] It should be noted that although the pre-trained DINOv2 model has powerful visual encoding capabilities, it still needs to be fine-tuned on the training set so that the fine-tuned DINOv2 model can more accurately extract the global feature vectors of street view images. Therefore, the present invention proposes to set an adapter including a fully connected layer and an activation layer in each transformer module in the pre-trained DINOv2 model. The fully connected layer projects the input data it receives down to a smaller dimension, and then activates it with the activation layer and projects it back up to the original dimension. By introducing the adapter, the DINOv2 model can adapt to the data in the training set while retaining the general knowledge from the pre-training and maintaining its original visual encoding capabilities. It should also be noted that when fine-tuning the pre-trained DINOv2 model using the training set, the original architecture parameters of the DINOv2 model need to be frozen, and only the parameters of the adapter set in each transformer module in the DINOv2 model are updated in each round of iterative training.

[0058] According to one embodiment of the present invention, in step S3, the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image is calculated as follows:

[0059]

[0060] in, Indicates the similarity between the global feature vector of the street view image to be located and the global feature vector of the current street view image, Represents the first global feature vector of the street view image to be located dimensional eigenvalues, Represents the first global feature vector of the current street view image dimensional eigenvalues.

[0061] Step S4

[0062] In step S4, all image-text matching candidate images and all image-image matching candidate images are unioned to obtain a target candidate set. The image-text similarity and image-image similarity corresponding to each street view image in the target candidate set are weightedly summed to obtain the comprehensive similarity between the street view image to be located and each street view image. The calculation process of the comprehensive similarity can be expressed as:

[0063]

[0064] in, Represents the street view image to be located and the target candidate set The comprehensive similarity of the street view images, Indicates the weight coefficient corresponding to the image-text similarity, Indicates the The image-text similarity corresponding to each street view image, Represents the weight coefficient corresponding to the graph similarity, Indicates the It should be noted that when calculating the comprehensive similarity, the image-image similarity is the main factor, and the image-text similarity is the auxiliary factor. Therefore, the weight coefficient is used when calculating the comprehensive similarity. The value must be greater than the weight coefficient For example, setting , .

[0065] 5. Step S5

[0066] In step S5, the comprehensive similarities corresponding to all street view images in the target candidate set are sorted in descending order, and one or more street view images ranked first are selected as target street view images, and the geographic coordinates corresponding to all target street view images are used as the positioning results of the street view images to be positioned.

[0067] Based on the aforementioned steps S2-S5, the present invention proposes a multimodal matching approach that combines image-text matching and image-image matching to achieve geolocation of street view images. This approach fully utilizes multiple information sources, including images, text, and geographic coordinates, improving the accuracy and robustness of geolocation. Compared to traditional single-modality matching methods, the present invention, supported by multimodal data, can reduce mismatches and improve positioning accuracy, particularly in situations with high visual variation and environmental complexity.

[0068] 6. Step S6

[0069] In step S6, a preset second multimodal model is used to match the description of the specific geographic entity in the text description corresponding to each target street view image with the specific region in the street view image to be located, thereby determining a matching relationship between the street view image to be located and the text description corresponding to each target street view image. The preset second multimodal model is a pre-trained CLIP model.

[0070] Specifically, step S6 includes: extracting the geographic entity description in the text description corresponding to each target street view image and inputting it into the CLIP model to generate a text feature vector; inputting the street view image to be located into the CLIP model to generate a feature vector sequence of the image block; and performing gradient weighting and correlation matrix iterative updating based on the text feature vector and the feature vector sequence of the image block through the fused multi-layer multi-head attention matrix in the CLIP model to determine the area in the street view image to be located that is sensitive to the geographic entity in the text description, and generating a saliency heat map through attention visualization to highlight the area in the street view image to be located that corresponds to the geographic entity in the text description.

[0071] In order to better understand the matching relationship between the street view image to be located and the text description corresponding to the target street view image, the following Figure 3 The matching example shown in .

[0072] Depend on Figure 3 It can be seen that after the street view image to be located is located, a target street view image is obtained. According to the aforementioned step S6, the text description corresponding to the street view image to be located and the target street view image is matched to obtain the following: Figure 3 The matching relationship between the street view image to be located and the corresponding text description of the target street view image is shown in the lower right corner. That is, the corresponding geographic entity in the street view image to be located is highlighted to match the corresponding geographic entity description in the text description of the target street view image.

[0073] Based on step S6, it can be seen that the present invention can intuitively display the geographic entity objects in the street view image to be located that match the text description of the target street view image through visualization means. While enhancing the credibility of geographic positioning, it also provides technical support for multimodal feature alignment and fine-grained geographic entity association.

[0074] 7. Step S7

[0075] In step S7, a pre-trained feature extractor is used to extract local feature vectors of the street view image to be located and each target street view image; wherein the local feature vector is composed of a grid map of fixed size in multiple dimensions, the grid map in each dimension includes multiple grids and each grid corresponds to a feature value, the grids at the same position in all dimensions constitute a key point, and each key point corresponds to a specific area in the street view image to be located or the target street view image; based on the local feature vectors of the street view image to be located and each target street view image, the local feature vectors of each key point in the local feature vector of the street view image to be located and each target street view image are calculated. The method further comprises the steps of: calculating the similarity of all key points in the local feature vector of the street view image to be located and the local feature vector of each target street view image to determine all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image; wherein the mutually matching key point pairs indicate that the two key points are mutually corresponding key points with the greatest similarity; and based on all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image, matching the specific area in the street view image to be located indicated by the key point pairs with the specific area in the corresponding target street view images to determine the matching relationship between the street view image to be located and each target street view image.

[0076] It should be noted that in step S7, the local feature vectors of the street view image to be located and each target street view image are processed by the backbone network and upsampling module in the pre-trained feature extractor (DINOv2 model). The upsampling module consists of two up-conv layers and a middle ReLU layer. The specific local feature vector extraction process can be expressed as follows:

[0077]

[0078] in, represents the street view image to be located or the local feature vector of the street view image, represents L2 normalization, represents the second convolutional layer, represents the activation function, represents the first convolutional layer, Represents the street view image to be located or the feature map obtained after the street view image is processed by the backbone network in the pre-trained feature extractor (DINOv2 model).

[0079] In order to better understand the matching relationship between the street view image to be located and the target street view image, Figure 3 The matching example shown in .

[0080] Depend on Figure 3It can be seen that after the street view image to be located is located, a target street view image is obtained. According to the above step S7, the street view image to be located is matched with the target street view image to obtain the following: Figure 3 The matching relationship between the street view image to be located and the target street view image is shown in the lower left corner, that is, the matching geographic entities in the street view image to be located and the target street view image are corresponded by lines of different colors.

[0081] Based on step S7, it can be seen that the present invention can match the street view image to be located with the target street view image through visualization means, and thus can clearly observe the matching geographic entities in the street view image to be located and the target street view image, making the geographic positioning result more accurate and interpretable.

[0082] Based on the aforementioned steps S1-S7, it can be seen that, unlike the prior art, the present invention introduces a text description containing all geographic entities in the image and the relationships between the geographic entities, as well as a language feature vector corresponding to the text description, into the street view image as prior knowledge to assist in geolocation, thereby avoiding the defect of focusing only on image features; image-image matching and image-text matching are also performed based on the visual features and language description features of the street view image itself, and the image-image matching results and the image-text matching results are combined to realize the inference of the geographic coordinates of the street view image to be located. This not only effectively combines image and text information to realize geolocation, but also improves the positioning accuracy of the street view image to be located; further, the street view image to be located is matched with the target street view image and text description through visualization means, so that the geolocation result is interpretable.

[0083] The beneficial effects of the present invention are as follows: (1) a text description containing all the relationships between geographic entities in the image and the language feature vectors corresponding to the text description is introduced into the street view image, so as to provide a variety of reliable data information for image-text matching and image-image matching, thereby achieving accurate geolocation; (2) geolocation is achieved by combining image-image matching and image-text matching, thereby improving the accuracy and robustness of geolocation; (3) a visualization means is provided to determine the matching relationship between the street view image to be located and the target street view image, as well as the matching relationship between the text description of the street view image to be located and the target street view image, so that the geolocation result is interpretable.

[0084] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0085] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0086] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0087] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal matching geolocation method for geolocating a street view image to be located by combining image information and text information, characterized in that: The method comprises: Step S1: Select multiple street view images from an existing database to construct a street view image dataset; wherein each street view image corresponds to geographic coordinates, a text description, and a language feature vector of the text description, and the text description indicates geographic entities contained in the street view image and the relationships between the geographic entities; Step S2: calculating the image-text similarity between the street view image to be located and the text description corresponding to each street view image, and selecting multiple street view images as image-text matching candidate images based on the image-text similarity corresponding to each street view image; Step S3: calculating the image-to-image similarity between the street view image to be located and each street view image, and selecting multiple street view images as image-to-image matching candidate images based on the image-to-image similarity corresponding to each street view image; Step S4: Calculate the union of all image-text matching candidate images and all image-image matching candidate images to obtain a target candidate set, and perform weighted summation of the image-text similarity and image-image similarity corresponding to each street view image in the target candidate set to obtain a comprehensive similarity between the street view image to be located and each street view image; Step S5: Sort the comprehensive similarities corresponding to all street view images in the target candidate set from high to low, select one or more street view images with the highest similarity as the target street view images, and use the geographic coordinates corresponding to all target street view images as the positioning results of the street view images to be positioned.

2. The method according to claim 1, characterized in that The step S2 comprises: Generate a visual feature vector of the street view image to be located using a preset first multimodal model, and calculate the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image to obtain the graphic and text similarity between the street view image to be located and the text description corresponding to each street view image; The image-text similarities corresponding to all street view images are sorted in descending order, and a plurality of street view images ranked first are selected as image-text matching candidate images based on a preset first value.

3. The method according to claim 2, characterized in that In step S2, the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to each street view image is calculated as follows: in, Represents the similarity between the visual feature vector of the street view image to be located and the language feature vector corresponding to the current street view image, Represents the visual feature vector of the street view image to be located, The language feature vector representing the current street view image.

4. The method according to claim 3, characterized in that The step S3 comprises: A pre-trained feature extractor is used to extract the global feature vectors of the street view image to be located and each street view image, and the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image is calculated to obtain the image-to-image similarity between the street view image to be located and each street view image; The image-to-image similarity corresponding to each street view image is sorted in descending order, and a plurality of street view images ranked first are selected as image-to-image matching candidate images based on a preset first value.

5. The method according to claim 4, characterized in that In step S3, the similarity between the global feature vector of the street view image to be located and the global feature vector of each street view image is calculated as follows: in, Indicates the similarity between the global feature vector of the street view image to be located and the global feature vector of the current street view image, Represents the first global feature vector of the street view image to be located dimensional eigenvalues, Represents the first global feature vector of the current street view image dimensional eigenvalues.

6. The method according to claim 5, characterized in that The method further comprises: Step S6: Using a preset second multimodal model, the description of the specific geographic entity in the text description corresponding to each target street view image is matched with the specific area in the street view image to be located, so as to determine a matching relationship between the street view image to be located and the text description corresponding to each target street view image.

7. The method according to claim 6, characterized in that The method further comprises: Step S7: Using a pre-trained feature extractor to extract local feature vectors for the street view image to be located and each target street view image; wherein the local feature vectors are composed of fixed-size grid maps in multiple dimensions, wherein each grid map in each dimension includes multiple grids, and each grid has a corresponding feature value, and grids at the same position in all dimensions constitute a key point, and each key point corresponds to a specific area in the street view image to be located or the target street view image; Based on the local feature vectors of the street view image to be located and each target street view image, calculating the similarity between each key point in the local feature vector of the street view image to be located and all key points in the local feature vector of each target street view image, so as to determine all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image; wherein a mutually matching key point pair indicates that the two key points are key points corresponding to each other with the greatest similarity; Based on all mutually matching key point pairs between the local feature vector of the street view image to be located and the local feature vector of each target street view image, the specific area in the street view image to be located indicated by the key point pairs is matched with the specific area in the corresponding target street view image to determine the matching relationship between the street view image to be located and each target street view image.

8. The method according to claim 7, characterized in that The pre-trained feature extractor is trained in the following way: Obtain a training set and a pre-trained DINOv2 model; wherein the DINOv2 model includes multiple transformer modules; Set an adapter in each transformer module of the DINOv2 model; the adapter includes a fully connected layer and an activation layer; The processed DINOv2 model is trained for multiple rounds of iterative training using the training set until convergence, and the converged DINOv2 model is used as the feature extractor. In each round of iterative training, the parameters of the adapter set in each transformer module in the model are updated using a preset loss function.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1-8.

10. An electronic device, characterized in that: include: One or more processors, and memory, wherein the memory is configured to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images