A street view image positioning method and system for multi-modal information integrated retrieval

Abstract

The application discloses a kind of multi-modal information comprehensive retrieval street view image positioning method and system, by establishing multi-modal information database, using visual feature extraction model and scene text recognition model establish multi-modal information database;By fusion feature retrieval, global feature and local feature are fused and preliminary recall search is carried out;By geometric verification rearrangement, the geometric verification algorithm based on local feature is used to rearrange preliminary recall search result, and obtain refined retrieval result;By scene text retrieval, the scene text set overlap degree of query picture and database picture is calculated, and the final retrieval result is obtained based on this score ranking. The application considers the comprehensive retrieval scheme of multi-modal information, can greatly improve the image positioning accuracy in practical application, and to a certain extent, solves the complete positioning failure problem of single modal information in some scenes.

Description

Technical Field

[0001] This invention relates to a method for locating street view images, and more specifically, to a method and system for locating street view images based on multimodal information retrieval. Background Technology

[0002] Image localization using deep learning techniques is typically achieved through image retrieval. A common approach involves pre-collecting a large number of street view images with geographic location annotations, then using convolutional neural networks to extract features from these images to create a feature database. The features of the street view image to be queried are then used to retrieve the most similar image from the feature database, and the location of the queried street view image is inferred from this database.

[0003] However, current industrial solutions often rely on initial recall based on global features, followed by geometric verification and rearrangement using local features. These solutions fail to consider the interaction and relationship between global and local visual features, significantly reducing image retrieval accuracy. Recent academic retrieval schemes attempt to orthogonally fuse global and local features to obtain new feature descriptors; however, they do not consider further geometric verification and rearrangement based on local features, thus failing to meet the demands of practical applications.

[0004] Meanwhile, both mature industrial solutions and the latest academic solutions, when using image retrieval for image localization, only consider the use of one modality: visual information. However, the images to be localized are often street view images, which contain rich scene text information such as road signs and shop signs. This text information is clearly crucial for image localization. Existing image localization methods do not utilize this high-value scene text information, relying solely on visual features. In large-scale real-world image localization applications, this often results in low localization accuracy and complete failure in certain scenarios (such as typical streets and residential building scenes). Summary of the Invention

[0005] The purpose of this invention is to address the problem that existing image retrieval-based graph localization methods cannot be truly implemented in large-scale image localization applications. It proposes a street view image localization method and system based on deep learning and multimodal information retrieval. This method can fully utilize the multimodal information of street view images for geographic location inference, effectively solving the problem that current graph localization methods are limited to a single modality. Especially in difficult scenarios lacking obvious visual features, it can greatly improve the localization accuracy of existing methods, enabling graph localization technology to be put into practical applications.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A street view image localization method based on multimodal information retrieval includes the following steps:

[0008] Collect street view images and label their geographical locations to build a street view image database; identify and extract the fusion features, local features and scene text of the street view images in the street view image database to establish a fusion feature database, a local feature database and a scene text database of street view images;

[0009] For a street view image to be queried, its fusion features are extracted, and the top few street view images with the closest fusion feature vectors are initially retrieved from the fusion feature database by feature comparison.

[0010] The system identifies and extracts the first few local features of the street view image to be queried, and obtains the local features of the first few most similar street view images from the local feature database. It performs feature matching calculation on these two types of local features to obtain the matching feature key points, and reorders the first few most similar street view images according to the number of matching feature key points.

[0011] The system identifies and extracts scene text from the street view image to be queried. If the number of identified scene texts is less than a preset threshold, the geographical location corresponding to the first reordered street view image is used as the geographical location of the street view image to be queried. If the number of identified scene texts is greater than the preset threshold, the system compares the identified scene text set with the scene text set of the most similar street view images in the scene text database. Based on the comparison result, the most similar street view images are reordered again, and the geographical location corresponding to the first reordered street view image is used as the geographical location of the street view image to be queried.

[0012] Preferably, the method for identifying and extracting fusion features includes: using the shallow layers of a deep convolutional neural network, namely the res1, res2, and res3 modules and the self-attention module of the resnet101 network, to extract low-level information features of the street view image, which include geometric, texture, and color information of the street view image; using the deep layers of a deep convolutional neural network, namely the res4 module and the gem pooling layer of the resnet101 network, to extract high-level semantic information features of the street view image; and orthogonally fusing the low-level information features and the high-level semantic information features to obtain fusion features.

[0013] Preferably, the method for identifying and extracting local features includes: using the self-attention module in a deep convolutional neural network to identify and extract the top few local features with the highest weights.

[0014] Preferably, the self-attention module adopts a spatial attention mechanism, which is implemented through two 1×1 convolutional layers.

[0015] Preferably, the method for identifying and extracting scene text includes: using a scene text detection and recognition network to identify and extract scene text contained in a street view image.

[0016] Preferably, feature matching is performed based on the cosine distance of the feature vectors.

[0017] Preferably, a K-nearest neighbor search supported by a KD tree is established for the local features of the most similar first few street view images, and subsequent feature matching calculations of the local features are performed.

[0018] Preferably, the step of obtaining the matched feature key points includes:

[0019] After performing feature matching calculations, preliminary matching feature key points are obtained;

[0020] The initially matched feature key points are then input into the Ransac algorithm to verify and filter the matched feature key points, thus obtaining the final matched feature key points.

[0021] Preferably, when performing feature matching calculations for local features, the distance between the vectors of two local features and the distance threshold are used to determine whether the features match. If the distance is less than the distance threshold, the feature is determined to match, and the preliminary matching feature key points are obtained.

[0022] Preferably, the scene text set comparison is performed by calculating the number of intersections according to the following formula:

[0023]

[0024] Among them, S i is the intersection score between the i-th scene text set in the scene text database and the text set of the street view image scene to be queried, M is the total number of scene texts in the street view image to be queried, and N is the number of texts in the i-th scene text set. q This refers to the q-th scene text identified in the street view image to be queried, text′ j It refers to the j-th scene text in the i-th scene text set, confidence(text) q () refers to the text provided by the scene text detection and recognition network. q Confidence level; indicator(text) q ,text′ j ) is an indicator function, if text q and text′ j If they are equal, the function value is 1; if they are not equal, the function value is 0.

[0025] A street view image localization system for multimodal information retrieval includes a memory and a processor. The memory stores a computer program, and the processor executes the program to implement the steps of the above-described method.

[0026] The advantages of the technical solution of the present invention are as follows:

[0027] 1. Image retrieval methods for image localization often rely on pre-established feature databases. This invention primarily aims to achieve city-level image localization by establishing a street view image database of millions of images, ensuring basic coverage of most areas of a city. Since this invention utilizes multiple modalities and types of features, it requires pre-extraction of features from each image in the database using a visual feature extraction model and a scene text recognition model, establishing three distinct feature databases: a fusion feature database, a local feature database, and a scene text database.

[0028] 2. Different layers of a deep convolutional neural network can extract different types of features. Shallow layers extract low-level information features, which are local features describing the image's geometry, texture, color, and other basic information. Deep layers extract high-level semantic information features, which are global features describing the image's high-level semantic information. This invention orthogonally fuses low-level and high-level semantic information features to obtain a fused feature. This feature includes both a macroscopic semantic understanding of the image and retains the basic geometric information at the lower levels. Using this fused feature for initial recall can maximize the quality of the initial recall results.

[0029] 3. Re-ranking based on local feature geometric verification is crucial for further improving the retrieval ranking results. Using the self-attention module in the neural network, attention scores are assigned to all local features. More discriminative local features receive higher attention scores; therefore, the top 1000 local feature descriptors with the highest attention scores are selected for further geometric verification. This method of selecting local feature descriptors reduces computational complexity while ensuring the effectiveness of the geometric verification algorithm.

[0030] 4. Scene text retrieval incorporates information from another modality into the overall solution. The effectiveness of this step is based on a simple observation: the probability of the same set of scene text appearing in different geographical locations is essentially zero. Therefore, based on this observation, the scene text sets of two images can be compared. If they are indeed two images taken from the same location but under different conditions, their scene text sets will show significant overlap; otherwise, they will not. Using this strategy for retrieval will solve the image localization problem in scenarios where visual features fail. The final ranking result based on scene text recognition considers text information from another modality, which will greatly enhance the robustness of the overall retrieval performance in scenarios where visual information retrieval fails. Attached Figure Description

[0031] Figure 1 This is an overall framework diagram of a street view image localization method for multimodal information comprehensive retrieval proposed in this invention.

[0032] Figure 2 These are examples of difficult samples that cause visual feature failure. Detailed Implementation

[0033] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0034] This invention proposes a street view image localization method based on multimodal information comprehensive retrieval, the processing flow of which is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0035] Step 1: Based on the target city, collect millions of street view images from the internet, densely sampling the city street views at 100m intervals to build a comprehensive city street view image library. Use a pre-trained deep convolutional neural network (DCNN) with feature fusion and an end-to-end scene text detection and recognition network to extract visual features and scene text from the city street view image library. The DCNN with feature fusion ultimately outputs a feature descriptor that fuses global and local features. The top 1,000 local features need to be selected based on the weights assigned to local features by the attention module in the network. This self-attention module uses a spatial attention mechanism, specifically implemented by two 1×1 convolutional layers. When using scene text detection and recognition, images that do not identify any scene text need to be filtered out and not added to the scene text database.

[0036] The second step, during the initial visual feature retrieval, involves extracting the fused features from the street view image to be queried. These fused features are then compared one-to-one with all features in the fused feature database. To improve efficiency, this process uses an accelerated feature retrieval library, and the specified comparison criterion is the cosine distance between two vectors. After the retrieval is complete, the top one hundred street view images whose fused feature vectors are most similar to the street view image to be queried are selected.

[0037] Step 3: Obtain the local features W of these one hundred street view images from the local feature database. Build a KD-tree for these one hundred street view images' local features to support efficient K-nearest neighbor search. Specifically, if the distance between a feature vector in the local features of the street view image to be queried and a vector in the local features of W is less than a set distance threshold, then the two feature vectors are considered to match, establishing a keypoint match. After establishing keypoint matches for two images, all matched keypoints need to be input into the Ransac algorithm for geometric verification to further remove incorrect matches and retain correct matches. The final number of matched keypoints will be used as the sorting criterion to finely reorder the preliminary search results from Step 2.

[0038] Step 4: First, identify the scene text in the street view image to be queried. If the number of scene texts is less than a preset threshold, the result of Step 3, i.e., the finely rearranged street view image, is used as the standard, and the labeled geographical location corresponding to the first street view image in the ranking is taken as the geographical location of the street view image to be queried. If the number of scene texts is greater than the preset threshold, the identified scene text set is compared with the scene text database. The intersection score between the scene text set corresponding to a certain image in the database and the scene text set of the street view image to be queried is calculated using the following formula:

[0039]

[0040] Among them, S i is the intersection score between the i-th scene text set in the scene text database and the scene text set of the street view image to be queried. M is the total number of scene texts in the street view image to be queried, and N is the number of texts in the i-th scene text set. q This refers to the q-th scene text identified in the street view image to be queried, text′ j This refers to the j-th scene text within the i-th scene text set. q () refers to the text provided by the scene text detection and recognition network. q Confidence level. indicator(text) q ,text′ j ) is an indicator function, if text q and text′ j If the text in two images are equal, the function value is 1; otherwise, it is 0. In summary, this formula calculates the degree of overlap between the text and scenes in the two images, and this score can be used as the basis for the final ranking of the rearrangement results in the third step. In scenarios where visual feature-based ranking fails, the fourth step, ranking based on text retrieval, will significantly improve the accuracy of image localization.

[0041] The following test was conducted on a city street view dataset. If the actual physical distance between the candidate image retrieved and the query image is within a certain range, the query image is considered to have been successfully located.

[0042] The number of street view images to be queried is 50, and the database contains 50,000 street view images. The location is performed using the method of this invention. The relevant data is shown in Table 1.

[0043] Table 1

[0044] 300m 500m 1000m 2000m accuracy 0.7 0.82 0.88 0.96 Average distance 330 330 330 330 Distance from median 41 41 41 41 Accuracy after rearrangement 0.84 0.94 0.96 1.0 Average distance after rearrangement 187 163 200 233 Distance from median after rearrangement 41 30 41 41

[0045] As shown in Table 1, when the positioning error is 300m, the positioning accuracy is 70% before local feature rearrangement, with an average error distance of 330m. After local feature rearrangement, the positioning accuracy increases to 84%, and the average error distance decreases to 187m. For other positioning error values, geometric verification based on local features can improve the positioning accuracy.

[0046] When applying the method of this invention to real-world big data, 20 difficult samples (such as images that report errors) are selected from the real-world data. Figure 2 The examples of difficult samples shown in (a) and (b) demonstrate that conventional visual feature models alone cannot locate these two images. However, when applied to scene text retrieval, the localization accuracy increased from 0% to 10%. This shows that even in difficult scenarios where visual features are completely ineffective, scene text retrieval can locate a portion of the images.

[0047] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A street view image localization method for multimodal information comprehensive retrieval, characterized in that, Includes the following steps: Collect street view images and label their geographical locations to build a street view image database; identify and extract the fusion features, local features and scene text of the street view images in the street view image database to establish a fusion feature database, a local feature database and a scene text database of street view images; For a street view image to be queried, its fusion features are extracted, and the top few street view images with the closest fusion feature vectors are initially retrieved from the fusion feature database by feature comparison. The system identifies and extracts the first few local features of the street view image to be queried, and obtains the local features of the first few most similar street view images from the local feature database. It performs feature matching calculation on these two types of local features to obtain the matching feature key points, and reorders the first few most similar street view images according to the number of matching feature key points. The scene text in the street view image to be queried is identified and extracted. If the number of identified scene texts is less than a preset threshold, the geographical location corresponding to the first reordered street view image is taken as the geographical location of the street view image to be queried. If it is greater than the preset threshold, the identified scene text set is compared with the scene text sets of the most similar street view images in the scene text database. The scene text set comparison is calculated based on the following formula to determine the number of intersections: Among them, S i is the intersection score between the i-th scene text set in the scene text database and the text set of the street view image scene to be queried, M is the total number of scene texts in the street view image to be queried, and N is the number of texts in the i-th scene text set. q This refers to the q-th scene text identified in the street view image to be queried, text′ j It refers to the j-th scene text in the i-th scene text set, confidence(text) q () refers to the text provided by the scene text detection and recognition network. q Confidence level; indicator(text) q ,text′ j ) is an indicator function, if text q and text′ j If they are equal, the function value is 1; if they are not equal, the function value is 0. Based on the comparison results, the top few most similar street view images are reordered, and the geographical location corresponding to the first reordered street view image is taken as the geographical location of the street view image to be queried.

2. The method as described in claim 1, characterized in that, The methods for identifying and extracting fusion features include: using the shallow layers of a deep convolutional neural network, namely the res1, res2, and res3 modules and the self-attention module of the resnet101 network, to extract the low-level information features of the street view image, which include the geometric, texture, and color information of the street view image; and using the deep layers of a deep convolutional neural network, namely the res4 module and the gem pooling layer of the resnet101 network, to extract the high-level semantic information features of the street view image. The underlying information features and the high-level semantic information features are orthogonally fused to obtain the fused features.

3. The method as described in claim 1, characterized in that, Methods for identifying and extracting local features include: using the self-attention module in a deep convolutional neural network to identify and extract the top few local features with the highest weights.

4. The method as described in claim 2 or 3, characterized in that, The self-attention module employs a spatial attention mechanism, implemented through two 1×1 convolutional layers.

5. The method as described in claim 1, characterized in that, Methods for identifying and extracting scene text include: using scene text detection and recognition networks to identify and extract scene text contained in street view images.

6. The method as described in claim 1, characterized in that, Feature fusion and comparison are performed based on the cosine distance of the feature vectors.

7. The method as described in claim 1, characterized in that, A K-nearest neighbor search supported by a KD-tree is established for the local features of the most similar street view images, and subsequent feature matching calculations of the local features are performed.

8. The method as described in claim 1, characterized in that, The steps to obtain the matching feature key points include: When performing feature matching calculations for local features, the distance between the vectors of two local features and the distance threshold are used to determine whether the features match. If the distance is less than the distance threshold, the features are considered to match, and the preliminary matching feature key points are obtained. The initially matched feature key points are then input into the Ransac algorithm to verify and filter the matched feature key points, thus obtaining the final matched feature key points.

9. A street view image localization system for multimodal information comprehensive retrieval, characterized in that, It includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the method according to any one of claims 1-8.

Images