Multi-point geolocation method and device for ground panoramic image

Abstract

This application relates to the field of computer vision technology, and in particular to a method and apparatus for multi-point geolocation of ground panoramic images. The method includes: acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database; performing cylindrical and cubic projections on all panoramic images; extracting cylindrical and cubic projection features respectively, and extracting all remote sensing image features; constructing a hypermap based on the cylindrical and cubic projection features; classifying and predicting the relative positions of the multiple panoramic images based on the hypermap; measuring distance based on the predicted relative positions, and combining the features of all remote sensing images to obtain the latitude, longitude, and altitude information of the best-matching remote sensing image. The embodiments of this application can achieve relative position prediction based on the block feature extraction of multi-point panoramic images and remote sensing images, thereby calculating distance metrics, matching the best remote sensing image to obtain the positioning result, improving the accuracy and robustness of the positioning result, and making it more accurate and practical.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for multi-point geolocation of ground panoramic images. Background Technology

[0002] Geo-localization technology can extract and match features from different images taken on the ground and in the sky to obtain the best matching sky image, and use the latitude, longitude and altitude information carried in the sky image to achieve positioning.

[0003] In related technologies, positioning can be achieved by matching a single ground view image with a single remote sensing image marked by GPS to determine the corresponding geographical location.

[0004] However, in related technologies, because the shooting range of remote sensing images is much larger than that of ground camera equipment, a single ground image appears in the field of view of multiple remote sensing images. This results in insufficient uniqueness of the remote sensing images retrieved from a single ground image, which cannot meet the actual positioning needs, reduces positioning accuracy and robustness, and is inaccurate, which urgently needs to be solved. Summary of the Invention

[0005] This application provides a multi-point geolocation method and apparatus for ground panoramic images to solve the problems in related technologies, such as the fact that the shooting range of remote sensing images is much higher than that of ground camera equipment, causing a single ground image to appear in the field of view of multiple remote sensing images, resulting in insufficient uniqueness of the remote sensing images retrieved from a single ground image, failing to meet actual positioning needs, reducing positioning accuracy and robustness, and being inaccurate.

[0006] The first aspect of this application provides a multi-point geolocation method for ground panoramic images, comprising the following steps: acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database; performing cylindrical and cubic projections on all panoramic images to obtain cylindrical and cubic projections of the panoramic images; extracting cylindrical and cubic projection features from the cylindrical and cubic projections of the panoramic images, respectively, and extracting all remote sensing image features from the candidate remote sensing image database; constructing a hypermap from the cylindrical and cubic projection features; classifying and predicting the relative positions of the multiple panoramic images based on the hypermap to obtain prediction results of the relative positions; measuring distances based on the prediction results of the relative positions, and combining the features of all remote sensing images to select the best-matching remote sensing image as the positioning result, and outputting the latitude, longitude, and altitude information of the remote sensing image.

[0007] Optionally, in one embodiment of this application, the step of acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database includes: recording the acquired ground images as panoramic images; using the offset of ground points on the remote sensing image as a position representation, discretizing the offset for each remote sensing image at a preset resolution, and specifying the initial position of the ground point discretization position in the first panoramic image; selecting and specifying the candidate remote sensing image database, such that the remote sensing images in the candidate remote sensing image database cover the input ground points to be located.

[0008] Optionally, in one embodiment of this application, the step of extracting the cylindrical projection features and cube projection features of the panoramic image and the panoramic image cube projection respectively, and extracting all remote sensing image features from the candidate remote sensing image database, includes: extracting features from the cylindrical projection of the panoramic image to obtain the cylindrical projection features of ground points; extracting features from each of the six faces of the cube projection of the panoramic image, and directionally decomposing the panoramic image of each cube projection to obtain the six faces, thereby obtaining the cube projection features of ground points; and performing block feature extraction on all remote sensing images in the remote sensing image database, representing each remote sensing image with a feature vector to obtain the total remote sensing image features.

[0009] Optionally, in one embodiment of this application, the construction of a hypergraph for the cylindrical projection features and the cube projection features includes: constructing a hypergraph using all panoramic features as nodes; constructing positional hyperedges for cylindrical projection features and cube projection features belonging to the same ground panoramic image; constructing directional hyperedges for the cube projection using the K-Hop nearest neighbor method; and aggregating the positional hyperedges and the directional hyperedges to obtain a hypergraph with positional hyperedges and directional hyperedges, wherein each hypergraph node is a feature of all panoramic images.

[0010] Optionally, in one embodiment of this application, the step of classifying and predicting the relative positions of multiple panoramic images based on the hypergraph to obtain the prediction results of the relative positions includes: performing HCNNConv+ convolution on the hypergraph node features under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes; performing hyperedge aggregation on the new features of all nodes under the guidance of the position hyperedges to obtain features of new dimensions; and converting the distribution of relative positions into the distribution probability of absolute positions by using the discretized positions of ground points in the first panoramic image and the features of the new dimensions.

[0011] Optionally, in one embodiment of this application, the distance is calculated using the following formula:

[0012]

[0013] Where d is the relative position, dij Let n be the Euclidean distance between the cylindrical surface feature of the i-th panoramic image and the feature of the j-th remote sensing patch, and n be the number of ground panoramic images.

[0014] Optionally, in one embodiment of this application, the step of using the best-matching remote sensing image as the positioning result and outputting the latitude, longitude, and altitude information of the remote sensing image includes: using the remote sensing image closest to the ground panoramic image group as the positioning result, and extracting the latitude, longitude, and altitude information carried by the remote sensing image as the final positioning result.

[0015] A second aspect of this application provides a multi-point geolocation device for ground panoramic images, comprising: an acquisition module for acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database; a projection module for performing cylindrical and cubic projections on all panoramic images to obtain cylindrical and cubic projections of the panoramic images; an extraction module for extracting cylindrical and cubic projection features of the cylindrical and cubic projections of the panoramic images, respectively, and extracting all remote sensing image features from the candidate remote sensing image database; a construction module for constructing a hypermap based on the cylindrical and cubic projection features; a prediction module for classifying and predicting the relative positions of the multiple panoramic images based on the hypermap to obtain prediction results of the relative positions; and a positioning module for measuring distances based on the prediction results of the relative positions, combining the features of all remote sensing images, selecting the best-matching remote sensing image as the positioning result, and outputting the latitude, longitude, and altitude information of the remote sensing image.

[0016] Optionally, in one embodiment of this application, the acquisition module includes: an acquisition unit for recording the acquired ground image as a panoramic image; a discretization unit for using the offset of ground points on the remote sensing image as a position representation, and for each remote sensing image, discretizing the offset at a preset resolution, and specifying the initial position of the ground point discretization position in the first panoramic image; and a selection unit for selecting and specifying the candidate remote sensing image database, such that the remote sensing images in the candidate remote sensing image database cover the input ground point to be located.

[0017] Optionally, in one embodiment of this application, the extraction module includes: a first extraction unit, configured to extract features from the cylindrical projection of the panoramic image to obtain the cylindrical projection features of ground points; a second extraction unit, configured to extract features from the six faces of the cube projection of the panoramic image, and to decompose the panoramic image of each cube projection in terms of direction to obtain the six faces, thereby obtaining the cube projection features of ground points; and a third extraction unit, configured to extract block features from all remote sensing images in the remote sensing image database, and to represent each remote sensing image with a feature vector to obtain the features of all remote sensing images.

[0018] Optionally, in one embodiment of this application, the construction module includes: a first construction unit, configured to construct a hypergraph using all panoramic image features as nodes; a second construction unit, configured to construct positional hyperedges from cylindrical projection features and cubic projection features belonging to the same ground panoramic image; a third construction unit, configured to construct directional hyperedges from the cubic projection using the K-Hop nearest neighbor method; and a first aggregation unit, configured to aggregate the positional hyperedges and the directional hyperedges to obtain a hypergraph with positional hyperedges and directional hyperedges, wherein each hypergraph node is a feature of all panoramic images.

[0019] Optionally, in one embodiment of this application, the prediction module includes: a convolution unit, used to perform HCNNConv+ convolution on the features of the hypergraph nodes under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes; a second aggregation unit, used to perform hyperedge aggregation on the new features of all nodes under the guidance of the position hyperedges to obtain features of a new dimension; and a transformation unit, used to transform the distribution of relative positions into the distribution probability of absolute positions by discretizing the positions of the ground points in the first panoramic image and the features of the new dimension.

[0020] Optionally, in one embodiment of this application, the distance is calculated using the following formula:

[0021]

[0022] Where d is the relative position, d ij Let n be the Euclidean distance between the cylindrical surface feature of the i-th panoramic image and the feature of the j-th remote sensing patch, and n be the number of ground panoramic images.

[0023] Optionally, in one embodiment of this application, the positioning module includes: a positioning unit, used to take the remote sensing image closest to the ground panoramic image group as the positioning result, and extract the latitude, longitude and altitude information carried by the remote sensing image as the final positioning result.

[0024] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-point geolocation method for ground panoramic images as described in the above embodiments.

[0025] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-point geolocation method for a panoramic ground image.

[0026] This application's embodiments can predict relative positions based on block feature extraction from multi-point panoramic images and remote sensing images, thereby calculating distance metrics and matching the best remote sensing image to obtain the positioning result. This improves the accuracy and robustness of the positioning result, making it more accurate and practical. Therefore, it solves the problems in related technologies where, because the shooting range of remote sensing images is much larger than that of ground camera equipment, a single ground image appears in the field of view of multiple remote sensing images, resulting in insufficient uniqueness of the remote sensing image retrieved from a single ground image, failing to meet actual positioning needs, and reducing positioning accuracy, robustness, and precision.

[0027] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a flowchart of a multi-point geolocation method for a ground panoramic image provided according to an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of a multi-point geolocation algorithm framework for a ground panoramic image according to an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of the hyperedge construction process of a hypergraph according to an embodiment of this application;

[0032] Figure 4 This is a schematic diagram illustrating the overall process of multi-point geolocation of a ground panoramic image according to an embodiment of this application.

[0033] Figure 5 This is a schematic diagram of the structure of a multi-point geolocation device for ground panoramic images according to an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0036] The following description, with reference to the accompanying drawings, illustrates a multi-point geolocation method and apparatus for ground panoramic images according to embodiments of this application. Addressing the issues raised in the background section regarding related technologies, where the range of remote sensing images is much greater than that of ground cameras, resulting in a single ground image appearing within the field of view of multiple remote sensing images, the uniqueness of the remote sensing image retrieved from a single ground image is insufficient, failing to meet practical positioning needs and reducing positioning accuracy, robustness, and precision. This application provides a multi-point geolocation method for ground panoramic images, which can predict relative positions based on block feature extraction from multi-point panoramic images and remote sensing images, thereby calculating distance metrics and matching the optimal remote sensing image to obtain the positioning result. This improves the accuracy and robustness of the positioning result, making it more accurate and practical. Thus, it solves the problems in related technologies where the range of remote sensing images is much greater than that of ground cameras, resulting in a single ground image appearing within the field of view of multiple remote sensing images, leading to insufficient uniqueness of the remote sensing image retrieved from a single ground image, failing to meet practical positioning needs, and reducing positioning accuracy, robustness, and precision.

[0037] Specifically, Figure 1 This is a flowchart illustrating a multi-point geolocation method for a ground panoramic image provided in an embodiment of this application.

[0038] like Figure 1 As shown, the multi-point geolocation method for this panoramic ground image includes the following steps:

[0039] In step S101, panoramic images of multiple points within the same area are acquired, and a candidate remote sensing image database is selected.

[0040] It is understood that, in the embodiments of this application, the panoramic image is a collection of images in different directions within a spherical space centered on the shooting point. Multiple ground panoramic images I1, I2, ..., I can be acquired for the area to be located. n And select a remote sensing image database S within the required positioning area. For example, an aerial satellite image can be used as a candidate remote sensing image database.

[0041] Optionally, in one embodiment of this application, acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database includes: recording the acquired ground images as panoramic images; using the offset of ground points on the remote sensing image as a position representation; for each remote sensing image, discretizing the offset at a preset resolution; specifying the initial position of the ground point discretization position in the first panoramic image; selecting and specifying a candidate remote sensing image database such that the remote sensing images in the candidate remote sensing image database cover the input ground point to be located.

[0042] In the actual execution of this process, the acquired panoramic ground images can be denoted as I1, I2, ..., I... nThe location of a point is represented by its offset on the remote sensing image. For each remote sensing image, the offset is discretized at a resolution of N×N. The initial position of the ground point in the first panoramic image is specified at the N×N discretized position. A remote sensing image database is selected and specified to ensure that the remote sensing images in the database provide complete and dense coverage of the input ground point to be located. When the resolution of the discrete position is N×N, the relative position has 4N(N-1)+1 possibilities.

[0043] In step S102, cylindrical and cubic projections are performed on all panoramic images to obtain the cylindrical projection and cubic projection of the panoramic images.

[0044] In actual execution, all acquired ground panoramic images I1, I2, ..., I can be processed. n Each panoramic image in the image is subjected to cylindrical and cubic projections to obtain a cylindrical projection ground panoramic image I. e1 ,I e2 ,...,I en And cube projection ground panoramic map I c1 ,I c2 ,...,I cn .

[0045] In step S103, the cylindrical projection features and cube projection features of the panoramic image cylindrical projection and the panoramic image cube projection are extracted respectively, and all remote sensing image features in the candidate remote sensing image database are extracted.

[0046] It is understood that, in the embodiments of this application, the cylindrical surface projection panoramic image I obtained in the above steps can be extracted. e1 ,I e2 ,...,I en Obtain the corresponding cylindrical surface projection features and extract the cube projection ground panorama I. c1 ,I c2 ,...,I cn The corresponding cube projection features are obtained, and the candidate remote sensing image database S is extracted to obtain all remote sensing image features.

[0047] Optionally, in one embodiment of this application, the cylindrical projection features and cube projection features of the panoramic image and the panoramic image cube projection are extracted respectively, and all remote sensing image features in the candidate remote sensing image database are extracted, including: extracting features from the cylindrical projection of the panoramic image to obtain the cylindrical projection features of the ground points; extracting features from the six faces of the cube projection of the panoramic image, and deconstructing the panoramic image of each cube projection in terms of direction to obtain the six faces, so as to obtain the cube projection features of the ground points; and extracting block features from all remote sensing images in the remote sensing image database, representing each remote sensing image with a feature vector to obtain all remote sensing image features.

[0048] In actual implementation, the cylindrical projection I of the collected ground panoramic image can be applied. e1 ,I e2 ,...,I en Feature extraction is performed to obtain the cylindrical panoramic image features F of ground points. e1 ,F e2 ,...,F en The cube projection I of the acquired panoramic ground image e1 ,I e2 ,...,I en Feature extraction is performed on each of the six faces of the cube. The panoramic image of each cube projection is decomposed into six faces to obtain the cube panoramic image features F of the ground point. c11 ,F c12 ,...,F c16 ,F c21 ,F c22 ,...,F c26 ,...,F cn1 ,F cn2 ,...,F cn6 Block feature extraction is performed on all remote sensing images in the selected remote sensing image database S, and each remote sensing image is represented by N×N 1×4096 feature vectors.

[0049] Specifically, such as Figure 2The diagram illustrates a multi-point geolocation algorithm framework for ground panoramic images according to an embodiment of this application. During the block feature extraction process, for the ground view branch, since features from one cylindrical projection image and six cube projection images for each query need to be extracted, and each input has multiple queries, ResNet can replace the SAFA's VGG (Visual Geometry Group) network, making it more lightweight. SAFA is a widely used module in current cross-view geolocation tasks. Its main idea is to reweight the embeddings based on their positions in the feature maps. Spatial-aware feature aggregation blocks significantly improve the performance of orientation information representation. The ground-view branch outputs 7n feature vectors of the same size, including 6n cube projection features and n cylindrical projection features, where n is the number of input panoramic images.

[0050] For the remote sensing image branch, the same twin backbone and aggregation as the ground view branch can be used. To find the best matching reference covering the relative positions of all multiple queries, it is necessary to enhance the focus on local features of the remote sensing image. Inspired by the cross-view metric localization method, spatial resolution can be improved by dividing the entire satellite feature volume into N×N sub-volumes, where N represents the number of sub-volumes in the length and width directions. After splitting, each sub-volume is fed into a feature aggregation module with shared weights, and the remote sensing image branch outputs N×N feature vectors of the same size.

[0051] In step S104, a hypergraph is constructed for the cylindrical projection features and the cubic projection features.

[0052] It can be understood that the hypergraph structure in the embodiments of this application can model multi-point relationships and extract their higher-order relationships. In addition to optical information, panoramic images also contain sufficient directional information. It is natural to analyze and construct their relative positions by extracting features from different inputs. Therefore, the features of panoramic images can be used to construct hypergraph structures for nodes.

[0053] Optionally, in one embodiment of this application, constructing a hypergraph for cylindrical projection features and cube projection features includes: constructing a hypergraph using all panoramic features as nodes; constructing positional hyperedges for cylindrical projection features and cube projection features belonging to the same ground panoramic image; constructing directional hyperedges for cube projections using the K-Hop nearest neighbor method; and aggregating the positional hyperedges and directional hyperedges to obtain a hypergraph with positional hyperedges and directional hyperedges, wherein each hypergraph node is a feature of all panoramic images.

[0054] Specifically, such as Figure 3The diagram shown illustrates the hyperedge construction process of a hypergraph according to an embodiment of this application. Since cylindrical projection projects a spherical surface onto a plane, it can more comprehensively represent the position information of ground points, and cubic projection expands the information of each face separately, it can more comprehensively represent the direction information. Therefore, two sets of hyperedges can be constructed to represent position and direction respectively. Features of the ground panorama can be used as nodes to construct the hypergraph, with a node count of 7n, where n is the number of input ground panoramas. Position hyperedges ε are constructed from the cylindrical projection and cubic projection features belonging to the same ground panorama. L , ε L ={(F e1 ,F c11 ,F c12 ,...,F c16 ),...,(F en ,F cn1 ,F cn2 ,...,F cn6 The cube projection is then processed using the K-Hop nearest neighbor method to construct directional hyperedges. This method aggregates features pointing in the same direction. The specific construction method for directional hyperedges is as follows: in The feature of a face representing the entire cube projection is used to finally merge the two constructed hyperedges to obtain a hyperedge with position ε. L and direction of the super-edge ε D The hypergraph, where each node represents all the features of the ground panorama.

[0055] In step S105, the relative positions of multiple panoramic images are classified and predicted based on the hypergraph to obtain the prediction results of the relative positions.

[0056] In actual implementation, the location of the constructed hypergraph and its hyperedges ε can be determined. L and direction of the super-edge ε D Based on the characteristics of the hypergraph nodes, the relative positions of the acquired panoramic images are classified and predicted to obtain the prediction results.

[0057] Optionally, in one embodiment of this application, the relative positions of multiple panoramic images are classified and predicted based on a hypergraph to obtain the prediction results of the relative positions, including: performing HCNNConv+ convolution on the features of the hypergraph nodes under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes; performing hyperedge aggregation on the new features of all nodes under the guidance of the position hyperedges to obtain features of new dimensions; and converting the distribution of relative positions into the distribution probability of absolute positions by discretizing the positions of the ground points of the first panoramic image and the features of the new dimensions.

[0058] Specifically, all hyperedges (positional hyperedges ε) of the hypergraph can be constructed in the above steps. Land direction of the super-edge ε D Under the guidance of HCNNConv+, the features of the hypergraph nodes are convolved to obtain new features for all 7n nodes. The new node features are then analyzed at the hyperedge ε. L Under the guidance of [the relevant parameters], feature hyperedge aggregation is performed, ultimately obtaining features of dimension n×(4N(N-1)+1), where each dimension of 4N(N-1)+1 represents the probability of the relative position of the ground point relative to the first ground point. By recording the position of the first ground point and obtaining the probabilities of the positions of other ground points relative to the first ground point, the distribution of relative positions is transformed into the distribution of absolute positions (represented in an N×N discretized position grid) probabilities.

[0059] In step S106, the distance is measured based on the prediction result of the relative position, and combined with all remote sensing image features, the best matching remote sensing image is used as the positioning result, and the latitude, longitude and altitude information of the remote sensing image is output.

[0060] It is understood that in this embodiment of the application, the relative distance can be calculated by the prediction result of the relative position obtained in the above steps, and combined with all remote sensing image features of the extracted candidate remote sensing image database, the remote sensing image that best matches the features can be obtained as the matching result, and the latitude, longitude and altitude information of the remote sensing image can be obtained as the positioning result.

[0061] Optionally, in one embodiment of this application, the distance is calculated using the following formula:

[0062]

[0063] Where d is the relative position, d ij Let n be the Euclidean distance between the cylindrical surface feature of the i-th panoramic image and the feature of the j-th remote sensing patch, and n be the number of ground panoramic images.

[0064] In actual implementation, based on the location information obtained from the above steps, all ground panoramic images can be grouped together and the distance between them and all remote sensing images in the candidate remote sensing image database can be calculated based on the above formula.

[0065] Optionally, in one embodiment of this application, the best-matching remote sensing image is used as the positioning result, and the latitude, longitude, and altitude information of the remote sensing image is output, including: using the remote sensing image closest to the ground panoramic image group as the positioning result, and extracting the latitude, longitude, and altitude information carried by the remote sensing image as the final positioning result.

[0066] Specifically, the remote sensing image that is closest to the ground panoramic image group, i.e., has the smallest Euclidean distance, can be used as the positioning result. The GPS marker on the remote sensing image can be obtained to get the latitude, longitude and altitude information it carries, which can then be used as the final positioning result.

[0067] The working content of the embodiments of this application will be described in detail below with a specific example.

[0068] like Figure 4 The diagram illustrates the overall process of multi-point geolocation of a ground panoramic image according to an embodiment of this application. It involves acquiring panoramic images of multiple points within the same area, specifying the discretized location of the first panoramic image, selecting a candidate remote sensing image database, and then performing cylindrical and cubic projections on all panoramic images to obtain two representations of the same panoramic image. Features from both the cylindrical and cubic projections of the panoramic images are extracted, along with features from all remote sensing images in the database. A hypermap is constructed based on the cylindrical and cubic projection features. The relative positions of the multiple panoramic images are classified and predicted based on the hypermap, and distance is measured based on the relative positions. The best-matching remote sensing image is used as the positioning result, and the latitude, longitude, and altitude information of the remote sensing image is output.

[0069] The multi-point geolocation method based on ground panoramic images proposed in this application can predict relative positions based on block feature extraction of multi-point panoramic images and remote sensing images, thereby calculating distance metrics and matching the best remote sensing image to obtain the positioning result. This improves the accuracy and robustness of the positioning result, making it more accurate and practical. Therefore, it solves the problems in related technologies where, because the shooting range of remote sensing images is much larger than that of ground camera equipment, a single ground image appears in the field of view of multiple remote sensing images, resulting in insufficient uniqueness of the remote sensing image retrieved from a single ground image, failing to meet actual positioning needs, reducing positioning accuracy and robustness, and causing insufficient accuracy.

[0070] Next, referring to the accompanying drawings, a multi-point geolocation device for ground panoramic images according to an embodiment of this application is described.

[0071] Figure 5 This is a schematic diagram of the structure of a multi-point geolocation device for ground panoramic images according to an embodiment of this application.

[0072] like Figure 5 As shown, the multi-point geolocation device 10 for the ground panoramic image includes: an acquisition module 100, a projection module 200, an extraction module 300, a construction module 400, a prediction module 500, and a positioning module 600.

[0073] The acquisition module 100 is used to acquire panoramic images of multiple points within the same area and select alternative remote sensing image databases.

[0074] The projection module 200 is used to perform cylindrical and cubic projections on all panoramic images to obtain cylindrical projections and cubic projections of panoramic images.

[0075] The extraction module 300 is used to extract the cylindrical projection features and cube projection features of the panoramic image cylindrical projection and the panoramic image cube projection respectively, and to extract all remote sensing image features from the candidate remote sensing image database.

[0076] Module 400 is used to construct a hypergraph from cylindrical and cubic projection features.

[0077] The prediction module 500 is used to classify and predict the relative positions of multiple panoramic images based on the hypergraph, and obtain the prediction results of the relative positions.

[0078] The positioning module 600 is used to measure distance based on the prediction results of relative position, and combines all remote sensing image features to take the best matching remote sensing image as the positioning result, and outputs the latitude, longitude and altitude information of the remote sensing image.

[0079] Optionally, in one embodiment of this application, the acquisition module 100 includes: an acquisition unit, a discrete unit, and a selection unit.

[0080] The acquisition unit is used to record the acquired ground images as panoramic images.

[0081] Discrete units are used to represent the position of ground points on the remote sensing image. For each remote sensing image, the offset is discretized at a preset resolution, specifying the initial position of the ground points on the discretized position of the first panoramic image.

[0082] The selection unit is used to select and specify alternative remote sensing image databases, such that the remote sensing images in the alternative remote sensing image databases cover the input ground points to be located.

[0083] Optionally, in one embodiment of this application, the extraction module 300 includes: a first extraction unit, a second extraction unit, and a third extraction unit.

[0084] The first extraction unit is used to extract features from the cylindrical projection of the panoramic image to obtain the cylindrical projection features of the ground points.

[0085] The second extraction unit is used to extract features from the six faces of the cube projection of the panoramic image, and to decompose the panoramic image of each cube projection in terms of orientation to obtain the six faces, so as to obtain the cube projection features of the ground points.

[0086] The third extraction unit is used to extract block features from all remote sensing images in the remote sensing image database, representing each remote sensing image with a feature vector to obtain the features of all remote sensing images.

[0087] Optionally, in one embodiment of this application, the building module 400 includes: a first building unit, a third building unit, and a first aggregation unit.

[0088] The first construction unit is used to construct a hypergraph using all panoramic features as nodes.

[0089] The second building unit is used to construct positional hyperedges from cylindrical and cubic projection features belonging to the same ground panorama.

[0090] The third building unit is used to construct directional hyperedges by projecting the cube onto the nearest neighbor of the cube using the K-Hop method.

[0091] The first aggregation unit is used to aggregate positional hyperedges and orientational hyperedges to obtain a hypergraph with positional hyperedges and orientational hyperedges, wherein each hypergraph node is a feature of the entire panorama.

[0092] Optionally, in one embodiment of this application, the prediction module 500 includes: a convolution unit, a second aggregation unit, and a transformation unit.

[0093] The convolutional unit is used to perform HCNNConv+ convolution on the features of the hypergraph nodes under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes.

[0094] The second aggregation unit is used to perform hyperedge aggregation of features of all nodes under the guidance of position hyperedges to obtain features of new dimensions.

[0095] The transformation unit is used to convert the distribution of relative positions into the distribution probability of absolute positions by discretizing the positions of ground points in the first panoramic image and using features of new dimensions.

[0096] Optionally, in one embodiment of this application, the distance is calculated using the following formula:

[0097]

[0098] Where d is the relative position, d ij Let n be the Euclidean distance between the cylindrical surface feature of the i-th panoramic image and the feature of the j-th remote sensing patch, and n be the number of ground panoramic images.

[0099] Optionally, in one embodiment of this application, the positioning module 600 includes a positioning unit.

[0100] The positioning unit is used to take the remote sensing image closest to the ground panoramic image group as the positioning result, and extract the latitude, longitude and altitude information carried by the remote sensing image as the final positioning result.

[0101] It should be noted that the foregoing explanation of the multi-point geolocation method embodiment for ground panoramic images also applies to the multi-point geolocation device for ground panoramic images in this embodiment, and will not be repeated here.

[0102] The multi-point geolocation device based on the ground panoramic image proposed in this application can predict relative positions based on the block feature extraction of multi-point panoramic images and remote sensing images, thereby calculating distance metrics and matching the best remote sensing image to obtain the positioning result. This improves the accuracy and robustness of the positioning result, making it more accurate and practical. Therefore, it solves the problems in related technologies where, because the shooting range of remote sensing images is much larger than that of ground camera equipment, a single ground image appears in the field of view of multiple remote sensing images, resulting in insufficient uniqueness of the remote sensing image retrieved from a single ground image, failing to meet actual positioning needs, reducing positioning accuracy and robustness, and causing insufficient accuracy.

[0103] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0104] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0105] When the processor 602 executes the program, it implements the multi-point geolocation method for ground panoramic images provided in the above embodiments.

[0106] Furthermore, electronic devices also include:

[0107] Communication interface 603 is used for communication between memory 601 and processor 602.

[0108] The memory 601 is used to store computer programs that can run on the processor 602.

[0109] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0110] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0112] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0113] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-point geolocation method for ground panoramic images.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0116] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0118] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multi-point geolocation method for ground panoramic images, characterized in that, Includes the following steps: Collect panoramic images of multiple points within the same area and select a database of alternative remote sensing images; Perform cylindrical and cubic projections on all panoramic images to obtain the cylindrical projection and cubic projection of the panoramic images. Extract the cylindrical projection features and cube projection features of the panoramic image cylindrical projection and the panoramic image cube projection respectively, and extract all remote sensing image features from the candidate remote sensing image database. A hypergraph is constructed for the cylindrical surface projection features and the cube projection features; Based on the hypergraph, the relative positions of multiple panoramic images are classified and predicted to obtain the prediction results of the relative positions; as well as The distance is measured based on the predicted result of the relative position, and combined with the features of all the remote sensing images, the best matching remote sensing image is taken as the positioning result, and the latitude, longitude and altitude information of the remote sensing image is output. The construction of a hypergraph for the cylindrical projection features and the cubic projection features includes: constructing a hypergraph using all panoramic features as nodes; constructing positional hyperedges for cylindrical projection features and cubic projection features belonging to the same ground panoramic image; constructing directional hyperedges for the cubic projection using the K-Hop nearest neighbor method; and aggregating the positional hyperedges and the directional hyperedges to obtain a hypergraph with both positional and directional hyperedges, wherein each hypergraph node is a feature of all panoramic images. The step of classifying and predicting the relative positions of multiple panoramic images based on the hypergraph to obtain the prediction results of the relative positions includes: performing HCNNConv+ convolution on the hypergraph node features under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes; performing hyperedge aggregation on the new features of all nodes under the guidance of the position hyperedges to obtain features of new dimensions; and converting the distribution of relative positions into the distribution probability of absolute positions by discretizing the ground points of the first panoramic image and the features of the new dimensions.

2. The method according to claim 1, characterized in that, The process of acquiring panoramic images of multiple points within the same area and selecting a candidate remote sensing image database includes: The collected ground images are recorded as panoramic images; The offset of ground points on the remote sensing image is used as the position representation. For each remote sensing image, the offset is discretized at a preset resolution, and the initial position of the ground points on the discretized position of the first panoramic image is specified. Select and specify the alternative remote sensing image database such that the remote sensing images in the alternative remote sensing image database cover the input ground point to be located.

3. The method according to claim 1, characterized in that, The process involves extracting the cylindrical projection features and cube projection features of the panoramic image's cylindrical projection and cube projection, respectively, and extracting all remote sensing image features from the candidate remote sensing image database, including: Feature extraction is performed on the cylindrical projection of the panoramic image to obtain the cylindrical projection features of the ground points; Feature extraction is performed on the six faces of the cube projection of the panoramic image, and the panoramic image of each cube projection is decomposed in direction to obtain the six faces, so as to obtain the cube projection features of the ground point. Block feature extraction is performed on all remote sensing images in the remote sensing image database, and each remote sensing image is represented by a feature vector to obtain the features of all remote sensing images.

4. The method according to claim 1, characterized in that, The formula for calculating the distance is: in, Relative position For the first The cylindrical surface features of the first panoramic image and the first The Euclidean distance of the features of a remote sensing patch This represents the number of ground panoramic images.

5. The method according to claim 1, characterized in that, The step of using the best-matching remote sensing image as the positioning result and outputting the latitude, longitude, and altitude information of the remote sensing image includes: The remote sensing image closest to the ground panoramic image set is used as the positioning result, and the latitude, longitude, and altitude information carried by the remote sensing image is extracted as the final positioning result.

6. A multi-point geolocation device for ground panoramic images, characterized in that, include: The acquisition module is used to acquire panoramic images of multiple points within the same area and select a database of alternative remote sensing images. The projection module is used to perform cylindrical and cubic projections on all panoramic images to obtain cylindrical and cubic projections of panoramic images. The extraction module is used to extract the cylindrical projection features and cube projection features of the panoramic image cylindrical projection and the panoramic image cube projection, respectively, and to extract all remote sensing image features from the candidate remote sensing image database. A construction module is used to construct a hypergraph from the cylindrical surface projection features and the cube projection features; The prediction module is used to classify and predict the relative positions of multiple panoramic images based on the hypergraph, and obtain the prediction results of the relative positions. as well as The positioning module is used to measure distance based on the predicted result of the relative position, and combine all the remote sensing image features to take the best matching remote sensing image as the positioning result, and output the latitude, longitude and altitude information of the remote sensing image. The construction module includes: a first construction unit for constructing a hypergraph using all panoramic image features as nodes; a second construction unit for constructing positional hyperedges from cylindrical and cubic projection features belonging to the same ground panoramic image; a third construction unit for constructing directional hyperedges from the cubic projection using the K-Hop nearest neighbor method; and a first aggregation unit for aggregating the positional and directional hyperedges to obtain a hypergraph with both positional and directional hyperedges, wherein each hypergraph node is a feature of all panoramic images. The prediction module includes: a convolution unit, used to perform HCNNConv+ convolution on the features of the hypergraph nodes under the guidance of all hyperedges of the hypergraph to obtain new features of all nodes; a second aggregation unit, used to perform hyperedge aggregation on the new features of all nodes under the guidance of the position hyperedges to obtain features of new dimensions; and a transformation unit, used to convert the distribution of relative positions into the distribution probability of absolute positions by discretizing the positions of ground points in the first panoramic image and the features of the new dimensions.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the multi-point geolocation method for ground panoramic images as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-point geolocation method for ground panoramic images as described in any one of claims 1-5.

Images