Cross-view cross-modal image geolocation method, device, equipment and medium

Abstract

The application discloses a cross-view cross-modal image geolocation method and device, equipment and medium, relates to the field of computer vision and geographic information processing. The method comprises the following steps: acquiring a query image and a reference image, the query image is a ground view, the reference image is a aerial view or a radar depth image, and the direction is unknown; extracting initial spatial features through a shared backbone network; extracting frequency-guided structural priori of the initial features to obtain multi-frequency structural features encoding global structural patterns; frequency-guided adaptive modulation is performed on the initial features based on the multi-frequency structural features to generate refined features; state space modeling is performed on the refined features, global information is propagated based on structural consistency to generate structure alignment feature representation; and the matching reference image is determined based on distance measurement in the shared embedding space to complete geolocation. The application significantly improves the positioning robustness and accuracy in complex environments.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and geographic information processing, and in particular to a cross-view, cross-modal image geolocation method, apparatus, device, and medium. Background Technology

[0002] Image geolocation aims to determine the precise geographic coordinates of a query image by matching it with a georeferenced database. As a supplementary solution to the Global Navigation Satellite System (GNSS), it plays a crucial role in enhancing the spatial awareness capabilities of autonomous systems in environments with degraded signal strength. This technology has been widely applied in fields such as UAV navigation, autonomous driving, and robot navigation.

[0003] In practical applications, image geolocation faces complex and uncontrollable data sources. Query images typically come from ground-level perspectives, while reference data may be satellite or aerial images, and may also involve different sensor modalities, such as RGB images or radar depth images. These cross-viewpoint and cross-modal situations often coexist, making models designed for single scenarios difficult to apply. Building separate models for each scenario would significantly increase system complexity and maintenance costs, limiting system scalability. Existing methods typically handle cross-viewpoint and cross-modal image geolocation separately, achieving good performance under their respective single task settings by designing rotation-invariant features, constructing spatial alignment mechanisms, or learning shared feature spaces. However, they struggle to handle both types of variations simultaneously within a unified framework.

[0004] The difficulty of geolocalization increases significantly when cross-viewpoint variations and cross-modal differences coexist. On the one hand, cross-viewpoint variations disrupt the spatial correspondence between images, making features dependent on spatial layout ineffective. On the other hand, cross-modal differences weaken visual appearance consistency, rendering appearance-based matching methods ineffective. Especially in real-world scenarios with unknown orientations, the combined effect of these two factors disrupts both spatial alignment and visual consistency, making it difficult for existing methods to learn representations with good generalization capabilities, resulting in a significant decline in localization performance. Therefore, there is an urgent need for an image geolocalization method that can simultaneously handle cross-viewpoint and cross-modal variations with unknown orientations within a unified framework. Summary of the Invention

[0005] The technical problem solved by the embodiments of the present invention is: how to achieve reliable cross-viewpoint and cross-modal image geolocation when the shooting direction of the query image is unknown and the viewpoint and modality of the reference image source are uncertain.

[0006] In a first aspect, embodiments of the present invention provide a cross-view, cross-modal image geolocation method, comprising: Obtain a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; Initial spatial feature maps of the query image and the reference image are extracted using a shared feature extraction backbone network. Frequency-guided structural prior extraction is performed on the initial spatial feature map. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structural features encoding global structural patterns are obtained. Based on the multi-frequency structural features, the initial spatial feature map is subjected to frequency-guided adaptive modulation. Through a spatial adaptive gating mechanism and learnable fusion weights, the prior structural information and the original spatial information are selectively integrated to generate a refined feature map. State space modeling is performed on the refined feature map, and global information is propagated along the spatial dimension with structural consistency as a condition to generate context-enhanced structural alignment feature representation; Based on the structural alignment feature representations of the query image and the reference image, distance measurement is performed in the shared embedding space to determine the reference image that matches the query image, thereby completing geolocation.

[0007] Secondly, embodiments of the present invention also provide a cross-view, cross-modal image geolocation device, comprising: The acquisition module is used to acquire a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; The feature extraction module is used to extract initial spatial feature maps of the query image and the reference image respectively through a shared feature extraction backbone network; The frequency-guided structure prior module is used to extract the structure prior from the initial spatial feature map by frequency guidance. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structure features that encode the global structure pattern are obtained. The frequency-guided modulation module is used to perform frequency-guided adaptive modulation on the initial spatial feature map based on the multi-frequency structural features. Through a spatial adaptive gating mechanism and learnable fusion weights, it selectively integrates structural prior information with the original spatial information to generate a refined feature map. The state space modeling module is used to perform state space modeling on the refined feature map, propagating global information along the spatial dimension with structural consistency as a condition, and generating a context-enhanced structural alignment feature representation. The geolocation module is used to perform distance measurement in a shared embedding space based on the structural alignment feature representation of the query image and the reference image to determine the reference image that matches the query image, thereby completing geolocation.

[0008] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned cross-view cross-modal image geolocation method.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described cross-view, cross-modal image geolocation method.

[0010] This invention introduces a frequency-guided structural prior and an adaptive modulation mechanism to transform the features of query and reference images from spatial domain representations that depend on local appearance and absolute spatial location to representations centered on globally stable structures. This effectively reduces sensitivity to changes in viewpoint, unknown orientation, and modal differences. Based on this, state-space modeling is combined with global information propagation under the condition of structural consistency. This allows the feature representation to maintain discriminative power across views and modalities even when spatial misalignment and appearance inconsistency coexist. Thus, within a unified framework, the robustness and accuracy of cross-viewpoint and cross-modal image geolocation under unknown orientations are significantly improved. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a cross-view, cross-modal image geolocation framework provided in an embodiment of the present invention; Figure 2 A schematic flowchart of the cross-view, cross-modal image geolocation method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the prior flow of the frequency guidance structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the feature fusion process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the refined feature map generation process provided in an embodiment of the present invention; Figure 6 A schematic diagram of the state-space modeling process provided in an embodiment of the present invention; Figure 7 A comparison chart showing the trade-off between accuracy and efficiency provided in embodiments of the present invention; Figure 8 A heatmap of the CVACT dataset under unknown input direction provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a cross-view, cross-modal image geolocation device provided in an embodiment of the present invention; Figure 10 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0018] like Figure 2As shown, this embodiment of the invention provides a cross-view, cross-modal image geolocation method, including steps S1-S6: S1. Obtain a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown.

[0019] In this step, the query image typically comes from ground-based imaging equipment, and its azimuth, elevation, and other directional information at the time of capture is unknown. The reference image comes from aerial or satellite remote sensing platforms (aerial images) or radar sensors (depth images). The two types of images differ across viewing angles and may also differ across imaging modalities. This setting simulates real-world scenarios where data sources are complex and directional information is lacking. After acquiring the images, both the query and reference images can be uniformly resized and augmented to improve the model's generalization ability. For example, for cross-view image geolocation tasks, satellite images can be resized to 384×384 pixels, and ground images to 140×768 pixels; for cross-modal image geolocation tasks, all input images can be resized to 288×384 pixels. Data augmentation strategies can use the same approach as Sample4Geo, including random horizontal flipping and color dithering.

[0020] S2. Using a shared feature extraction backbone network, extract the initial spatial feature maps of the query image and the reference image respectively.

[0021] In this step, such as Figure 1 As shown, a two-stream Siamese network architecture is employed, using a weight-sharing ConvNeXt-B backbone network to extract features from the query and reference images. ConvNeXt-B is a pure convolutional architecture that effectively captures local appearance patterns and spatial layout information. However, these initial spatial feature maps still heavily rely on local texture and absolute spatial location. When faced with spatial mismatches caused by unknown orientations and appearance inconsistencies arising from cross-modal interactions, local patterns may no longer be reliable, and directly matching based on such features leads to a significant performance degradation. Therefore, a subsequent frequency-guided state-space model is needed to further refine the features.

[0022] S3. Frequency-guided structural prior extraction is performed on the initial spatial feature map. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structural features of the encoded global structural pattern are obtained.

[0023] In this step, a Frequency-Guided Structure Prior (FGSP) module is introduced. Its purpose is to explicitly separate and enhance globally stable structural patterns under cross-viewpoint and cross-modal conditions, suppressing dependence on local appearance and absolute spatial layout. For example... Figure 3 As shown, this process extracts multi-frequency structural features through frequency domain decomposition, and gradually transforms spatially sensitive features into a structure-centered representation.

[0024] In a specific embodiment, S3 further includes steps S31-S35, such as... Figure 3 As shown: S31. Perform a two-dimensional discrete Fourier transform on the initial spatial feature map to obtain a frequency domain representation.

[0025] In this step, the feature map output by the backbone network is given. The spatial domain features are transformed to the frequency domain by applying the two-dimensional discrete Fourier transform (DFT2) to obtain the frequency domain representation. This transformation reorganizes spatial information into frequency components, providing an alternative representation where different frequency bands capture complementary features of the scene: low-frequency components encode coarse spatial variations related to global layout and geometry, while high-frequency components capture fine details such as texture and edges. This decomposition allows the model to explicitly handle signals with varying sensitivities to geometric and appearance changes, avoiding confusion and thus improving generalization ability.

[0026] S32. In the frequency domain, the frequency domain representation is separated into low-frequency components and high-frequency components using a preset frequency mask. The low-frequency components correspond to global layout and geometric information, while the high-frequency components correspond to texture and edge detail information.

[0027] In this step, frequency separation is performed using a binary mask defined by a fixed cutoff frequency ratio. Specifically, the low-frequency region is selected from the upper left corner of the spectrum. The remaining components are considered high-frequency regions. After masking, the following is obtained: and This decomposition forces a structured decomposition of the feature representation, separating information at different spatial frequencies. It prevents the model from implicitly mixing signals with different sensitivities to geometric transformations and appearance changes, and helps to enhance the model's robustness to unknown orientations and modal differences.

[0028] S33. Apply learnable scalar weights to the low-frequency component and the high-frequency component respectively to adaptively adjust the contribution of each frequency band component.

[0029] In this step, learnable scalar weights are introduced. and The separated low-frequency and high-frequency components are weighted element-wise to obtain the weighted low-frequency component. and weighted high-frequency frequency domain components The learned weights enable an adaptive balance between complementary components. The model no longer assumes fixed preferences but adjusts the contribution of each frequency band based on the input content and task requirements. For example, in cases of severe spatial misalignment or modal absence, the model can rely more on globally consistent low-frequency patterns; while selectively emphasizing high-frequency components when fine details are still informative. This flexibility is crucial for handling various geolocation conditions within a unified framework.

[0030] S34. Map the weighted frequency domain components back to the spatial domain through inverse two-dimensional discrete Fourier transform to obtain low-frequency spatial features and high-frequency spatial features.

[0031] In this step, the weighted low-frequency frequency domain components... and high frequency domain components Perform inverse two-dimensional discrete Fourier transform (IDFT2) on each side to obtain the low-frequency spatial features. and high-frequency spatial characteristics The frequency domain representation is transformed into a spatial feature map aligned with the original feature space. The inverse transform allows the frequency information to be reintegrated into the spatial processing while preserving the structural differences introduced by the decomposition: low-frequency spatial features focus on global layout, while high-frequency spatial features focus on edges and texture details.

[0032] S35. The initial spatial feature map, the low-frequency spatial feature, and the high-frequency spatial feature are fused to obtain the multi-frequency structural feature. Specifically, this includes: S351. The initial spatial feature map, the low-frequency spatial feature, and the high-frequency spatial feature are spliced ​​together along the channel dimension.

[0033] S352. Perform linear projection, nonlinear activation, and linear projection again on the spliced ​​features in sequence to obtain the multi-frequency structural features.

[0034] In this step, the fusion process aggregates complementary information from three sources: original spatial features preserve the original representation, low-frequency spatial features capture global organization, and high-frequency spatial features preserve detailed variations. For example... Figure 4 As shown, the specific implementation of the fusion is as follows: First, the three features are concatenated along the channel dimension (S351). Then, the concatenated features are sequentially subjected to linear projection, ReLU nonlinear activation, and linear projection again (S352) to obtain multi-frequency structural features. Learning in a data-driven manner integrates these signals into a unified multi-frequency structural feature without making artificial assumptions about the relative importance of various features. The resulting representation explicitly encodes multi-scale structural information and adaptively balances complementary cues, making it more robust to changes in perspective, directional uncertainty, and modal differences.

[0035] S4. Based on the multi-frequency structural features, the initial spatial feature map is subjected to frequency-guided adaptive modulation. Through a spatial adaptive gating mechanism and learnable fusion weights, the prior structural information and the original spatial information are selectively integrated to generate a refined feature map.

[0036] In this step, a Frequency Guided Modulation (FGM) module is introduced, aiming to adaptively optimize spatial features using frequency-guided structural representations. While FGSP provides structure-aware features... However, this structural and local information is not always spatially aligned, thus requiring adaptive modulation mechanisms to selectively integrate complementary cues, suppressing unstable local responses while emphasizing reliable structural cues. For example... Figure 5 As shown, FGM generates refined feature maps that are more suitable for subsequent long-range dependency modeling through content-aware gating and adaptive weighted fusion.

[0037] In a specific embodiment, S4 further includes steps S41-S45, such as... Figure 5 As shown: S41. The multi-frequency structural features are sequentially convolved, nonlinearly activated, and then convolved again to generate a spatial adaptive gating graph, and the gating value is restricted to a preset range by an activation function.

[0038] In this step, a content-aware gating mechanism is used to first analyze the multi-frequency structural features of the FGSP output. The model sequentially performs convolution, ReLU nonlinear activation, and further convolution to capture the local spatial context within the frequency-enhanced features, enabling the model to recognize spatially coherent patterns. Subsequently, the results are mapped to the (0,1) interval using the Sigmoid activation function to generate a spatially adaptive gating map. This gating map allows for the continuous modulation of feature responses, rather than applying them uniformly across the entire feature map. This enables the model to selectively emphasize regions richer in structural cue information while attenuating responses that may be obscure or less reliable. This spatial selectivity is particularly important when dealing with changes in viewpoint, orientation, and modal differences, as these variations have a non-uniform effect across different regions.

[0039] S42. The multi-frequency structural features are modulated element-by-element using the spatial adaptive gating graph to obtain the gating structural features.

[0040] In this step, the spatial adaptive gating graph is multiplied element-wise with the multi-frequency structural feature f_out to obtain the gating structural feature. This gating step can be viewed as a soft filtering process used to refine the structural representation generated by FGSP, preventing unfiltered frequency enhancement features from propagating directly and introducing misaligned or low-correlation signals in subsequent processing stages.

[0041] S43. Add the residuals of the gated structure features and the initial spatial feature map to obtain the combined features.

[0042] In this step, residual fusion is used to integrate the gated structure features. With original spatial features Add them together to obtain the combined features. This design incorporates structure-aware correction while preserving the original spatial representation. Retaining the original features as a reference is crucial because, although local appearance cues may be unstable, they can still provide strong discriminative signals in certain regions. The additive formula ensures that the model refines rather than replaces spatial features, thereby avoiding information loss and improving stability during training.

[0043] S44. After performing global average pooling on the combined features, convolution, nonlinear activation, re-convolution and normalized exponential function processing are performed in sequence to generate adaptive fusion weights corresponding to structural branches and spatial branches respectively.

[0044] In this step, the combined features are first processed using global average pooling. By aggregating contextual information into scene-level data, the model can evaluate the overall reliability of different feature types for the current input. Subsequently, through convolution, ReLU activation, further convolution, and softmax operations, two non-negative adaptive fusion weights that sum to 1 are generated. and ,Right now ,satisfy And α, This adaptive weighting mechanism allows the model to dynamically adjust its dependence on structural or local cues. For example, in cases of severe spatial misalignment or large cross-modal gaps, the model can increase the contribution of structure-aware features. Increased); while in scenarios where local details are reliable, a stronger spatial response can be preserved ( (Increase).

[0045] S45. Based on the adaptive fusion weights, the gated structural features and the initial spatial feature map are weighted and summed to obtain the refined feature map.

[0046] In this step, according to The calculation method is based on The weights are multiplied by the gating structure features. ,by The weights are multiplied by the original spatial features. The two are then added together to obtain a refined feature map. This weighted fusion balances globally consistent structural information and locally discriminative details in a content-relevant manner. It not only integrates complementary cues but also aligns them into a unified feature space, reducing the inconsistencies introduced by frequency decomposition. Through these steps, FGM performs fine-grained, content-aware feature refinement, bridging the gap between frequency domain structural cues and spatial domain representations, and generating well-suited conditional features for subsequent state-space modeling.

[0047] S5. Perform state-space modeling on the refined feature map, propagate global information along the spatial dimension with structural consistency as a condition, and generate a context-enhanced structural alignment feature representation.

[0048] In this step, a selective state-space model (SSM) based on Mamba is introduced to model long-range dependencies in the refined feature map refined by FGM. Unlike the classic linear SSM with fixed transition dynamics, the selective SSM dynamically generates both input and output projection parameters based on the current input, enabling adaptive information propagation. Since the input features have been refined into a representation centered on structural consistency through FGSP and FGM, this embodiment can effectively propagate information using a single-directional scan, eliminating the need for complex multi-directional scans. Figure 6 As shown, the SSM receives a refined feature map and outputs a structure-aligned feature representation that includes global context, which is inherently robust to orientation shifts and modal differences.

[0049] In a specific embodiment, S5 further includes steps S51-S54, such as... Figure 6 As shown: S51. A sequence modeling mechanism based on a selective state-space model is adopted, wherein the state transition matrix is ​​a learnable parameter, and the input projection matrix and the output projection matrix are dynamically generated based on the current input features.

[0050] In this step, given a refined feature map, the model will serialize it based on a selective SSM mechanism. The state transition matrix A consists of learnable parameters, while the input projection matrix... and output projection matrix All are related to the current input The function can be dynamically generated. Furthermore, a learnable step size is predicted for each position. This is used to adaptively control the degree of state updates. This design enables SSM to flexibly decide whether to retain or forget information based on the input content, achieving global reasoning on structured features.

[0051] S52. The refined feature map is expanded into a one-dimensional feature sequence according to a preset spatial scanning order.

[0052] In this step, a row-first scanning strategy is used to unfold the two-dimensional refined feature map into a one-dimensional sequence. Row-first scanning refers to arranging the feature vectors row by row from left to right and top to bottom, starting from the top left corner of the feature map, to form a sequence. Since the input features have been thoroughly structured and refined in previous modules, this single forward scan is sufficient to effectively propagate global information without the need for complex multi-directional scanning strategies, thus balancing efficiency and performance.

[0053] S53. Update the hidden state sequentially along the positive direction of the one-dimensional feature sequence, and adaptively predict the state update step size at each step based on the current input features.

[0054] In this step, the state transition equations are passed position by position along the positive direction of the sequence. Update the hidden state and adaptively predict the state and update the step size at each step based on the current input features. This is to control the degree to which new information is incorporated. Based on this, an output projection matrix conditioned on the input is used. Combined with the updated hidden state Generate the output vector at the current position. This dynamic state update and output generation mechanism based on input conditions enables the model to effectively capture long-range structural dependencies in the spatial dimension, rather than relying on fixed absolute spatial locations, thus maintaining the stability of information propagation even when faced with directional shifts and spatial mismatches.

[0055] S54. Reorganize the output vectors of the one-dimensional feature sequence at all positions into a two-dimensional feature map, which serves as the structural alignment feature representation.

[0056] In this step, the output vectors at all positions The original spatial scanning order is rearranged into two-dimensional feature maps to obtain the final structure-aligned feature representation. Based on the multi-frequency structure prior extracted by FGSP and the adaptive refinement of FGM, this representation is injected with global contextual information and long-range structural dependencies through SSM. It has joint robustness to unknown orientation, cross-viewpoint and cross-modal changes, and can provide highly discriminative features for subsequent geolocation.

[0057] S6. Based on the structural alignment feature representation of the query image and the reference image, perform distance measurement in the shared embedding space to determine the reference image that matches the query image, and complete the geolocation.

[0058] In this step, the query image is obtained through the aforementioned steps. and reference image After representing the structural alignment features, the distance between the two images is calculated in the shared embedding space. The distance metric can be Euclidean distance or cosine similarity. The goal of geolocation is to find the distance between positive sample pairs (the query image and its corresponding reference image). Less than the distance from the query image to any negative sample (not the corresponding reference image). .in, Indicates a query for an image. Indicates the corresponding reference image, Indicates a mismatched reference image. It is a distance metric embedded in space.

[0059] During training, the Information Noise Contrast Estimation (InfoNCE) loss function was used for optimization, the label smoothness was set to 0.1, and a learnable temperature parameter was introduced. Temperature parameters Automatically adjusted during training to control the sharpness of the feature distribution: smaller This makes the distribution more concentrated, and the model has a stronger ability to distinguish difficult samples; a larger Smoother distribution improves training stability. For cross-view tasks, the model is trained with a learning rate of 0.00025, a batch size of 32, and 60 epochs; for cross-modal tasks, the learning rate is 0.000125, the batch size is 32, and 40 epochs are used. During the retrieval phase, the distance between the query image and all reference images is calculated, sorted in ascending order of distance, and the reference image with the smallest distance or the highest ranking is selected as the matching result to determine the geographical location of the query image.

[0060] Through the overall design of the frequency-guided state space model described above, this invention gradually transforms spatially sensitive features into structure-centered embeddings and performs global reasoning on them, effectively solving the problem of image geolocation when cross-viewpoint and cross-modal conditions exist simultaneously under unknown directions within a unified framework.

[0061] In terms of cross-view image geolocation in unknown directions, as shown in Table 1, this invention ranks second in all metrics on the CVUSA dataset, with an R@1 of 95.26%. On the larger and more challenging CVACT dataset, the validation set R@1 reaches 86.53%, and the test set R@1 reaches 69.81%, which are improvements of 3.53% and 4.98% over ConGeo, respectively, and improvements of 4.13% and 5.93% over Sample4Geo, respectively, demonstrating better handling of spatial misalignment and stronger discrimination ability in visually similar locations.

[0062] Table 1 In terms of cross-modal image geolocation, as shown in Table 2, the present invention achieves an R@1 of 38.21% on the GRAL dataset, which is 3.36% higher than FOENet; R@5 and R@10 are also improved by 2.12% and 1.51%, respectively. This improvement is attributed to the frequency domain representation emphasizing a stable global structure, frequency-guided modulation suppressing inconsistent local features across modalities, and state-space modeling capturing long-range dependencies in structural alignment representations.

[0063] Table 2 In ablation experiments, as shown in Table 3, ablation analysis was performed on the three core components, FGSP, FGM, and SSM, on the CVACT dataset with unknown input directions. The baseline model (using only the ConvNeXt-B backbone network) had an R@1 of 82.40%, which increased to 84.76% after adding SSM, indicating that state-space modeling can effectively improve discriminative ability when global spatial layout changes significantly by capturing long-range dependencies. After further introducing FGSP on top of SSM, R@1 slightly decreased to 84.19%. This is because although the frequency-guided structural prior extracted by FGSP can capture globally consistent patterns, the injected structural information may interfere with the original feature distribution in the absence of an explicit reconciliation mechanism. When FGM was fully introduced, R@1 jumped to 86.53%, and R@5 reached 93.12%, indicating that FGM, through adaptive gating and learnable fusion weights, achieved effective interaction and alignment between structural priors and spatial features, allowing the structural information extracted by FGSP to be correctly integrated into the feature space and more effectively utilized by SSM. The ablation results validated the rationality and necessity of the collaborative design logic of FGSP providing structural clues, FGM ensuring correct integration of clues, and SSM achieving long-range modeling based on refined features.

[0064] Table 3 Regarding generalization ability, as shown in Table 4, in the CVUSA→CVACT cross-dataset generalization scenario, the R@1 of this invention reaches 10.19%, which is more than 2.4 times that of ConGeo; in the CVACT→CVUSA scenario, the R@1 is 10.79%, also significantly higher. This indicates that this invention can learn more transferable representations and effectively generalize across different geographical regions, scene layouts, and imaging conditions.

[0065] Table 4 Regarding robustness, as shown in Table 5, this invention achieves state-of-the-art performance on the three damaged datasets: CVUSA-C-ALL, CVACT_val-C-ALL, and CVACT_test-C-ALL. On the most challenging CVACT_test-C-ALL dataset, R@1_all reaches 69.42%, a 3.53% improvement over Sample4Geo and a 7.15% improvement over ConGeo, demonstrating strong performance even under severe visual distortion. This is attributed to the frequency-guided structural prior reducing the impact of localized damage and maintaining consistent representations across different damage types.

[0066] Table 5 In terms of efficiency, such as Figure 7 As shown, this invention achieves the highest Recall@1 while maintaining competitive single-batch inference time. Although the FGSSM module introduces additional parameters, the overall computational overhead remains moderate compared to methods with higher inference times and FLOPs, such as ConGeo. This favorable trade-off stems from improvements in feature representation rather than increased matching complexity: frequency-guided structural modeling and efficient long-range dependency learning enhance the discriminative power of features and reduce reliance on unstable local patterns, thereby achieving superior localization accuracy while maintaining efficient inference.

[0067] In terms of visualization analysis, to further study the behavior of FSGeo, heatmaps were visualized on the CVACT dataset, and FSGeo was compared with baseline models. Figure 8 As shown, on the CVACT dataset with unknown input orientation, baseline models tend to focus on locally salient regions with strong appearance contrast, such as textured ground patches or high-response areas. However, these regions do not correspond to stable structures under different viewpoints. In unknown orientation settings, the spatial correspondence between ground and satellite images is disrupted, and such local appearance cues often fail to be maintained across viewpoints, leading to erroneous retrieval. In contrast, this invention exhibits different behavior by focusing on structurally consistent regions: the model does not rely on unstable local textures but focuses on geometric elements such as road boundaries and stair structures, which remain consistent across different viewpoints and modalities. This behavior is achieved through the FGSSM module, where frequency-guided structural priors suppress unreliable high-frequency variations, and adaptive modulation further optimizes the spatial response, making the learned representation emphasize stable structural patterns. This results in accurate alignment between ground and satellite views, achieving correct retrieval results even in the absence of orientation information.

[0068] This invention also provides a cross-view, cross-modal image geolocation device, such as... Figure 9As shown, it includes: The acquisition module 10 is used to acquire a query image and a reference image, wherein the query image is a ground view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; Feature extraction module 20 is used to extract initial spatial feature maps of the query image and the reference image respectively through a shared feature extraction backbone network; The frequency-guided structure prior module 30 is used to extract the structure prior of the initial spatial feature map by frequency guidance. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structure features of the encoded global structure pattern are obtained. The frequency-guided modulation module 40 is used to perform frequency-guided adaptive modulation on the initial spatial feature map based on the multi-frequency structural features. Through a spatial adaptive gating mechanism and learnable fusion weights, it selectively integrates the structural prior information with the original spatial information to generate a refined feature map. The state space modeling module 50 is used to perform state space modeling on the refined feature map, propagate global information along the spatial dimension with structural consistency as a condition, and generate a context-enhanced structural alignment feature representation. The geolocation module 60 is used to perform distance measurement in a shared embedding space based on the structural alignment feature representation of the query image and the reference image to determine the reference image that matches the query image, thereby completing geolocation.

[0069] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned cross-view cross-modal image geolocation device and its various units can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0070] The aforementioned cross-view, cross-modal image geolocation device can be implemented as a computer program, which can, for example... Figure 10 It runs on the computer device shown.

[0071] Please see Figure 10 , Figure 10 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0072] See Figure 10The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0073] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to perform a cross-view, cross-modal image geolocation method.

[0074] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0075] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a cross-view, cross-modal image geolocation method.

[0076] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0077] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: S1. Obtain a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; S2. Using a shared feature extraction backbone network, extract the initial spatial feature maps of the query image and the reference image respectively; S3. Frequency-guided structural prior extraction is performed on the initial spatial feature map. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structural features that encode the global structural pattern are obtained. S4. Based on the multi-frequency structural features, the initial spatial feature map is subjected to frequency-guided adaptive modulation. Through a spatial adaptive gating mechanism and learnable fusion weights, the prior structural information and the original spatial information are selectively integrated to generate a refined feature map. S5. Perform state-space modeling on the refined feature map, propagate global information along the spatial dimension with structural consistency as a condition, and generate a context-enhanced structural alignment feature representation. S6. Based on the structural alignment feature representation of the query image and the reference image, perform distance measurement in the shared embedding space to determine the reference image that matches the query image, and complete the geolocation.

[0078] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0079] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0080] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the following steps: S1. Obtain a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; S2. Using a shared feature extraction backbone network, extract the initial spatial feature maps of the query image and the reference image respectively; S3. Frequency-guided structural prior extraction is performed on the initial spatial feature map. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structural features that encode the global structural pattern are obtained. S4. Based on the multi-frequency structural features, the initial spatial feature map is subjected to frequency-guided adaptive modulation. Through a spatial adaptive gating mechanism and learnable fusion weights, the prior structural information and the original spatial information are selectively integrated to generate a refined feature map. S5. Perform state-space modeling on the refined feature map, propagate global information along the spatial dimension with structural consistency as a condition, and generate a context-enhanced structural alignment feature representation. S6. Based on the structural alignment feature representation of the query image and the reference image, perform distance measurement in the shared embedding space to determine the reference image that matches the query image, and complete the geolocation.

[0081] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0083] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0084] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0087] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-view, cross-modal image geolocation method, characterized in that, include: Obtain a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; Initial spatial feature maps of the query image and the reference image are extracted using a shared feature extraction backbone network. Frequency-guided structural prior extraction is performed on the initial spatial feature map. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structural features encoding global structural patterns are obtained. Based on the multi-frequency structural features, the initial spatial feature map is subjected to frequency-guided adaptive modulation. Through a spatial adaptive gating mechanism and learnable fusion weights, the prior structural information and the original spatial information are selectively integrated to generate a refined feature map. State space modeling is performed on the refined feature map, and global information is propagated along the spatial dimension with structural consistency as a condition to generate context-enhanced structural alignment feature representation; Based on the structural alignment feature representations of the query image and the reference image, distance measurement is performed in the shared embedding space to determine the reference image that matches the query image, thereby completing geolocation.

2. The cross-view, cross-modal image geolocation method according to claim 1, characterized in that, The frequency-guided structural prior extraction of the initial spatial feature map, by transforming the initial spatial feature map to the frequency domain and separating different frequency band components, obtains multi-frequency structural features encoding the global structural pattern, including: Perform a two-dimensional discrete Fourier transform on the initial spatial feature map to obtain a frequency domain representation; In the frequency domain, a preset frequency mask is used to separate the frequency domain representation into low-frequency components and high-frequency components, wherein the low-frequency components correspond to global layout and geometric information, and the high-frequency components correspond to texture and edge detail information. Learnable scalar weights are applied to the low-frequency components and the high-frequency components respectively to adaptively adjust the contribution of each frequency band component. The weighted frequency domain components are mapped back to the spatial domain through the inverse two-dimensional discrete Fourier transform to obtain low-frequency spatial features and high-frequency spatial features. The initial spatial feature map, the low-frequency spatial feature, and the high-frequency spatial feature are fused to obtain the multi-frequency structural feature.

3. The cross-view, cross-modal image geolocation method according to claim 2, characterized in that, The step of fusing the initial spatial feature map, the low-frequency spatial features, and the high-frequency spatial features includes: The initial spatial feature map, the low-frequency spatial feature, and the high-frequency spatial feature are concatenated along the channel dimension. The spliced ​​features are sequentially subjected to linear projection, nonlinear activation, and then linear projection again to obtain the multi-frequency structural features.

4. The cross-view, cross-modal image geolocation method according to claim 1, characterized in that, The frequency-guided adaptive modulation of the initial spatial feature map based on the multi-frequency structural features, through a spatial adaptive gating mechanism and learnable fusion weights, selectively integrates prior structural information with the original spatial information to generate a refined feature map, including: The multi-frequency structural features are sequentially convolved, nonlinearly activated, and then convolved again to generate a spatial adaptive gating graph, and the gating value is restricted to a preset range by an activation function; The multi-frequency structural features are modulated element-wise using the spatial adaptive gating graph to obtain the gated structural features. The gated structural features are added to the initial spatial feature map by residual addition to obtain combined features; After performing global average pooling on the combined features, convolution, nonlinear activation, re-convolution and normalized exponential function processing are performed in sequence to generate adaptive fusion weights corresponding to structural branches and spatial branches respectively. Based on the adaptive fusion weights, the gated structural features and the initial spatial feature map are weighted and summed to obtain the refined feature map.

5. The cross-view, cross-modal image geolocation method according to claim 1, characterized in that, The process of performing state-space modeling on the refined feature map, propagating global information along the spatial dimension under the condition of structural consistency, and generating context-enhanced structural alignment feature representations includes: A sequence modeling mechanism based on a selective state-space model is adopted, in which the state transition matrix is ​​a learnable parameter, and both the input projection matrix and the output projection matrix are dynamically generated based on the current input features. The refined feature map is unfolded into a one-dimensional feature sequence according to a preset spatial scanning order; The hidden state is updated sequentially along the positive direction of the one-dimensional feature sequence, and the step size is updated adaptively based on the current input features to predict the state at each step. The output vectors of the one-dimensional feature sequence at all positions are recombined into a two-dimensional feature map, which serves as the structural alignment feature representation.

6. The cross-view, cross-modal image geolocation method according to any one of claims 1 to 5, characterized in that, The shared feature extraction backbone network is a ConvNeXt-B network; the distance metric is calculated as the Euclidean distance or cosine similarity between feature vectors in the shared embedding space; the method uses information noise contrastive estimation loss for optimization training.

7. The cross-view, cross-modal image geolocation method according to any one of claims 1 to 5, characterized in that, After obtaining the query image and the reference image, the process further includes: performing uniform size adjustment and data enhancement processing on the query image and the reference image.

8. A cross-view, cross-modal image geolocation device, characterized in that, include: The acquisition module is used to acquire a query image and a reference image, wherein the query image is a ground-view image, the reference image is an aerial view image or a radar depth image, and the shooting directions of the query image and the reference image are unknown; The feature extraction module is used to extract initial spatial feature maps of the query image and the reference image respectively through a shared feature extraction backbone network; The frequency-guided structure prior module is used to extract the structure prior from the initial spatial feature map by frequency guidance. By transforming the initial spatial feature map to the frequency domain and separating different frequency band components, multi-frequency structure features that encode the global structure pattern are obtained. The frequency-guided modulation module is used to perform frequency-guided adaptive modulation on the initial spatial feature map based on the multi-frequency structural features. Through a spatial adaptive gating mechanism and learnable fusion weights, it selectively integrates structural prior information with the original spatial information to generate a refined feature map. The state space modeling module is used to perform state space modeling on the refined feature map, propagating global information along the spatial dimension with structural consistency as a condition, and generating a context-enhanced structural alignment feature representation. The geolocation module is used to perform distance measurement in a shared embedding space based on the structural alignment feature representation of the query image and the reference image to determine the reference image that matches the query image, thereby completing geolocation.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the cross-view, cross-modal image geolocation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the cross-view, cross-modal image geolocation method as described in any one of claims 1 to 7.

Images